Andrei Lopatenko discussed how search engines, recommendations, and conversational interfaces can be improved to enhance customer experience and drive business revenues. Key investments include building a high-quality engineering team with diverse skills, establishing strong business relationships, developing an experimentation platform, and implementing a continuous improvement process to evaluate and update all aspects of the system. The goal is to create a science-driven culture focused on continuously measuring metrics and making data-backed improvements.

1. Andrei Lopatenko, VP Eng Zillow
Driving Customer Experience and
Business Revenues Through Search
Engines, Recommendations and
Conversational Interfaces

2. ©CX VIRTUAL SUMMIT
What will you learn from this talk
• How to drive continuous circle of improvements of your
search/recommendation/personalization engines or
conversational system leading to improvement of customer
experience and business revenues
• What investments in this area deliver business value and visible
as investments improving business and customer experience

3. ©CX VIRTUAL SUMMIT
Search, Recommendation, Personalization
• In this talk, we are talking about search, recommendation,
personalization engines driving your business, key gateways
letting your customers to find and discover your business items
to buy, book, rent, etc them
• We do not talk about small secondary search engines (search in
forums of your web site, etc)

4. ©CX VIRTUAL SUMMIT
Engines
• We put Search, Recommendation, Personalization engine into one
category - SRP engine
• Despite very different product functions, under the hood, they
implement similar functionality, letting user to express their interests
(‘query’) explicitly or implicitly, and giving user a set of results relevant
to the user’s interest. But ‘query’ can be a text (search) query, user id
(personalization) , location, item id etc (recommendation) ,
• Examples of SRP engines: Google.com, Yelp (localsearch and local
recommendation), Apple Appstore, Amazon or Walmart online
(search, item recommendation, personalization), Zillow.com,
• Despite they are different, they are a lot of similarities and
recommendations of this talk are valid for all of them

5. ©CX VIRTUAL SUMMIT
SRP Engines
• Typically, recommendation and personalization engines are
smaller siblings of search engines, since they get 10X less
customers, and produce 10X less revenue
• But nevertheless, they are very impactful revenue generators
and affect customer experience in a significant way after search
engines

6. ©CX VIRTUAL SUMMIT
Conversational Interfaces
• Conversational interfaces are voice based or text based
interfaces letting user to interactivity communicate with business
services (voice search-> buy, ask questions -> get answers,
voice based customer support)
• Some of them utilize search engines to answer users’ questions.
• They have a lot tech shared with SRP, but a lot differences in
science and technologies
• Nevertheless, key decisions how to develop and run them are
shared with SRP, so most of focus of this talk is about SRP (due
to time limit), but recommendations are applicable to
Conversational Interfaces

7. ©CX VIRTUAL SUMMIT
Customer experience
• Frequently, we separate customer experience (CX) metrics
(typically measured through various behavior metrics) from
business metrics (such as revenue per sessions, revenue per
month, GMV).
• CX and business are related but not directly.
• Good customer experience leads to growth (new users stick with
your business, existing users do not move to competitors)
• Customer experience metrics must be designed for your
business, few CX metrics are applicable to every SRP engine.
CX metrics reflect how users interact with your business site/app

8. ©CX VIRTUAL SUMMIT
Revenue/Business
• SRP engines drive your business since customers buy, book,
rent through SRP engines
• SRP engines produce cash flow
• And they produce expenses (operational support, development
support, data acquisition, software costs)

9. ©CX VIRTUAL SUMMIT
Typical architecture of SRP Engine

10. ©CX VIRTUAL SUMMIT
Typical architecture
• There are a lot of variations in that diagram: but overall it’s the
same flow: search assistance -> query understanding ->
retrieval -> ranking -> presentation generator
• Where different technologies can be used at every layer: either
traditional space feature IR or dense representation retrieval or
similarity search etc but still high level architecture is very
similar across systems

11. ©CX VIRTUAL SUMMIT
Typical Entry points of Customer Experience
• Autocomplete/Autosuggest
• Filters/Facets
• Refinements/Query suggestions
• Search box
• Search Results page
• Recommendations on search result pages
• Personalizated lists

12. ©CX VIRTUAL SUMMIT
Customer experience Metrics
• Many way to measure satisfaction of customers
• From offline - questionaries, user studies to live traffic/behavior
based
• Click through rates and shallow engagement metrics
• Conversion rates and deep engagement metrics
• Subscription/unsubscription rates

13. ©CX VIRTUAL SUMMIT
Customer Experience Metrics
• Longitudinal studies: when customer come back, behavior shifts
over time
• Longitudinal studies - How frequently customer use SRP engine
/ the number of sessions per week/month/year, why they stay or
leave
• Abandonment rates, queries with no results, session with no
clicks
• Many other metrics, frequently needs to be designed for your
business and your customers
• Many of metrics should be studied by subcategories of services,
query types, customer types

14. ©CX VIRTUAL SUMMIT
Business Metrics
• Revenue based: conversion per query and conversion per
session
• Total Revenue per week/month/year , Gross Merchandise Value
in C2C case
• Conversion per customer
•

15. ©CX VIRTUAL SUMMIT
Business Metrics
• Other business targets
• Revenue/number of engaged customer per certain business
segments
• Cross category basket size
• Depends on business goals
• Should be measured and used to turn models, build systems to
develop business in this way

16. ©CX VIRTUAL SUMMIT
Operational Cost Metrics
• Direct operational costs to support search engine running: cost
of datacenter or cloud costs, cost of data acquisitions
• Sometime hard to compute since many systems are shared with
other software systems: proper attribution models are required
• Salaries of the team developing and fsupporting SRP

17. ©CX VIRTUAL SUMMIT
Goal: metrics improvement
• Typical goals: improve customer experience and business
• Subgoals: reduce operational costs
• Required metrics prioritization to drive the development in the
right direction. One metrics - too little, many metrics -> too
chaotic, no focus
• Top down system of OKRs is proven as a good way to drive
search engine development

18. ©CX VIRTUAL SUMMIT
Continuous Circle of Improvement
• Ref: Architecture diagram
• Every subsystem has impact on business, customer experience
and operation costs
• Most of them produce very substantial impact
• The continuous circle: each of system must be continuously
improved

19. ©CX VIRTUAL SUMMIT
High level - Circles of improvement
• Design metrics, work on new models, features, return, resample,
to improve
• Analyze user behavior data to find cases when users have bad
experience,
• Analyze root causes of bad CX
• work to resolve these cases (new data, new features, new
software systems, new models, )

20. ©CX VIRTUAL SUMMIT
Examples of improvement
• Search Assistance
• Type ahead helps users to type their queries. 20-30% of queries
are from type ahead.
• Improvements of typeahead may cause double digit %
improvement in business revenues
• Large improvement in customer satisfaction / easy to use the
system

21. ©CX VIRTUAL SUMMIT
Examples of improvement
• Query Understanding
• Deeper understanding of meaning of queries both in
compositional semantics (natural language search) and other
linguistic features, spell correction (reducing customer errors),
meaning affecting ranking results (strawberry in grocery search -
> fresh produce rather than jams)

22. ©CX VIRTUAL SUMMIT
Examples of improvement
• Ranking
• Despite, presentations about search improvements frequently
focused on query understanding, because it’s easily visible
search improvement, relevance, conversion etc ranking gains
are big factor of search improvements - with big impact on
business and customer satisfaction

23. ©CX VIRTUAL SUMMIT
Examples of improvement
• Search presentation
• How results are presented to customers. What title is shown.
What snippet is shown. Layout of the page. Map. How outputs of
several search engines are mixed
• Double digit % revenue, business impact and customer
experience

24. ©CX VIRTUAL SUMMIT
Key technology stack
• NLP: query understanding, search assistance
• ML: ranking (learning to rank, various techniques from GBTs to
neural methods)
• Data Science, Statistics: analysis of experiments

25. ©CX VIRTUAL SUMMIT
Engineering Quality as investment
• SRP engines are software systems under high load (~ 10^5 qps
, billions items, 100ms latency times)
• Any downtime costs huge money to business due to lost
customers and big loss to customer experience
• Any malfunction - loss of sales, loss of customer experience
• High quality software engineering is crucial part of operations
• Investments into software engineering culture, good SRE and
DevOps - payback well both in short and even more in long
terms

26. ©CX VIRTUAL SUMMIT
Team
• SRP engine team quality
• Important to have a team with good set of diverse talents from
high performance system engineering to machine learning / AI
(multiple functions: NLP, ranking, many others), design and
analysis of experiments/statisticians, SRE, product managers,
•

27. ©CX VIRTUAL SUMMIT
Business Relations
• Productive relationship between business teams and search
engine team
• Product managers which understand business and
understanding engineerings (tech savvy PMs are necessary)
• Analytical and experimental frameworks to evaluate business
ideas, measure their impact key business and customer metrics
• Fast experimentation (engineering quality!) to experiment rapidly
with new search ideas to show their value to business

28. ©CX VIRTUAL SUMMIT
Ops
• Typical SRM engine runs under 24*7 load with hundreds
millions users per month
• Downtime of the engine and any component is not an option
• High quality ops - important investment
• DevOps for canary testing and other scenarios is important
investment
• Observability, - search engine is very complex systems of
hundreds/thousand components

29. ©CX VIRTUAL SUMMIT
Experimentation platform
• To support improvement circle
• to evaluate changes: ranking, query understanding,
autocomplete, user interface, snippets,
• to measure metrics impact
• How to continuously evaluate quality and impact of already
developed components: (it changes over time, some positive
impact component can become negative impact ones)

30. ©CX VIRTUAL SUMMIT
Summary
• Key investments:
• High quality engineering culture, engineering productivity
• Team (wide and diverse)
• Business relations
• Experimentation platform and culture
• Science culture and tools
• Continuous improvement circle and tools to support it at any part
search engine stack
• Operations