Counterintuitive observations in optimizing search engine algorithms. Don't get fooled if your user satisfaction goes up - it might not be all good... Topics covered include: 1. How to test end user satisfaction and dissatisfaction. 2. How do you optimize your search engine algorithms. Intuitive and counterintuitive learnings from optimizing. What is best? General purpose vs. specializing your search engine on focus areas? Optimizing search for each user vs. users grouped in cohorts? How relevant are individual wrong results in a list of good query results? What is the impact of usage scenarios, target devices & environment? ... 3. Resulting strategies

1. Developing and Testing Search
Engine Algorithms –
Counterintuitive observations due to end
user behavior. Suggestions.
(This presentation is not representing views of my current employer)
CHRISTIAN VON REVENTLOW (VONREVENTLOW)
VONREVENTLOW@YAHOO.COM, +1 201 259 5973

2. Use a data driven feedback loop to
evolve and test search engine algorithms
Search Engine
Algorithm
Queries
Query
results Tester
or
End user
Search
End-User
Interface
Queries
Results
Rate results
Data Scientists,
Algorithm- and
Software
Developers
Software
Compute
Search
Result
Quality
Statics
Search Quality Metrics

3. Measuring search performance
• Understanding behavior and needs of satisfied and unsatisfied search users is key for improving the
users search experience [0]
• Satisfaction/dissatisfaction data is used to evolve and optimize the search algorithms [1].
• Traces from end users or a subset thereof
• Testers creating sample queries.
• Metrics like MAP (mean average precision) or NDCG (normalized discounted cumulative gain) had
been used to measure search quality. [2]. A Click-through was used to judge relevance of results.
• Nowadays metrics use the entire sequence of events in a search. An example is modelling search
logs in Markov Models to get estimators of user satisfaction or dissatisfaction [1,3].
[0] A Dasdan, K Tsioutsiouliklis, E. Velipasaoglu. “Web Search Engine Metrics”, WWW2010
[1] A Hassan, Y Song, Li-Wei He. “A Task Level Metric for Improving Web Search Satisfaction and its Application on Improving Relevance Estimation”, ACM CIKM’11 Oct 2011
[2] K. Jaervelin, J. Kekalainen. “Cumulated gain based evaluation of IR techniques”, ACM TOIS 2002
[3] A. Hassan, R. White. “Personalized Models of Search Satisfaction”, CIKM Nov 2013

4. Specialization improves end user
satisfaction in search
• Economic theory: specialized search engines deliver an
advantage – specifically when its not only about attracting
as many searchers but satisfying as many of them [1].
• Real data: shows user satisfaction increases when grouping
users in cohorts with similar topical interests and optimize
for each relevant cohort [2].
• Best by combining search results that are valid for
everybody (Global) and the specific cohort only (personal).
• The Counterintuitive: User satisfaction is better when
optimizing search for the cohort – vs. optimizing for each
individual.
Users profit from the larger feedback dataset to the search
algorithms provided by a cohort of similar people.
[1] D. Kempe, B. Lucier. “User Satisfaction in Competitive Sponsored Search”. Cornell University.
arXiv:1310.4098v1 [cs.GT]
[2] A. Hassan, R. White. “Personalized Models of Search Satisfaction”. CIKM Nov 2013
6.00%
5.00%
4.00%
3.00%
2.00%
1.00%
0.00%
Example percentage gain in accuracy
vs optimizing for single target audience
Optimize by Topic Optimize by larger
Cohort
Optimize by smaller
Cohort
Global & Personal combined Personal only

5. Its not sufficient that “the right result” is
part of the results list
• Studies have shown that users are only interested in
the first few results – thus high accuracy is desirable
[1]
• Temporal and geographic relevance, coverage,
comprehensiveness, rapid discovery of new content,
content freshness and diversity are vectors relevant
Result 1- relevant
for users [2]
Result 2 - right results
Result 3 – wrong/irrelevant
• Users search environment has a major impact – like
search on a mobile device vs. search from a tablet –
Result 4 – relevant
requiring specialization.
Result 5 - relevant
• The counterintuitive: Even if the “right result” is part
of the first few results - having irrelevant/perceived
wrong results makes the user disbelief in the
correctness of ALL results.
Query results
[1] R.W. White. D. Morris. “Investigating the querying and browsing behavior of advanced search engine users” Proc. SIGIR
[2] A Dasdan, K Tsioutsiouliklis, E. Velipasaoglu. “Web Search Engine Metrics”, WWW2010

6. Don’t get fooled if your user satisfaction
goes up – it might not be all good..
• Behavioral differences have been shown between novice
and expert searchers [1]
• Optimizing differently for experts and casual users increases
user satisfaction. [2]
• The Counterintuitive: User satisfaction goes up over time
even if you do not modify the algorithms.
• Why:
• Users learn how to query best (i.e. become mature users)
• Learned what not to ask – i.e. intuitively restrict the usage space
• Or worse: defect to other search engines. 60% of switches to a
different engine are caused by dissatisfaction [3]
• So don’t get fooled – understand why your satisfaction went
up…
[1] R.W. White. D. Morris. “Investigating the querying and browsing behavior of advanced search engine
users” Proc. SIGIR
[2] A. Hassan, R. White. “Personalized Models of Search Satisfaction”. CIKM Nov 2013
[3] Q. Guo, R.W. White, Y. Zhang, B. Anderson, S Dumais. “Why searchers switch: understanding and
predicting engine switching rationales”. Proc. SIGIR 2011
6.00%
5.00%
4.00%
3.00%
2.00%
1.00%
0.00%
Example percentage gain in accuracy
vs optimizing for single target audience
Optimize by Topic Optimize for expert
vs. casual user
Optimize by larger
Cohort
Global & Personal combined Personal only

7. Resulting Strategies
• Specialization/Focus: Get clarify of what your search engine is targeted for – vs. a general
purpose web search. Examples are places on a map, images, research papers.
• Cohorts: Segment your user base in cohorts and optimize for each of them.
• Start with expert and casual user.
• Interview users, analyze search traces, … to identify other larger cohorts.
• Usage: Optimize for usage environment & target device.
• Smartphone, Tablet, PC, Professional multiscreen office setup.
• Correctness: Carefully evaluate the dissatisfying query results. And minimize them.
• Fresh end user participants for testing: Regularly recruit new groups of users to optimize your
algorithms – specifically people who have never used your search before.