1) The document examines the impact of response latency on user behavior in web search through a user study and large-scale query log analysis. 2) The user study found that delays under 500ms are not easily noticeable, delays over 1000ms are noticed with high likelihood, and perception of latency varies across users. 3) The query log analysis revealed that users are more likely to click on search results with lower latency, and prefer clicking more results over submitting new queries when the latency difference exceeds 1250ms.

1. Impact of Response Latency on
User Behavior in Web Search
Ioannis Arapakis, Xiao Bai, B. Barla Cambazoglu
Yahoo Labs, Barcelona

2. Background Information
§ The core research in IR has been on improving the quality of
search results with the eventual goal of satisfying the
information needs of users
§ This often requires sophisticated and costly solutions
• more information stored in the inverted index
• machine-learned ranking strategies
• fusing results from multiple resources
Page 2 / 29

3. Trade-off between the speed of a search system and
the quality of its results
Too slow or too fast may result in financial consequences
for the search engine
Page 3 / 29

4. User Side
§ Web users
• are impatient
• have limited time
• expect sub-second response times
§ High response latency
• can distract users
• results in fewer query submissions
• decreases user engagement over
time
Page 4 / 29

5. Search Engine Side
§ Search engines
• have large user bases and query
volumes
• make heavy investments on H/W
infrastructure
• try to maintain query response
times at reasonable levels
§ These investments
• incur a financial burden on
search engine companies
• result in financial losses
Page 5 / 29

6. Research Questions
1. What are the main components in the response
latency of a search engine?
2. How sensitive are users to response latency?
3. How much does response latency affect user
behavior?
§ We conduct two studies
• a small-scale user study
• a large-scale query log analysis
Page 6 / 29

7. Components of User-Perceived Response Latency
§ network latency: tuf + tfu
§ search engine latency: tpre + tfb + tproc + tbf + tpost
§ browser latency: trender
User
Search
frontend
Search
backend
tpre tproc
tpost
tfb
tbf
tuf
tfu
trender
Page 7 / 29

8. Distribution of Latency Values
0
0.005
0.010
0.015
0.020
0.025
0.030
0.035
0.040
0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 2.4
0
0.2
0.4
0.6
0.8
1.0
Fractionofqueries
Cumulativefractionofqueries
Latency (normalized by the mean)
Page 8 / 29

9. Contribution of Latency Components
0
20
40
60
80
100
0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 2.4
Contributionpercomponent(%)
Latency (normalized by the mean)
search engine latency
network latency
browser latency
Page 9 / 29

10. Study 1:
User Sensitivity to Latency

11. Experimental Method (Task 1 & 2)
§ Two independent variables
• Search latency (0 – 2750ms)
• Search site speed (slow, fast)
§ 12 participants (female=6, male=6)
• Studying (33.3%)
• Studying while working (54.3%)
• Full-time employees (16.6%)
Page 11 / 29

12. Task 1: Procedure
§ Participants submitted 40 navigational queries
§ After submitting each query, they were asked to report if the
response of the search site was “slow” or “normal”
§ For each query we increased latency by a fixed amount (0 –
1750ms), using a step of 250ms
§ Each latency value (e.g., 0, 250, 500) was introduced 5 times,
in a random order
Page 12 / 29

13. Task 1: Results
250 750 1250 1750
Added latency (ms)
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Likelihoodoffeelingaddedlatency
Slow SE (base)
Slow SE
Fast SE (base)
Fast SE
250 750 1250 1750
Added latency (ms)
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Increaserelativetobaselikelihood
Slow SE
Fast SE
§ Delays <500ms are not
easily noticeable
§ Delays >1000ms are
noticed with high likelihood
Page 13 / 29

14. Task 2: Procedure
§ Participant submitted 50 navigational queries
§ After each query submission they provided an estimation
of the search latency in milliseconds
§ For each query we increased latency by a fixed amount
(500 – 2750ms), using a step of 250ms
§ Each latency value (e.g., 0, 250, 500) was introduced 5
times, in a random order
§ A number of training queries was submitted without any
added delay
Page 14 / 29

15. Task 2: Results
Fig. 1: Slow search engine
1750 2000 2250 2500 2750
Actual latency (ms)
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Predictedlatency(ms)
Actual
Males
Females
Average
750 1000 1250 1500 1750 2000 2250 2500 2750
Actual latency (ms)
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Predictedlatency(ms)
Actual
Males
Females
Average
Fig. 2: Fast search engine
Page 15 / 29

16. Study 2:
Impact of Latency on Search Experience

17. Experimental Design
§ Investigate the effects of response latency on the user
engagement and satisfaction
§ Two independent variables
• Search latency (0, 750, 1250, 1750)
• Search site speed (slow, fast)
§ Search latency was set to desired amount using a custom-made
javascript deployed through the Greasemonkey extension
§ 20 participants (female=10, male=10)
Page 17 / 29

18. Procedure
§ Participants performed four search tasks
• Evaluate the performance of four different backend search systems
• Submit as many navigational queries from a list of 200 randomly sampled
web domains
• For each query they were asked to locate the target URL among the first
ten results of the SERP
§ Training queries were used to allow participants to familiarize
themselves with the “default” search site speed
Page 18 / 29

19. Questionnaires
• User Engagement Scale (USE)
• Positive affect (PAS)
• Negative affect (NAS)
• Perceived usability
• Felt involvement and focused attention
• IBM’s Computer System Usability Questionnaire (CSUQ)
• System usefulness (SYSUSE)
• Custom statements
• Custom-1: “This search site was fast in responding to my queries”
• Custom-2: “This search site helped me accomplish my task in a reasonable amount of time”
• Custom-3: “I feel satisfied with the retrieved results”
Page 19 / 29

20. Descriptive Statistics (M) for UE and SYSUSE
SEslow latency SEfast latency
0ms 750ms 1250ms 1750ms 0ms 750ms 1250ms 1750ms
Post-Task Positive Affect 16.20 14.50 15.50 15.20 20.50 19.00 20.80 19.30
Post-Task Negative Affect 7.00 6.80 7.60 6.90 6.80 7.40 7.40 7.20
Frustration 3.20 3.10 2.90 3.30 2.80 3.00 3.50 2.60
Focused Attention 22.80 22.90 19.90 22.20 27.90 26.60 23.90 29.50
SYSUS 32.80 28.90 29.80 27.90 35.20 31.30 29.80 33.20
§ Positive bias towards SEfast
§ SEfast participants were more deeply engaged
§ SEfast participants’ usability perception was more tolerant to delays
Page 20 / 29

21. Correlation Analysis of Beliefs and Reported Scales
Beliefs postPAS postNAS FA CSUQ-SYSUS custom-1 custom-2 custom-3
SEslow will respond fast to my queries .455** .041 0.702** .267 .177 .177 .082
SEslow will provide relevant results .262 -.083 .720** .411** .278 .263 .232
SEfast will respond fast to my queries -.051** .245 .341* .591** .330* .443** .624**
SEfast will provide relevant results -.272 .133 -.133 .378* .212 .259 .390*
*. Correlation is significant at the .05 level (2-tailed). **. Correlation is significant at the .01 level (2-tailed)
Page 21 / 29

22. Query Log Analysis

23. Query Log and Engagement Metric
§ Random sample of 30m web search queries obtained from Yahoo
§ We use the end-to-end (user perceived) latency values
§ To control for differences due to geolocation or device, we select
queries issued:
• Within the US
• To a particular search data center
• from desktop computers
§ We quantify engagement using the clicked page ratio metric
Page 23 / 29

24. 0.0
0.2
0.4
0.6
0.8
1.0
0.4 0.6 0.8 1.0 1.2 1.4
Clickedpageratio(normalizedbythemax)
Latency (normalized by the mean)
Variation of Clicked Page Ratio Metric
Page 24 / 29

25. 0.02
0.04
0.06
0.08
0.10
0.12
0 250 500 750 100012501500 17502000
0.8
0.9
1.0
1.1
1.2
1.3
Fractionofquerypairs
Click-on-fast/Click-on-slow
Latency difference (in milliseconds)
Click-on-fast
Click-on-slow
Ratio
Eliminating the Effect of Content
§ 500ms of latency difference is
the critical point beyond
which users are more likely to
click on a result retrieved with
lower latency
Page 25 / 29

26. Eliminating the Effect of Content
0.02
0.04
0.06
0.08
0.10
0.12
0 250 500 750 100012501500 17502000
0.8
0.9
1.0
1.1
1.2
1.3
Fractionofquerypairs
Click-more-on-fast/Click-more-on-slow
Latency difference (in milliseconds)
Click-more-on-fast
Click-more-on-slow
Ratio
§ Clicking on more results
becomes preferable to
submitting new queries when
the latency difference exceeds
a certain threshold (1250ms)
Page 26 / 29

27. Conclusions
§ Up to a point (500ms) added response time delays are not
noticeable by the users
§ After a certain threshold (1000ms) the users can feel the added
delay with very high likelihood
§ Perception of search latency varies considerably across the
population!
§ The tendency to overestimate or underestimate system
performance biases users’ interpretations of search interactions
and system usability
Page 27 / 29

28. Conclusions
§ Given two content-wise identical result pages, users are more
likely to click on the result page that is served with lower latency
§ 500ms of latency difference is the critical point beyond which
users are more likely to click on a result retrieved with lower
latency
§ Clicking on more results becomes preferable to submitting new
queries when the latency difference exceeds a certain threshold
(1250ms)
Page 28 / 29

29. Thank you for your attention!
arapakis@yahoo-inc.com
iarapakis
http://www.slideshare.net/iarapakis/sigir2014