From Velocity 2012 in Santa Clara, CA. Buddy Brewer, Philip Tellis, and Carlos Bueno talk about real user measurement collection, analysis, and insights.
6. Carlos built the dns plugin
Velocity 2012 / 2012-06-26 RUM for Breakfast 6
7. Buddy built the navtiming plugin
Velocity 2012 / 2012-06-26 RUM for Breakfast 7
8. tl;dr
1 Measure a bunch of stuff in the browser
2 Use high school stats that we vaguely remember
3 Randomly invent insights
Velocity 2012 / 2012-06-26 RUM for Breakfast 8
9. 1
Measure
Velocity 2012 / 2012-06-26 RUM for Breakfast 9
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36. 2
Analyze
Velocity 2012 / 2012-06-26 RUM for Breakfast 10
38. Log-Normal Distribution
The logarithm of the x-axis follows a Normal distribution
Velocity 2012 / 2012-06-26 RUM for Breakfast 11
39. Log-Normal Distribution
Use the Geometric Mean for pure Log-Normal distributions
Velocity 2012 / 2012-06-26 RUM for Breakfast 12
40. Log-Normal Distribution
Performance data does not always follow a "pure" Log-Normal
distribution
Velocity 2012 / 2012-06-26 RUM for Breakfast 13
41. Look at the entire spread
...
Velocity 2012 / 2012-06-26 RUM for Breakfast 14
42. Look at the entire spread
which often approaches an infinite width
Velocity 2012 / 2012-06-26 RUM for Breakfast 14
43. Distill
Velocity 2012 / 2012-06-26 RUM for Breakfast 15
44. • 0.8% of hits are fake/abusive
• 0.2-0.5% of hits are from a stale cache
• 0.1% of hits are absurd
• Timestamps in the future (or past depending on how you
interpret it)
• Bots ignore robots.txt across domains
• "Interesting" caches/copies
Velocity 2012 / 2012-06-26 RUM for Breakfast 16
45. Even with beacons, you need to sanitize your input
Velocity 2012 / 2012-06-26 RUM for Breakfast 17
47. Band-pass filtering
• Strip everything outside a reasonable range
• Bandwidth range: 4kbps - 4Gbps
• Page load time: 0ms - 600s
• You may need to relook at the ranges all the time
Velocity 2012 / 2012-06-26 RUM for Breakfast 18
48. IQR filtering
Velocity 2012 / 2012-06-26 RUM for Breakfast 19
49. IQR filtering
Derive the range from the data
Velocity 2012 / 2012-06-26 RUM for Breakfast 19
50. Sampling
Velocity 2012 / 2012-06-26 RUM for Breakfast 20
51. Margin of Error
σ
±1.96 √n
Velocity 2012 / 2012-06-26 RUM for Breakfast 21
52. MoE & Sample size
There is an inverse square root correlation between sample size
and margin of error
Velocity 2012 / 2012-06-26 RUM for Breakfast 22
53. How big a sample is representative?
Select nsuch that
σ
1.96 √n ≤ 5%µ
Velocity 2012 / 2012-06-26 RUM for Breakfast 23
54. This needs to be at your lowest drilldown level
Velocity 2012 / 2012-06-26 RUM for Breakfast 24
55. 3
Insight
Velocity 2012 / 2012-06-26 RUM for Breakfast 25
61. bounce rate vs. front end time
80.00%
60.00%
40.00%
20.00%
0%
0.5 2 3.5 5 6.5 8 9.5 11 12.5 14 15.5 17 18.5 20 21.5 23 24.5 26 27.5 29
62. is my web site performance toxic to my
users?
http://www.flickr.com/photos/21560098@N06/3796822070
63. LD50 - when do half the users bounce?
http://www.flickr.com/photos/thecosmopolitan/6117530924
64. Bounce rate =50%
Back end time 1.7 sec
DOM Loading 1.8 sec
DOM Interactive 2.75 sec
Front end time 3.5 sec
DOM Complete 4.75 sec
Load event 5.5 sec
65. Future directions
What is the LD50 for your site?
Other bounce rates? 40%? 30%?
Other variables? (critical content
visible, etc)
Other behaviors? Conversions,
revenue, pages per session, actions,
when do people make tea?