The document discusses analyzing real user monitoring (RUM) data to gain insights into website performance and user behavior. It describes building plugins to collect navigation and timing data from browsers. Various statistical techniques for analyzing the data are covered, including log-normal distributions, filtering outliers, sampling, and correlating metrics like page load time and bounce rates. The analysis of an example 8 million page dataset suggests very fast or slow page loads are associated with higher bounce rates, and thresholds for user-unfriendly performance are proposed based on bounce rates exceeding 50%.

1. Velocity 2012 / 2012-06-26 RUM for Breakfast 1

2. RUM for Breakfast
Buddy Brewer, Carlos Bueno, Philip Tellis
Velocity 2012 / 2012-06-26
Velocity 2012 / 2012-06-26 RUM for Breakfast 2

3. Real Users
Velocity 2012 / 2012-06-26 RUM for Breakfast 3

4. Real Browsers
Velocity 2012 / 2012-06-26 RUM for Breakfast 4

5. https://github.com/lognormal/boomerang/
Velocity 2012 / 2012-06-26 RUM for Breakfast 5

6. Carlos built the dns plugin
Velocity 2012 / 2012-06-26 RUM for Breakfast 6

7. Buddy built the navtiming plugin
Velocity 2012 / 2012-06-26 RUM for Breakfast 7

8. tl;dr
1 Measure a bunch of stuff in the browser
2 Use high school stats that we vaguely remember
3 Randomly invent insights
Velocity 2012 / 2012-06-26 RUM for Breakfast 8

9. 1
Measure
Velocity 2012 / 2012-06-26 RUM for Breakfast 9

36. 2
Analyze
Velocity 2012 / 2012-06-26 RUM for Breakfast 10

37. Log-Normal Distribution
Velocity 2012 / 2012-06-26 RUM for Breakfast 11

38. Log-Normal Distribution
The logarithm of the x-axis follows a Normal distribution
Velocity 2012 / 2012-06-26 RUM for Breakfast 11

39. Log-Normal Distribution
Use the Geometric Mean for pure Log-Normal distributions
Velocity 2012 / 2012-06-26 RUM for Breakfast 12

40. Log-Normal Distribution
Performance data does not always follow a "pure" Log-Normal
distribution
Velocity 2012 / 2012-06-26 RUM for Breakfast 13

41. Look at the entire spread
...
Velocity 2012 / 2012-06-26 RUM for Breakfast 14

42. Look at the entire spread
which often approaches an infinite width
Velocity 2012 / 2012-06-26 RUM for Breakfast 14

43. Distill
Velocity 2012 / 2012-06-26 RUM for Breakfast 15

44. • 0.8% of hits are fake/abusive
• 0.2-0.5% of hits are from a stale cache
• 0.1% of hits are absurd
• Timestamps in the future (or past depending on how you
interpret it)
• Bots ignore robots.txt across domains
• "Interesting" caches/copies
Velocity 2012 / 2012-06-26 RUM for Breakfast 16

45. Even with beacons, you need to sanitize your input
Velocity 2012 / 2012-06-26 RUM for Breakfast 17

46. Band-pass filtering
Velocity 2012 / 2012-06-26 RUM for Breakfast 18

47. Band-pass filtering
• Strip everything outside a reasonable range
• Bandwidth range: 4kbps - 4Gbps
• Page load time: 0ms - 600s
• You may need to relook at the ranges all the time
Velocity 2012 / 2012-06-26 RUM for Breakfast 18

48. IQR filtering
Velocity 2012 / 2012-06-26 RUM for Breakfast 19

49. IQR filtering
Derive the range from the data
Velocity 2012 / 2012-06-26 RUM for Breakfast 19

50. Sampling
Velocity 2012 / 2012-06-26 RUM for Breakfast 20

51. Margin of Error
σ
±1.96 √n
Velocity 2012 / 2012-06-26 RUM for Breakfast 21

52. MoE & Sample size
There is an inverse square root correlation between sample size
and margin of error
Velocity 2012 / 2012-06-26 RUM for Breakfast 22

53. How big a sample is representative?
Select nsuch that
σ
1.96 √n ≤ 5%µ
Velocity 2012 / 2012-06-26 RUM for Breakfast 23

54. This needs to be at your lowest drilldown level
Velocity 2012 / 2012-06-26 RUM for Breakfast 24

55. 3
Insight
Velocity 2012 / 2012-06-26 RUM for Breakfast 25

56. How does performance
impact human behavior?

57. 8 million pages
1.5 million visits
50 different dimensions

58. very fast sessions had high bounce rates
70.00%
52.50%
35.00%
17.50%
0%
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

59. bounce rate vs. load time
70.00%
52.50%
35.00%
17.50%
0%
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

60. bounce rate vs. DOM interactive
70.00%
52.50%
35.00%
17.50%
0%
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5 11 11.5 12 12.5

61. bounce rate vs. front end time
80.00%
60.00%
40.00%
20.00%
0%
0.5 2 3.5 5 6.5 8 9.5 11 12.5 14 15.5 17 18.5 20 21.5 23 24.5 26 27.5 29

62. is my web site performance toxic to my
users?
http://www.flickr.com/photos/21560098@N06/3796822070

63. LD50 - when do half the users bounce?
http://www.flickr.com/photos/thecosmopolitan/6117530924

64. Bounce rate =50%
Back end time 1.7 sec
DOM Loading 1.8 sec
DOM Interactive 2.75 sec
Front end time 3.5 sec
DOM Complete 4.75 sec
Load event 5.5 sec

65. Future directions
What is the LD50 for your site?
Other bounce rates? 40%? 30%?
Other variables? (critical content
visible, etc)
Other behaviors? Conversions,
revenue, pages per session, actions,
when do people make tea?

66. Thank you
Velocity 2012 / 2012-06-26 RUM for Breakfast 26

67. Questions?
Buddy Brewer @bbrewer Carlos Bueno @archivd
Philip Tellis @bluesmoon

68. Photo credits
• Rum on Ice – wiserbailey on flickr
• Laptop-top Cat – wabisabi2015 on flickr
• About NCSA Mosaic – ncsa
• Distilled – Lost Albatross on flickr
• Anthon Berg Chocolates – ulterior epicure on flickr
• KilroySchematic – on WikiPedia
• Boxplot vs PDF – on WikiPedia
Velocity 2012 / 2012-06-26 RUM for Breakfast 27