SlideShare une entreprise Scribd logo
1  sur  70
Télécharger pour lire hors ligne
Velocity 2012 / 2012-06-26   RUM for Breakfast   1
RUM for Breakfast

Buddy Brewer, Carlos Bueno, Philip Tellis



            Velocity 2012 / 2012-06-26




Velocity 2012 / 2012-06-26   RUM for Breakfast   2
Real Users




             Velocity 2012 / 2012-06-26   RUM for Breakfast   3
Real Browsers




            Velocity 2012 / 2012-06-26   RUM for Breakfast   4
https://github.com/lognormal/boomerang/




            Velocity 2012 / 2012-06-26   RUM for Breakfast   5
Carlos built the dns plugin




             Velocity 2012 / 2012-06-26   RUM for Breakfast   6
Buddy built the navtiming plugin




             Velocity 2012 / 2012-06-26   RUM for Breakfast   7
tl;dr




        1   Measure a bunch of stuff in the browser
        2   Use high school stats that we vaguely remember
        3   Randomly invent insights




                  Velocity 2012 / 2012-06-26   RUM for Breakfast   8
1
                    Measure



Velocity 2012 / 2012-06-26   RUM for Breakfast   9
2
                     Analyze



Velocity 2012 / 2012-06-26   RUM for Breakfast   10
Log-Normal Distribution




            Velocity 2012 / 2012-06-26   RUM for Breakfast   11
Log-Normal Distribution




     The logarithm of the x-axis follows a Normal distribution




             Velocity 2012 / 2012-06-26   RUM for Breakfast      11
Log-Normal Distribution




    Use the Geometric Mean for pure Log-Normal distributions




             Velocity 2012 / 2012-06-26   RUM for Breakfast    12
Log-Normal Distribution




   Performance data does not always follow a "pure" Log-Normal
                          distribution




             Velocity 2012 / 2012-06-26   RUM for Breakfast      13
Look at the entire spread

                             ...




Velocity 2012 / 2012-06-26   RUM for Breakfast   14
Look at the entire spread

which often approaches an infinite width




Velocity 2012 / 2012-06-26   RUM for Breakfast   14
Distill




          Velocity 2012 / 2012-06-26   RUM for Breakfast   15
• 0.8% of hits are fake/abusive
• 0.2-0.5% of hits are from a stale cache
• 0.1% of hits are absurd
• Timestamps in the future (or past depending on how you
  interpret it)
• Bots ignore robots.txt across domains
• "Interesting" caches/copies




          Velocity 2012 / 2012-06-26   RUM for Breakfast   16
Even with beacons, you need to sanitize your input




    Velocity 2012 / 2012-06-26   RUM for Breakfast   17
Band-pass filtering




            Velocity 2012 / 2012-06-26   RUM for Breakfast   18
Band-pass filtering




     • Strip everything outside a reasonable range
          • Bandwidth range: 4kbps - 4Gbps
          • Page load time: 0ms - 600s
     • You may need to relook at the ranges all the time




               Velocity 2012 / 2012-06-26   RUM for Breakfast   18
IQR filtering




               Velocity 2012 / 2012-06-26   RUM for Breakfast   19
IQR filtering




                       Derive the range from the data




               Velocity 2012 / 2012-06-26   RUM for Breakfast   19
Sampling




           Velocity 2012 / 2012-06-26   RUM for Breakfast   20
Margin of Error




                                               σ
                                         ±1.96 √n




            Velocity 2012 / 2012-06-26     RUM for Breakfast   21
MoE & Sample size




   There is an inverse square root correlation between sample size
                         and margin of error




             Velocity 2012 / 2012-06-26   RUM for Breakfast          22
How big a sample is representative?




                              Select nsuch that
                                   σ 
                             1.96 √n  ≤ 5%µ




            Velocity 2012 / 2012-06-26   RUM for Breakfast   23
This needs to be at your lowest drilldown level




  Velocity 2012 / 2012-06-26   RUM for Breakfast   24
3
                       Insight



Velocity 2012 / 2012-06-26   RUM for Breakfast   25
How does performance
impact human behavior?
8 million pages

1.5 million visits

50 different dimensions
very fast sessions had high bounce rates
70.00%




52.50%




35.00%




17.50%




   0%
         0   2    4   6   8   10   12   14   16   18   20   22   24   26   28   30
bounce rate vs. load time
70.00%




52.50%




35.00%




17.50%




   0%
         1   3   5   7    9   11   13   15   17   19   21   23   25   27   29
bounce rate vs. DOM interactive
70.00%




52.50%




35.00%




17.50%




   0%
         1   1.5   2   2.5   3    3.5   4   4.5   5   5.5   6   6.5   7   7.5   8   8.5   9   9.5   10   10.5   11   11.5   12   12.5
bounce rate vs. front end time
80.00%




60.00%




40.00%




20.00%




   0%
         0.5   2   3.5   5   6.5   8   9.5   11   12.5   14   15.5   17   18.5   20   21.5   23   24.5   26   27.5   29
is my web site performance toxic to my
                   users?

http://www.flickr.com/photos/21560098@N06/3796822070
LD50 - when do half the users bounce?




http://www.flickr.com/photos/thecosmopolitan/6117530924
Bounce rate =50%
 Back end time    1.7 sec

 DOM Loading      1.8 sec

DOM Interactive   2.75 sec

Front end time    3.5 sec

DOM Complete      4.75 sec

  Load event      5.5 sec
Future directions
What is the LD50 for your site?

Other bounce rates? 40%? 30%?

Other variables? (critical content
visible, etc)

Other behaviors? Conversions,
revenue, pages per session, actions,
when do people make tea?
Numbers don’t lie




            Velocity 2012 / 2012-06-26   RUM for Breakfast   26
Questions?



Buddy Brewer @bbrewer      Carlos Bueno @archivd
Philip Tellis @bluesmoon
Thank you




Velocity 2012 / 2012-06-26   RUM for Breakfast   27
RUM for Breakfast - distilling insights from the noise
RUM for Breakfast - distilling insights from the noise

Contenu connexe

Similaire à RUM for Breakfast - distilling insights from the noise

Open Source Lambda Architecture with Hadoop, Kafka, Samza and Druid
Open Source Lambda Architecture with Hadoop, Kafka, Samza and DruidOpen Source Lambda Architecture with Hadoop, Kafka, Samza and Druid
Open Source Lambda Architecture with Hadoop, Kafka, Samza and Druid
DataWorks Summit
 

Similaire à RUM for Breakfast - distilling insights from the noise (14)

Performance Warrior Tales: Cloud Load Testing the Retail Giants
Performance Warrior Tales: Cloud Load Testing the Retail Giants Performance Warrior Tales: Cloud Load Testing the Retail Giants
Performance Warrior Tales: Cloud Load Testing the Retail Giants
 
Performance Warrior Tales: Cloud Load Testing the Retail Giants
Performance Warrior Tales: Cloud Load Testing the Retail Giants Performance Warrior Tales: Cloud Load Testing the Retail Giants
Performance Warrior Tales: Cloud Load Testing the Retail Giants
 
Open Source Lambda Architecture with Hadoop, Kafka, Samza and Druid
Open Source Lambda Architecture with Hadoop, Kafka, Samza and DruidOpen Source Lambda Architecture with Hadoop, Kafka, Samza and Druid
Open Source Lambda Architecture with Hadoop, Kafka, Samza and Druid
 
Usability Study Mouse Zoom
Usability Study Mouse ZoomUsability Study Mouse Zoom
Usability Study Mouse Zoom
 
Learning from failures
Learning from failuresLearning from failures
Learning from failures
 
STPCon fall 2012: The Testing Renaissance Has Arrived
STPCon fall 2012: The Testing Renaissance Has ArrivedSTPCon fall 2012: The Testing Renaissance Has Arrived
STPCon fall 2012: The Testing Renaissance Has Arrived
 
Marrying CDNs with Front-End Optimization
Marrying CDNs with Front-End Optimization Marrying CDNs with Front-End Optimization
Marrying CDNs with Front-End Optimization
 
EMEA Webinar - An Introduction to Real User Measurement
EMEA Webinar - An Introduction to Real User Measurement EMEA Webinar - An Introduction to Real User Measurement
EMEA Webinar - An Introduction to Real User Measurement
 
EMEA Webinar - An Introduction to Real User Measurement
EMEA Webinar - An Introduction to Real User Measurement EMEA Webinar - An Introduction to Real User Measurement
EMEA Webinar - An Introduction to Real User Measurement
 
PEnDAR webinar 2 with notes
PEnDAR webinar 2 with notesPEnDAR webinar 2 with notes
PEnDAR webinar 2 with notes
 
Meta edit calc execution v3
Meta edit calc execution v3Meta edit calc execution v3
Meta edit calc execution v3
 
Java PerformanceChapter_7
Java PerformanceChapter_7Java PerformanceChapter_7
Java PerformanceChapter_7
 
SOASTA mPulse: Delivering the Real in Real User Measurement (RUM)
SOASTA mPulse: Delivering the Real in Real User Measurement (RUM)SOASTA mPulse: Delivering the Real in Real User Measurement (RUM)
SOASTA mPulse: Delivering the Real in Real User Measurement (RUM)
 
SOASTA mPulse: Delivering the Real in Real User Measurement (RUM)
SOASTA mPulse: Delivering the Real in Real User Measurement (RUM)SOASTA mPulse: Delivering the Real in Real User Measurement (RUM)
SOASTA mPulse: Delivering the Real in Real User Measurement (RUM)
 

Dernier

Dernier (20)

Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 

RUM for Breakfast - distilling insights from the noise

  • 1. Velocity 2012 / 2012-06-26 RUM for Breakfast 1
  • 2. RUM for Breakfast Buddy Brewer, Carlos Bueno, Philip Tellis Velocity 2012 / 2012-06-26 Velocity 2012 / 2012-06-26 RUM for Breakfast 2
  • 3. Real Users Velocity 2012 / 2012-06-26 RUM for Breakfast 3
  • 4. Real Browsers Velocity 2012 / 2012-06-26 RUM for Breakfast 4
  • 5. https://github.com/lognormal/boomerang/ Velocity 2012 / 2012-06-26 RUM for Breakfast 5
  • 6. Carlos built the dns plugin Velocity 2012 / 2012-06-26 RUM for Breakfast 6
  • 7. Buddy built the navtiming plugin Velocity 2012 / 2012-06-26 RUM for Breakfast 7
  • 8. tl;dr 1 Measure a bunch of stuff in the browser 2 Use high school stats that we vaguely remember 3 Randomly invent insights Velocity 2012 / 2012-06-26 RUM for Breakfast 8
  • 9. 1 Measure Velocity 2012 / 2012-06-26 RUM for Breakfast 9
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36. 2 Analyze Velocity 2012 / 2012-06-26 RUM for Breakfast 10
  • 37. Log-Normal Distribution Velocity 2012 / 2012-06-26 RUM for Breakfast 11
  • 38. Log-Normal Distribution The logarithm of the x-axis follows a Normal distribution Velocity 2012 / 2012-06-26 RUM for Breakfast 11
  • 39. Log-Normal Distribution Use the Geometric Mean for pure Log-Normal distributions Velocity 2012 / 2012-06-26 RUM for Breakfast 12
  • 40. Log-Normal Distribution Performance data does not always follow a "pure" Log-Normal distribution Velocity 2012 / 2012-06-26 RUM for Breakfast 13
  • 41. Look at the entire spread ... Velocity 2012 / 2012-06-26 RUM for Breakfast 14
  • 42. Look at the entire spread which often approaches an infinite width Velocity 2012 / 2012-06-26 RUM for Breakfast 14
  • 43. Distill Velocity 2012 / 2012-06-26 RUM for Breakfast 15
  • 44. • 0.8% of hits are fake/abusive • 0.2-0.5% of hits are from a stale cache • 0.1% of hits are absurd • Timestamps in the future (or past depending on how you interpret it) • Bots ignore robots.txt across domains • "Interesting" caches/copies Velocity 2012 / 2012-06-26 RUM for Breakfast 16
  • 45. Even with beacons, you need to sanitize your input Velocity 2012 / 2012-06-26 RUM for Breakfast 17
  • 46. Band-pass filtering Velocity 2012 / 2012-06-26 RUM for Breakfast 18
  • 47. Band-pass filtering • Strip everything outside a reasonable range • Bandwidth range: 4kbps - 4Gbps • Page load time: 0ms - 600s • You may need to relook at the ranges all the time Velocity 2012 / 2012-06-26 RUM for Breakfast 18
  • 48. IQR filtering Velocity 2012 / 2012-06-26 RUM for Breakfast 19
  • 49. IQR filtering Derive the range from the data Velocity 2012 / 2012-06-26 RUM for Breakfast 19
  • 50. Sampling Velocity 2012 / 2012-06-26 RUM for Breakfast 20
  • 51. Margin of Error σ ±1.96 √n Velocity 2012 / 2012-06-26 RUM for Breakfast 21
  • 52. MoE & Sample size There is an inverse square root correlation between sample size and margin of error Velocity 2012 / 2012-06-26 RUM for Breakfast 22
  • 53. How big a sample is representative? Select nsuch that σ 1.96 √n ≤ 5%µ Velocity 2012 / 2012-06-26 RUM for Breakfast 23
  • 54. This needs to be at your lowest drilldown level Velocity 2012 / 2012-06-26 RUM for Breakfast 24
  • 55. 3 Insight Velocity 2012 / 2012-06-26 RUM for Breakfast 25
  • 56. How does performance impact human behavior?
  • 57. 8 million pages 1.5 million visits 50 different dimensions
  • 58. very fast sessions had high bounce rates 70.00% 52.50% 35.00% 17.50% 0% 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
  • 59. bounce rate vs. load time 70.00% 52.50% 35.00% 17.50% 0% 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29
  • 60. bounce rate vs. DOM interactive 70.00% 52.50% 35.00% 17.50% 0% 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5 11 11.5 12 12.5
  • 61. bounce rate vs. front end time 80.00% 60.00% 40.00% 20.00% 0% 0.5 2 3.5 5 6.5 8 9.5 11 12.5 14 15.5 17 18.5 20 21.5 23 24.5 26 27.5 29
  • 62. is my web site performance toxic to my users? http://www.flickr.com/photos/21560098@N06/3796822070
  • 63. LD50 - when do half the users bounce? http://www.flickr.com/photos/thecosmopolitan/6117530924
  • 64. Bounce rate =50% Back end time 1.7 sec DOM Loading 1.8 sec DOM Interactive 2.75 sec Front end time 3.5 sec DOM Complete 4.75 sec Load event 5.5 sec
  • 65. Future directions What is the LD50 for your site? Other bounce rates? 40%? 30%? Other variables? (critical content visible, etc) Other behaviors? Conversions, revenue, pages per session, actions, when do people make tea?
  • 66. Numbers don’t lie Velocity 2012 / 2012-06-26 RUM for Breakfast 26
  • 67. Questions? Buddy Brewer @bbrewer Carlos Bueno @archivd Philip Tellis @bluesmoon
  • 68. Thank you Velocity 2012 / 2012-06-26 RUM for Breakfast 27