Measuring what matters

1. Measuring What Matters

2. © AKAMAI - EDGE 2016 Introductions Cliff Crocker Akamai Buddy Brewer SOASTA

3. © AKAMAI - EDGE 2016 Three eras of modern performance measurement 2001-2006 Basically one browser 2007-2010 Transitional period 2011-2016 WTF just happened

4. © AKAMAI - EDGE 2016 2001-2006: Internet Explorer 6 2001 - 2006

5. © AKAMAI - EDGE 2016 2007-2010: Transitional period 2007 IE7 iPhone Firefox 2 2008 2009 2010

6. © AKAMAI - EDGE 2016 2007-2010: Transitional period 2007 IE7 iPhone Firefox 2 2008 IE7 iPhone 3G Firefox 3 Android Chrome 1 2009 2010

7. © AKAMAI - EDGE 2016 2007-2010: Transitional period 2007 IE7 iPhone Firefox 2 2008 IE7 iPhone 3G Firefox 3 Android Chrome 1 2009 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 2010

8. © AKAMAI - EDGE 2016 2007-2010: Transitional period 2007 IE7 iPhone Firefox 2 2008 IE7 iPhone 3G Firefox 3 Android Chrome 1 2009 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 2010 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 iPad Nav Timing

9. © AKAMAI - EDGE 2016 2011-2016: WTF just happened ? How many browsers in a typical Thanksgiving weekend?

10. © AKAMAI - EDGE 2016 2011-2016: WTF just happened 572 How many browsers in a typical Thanksgiving weekend?

11. © AKAMAI - EDGE 2016 Meanwhile...

12. © AKAMAI - EDGE 2016

13. © AKAMAI - EDGE 2016 So what can I measure?

14. © AKAMAI - EDGE 2016 What is available today - Synthetic Tooling Pros CPU metrics Test environments & other places without real users Access to everything including headers Cons Gaps in: Geos, browsers, pages Impossible to precisely match user demo- & technographics No way to realistically measure browser cache impact

15. © AKAMAI - EDGE 2016 What is available today - RUM Pros Near perfect coverage of browsers, geos, pages Ability to understand impact to performance Realistic measurement of browser cache impact Cons Gaps in: Geos, browsers, pages Impossible to precisely match user demo- & technographics No access to HTTP header data

16. © AKAMAI - EDGE 2016 Engagement Revenue Coverage Caching CPU Pre-prod Availability Page details Resources Headers SyntheticReal

18. © AKAMAI - EDGE 2016 https://www.w3.org/wiki/Web_Performance/Publications W3C - Performance Working Group

19. © AKAMAI - EDGE 2016 Navigation Timing API

20. © AKAMAI - EDGE 2016 Resource Timing API

21. © AKAMAI - EDGE 2016 Coming soon to a browser near you! Measure the size of assets in addition to performance. Resource Timing (2.0) - Transfer Size

22. © AKAMAI - EDGE 2016 ● Measure performance using high-precision timestamps ● Consists of “marks” and “measures” ● PerformanceMark: Timestamp ● PerformanceMeasure: Duration between two given marks Browser Support - User Timing API

23. © AKAMAI - EDGE 2016 Browser Support - Navigation Timing API

24. © AKAMAI - EDGE 2016 Browser Support - Resource Timing API

25. © AKAMAI - EDGE 2016 Browser Support - User Timing API

26. © AKAMAI - EDGE 2016 So what should I measure?

28. © AKAMAI - EDGE 2016 What matters depends on you

29. © AKAMAI - EDGE 2016 What matters to an ops engineer? ● Preference for binary metrics ● Looks for indicators of system performance ● Affinity to network and back-end metrics MTTR

30. © AKAMAI - EDGE 2016 What matters to an ops engineer? Network Server

31. © AKAMAI - EDGE 2016 What matters to an ops engineer? Network Server function getPerf() { var timing = window.performance.timing; return { dns: timing.domainLookupEnd - timing.domainLookupStart, connect: timing.connectEnd - timing.connectStart }; } ttfb: timing.responseStart - timing.connectEnd};

32. © AKAMAI - EDGE 2016 Help answers questions like... ● Do I have an issue with an upstream DNS resolver? ● Am I having Tier1 connectivity issues? ● How fast is my application server responding?

33. © AKAMAI - EDGE 2016 What matters to an ops engineer? Don’t be misled by “super fast” responses ● DNS Caching ● Prefetching ● Connection reuse ● Pages served from browser cache

34. © AKAMAI - EDGE 2016 What matters to an ops engineer? Object Level Timing ● Resource timing API ● Binary metric for testing infrastructure ● Measure performance of a third-party ● CDN healthcheck

35. © AKAMAI - EDGE 2016 What matters to an ops engineer? Object Timing var rUrl = ‘https://edge.akamai.com/ec/us/multimed ia/images/edge/akamai-edge-conference-s peaker-buddy-brewer.jpg’; var me = performance.getEntriesByName(rUrl)[0]; var timings = { loadTime: me.duration, dns: me.domainLookupEnd - me.domainLookupStart, tcp: me.connectEnd - me.connectStart, waiting: me.responseStart - me.requestStart, fetch: me.responseEnd - me.responseStart }

36. © AKAMAI - EDGE 2016 What matters to a front-end developer? Optimized Load Times ● “Do no harm” ● Looks for indicators of application performance ● Looks to measure a “fully loaded” page ● Affinity for front-end metrics

37. © AKAMAI - EDGE 2016 What matters to a front-end developer? Client

38. © AKAMAI - EDGE 2016 What matters to a front-end developer?

39. © AKAMAI - EDGE 2016 What matters to a front-end developer? function getPerf() { var timing = window.performance.timing; return { Basepage: timing.responseEnd - timing.responseStart, DLoading: timing.domLoading – timing.navigationStart, DInt: timing.domIneractive – timing.navigationStart, DContLoaded: timing.domContentLoadedEventEnd – timing.navigationStart, DContLoadTime: timing.domContentLoadedEventEnd – timing.domContentLoadedEventStart, DComplete: timing.domComplete - timing.navigationStart, PLoad: timing.loadEventStart - timing.navigationStart }; } Client

40. © AKAMAI - EDGE 2016 Be wary of onload... ~1s OnLoad ~3.5s visibly complete

41. © AKAMAI - EDGE 2016 What matters to a designer? UX ● Performance by design (D4P) ● Looks for indicators of human perception ● How quickly the page renders ● Affinity for custom metrics and render metrics

42. © AKAMAI - EDGE 2016 RUM first paint and start render Chrome - “First Paint” Visual Start Render

43. © AKAMAI - EDGE 2016 User perception Courtesy of the filament group: https://www.filamentgroup.com/lab/weight-wait.html

45. © AKAMAI - EDGE 2016 Custom Metrics - User Timing API performance.mark(“startTask”); //Some stuff you want to measure happens here performance.mark(“stopTask”); //Measure the duration between the two marks performance.measure(“taskDuration”,“startTask”,“stopTask”); //Measure the load of an image <img src=”image1.gif” onload=”performance.mark(‘image1’)”>

46. © AKAMAI - EDGE 2016 What matters to a business executive? $$$$$$$ ● Business impact analysis ● Simplified/lean metrics ● Actionable data

51. © AKAMAI - EDGE 2016 Don’t create unintended silos The success of my business is a reflection of... The digital experience I provide, which depends on... how quickly I can render content in the browser, which is impacted by... the performance and stability of my infrastructure. 1 2 3 4 Biz Design Ops Dev

52. © AKAMAI - EDGE 2016 “I just want to buy my Mom’s birthday present.” “I just want to know if it will rain today.” “I just want to share pictures of my vacation”

64. © AKAMAI - EDGE 2016 Other opportunities for bots... 1. How does performance impact bot behavior? 2. Are bots skewing the numbers I’m reporting? 3. Am I providing an optimal experience for search bots? 4. Does any of this impact SEO?

65. © AKAMAI - EDGE 2016 Summary ● We have amazing tools at our disposal - your users are wired for sound! ● Measuring what matters is situation independent ● Important to look at all metrics from the same context ● Use robust statistics when looking at your data