Slides for my tutorial from Velocity 2014 on some of the more advanced features in WebPagetest. Video is available on Youtube: Part 1: http://youtu.be/6UeRMMI_IzI Part 2: http://youtu.be/euVYHee1f1M

1. WebPagetest Power Users
Patrick Meenan
pmeenan@webpagetest.org
@patmeenan

2. The Basics
● Presentation is available on slideshare
○ http://www.slideshare.net/patrickmeenan
○ Mostly contains links to everything I’ll be showing
● Video will be uploaded to YouTube
○ https://www.youtube.com/user/patmeenan
● I’ll send links and announcements of both to Twitter and Google+
○ https://twitter.com/patmeenan - @patmeenan
○ https://plus.google.com/u/0/+PatrickMeenan/posts

3. What we’re covering
1. Multiple Runs
2. medianMetric
3. Visual Comparisons
4. Filmstrip URL Manipulation
5. Video Comparison View
6. Better Screen Shots
7. Chrome timelines
8. Mobile device vs emulation (data reduction)
9. bodies/tcpdump/cloudshark
10. Chrome Tracing
11. Images
12. Keep UA String
13. Resource blocking
14. SPOF testing
15. Filmstrip comparison debugging/overlays
16. Scripting multi-step/authentication
17. Scripting request manipulation (headers, DNS, command-line)
18. User Timing
19. Custom metrics/success criteria
20. API
21. CI - node wrapper, Travis, Jenkins, Grunt budget
22. Bulk tests (url/host substitution)
23. Results logging (splunk, logster, etc)
24. Benchmarks
Using WebPagetest
The API
Private Instances

4. Plot Full Results
● Test with > 1 run
● Click “Plot Full Results” link below data table

5. medianMetric
● Defaults to onload (“loadTime”)
● Can be any metric by adding &medianMetric=<blah> to URL
● Valid metric names are the identifiers returned in the API

6. Visual Comparisons
● Log in
● Run Individual tests with the options you want
○ For best results, label each test run
● Go to your Test History
● Select the tests you want to compare
● Click Compare

7. Filmstrip URL Manipulation
http://www.webpagetest.org/video/compare.php?tests=140609_KC_P6Z,140609_Z5_NDE

8. Filmstrip URL Manipulation
http://www.webpagetest.org/video/compare.php?tests=140609_KC_P6Z,140609_Z5_NDE
● Test ID’s for the tests that are being compared
● In the order to be displayed
● Comma-delimited
● Selects the First View run with the median onload time

9. Each test can be configured
140609_KC_P6Z-l:First+View-r:3-c:0-e:7.3
● -l: (Label)
● -r: (Run)
● -c: (Cached? 0 = First View, 1 = Repeat View)
● -e: (end time in seconds or category)
○ doc (onload)
○ full (fully loaded)
○ visual (visually complete - first time it gets to 100%)
○ all (last visual change)

10. Video Comparison View
● Get the filmstrips configured the way you want
● Click “Create Video”
● Click “View as data comparison”

11. Better Screenshots
● PNG Screen Shots
● Control JPEG compression level
● Usually API-only, but also available through hidden parameters
pngss= 0/1, enables full-resolution PNG screen shots
(not video frames)
iq= <jpeq quality level>, sets compression level for all JPEG images
(screen shots and video frames)
http://www.webpagetest.org/?pngss=1&iq=75
Regular Resolution vs Improved

12. Chrome Timelines
● Advanced Settings -> Chrome -> Timeline
○ Always enabled on Android with video capture
○ Call stacks are optional (but helpful for debugging)
● Processing Breakdown link
○ top of details page
○ left of waterfall thumbnail
● Timeline CPU Time below waterfall including in filmstrip
● Timeline links to the left of the waterfall (download or view)

13. CPU Impact - www.amazon.com
Nexus 5
Desktop VM
Moto G
Moto E
200ms ticks
Filmstrip Comparison

14. CPU Impact - www.google.com/search?q=flowers
Nexus 5
Desktop VM
Moto G
Moto E
200ms ticks
Nexus 7 (2013)
Filmstrip Comparison

15. Mobile Device vs Emulation - http://www.theguardian.
com/uk?view=mobile
Nexus 5
Desktop VM
Moto G
Moto E
Nexus 7 (2013)
500ms ticks
Filmstrip Comparison

16. Data Reduction Proxy
● Android-specific
● Only available with actual devices
● Advanced Settings -> Chrome
● Sample Test and Comparison

17. Moar Data!
● Response Bodies
○ Desktop-only (or mobile emulation), all browsers
○ Stores text resources served to test browser (HTML, CSS, JS)
○ Available in “object” tab of waterfall dialog
○ Also as download from request detail
● TCP Dump
○ Desktop and Android Mobile
○ pcap download left of waterfall
○ Sharable through cloudshark (view link)
■ Filter “http”
■ Find request
■ Analysis tools->Follow TCP Stream
■ Show only this stream

18. Chrome Tracing
● Filing a Chrome Performance bug (crbug.com)?
● Debugging a performance issue and need more detail?
○ Miss the days of writing raw assembly? yeah, it’s like that
● Provides tracing of internal browser activity
● Downloadable or viewable directly in the browser
○ WASD to zoom/pan

19. Images
● View all images link below waterfall in detail view (main test)
● Images displayed in the order they were requested
● Look for:
○ visible images loading before slideshow/lower images
○ things that should be in sprites
○ Images with lots of savings
○ PNGs that should be JPEGs
○ Anything else “odd”
○ Decide if you really need to deliver all of those images while loading
● JPEG Analysis view

20. Preserve Original UA String
● Advanced Settings -> “Preserve Original User Agent String”
● Usually Appends PTST <version> to UA string
○ Identify/filter synthetic tests
● Sometimes sites do weird things (sometimes on purpose)
● AOL blocks ads for example

21. Resource Blocking - Desktop Only
● Good for what-if testing
○ How does the site load without my javascript?
○ How does it load without ads?
● Only works for external resources
● Substring match
● “Fails Fast” - returns an immediate error
● Combine with visual comparison for great effect

22. SPOF Testing
● Good for testing impact of 3rd-party outages
● Blackhole FQDN’s only (and entirely)
○ Coming soon, “SPOF all domains except…”
● Goes directly to a Filmstrip comparison
○ Comparing Original vs SPOF
● SPOF routes given domains to blackhole.webpagetest.org
○ blackhole.webpagetest.org is just a host with a firewall rule to drop all
incoming packets
● Non-Responding behaves VERY differently from Fast-Fail
● Would your monitoring detect it?

23. Test Comparison
● Waterfall overlay/slider
● Comparison Graphs

24. Scripting - Multi-step
● Simplest case is multiple distinct pages
● logData 0/1
○ logData 0
○ navigate www.amazon.com
○ logData 1
○ navigate http://www.amazon.com/High-Performance-Web-Sites-Essential-ebook/dp/B0028N4WHY/ref=sr_1_2?ie=UTF8&qid=1402339606&sr=8-2&keywords=even+faster+websites
● Test with shared site resources
● vs combineSteps
○ combineSteps
○ navigate www.amazon.com
○ navigate http://www.amazon.com/High-Performance-Web-Sites-Essential-ebook/dp/B0028N4WHY/ref=sr_1_2?ie=UTF8&qid=1402339606&sr=8-2&keywords=even+faster+websites

25. Scripting - Forms
● Common use is testing pages behind an auth curtain
○ Check the “sensitive data” box to have cookies stripped
● Multiple ways to automate:
○ Automate login form
○ Re-use existing session cookie
● Script Structure
navigate <landing page>
xxx (not measured)
xxx (not measured)
xxxAndWait (measured)
● WPT depends on network activity for end state

26. Scripting - Form Commands
● setValue
● setInnerText
● setInnerHtml
● selectValue
● click, sendClick
● sendKeyDown, sendKeyUp, sendKeyPress
● submitForm
● exec

27. exec(AndWait)
● Executes arbitrary javascript in the context of the page
● Best to keep it to plain javascript (don’t depend on in-page code)
● Debug scripts with local dev tools console before submitting

28. Script - Get the Weather in Santa Clara
logData 0
navigate http://www.weather.com/
exec document.getElementById("typeaheadBox").value = 95054
logData 1
execAndWaitdocument.getElementById("headerSearchForm").submit()
Result - Filmstrip

29. Scripting - Request Manipulation
● setDns, setDnsName, overrideHost
● setCookie
● setHeader, addHeader
● block

30. Scripting - Misc
● setViewportSize
● sleep
Full scripting docs:
https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/scripting

31. Custom Browser Settings
● Chrome
○ Custom command-line options (advanced settings)
■ Specify PAC scripts (proxy)
■ Custom host override
■ Disable SPDY
■ Toggle features
● Firefox
○ firefoxPref script command
■ Sets any firefox pref value before launching browser

32. User Timing
● performance.mark(‘label’)
● Exposed as metrics in WebPagetest results
○ In raw data
○ As purple vertical lines in waterfall
○ Last mark exposed as a unique metric
● Trivial polyfill for older browsers
● Validate RUM markers using video capture/filmstrip

33. Start Render From JS
Light green line in waterfall
Chrome:
var startTime = window.chrome.loadTimes()['requestTime'] ?
window.chrome.loadTimes()['requestTime'] :
window.chrome.loadTimes()['startLoadTime'];
var render = window.chrome.loadTimes().firstPaintTime - startTime;
IE:
var render = window.performance.timing.msFirstPaint;

34. Custom Metrics
● Runs arbitrary javascript snippet(s) at the end of the test
● Defined on a per-test basis or instance-wide
○ New HTTP Archive metrics (DOM depth, iFrame count, etc)
● Can be simple one-liners or more complex multi-function
operations
● Can override existing metrics
● Can return numeric or string values
● Lots of uses:
○ Validate expected page content
○ Collect RUM beacon results
○ Separate results based on A/B or type of ad served
○ Count widgets, etc

35. Custom Metrics - Simple
[iframe-count]
return document.getElementsByTagName("iframe").length;

36. Custom Metrics - Validate Content
[result]
var result = 499;
try {
var copyright = document.getElementsByClassName('copy-text')[0].innerText.slice(-8);
result = copyright == "AOL Inc." ? 0 : 498;
} catch(e) {
}
return result;
Changes test result to:
● 499 if the copyright block doesn’t exist
● 498 if the copyright block exists but the values don’t match
● Leaves it as a success (0) if it exists and matches

37. API
https://sites.google.com/a/webpagetest.org/docs/advanced-features/webpagetest-restful-apis
1. Submit Test
2. Poll for test status
3. Get test result

38. API - Submitting a test
● API key required for public instance
○ email me (pmeenan@webpagetest.org) for key
○ keys are quite limited (~200 “page loads” per day)
○ mostly for proof of concept, early dev before deploying private
instance
● Anything you can do with the UI you can do with the API
○ If a field isn’t documented, just inspect the HTML on the UI to get the
field name :-)
● Make sure to encode parameters
○ urlencode if GET
○ form encode if POST

39. API Response
http://www.webpagetest.org/runtest.php?url=www.aol.com&f=json
{
"statusCode":200,
"statusText":"Ok",
"data":{
"testId":"140610_FY_N2F",
"ownerKey":"04121a1e5b17a59ba9ddd666ae020cf52cee4abe",
"jsonUrl":"http://www.webpagetest.org/jsonResult.php?test=140610_FY_N2F",
"xmlUrl":"http://www.webpagetest.org/xmlResult/140610_FY_N2F/",
"userUrl":"http://www.webpagetest.org/result/140610_FY_N2F/",
"summaryCSV":"http://www.webpagetest.org/result/140610_FY_N2F/page_data.csv",
"detailCSV":"http://www.webpagetest.org/result/140610_FY_N2F/requests.csv"
}
}

40. Polling status
● Periodically poll the JSON/XML URL
● statusCode = 200 : Test Done
○ Test itself may still have failed
● statusCode = 1xx : Test pending/waiting
● statusCode = 4xx : Error locating the test

41. API test result
http://www.webpagetest.org/xmlResult.php?test=140610_FY_N2F&requests=1&breakdown=1

42. Bulk testing
● Submit a bunch of tests
● Periodically check status
● Do “something interesting” with the results

43. Bulk Testing
● PHP
○ https://github.com/WPO-Foundation/webpagetest/tree/master/bulktest
● Python
○ https://github.com/WPO-Foundation/webpagetest/tree/master/batchtool
● HTTP Archive
○ http://httparchive.org/
○ https://github.com/HTTPArchive/httparchive

44. Node API Wrapper
Created by Marcel Duran (@marcelduran)
https://github.com/marcelduran/webpagetest-api
$ npm install webpagetest -g
webpagetest test -k <key> http://twitter.com/marcelduran
webpagetest status 140610_F0_T5K
webpagetest results 140610_F0_T5K
webpagetest test http://twitter.com/marcelduran --poll 5 --timeout 60

45. CI Integration
https://github.com/marcelduran/webpagetest-api/wiki/Test-Specs
Match result to provided specs
{
"median": {
"firstView": {
"requests": 20,
"render": 400,
"loadTime": 3000,
"score_gzip": {
"min": 90
}
}
}
}

46. CI Integration cont.
Uses Mocha with reporters suitable for integration with:
● Jenkins
● Travis-CI
● Drone.io
● Just about anything else you can imagine

47. grunt-perfbudget
Created by Tim Kadlec (@tkadlec)
https://github.com/tkadlec/grunt-perfbudget

48. Bulk Testing in the UI (Private Instances)
● Submit list of URLs to test
● UI options apply to each test
● Scripts can be run against each URL
○ %URL% in script gets replaced with test URL
○ %HOST% in script gets replaced with host from test URL
○ %HOSTR% in script gets replaced with host from URL after redirects
● Aggregate results
○ Download as CSV (subset of data)
○ Compare tests against each other

49. Results logging (Private Instances)
● As each test comes in, full result data is written to log files
○ Page Data (load time, Speed index, requests, custom metrics, etc)
○ Request Data (details for every individual request)
● Each test run or request is on it’s own log file line
● JSON-formatted
● Config information is included
○ Test URL
○ Run #
○ First/Repeat View
○ Test label
○ Location
○ Browser
○ Connectivity
○ Test ID
○ URL to test result

50. Results logging - integration
● Splunk
● Logster
○ Logster -> StatsD = Arbitrary metrics trending automatically
● Track individual requests across all pages
○ ads JS performance
○ CDN response times
○ Effectiveness of image compression
○ so much more

51. Benchmarks (Private Instances)
● Still pretty experimental
● Recurring tests run automatically
○ Configuration is a bit ugly but flexible
● Tests are scheduled, run and aggregated automatically
○ Top-level trended aggregate view
○ All-metrics trended aggregate view
○ Per-page trended view
○ Scatter Plot for a given run
○ Filmstrip comparisons

52. bit.ly/usingwebpagetest

53. Google Booth
Wednesday 6/25
● Lightning Talks
○ 10:45 am - Meet4Speed, Pat Meenan
○ 2:40 pm - The PageSpeed Optimization Platform - Joshua Marantz
● Meet the Speaker
○ 2:40 pm - WebPagetest Q&A, Pat Meenan
Thursday, 6/26
● Lightning Talks
○ 10:45 am The Case for Tunneling Sockets Over HTTP, Wenbo Zhu
○ 2:40 pm - Optimize For Bandwidth - the what and why, Matt Atterbury
● Meet the Speaker
○ 2:40 pm - Making the Web POSH - Q&A, Jan Maessen

54. Feedback Please!
http://velocityconf.com/velocity2014/public/content/mobile-app

55. Thank You!