How to Troubleshoot Apps for the Modern Connected Worker
Measuring the web with boomerang
1. Why measure?
boomerang
data data data
Measuring the web with boomerang
Philip Tellis / philip@bluesmoon.info
SF Performance Meetup / 2010-08-11
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
2. Why measure?
boomerang
data data data
$ finger philip
Philip Tellis
philip@bluesmoon.info
@bluesmoon
yahoo
geek
http://bluesmoon.info/
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
3. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Where does all the time go?
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
4. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Who controls it?
Some of this we control and some of it we don’t
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
5. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Back end
Measuring and improving back end performance can be done
during development
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
6. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
80-20
Turns out that less than 20% of the time is spent on the back
end
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
7. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Front end
It’s what we can’t control that bites us
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
8. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Too many variations
browsers
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
9. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Too many variations
plugins
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
10. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Too many variations
OSes
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
11. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Too many variations
viruses
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
12. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Too many variations
antiviruses
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
13. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Too many variations
microwaves
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
14. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Too many variations
baby monitors
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
15. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Too many variations
naughty neighbours
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
16. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Too many variations
file shares
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
17. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Too many variations
governments
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
18. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Too many variations
rodents
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
19. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Too many variations
Try simulating all that in the lab!
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
20. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
We need to measure real end-user performance
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
21. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
We need to measure it from the real end-user’s box
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
22. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Ask the user?
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
23. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Bias
While this might work, it isn’t necessarily representative
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
24. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
A/B testing
You also want to be able to dynamically tune which users get
which tests
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
25. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Phone home
It’s most useful if you can send these measurements back to
your server for analysis
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
26. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Mostly ubiquitous
We know that javascript is available on almost every browser
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
27. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Rich pages
We really want to measure the performance of rich pages
which depend on javascript already
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
28. Why measure? The slow web
boomerang Measurements
data data data Measuring with javascript
Limited
But javascript can’t measure everything... we get as close as
we can
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
29. Why measure? What?
boomerang How does it work?
data data data Accuracy
A piece of javascript that you add to your web page where it
measures and beacons back to you the end user’s perceived
performance of your page
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
30. Why measure? What?
boomerang How does it work?
data data data Accuracy
How?
<script src="boomerang.js" type="text/javascript"></
<script type="text/javascript">
BOOMR.init({
user_ip: "<user’s ip address>",
beacon_url: "http://yoursite.com/path/to/beacon.php"
});
</script>
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
31. Why measure? What?
boomerang How does it work?
data data data Accuracy
What does it do?
About once a week, measures user’s bandwidth and
latency to your server
On (almost) every request, measures the time it took to
load the current page
Beacons these results back to your server
Other stuff based on plugins
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
32. Why measure? What?
boomerang How does it work?
data data data Accuracy
How does it do it?
Let’s take that one at a time
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
33. Why measure? What?
boomerang How does it work?
data data data Accuracy
How do we measure latency?
Download a 32 byte gif 10 times in sequence
Measure the time to download each
Discard the first measurement because it’s overpriced
Calculate the arithmetic mean, standard deviation and
margin of error of the remaining
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
34. Why measure? What?
boomerang How does it work?
data data data Accuracy
How do we measure bandwidth?
After the latency test is done, we download progressively
larger images
Stop at the first image that times out
Redownload that image a few more times
Calculate the median, standard deviation and margin of
error of the largest images
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
35. Why measure? What?
boomerang How does it work?
data data data Accuracy
How do we measure page load time?
In the onbeforeunload event, measure the time and
store it in a cookie
In the onload event, check the cookie, and measure the
difference with the current time
We also make sure that the page that set the cookie is the
referrer of the current page
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
36. Why measure? What?
boomerang How does it work?
data data data Accuracy
What? Two pages?
Yes, this needs two pages and cookies. If those aren’t
supported, we try to use the WebTiming API.
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
37. Why measure? What?
boomerang How does it work?
data data data Accuracy
How accurate is it?
Latency measurements are very accurate (±1%)
Bandwidth is to an order of magnitude. For bad
connections can be ±30%
Page load time sometimes has outliers, you need
post-filtering
The margin of error tells you how good your data is
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
38. Why measure? Filtering
boomerang Grouping
data data data More...
What do we do with the data?
Sanity checking to:
Remove fake data
Remove abusive data
Maybe just rate limiting
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
39. Why measure? Filtering
boomerang Grouping
data data data More...
What do we do with the data?
Statistical analysis to:
Remove outliers
Aggregate based on bandwidth blocks
Measure trends over time and correlate them with code
changes
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
40. Why measure? Filtering
boomerang Grouping
data data data More...
Bandwidth slabs
0-100 kbps
100-300 kbps
300-2000 kbps
2-6 Mbps
6+ Mbps
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
41. Why measure? Filtering
boomerang Grouping
data data data More...
Bandwidth slabs
Group page load times based on bandwidth slab
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
42. Why measure? Filtering
boomerang Grouping
data data data More...
Bandwidth slabs
Data points from some countries may require smaller slabs
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
43. Why measure? Filtering
boomerang Grouping
data data data More...
Geographic data
Looking at latency from different geographic locations can tell
you where to put your next CDN
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
44. Why measure? Filtering
boomerang Grouping
data data data More...
ISPs
Grouping data by ISP can tell you who’s behaving badly
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
45. Why measure? Filtering
boomerang Grouping
data data data More...
More data
Write plugins to get more performance data
We already have a DNS plugin
I’m thinking of an IPv6 v/s IPv4 plugin
What about a full WebTiming plugin?
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
46. Why measure? Filtering
boomerang Grouping
data data data More...
You decide
Once you have the data, you can do anything with it
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
47. Why measure? Filtering
boomerang Grouping
data data data More...
Thank you
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
48. Why measure? Filtering
boomerang Grouping
data data data More...
Photo credits
flickr.com/photos/21233184@N02/4389412851
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
49. Why measure? Filtering
boomerang Grouping
data data data More...
Contact me
Philip Tellis
yahoo
geek
@bluesmoon
http://bluesmoon.info/
slideshare.net/bluesmoon
philip@bluesmoon.info
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
50. Why measure? Filtering
boomerang Grouping
data data data More...
References
github.com/yahoo/boomerang
More bandwidth doesn’t matter (much) – Mike Belshe
Analysing Bandwidth & Latency – YUI Blog
It’s the latency, stupid – Stuart Cheshire
The statistics of web performance
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang