Web PerformanceCulture and Toolsat EtsyMike
BrittainDir. of Engineering, InfrastructureEtsyBoston Web Performance MeetupAug 24, 2011

OverviewEtsy and EngineeringMake PerformanceMatterTools and
Process

http://qacreate.etsy.com

MonthlyWeather Report$38 MM in sales1.9
MM items soldhttp://etsy.me/weather-report-june-2011

MonthlyWeather Report990 MM page viewshttp://etsy.me/weather-report-june-2011

EngineeringTechnologiesLinux, Apache, MySQL,PHP, MemcacheSolr, Squid,
Hadoop, Amazon S3, EC2, EMRGanglia, Cacti, Nagios, Graphite, Splunk, andsome of our own...

EngineeringTeams90 Engineers3-6 engineers+ product developer
and/or designer

EngineeringContinuousDeployment~40 releases per day inc.
app code and config changes

1. Write code2. Code review3.
Automated tests

1. Write code2. Code review3.
Automated tests4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod

1. Write code2. Code review3.
Automated tests4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod5. Monitor!6. Monitor!7. Monitor!

EngineeringData-DrivenDevelopment45,000+ metrics50+ dashboards

MakePerformanceMatterHave a story

Business ImpactMeasure“Our own years of
testing have conclusively shown that when speed of a feature or product improves, usage, quite simply, goes up.” ~GoogleAbundance of research from Google, Bing,AOL, Amazon, Shopzilla, etc.http://googleblog.blogspot.com/2009/12/this-week-in-search-121809.html

Business ImpactMeasureBounce rateSearch conversion

Business ImpactMeasurePurchase funnelAd impressions, page
views andtracking discrepanciesSocial, engagement

OperationsSite StabilityMeasure perf for discrete
piecesof your infrastructureContention for shared resources likedatabase, memcache, solr, or even webserver processes

MoraleHappy Engineers

Getting to workWhere toStart?Focus your
efforts where it makessense for your business

Getting to workWhere toOptimize?Tiers of
serviceSLAs

Getting to workFocusBeware of CTS:Constant
Tweaking Syndrome

Getting to workFocusFriends don’t let
friendstweak without graphing

MakePerformanceMatterHave your story

Tools andProcess atEtsy

ProcessEtsy’sPerf TeamStandardize patternsCreate tools and
reportsCoordinate efforts

ProcessServer-sidePerformanceThere is nothing worsethan a
blank page or a spinningthing-a-ma-jigger

ProcessServer-sidePerformance95th Percentile> 800 ms is
a P2 bug

ProcessServer-sidePerformance

ToolsLoggingPerf-related data belongs inyour server
logsapache_note()

ToolsLogging$timer_start = microtime(true); ...$timer_diff =
microtime(true) - $timer_start;register_shutdown_function()apache_note(php_microsec, $timer_diff);

ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t
"%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n %{php_microsec}n%D" combined

ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t
"%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n %{php_microsec}n%D" combined

ToolsLoggingLogFormat "%{True-Client-IP}i %l %u %t
"%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n %{php_microsec}n%D" combined

ToolsLoggingweb0060 66.249.71.110 - - [24/Aug/2011:04:16:52+0000]
"GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - 13399576 505780554876

ToolsAnalyzegrep ... access.log | awk
...grep "/listing/" access.log | awk {sum=sum+$(NF-1)} END {print sum/NR}grep "/listing/" access.log | awk BEGIN {max=0} {if ($(NF-1)>max) max=$(NF-1)}END {print max}

ToolsAnalyze

ToolsAnalyze1. Capture perf data in
logs page gen., Boomerang, Gomez2. Aggregate Splunk, Logster, StatsD3. Record in Graphite http://graphite.wikidot.com

ToolsAnalyze

ToolsLogster1. Read new log entries
every minute2. Parse and aggregate useful numbers3. Send to Graphite http://github.com/etsy

web0001 [04:28:54 2011] [error] [client
10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0201 [04:28:54 2011] [error] [client 10.101.x.x] Youve been eaten by a gweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0201 [04:28:54 2011] [error] [client 10.101.x.x] Youve been eaten by a gweb0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo

Fatals Errors Warnings

:)

ToolsStatsDCollects metrics from your appcode
and sticks them in GraphiteStatsD::increment("logins.success");StatsD::timing("gearman.time", $msec);

ToolsStatsD 90th pct average lower

:) :)

ToolsPerf Dashboard

ToolsI/O ProfilerLightweight, inline profilingStart and
end times wrappedaround service calls databases, memcache, apc, etc.

ToolsI/O Profiler

ToolsI/O Profiler

ToolsClient-sideTestingGomez (API), Boomerang,WebPagetest

ToolsShowSlowTracks front-end best practices

ToolsShowSlowInternal instanceAutomated testingEnd-point for I/O
ProfilerTrending on individual rules http://showslow.com

Other thoughtsYSlow &Page SpeedMind your
development process3rd-party content and ga.js“Use a CDN”

Other thoughtsDevice-SpecificDesignThe mobile web is
very muchabout designing for performanceScreen size, pixel density, connection speed,wi-fi vs. cellular, browser cache size, localstorage, connections per host, meteredpricing, etc.

MakePerformanceMatterHave your storyUse tools and
processto focus your efforts

We areHiringhttp://etsy.com/careersSoftware Engineering positions available
ina number of teams, including Analytics,Operations, Web Performance, Payments,Core Platform, Front-End, Internal Apps,Search, Security, and more...

ThankYouMike Brittainmike@etsy.com@mikebrittainCodeAsCraft.etsy.comgithub.com/etsygraphite.wikidot.comshowslow.com