Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1v52uOK. Nellwyn Thomas discusses how Etsy is using A/B testing, how Etsy's data-driven culture has evolved over time and how continuous delivery and big data can coexist. Filmed at qconnewyork.com. Nellwyn Thomas is Director of Analytics at Etsy. She leads a team of data analysts who partner with product, marketing, and engineering to scout, build, instrument and improve Etsy's product portfolio. Before Etsy, she worked on analytics and product teams at two small NYC start-ups. She is a graduate of Harvard College, and holds a master's degree from the University of Pennsylvania.

1. Continuous delivery,
A/B testing + everyday
launch decisions at Etsy
QCon NY 2014
Nellwyn Thomas
Director of Analytics, Etsy
nellwyn@etsy.com
@nellwyn

2. Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/ab-testing-continuous-delivery
InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. etsy, briefly
deploying
experimenting
making decisions, everyday
takeaways
!

5. etsy, briefly
deploying
experimenting
making decisions, everyday
takeaways

7. The world’s handmade and vintage marketplace.

14. Over 25M items from 1M shops in 150 countries.

15. $1.35B sales in 2013
!

16. 30 deploys by 200+ people per day
!
!
!
!
This is continuous delivery at Etsy

17. 3 dozen experiments running simultaneously
!
!
!
!
This is experimentation at Etsy

19. etsy, briefly
deploying
experimenting
making decisions, everyday
takeaways

21. Basic principles
• A single source repository
• Automate the build
• Automate the testing
• Automate deployment
• Always ready to deploy quality software

25. Why continuous delivery?
• Move faster
• Fail faster
• More empowering
• Breeds confidence

26. Why continuous delivery?
“Because you’re integrating so frequently,
there is significantly less back-tracking to
discover where things went wrong, so
you can spend more time building
features.”
! http://www.thoughtworks.com/continuous-integration

29. config.php
template.tpl

32. CI Status (1 hour)
End-User (1 hour)
Error Logs (All) (1 hour)
..................................................................
Deploy Dashboard Git Commits Princess Supergrep Supergrep Forum Watcher
Time: 1 hour Until: Now Hide Deploys Deploys hidden: Config Web / Api / Atlas Search Photos Blog Chef (prod) Schemanator
Push
rbateman
on qa
6m since 7, 2012 9:54
Three-Armed Sweaters
error pages · historical
clip Akamai Whoopsies
origin failures · historical
clip Edgecast Whoopsies
origin failures · historical
clip Screwed Users
unique users · historical
clip
All
File or Script Not Found
current · historical
clip Exceptions
current · historical
clip Fatals
current · historical
clip Errors
current · historical
clip Warnings
current · historical
clip
Segmentation Faults
current · historical
clip PHP Allowed Memory
Size Exhausted
current · historical
clip PHP Maximum Execution
Time
current · historical
clip Memcache Connection
Failures
current · historical
clip Thrift Exceptions
current · historical
clip

33. Segmentation Faults
current · historical
clip PHP Allowed Memory
Other Errors (1 hour)
Size Exhausted
current · historical
Business Graphs (1 hour)
clip PHP Maximum Execution
Time
current · historical
clip Memcache Connection
Failures
current · historical
clip Thrift Exceptions
current · historical
clip
StatsD
packet receive errors · historical
clip CSRF Mismatch
current · historical
clip CSRF Missing
current · historical
clip Client Side (JS) Errors
current · historical
clip Empty Image URLs
grey.gif · Default Avatars
clip
Registrations
facebook · facebook (historical) · other · other (historical)
clip Logins (82.00%)
logins · logins (historical) · errors · errors (historical)
clip Checkouts (97.81%)
success · failure
clip
Post Checkouts
current · historical
clip Mobile Checkouts
iOS · mobile web
clip Direct Checkout Checkouts (98.00%)
success · failure
clip

34. Post Checkouts
current · historical
clip Mobile Checkouts
iOS · mobile web
Page Performance by Site Section (1 hour)
clip Direct Checkout Checkouts (98.00%)
success · failure
clip
Listings Creation/Renewal
New & Copied · Renewed
clip Requests
logged-out · logged-in
clip CDN Requests Balance
akamai · edgecast · direct-origin
clip
Search Ads
Requests · Responses · Logged
clip Forum Posts
bugs forum · help forum
clip Seller Registration (100.00%)
success · failure
clip
PHP time (median)
listing · shop · profile · search · homepage
clip PHP time (95th percentile)
listing · shop · profile · search · homepage
clip webpagetest front-end time
listing · shop · profile · search · homepage
clip lognormal load time
listing · shop · profile · search · homepage
clip

35. Page Performance by Site Section (1 hour)
PHP time (median)
listing · shop · profile · search · homepage
Statistics displayed represent signed in page requests. See Page Performance Dashboard for more detail.
Web Cluster (1 hour)
API Cluster (1 hour)
clip PHP time (95th percentile)
listing · shop · profile · search · homepage
clip webpagetest front-end time
listing · shop · profile · search · homepage
clip lognormal load time
listing · shop · profile · search · homepage
clip

36. API Cluster (1 hour)
Gearman (1 hour)
Memcache (1 hour)

37. Memcache (1 hour)
Master01 Database Activity (1 hour)
DBShard Database Activity (1 hour)

38. DBShard Database Activity (1 hour)
For more detailed shard information see the MySQL Shards Dashboard
DBIndex Database Activity (1 hour)

39. DBIndex Database Activity (1 hour)
Outgoing Bandwidth (1 hour)

40. etsy, briefly
deploying
experimenting
making decisions, everyday
takeaways

41. Experiments are easy
• Experiments need large samples, internet “big
data” is all about large samples
• Visitors are free
• Websites are built for iterative change, an
experiment is an iterative change

42. And they help.
• Fail faster
• Fights confirmation bias
• Forces you to tie product to goals (hypotheses!)
• Helps you understand your users

43. Why experiment?
“Because you’re integrating so frequently,
there is significantly less back-tracking to
discover where things went wrong, so
you can spend more time building
features.”
! http://www.thoughtworks.com/continuous-integration

44. How do experiment?
exactly the same way we deploy.
!

47. Bucketing
listing
page
cart
page checkout

49. Logging
hadoop cluster
mysql db
internal
dashboard

51. reusing config,
automated bucketing,
automated analysis & graphing
!
with no additional effort
!

52. on
off

53. etsy, briefly
deploying
experimenting
making decisions, everyday
takeaways

54. Buy Sell Registry Community Blogs Mobile Gift Cards Help
Reeggiisstteerr Siiggnn IInn Search for items and shops Seeaarrcchh Caarrtt
Home › BayTowneLeatherUSA › IPAD Cases
Leather Ipad Sleeve/ Case/ Envelope/ Folio for IPAD 2 or 3. Full
Grain Veg Tan Leather. Handmade by BayTowneLeatherUSA.
This luxurious/rugged and well built IPAD case will protect your IPAD in style. Or you can use
it as a folio to carry your necessities . Luxurious enough for use at the office, and rugged
enough for use on the trail.
Made of beautiful - rugged 5/6 oz. Premium veg tan leather (which will break in beautifully) ; it
will develop a wonderful natural patina over time as only full grain veg tan leather can do.
(Gets better with age)
***I make these cases ONE BY ONE as orders are received***
Only 1 available
Like this item?
Add it to your favorites Favorite to revisit it later.
$74.00 USD
Add to Cart
Favorite
Shop
BayTowneLeatherUSA
BayTowneLeatherUSA
Add shop to favorites
See who favorites this shop
38
items
Shop owner
Roger
Marion, Illinois, United States
Favorites
Circle
Feedback: 27, 100% pos.
Contact
Item
Add item to favorites
See who favorites this item

58. All credit: John Allspaw

59. All credit: John Allspaw

60. All credit: John Allspaw

61. All credit: John Allspaw

62. All credit: John Allspaw

63. All credit: John Allspaw

64. All credit: John Allspaw

65. All credit: me
change
time
direction

69. etsy, briefly
deploying
experimenting
making decisions, everyday
takeaways

70. validate early
and often.

71. leverage
existing
workflows.
“Changing behavior works only if it
can be based on the existing ‘culture.’”
Peter Drucker

72. don’t confuse
conflate
iteration with
“tweaking pixels”
the importance
of the change
with the size of
the change

73. More Goodness
John Allspaw on CD
http://www.kitchensoap.com/category/
talks/
!
Mike Brittain on CD
www.mikebrittain.com/blog/talks/
!
Dan McKinley on CE
http://mcfunley.com/design-for-continuous-
experimentation
!

74. The End
Learn more
etsy.com/careers
github.com/etsy
!
Nellwyn Thomas
nellwyn@etsy.com
@nellwyn
!
!

75. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/ab-testing-
continuous-delivery