Big data at Etsy began in early 2010 and has since grown to power applications as diverse as ETL, A/B testing, recommender systems, and search indexing. Join us at this talk for an amusing tour through the history of big data at Etsy going back to the roots of our mission-critical A/B testing approach followed by a dive into a selection of the technologies that power such applications today.

1. Patchwork Data at Etsy
Matt Walker

4. Etsy
June
2005 2007 2009 2011 2013

5. What happened?

6. We don’t like to talk about it

7. Okay, we do
• http://codeascraft.etsy.com
• https://www.etsy.com/codeascraft/talks
• http://kongscreenprinting.com

8. Catch Phrases
• Continuous deployment
• Blameless postmortems
• Measure everything
• Continuous experimentation

9. Metrics-Driven Development
• Ganglia
• StatsD/Graphite
• Splunk

10. Scaling a Traditional RDBMS
• Sharded MySQL
• memcached
• Object-relational mapping in PHP

11. December
2005 2007 2009 2011 2013

12. Adtuitive
• Online advertising network
• Match forum post with rich product advertisements
• Unafraid of scaling across Etsy sellers

13. Adtuitive
• Amazon Web Services
• JRuby
• Rails

15. LAMP Stack for Big Data
• HDFS • Pig
• MapReduce • Oozie
• HBase • Avro
• Hive • Zookeeper
• Flume
• JDBC/ODBC http://gigaom.com/2010/08/01/meet-big-data-equivalent-of-the-lamp-stack/
• Hue

16. LAMP Stack for Big Data
• HDFS • Pig
• MapReduce • Oozie
• HBase • Avro
• Hive • Zookeeper
• Flume
• JDBC/ODBC http://gigaom.com/2010/08/01/meet-big-data-equivalent-of-the-lamp-stack/
• Hue

17. LAMP Stack for Big Data
• HDFS S3 • Pig Cascading
• MapReduce (Elastic) • Oozie
• HBase • Avro TupleSerialization
• Hive • Zookeeper
• Flume
• JDBC/ODBC
• Hue

18. Powered by MapReduce
• ETL
• Analytics
• A/B testing
• Recommenders
• Search

19. Applications
• Log ETL • A/B Analyzer
• Database snapshotter • Catapult
• TasteTest • Distributed search indexing
• Facebook Gift Recommender • Fast Game (search index)
• Complimentary/similar listings • Search autosuggest
• Funnel Cake • SearchAds
• Feature Funnel • SCRAM ETL (fraud detection)

20. Applications
• Log ETL • A/B Analyzer
• Database snapshotter • Catapult
• TasteTest • Distributed search indexing
• Facebook Gift Recommender • Fast Game (search index)
• Complimentary/similar listings • Search autosuggest
• Funnel Cake • SearchAds
• Feature Funnel • SCRAM ETL (fraud detection)

21. Catapult
• End-to-end success story
• Extremely valuable for a web shop

22. Relevancy Thursdays
January
2005 2007 2009 2011 2013

23. Relevancy Thursdays
• Switch default sort order to relevance
• Each Thursday in January

24. Relevancy Thursdays
• Default search order was recency
• Relisting was our equivalent of advertising
• $0.20 updated your listing’s timestamp

25. Relevancy Thursdays
• Recency was meant to support “freshness” in search results
• Search originated as PostgreSQL query
• Converted to Solr to scale

26. What happens if we switch to
relevance?

27. Relevancy Thursdays
• No A/B testing framework
• No event logs
• Limping along with Google Analytics

30. First Log Analysis
February
2005 2007 2009 2011 2013

31. First Log Analysis
• Raw web access logs
• URL- and ref tag-based
• Regex parser

34. Heyday of Tooling
• A/B framework
• Front end event logger
• Database snapshotter
• Barnum and Bailey
• Custom operator library
• Loaders

35. LAMP Stack for Big Data
• HDFS S3 • Pig Cascading
• MapReduce (Elastic) • Oozie
• HBase • Avro TupleSerialization
• Hive • Zookeeper
• Flume
• JDBC/ODBC
• Hue

36. LAMP Stack for Big Data
• HDFS S3 • Pig Cascading
• MapReduce (Elastic) • Oozie Barnum
• HBase • Avro TupleSerialization
• Hive • Zookeeper
• Flume Akamai
• JDBC/ODBC snapshotter/loaders
• Hue

37. A/B Framework
• Ramp-ups + A/B testing
• Feature flag development

38. Self-service analytics for any A/B
test on the site

39. A/B Framework
June
2005 2007 2009 2011 2013

40. A/B Analyzer
November
2005 2007 2009 2011 2013

41. Why did it take so long?
• Non-web developers learning the PHP stack
• Failed experiments with “easier to use” MapReduce tools
• Realizing self-service analytics was what Etsy needed

44. Catapult
February
2005 2007 2009 2011 2013

45. Catapult
• A/B Analyzer + Launch Calendar
• Full product lifecycle

49. LAMP Stack for Big Data
• HDFS S3 • Pig Cascading
• MapReduce (Elastic) • Oozie Barnum
• HBase • Avro TupleSerialization
• Hive • Zookeeper
• Flume Akamai
• JDBC/ODBC snapshotter/loaders
• Hue

50. LAMP Stack for Big Data
• HDFS • Pig Cascading
• MapReduce • Oozie
• HBase • Avro TupleSerialization
• Hive Vertica • Zookeeper
• Flume logrotate
• JDBC/ODBC snapshotter/loaders
• Hue

51. Computation Models
• Batch
• Interactive
• Streaming

53. Batch

54. Cascading

55. RDBMS / Cascading
SQL cascading.jruby
Query Planner/Optimizer Cascading
Execution Engine MapReduce
Storage HDFS

56. cascading.jruby

57. cascading.jruby
• Productivity: no compile
• Reuse: factor out structure
• Efficiency: no JRuby runtime
• Optimization: move aggregations map-side

58. A nice constructor

59. cascading.jruby

60. Productivity
• Job templates
• Reloader
• Cascading local mode
• Sampled data

61. Reuse

62. Reuse

63. Field Names

65. Efficiency
• Just a constructor
• Calls into Cascading API
• No JRuby runtime on cluster

66. Optimization

67. Tuple Data Model

68. UDFs

69. Scalding
• Distributed collections
• Function literals replace UDFs

73. Interactive

74. Vertica

75. Sharded MySQL
• Borrowed from Flickr
• Works

76. Thou Shalt Not Join

77. Hive
January
2005 2007 2009 2011 2013

78. Hive Turned Off
April
2005 2007 2009 2011 2013

79. Hive
• Slow
• Sensitive
• Operational burden
• Educational burden

80. Vertica
• Offline copy of shards, master, auxiliary databases
• Joins are easy
• Reasonable latency

81. Vertica
November
2005 2007 2009 2011 2013

82. Vertica
• Game changer at Etsy
• High demand for joins
• Rapid prototyping data pipelines

84. RDBMS / Cascading
SQL cascading.jruby
Query Planner/Optimizer Cascading
Execution Engine MapReduce
Storage HDFS

85. Back to MapReduce
• Event logs
• Schedule
• Load data in prod
• Scale

86. Vertica
• Not Hive, Impala, Shark, etc.
• May change our minds

87. Streaming

88. Not Powered by MapReduce
• Activity Feed
• Shop Stats

89. Etsyweb
• memcached
• Gearman
• Sharded MySQL

90. Usecases
• Trending
• Fraud detection
• ?

92. Turns out people don’t make
product decisions in real time
http://mcfunley.com/whom-the-gods-would-destroy-they-first-give-real-time-analytics

93. Summing Up
• Be glad you’re living in the future
• Automated tools for the common case
• Don’t be afraid to experiment

94. Image Credits
• http://kongscreenprinting.com/what-we-do- • http://www.globaltimes.cn/
showcase SPECIALCOVERAGE/Top10Peopleof2011.aspx
• http://animal.discovery.com • http://www.theculturemap.com/scream-time-
edvard-munch-museum/
• http://www.rallyrace.com/turning-over-the-
stone-event-production-basics/ • http://www.repentamerica.com/webelieve.html
• http://www.flickr.com/photos/bbalaji/ • https://soundcloud.com/tearland/tl-hive
2443820505/
• http://pocketnow.com/2012/08/02/wifi-vs-data-
• http://www.madeyoulaugh.com/funny_photos/ speed-vs-battery-life/bush-scratching-head
caveman_harley/caveman_harley.jpg
• http://theundercoverrecruiter.com/6-ways-
catapult-your-job-search-after-layoff/

95. Contact / Reference
• Matt Walker
• @data_daddy
• http://codeascraft.etsy.com/
• http://www.etsy.com/codeascraft/talks