The document discusses the challenges faced by Shopify in using its existing data warehouse and ETL processes due to increasing data volume and complexity. It describes Shopify's attempts to use Pig and Luigi as well as Platfora to address these issues, but notes they did not meet Shopify's needs. Shopify then moved to using Spark due to its fast performance, nice development model using Python, and ability to better handle their data and query complexity. The summary provides an overview of why Shopify changed its data warehousing approach and the key technology it adopted.

1. The Little Warehouse That
Couldn’t Or: How We Learned to
Stop Worrying and Move to Spark
1
Yandu Oppacher (@yandu)
Data Infrastructure

2. 2
Shopify Stores

3. ETL Warehouse Reporting
August 2013
TilllerRuby Vertica
3

4. Why we had to move
• Data volume
• Data/Query complexity
• Performance issues
4

5. Couple of false starts
5
Pig + Luigi
Pig + Oozie
Platfora

6. –platfora.com
“Without coding or ETL, data
warehousing, BI tools, or breaking a
sweat.”
6

7. Enter Spark
• Fast
• Nice development model
• Python
7

8. 88
The Good Book

9. GMV
A Case Study
9
165,000+
ACTIVE SHOPIFY MERCHANTS
$8 BILLION+
CUMULATIVE GMV

10. Growing pains
• Joins
• Groupings
• General data skew
• Getting to know python’s performance quirks
10

11. Starscream
11
• specialized joins
• resolvers
• range
• cassandra
• overby
• contracts
• incrementalized fact
builds

12. Our current stack
12
Kafka
OLTP
HDFS
Cassandra
Spark
FrontroomBackroom
Redshift
Tableau

13. Thank you
13
Yandu Oppacher (@yandu)
Data Infrastructure