Growing a Data Pipeline for Analytics

1. Growing a Data Pipeline
for Analytics
Roberto Vitillo, Staff Data Engineer @ Mozilla
26th PyData London Meetup

4. brew install apache-spark

6. Don’t do it yourself!

7. Input OutputETL
Storage

8. JSON
JSON?

13. JSON
Parquet
Spark,
Hive, Pig …

14. JSON
Parquet
Spark,
Hive, Pig …
???

15. “The easier it is to ask questions,
the more questions will be asked”

17. Modern SQL supports Map,
Arrays & Structs

19. JSON
Parquet
Spark,
Hive, Pig …
Presto,
Re:dash

20. TLDR;
• Don’t build your own pipeline unless you
really have to
• Use schemas
• Exploit columnar storage
• Use SQL