Given at Data Day Texas 2016. Apache Spark has been hailed as a trail-blazing new tool for doing distributed data science. However, since it's so new, it can be difficult to set up and hard to use. In this talk, I'll discuss the journey I've had using Spark for data science at Bitly over the past year. I'll talk about the benefits of using Spark, the challenges I've had to overcome, the caveats for using a cutting-edge technology such as this, and my hopes for the Spark project as a whole.

1. Spark:
The Good, the Bad, and the Ugly
Sarah Guido
Data Day Texas 2016

2. About
this talk
About me & Bitly
Spark journey to date
The future!
The good, the bad, and the ugly

3. About this talk
This talk is
• My own experience using Spark
• New toys come with caveats
This talk is not
• Ground truth
• An argument for or against using Spark

4. About me!
• Lead data scientist at Bitly (bitly.is/hiring)
• NYC Python meetup organizer
• Spark user
• @sarah_guido

5. SPARK JOURNEY

6. The stage
“Don’t you guys just, like, shorten links?”

7. The stage

8. The stage
• Need for big data analysis tools
• Iterate/prototype quickly
• Overall goal: understand how people use not
only our app, but the Internet!

9. 1 HOUR OF DECODES
10 GB
1 DAY OF DECODES
240 GB
1 MONTH OF DECODES
~7 TB

10. Data
{"a": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/600.4.10 (KHTML, like Gecko) Version/8.0.4
Safari/600.4.10",
"c": "US",
"nk": 0,
"tz": "America/Los_Angeles",
"g": "1HfTjh8",
"h": "1HfTjh7",
"u": "http://www.nytimes.com/2015/03/22/opinion/sunday/why-health-
care-tech-is-still-so-bad.html?smid=tw-share",
"t": 1427288425,
"cy": "Seattle"}

11. Spark: the Why
• Fast. Really fast.
• SQL layer – kind of like Hive
• Distributed scientific tools
• Python! Sometimes.
• Cutting edge technology

12. Spark: the What
• Large-scale distributed data processing tool
• SQL and streaming tools
• Faster than Hadoop
• Python, Scala, Java, R APIs

13. Spark: the How
• Partitions your data to operate over in parallel
• Capability to add map/reduce features
• Lazy – only operates when a method is called (ex.
collect()/or writing to file)

14. First exploration of Spark during hack week
SPARK 1.2
DataFrames, AWS config
SPARK 1.3
Decision to use Scala, official AWS support
SPARK 1.4
Prototype of trend detection
SPARK 1.5

15. Spark environment
• From Hadoop to AWS
• Runs on AWS EMR clusters
• Reads data from S3
• Special config

16. THE GOOD

17. Speed
Both development
time and data
extraction
Hadoop vs. Spark

18. Speed

19. Easy to prototype
Building a working Spark program is relatively
simple.*
* In simple circumstances.

20. Integration with Jupyter Notebook

21. THE BAD

22. Updates are major and frequent

23. AWS Spark version lag

24. Documentation
• Lack of project maturity
• Simple examples
• Books end up being obsolete
• Can’t just Stackoverflow it!

25. THE UGLY

26. DataFrames
DataFrames are appealing to data scientists
familiar with Python/Pandas, but their lack of
flexibility makes them difficult to use.

27. Python vs. Scala: the Saga

28. Python vs. Scala: the Saga
Python (I know Python!)
Scala (Python API lacks parity with Scala)
Python (But I KNOW Python and can code faster!)
Scala (Lack of parity, weird undebuggable Python errors?!)

29. Python vs. Scala: the Saga

30. THE FUTURE

31. My hopes
• Python catches up – barrier to entry
• Community (and therefore knowledge base)
grows
• More people talk about using Spark
• Best practices
• Used in projects/stack at Bitly

32. I want to try
• Layer between NSQ and Spark – Cassandra?
• Tool like H2O
• Zeppelin (again)
• Spark slack channel?

33. Resources
• spark.apache.org - documentation
• Databricks blog
• Cloudera blog
• Other Spark users

34. THANK YOU
@sarah_guido