Slides from the Apache Spark Meetup in Melbourne discussing Spark DataFrames and Machine Learning.

1. Melbourne / April 2015
Introducing DataFrames
R on Spark
Large Scale Machine Learning
Speakers
Ned Shawa
Mark Moloney
Dr. Zhen He

2. Agenda
• Apache Spark 1.3 Overview 6:15 – 6:30 (Ned)
• DataFrames for Apache Spark 6:30 – 7:00 (Ned)
• R on Apache Spark 7:00 – 7:30 (Mark)
• Large Scale Machine Learning on Spark 7:30 – 8:15 (Zhen)

3. News
• Hadoop + Strata
• Jobs
• Meetup Update
• Personal Announcement
• Call for Contribution

4. Contributions so far…..
• Mark with R on Spark, Scala 101
• Tim with Building Spark on IDEs
• Con with building Spark with Gradle
• More?

5. Whats new in Spark 1.3
• Multi Level Aggregation Trees
• Improved Error reporting
• SSL Encryption for Control messages and WebUI
• DataFrames API
• Backward compatibility for HIVE
• Writing data in source
• JDBC Driver
• New Algorithms for MLLIB
• Direct KAFKA API

6. More Data Sources APIs
04/13/15

7. What are DataFrames?
• Distributed Collection of Data organized in Columns
• Equivalent to Tables in Databases or DataFrame in R/PYTHON
• Much richer optimization than any other implementation of DF
• Can be constructed from a wide variety of sources and APIs

8. Writing a DataFrame
val df = sqlContext.jsonFile("/home/ned/attendees.json")
df.show()
df.printSchema()
df.select ("First Name").show()
df.select("First Name","Age").show()
df.filter(df("age")>40).show()
df.groupBy("age").count().show()

9. DataFrame with RDD
case class attendees_class (first_name: String, last_name:String, age:Int)
Val attendees=sc.textFile("/home/ned/attendees.csv").map(_.split(",")).map(p=>attendees_class(p(0),p(1),p(2).trim.toInt)).toDF()
people.registerTempTable("attendees")
val youngppl=sqlContext.sql("select first_name,last_name from attendees where age <35")
youngppl.map(t=>"Name: " +t(0)+ " " + t(1)).collect().foreach(println)

10. DataFrames and Parquet
attendees.saveAsParquetFile("/home/ned/attendees.parquet")
val pfile = sqlContext.parquetFile("/home/ned/attendees.parquet")
pfile.printSchema()
pfile.registerTempTable("attendees_parquet")
val old_ppl=sqlContext.sql("select first_name,last_name,age from attendees_parquet where age >=35 order by age desc")
old_ppl.map(t=>"Name: " + t(0)+" "+t(1)+ " Age " +t(2)).collect().foreach(println)

11. DataFrames and JDBC
val jdbc_attendees = sqlContext.load("jdbc", Map("url" -> "jdbc:mysql://localhost:3306/db1?
user=root&password=xxx","dbtable" -> "attendees"))
jdbc_attendees.show()
jdbc.attendees.count()
jdbc_attendees.registerTempTable("jdbc_attendees")
val countall = sqlContext.sql("select count(*) from jdbc_attendees")
countall.map(t=>"Records count is "+t(0)).collect().foreach(println)

12. Dataframes for Apache Spark

13. Spark Components

14. Introduction to SparkR
Mark Moloney
April 2015
https://github.com/markmo/sparkr-meetup-sparkr-demo

15. R
• A language that targets statistical and general data analysis
• A package for nearly everything in this space
• Great for exploratory analysis – rapid statistics and plots
• Single threaded
• Datasets limited to memoryTool Selection Primary Analytic Tool
18%
15%
Cost is important
A
B
E
Ease-of-use
& interface
Ability to
write one’s
Everything is important
Data miners are a diverse group who are looking for
different things from their data mining tools. They report
using multiple tools to meet their analytic needs, and
even the most popular tool is identified as their primary
tool by just 24% of data miners. Over the years, R and
Rapid Miner have shown substantial increases.
Cluster analysis* reveals that, in their tool-selection
preferences, data miners fall into 5 groups. The primary
dimensions that distinguish them are price sensitivity and
code-writing / interface / ease-of-use preferences.
2013 Rexer Analytics Survey of 1,259 analytics professionals from 75 countries.

16. Spark
• An evolutionary step up from Map-Reduce programming on Hadoop
• Do more with less work
• Simpler API than Map-Reduce. Apply functional transforms to datasets. The
framework takes care of distribution of work across multiple machines.
• Can cache interim results in memory, which speeds up iterative procedures

17. SparkR
• An R API to Spark’s RDDs (Resilient Distributed Datasets)
• Work with massive datasets in R
• Works on top of YARN or Mesos
• Interactively run jobs on a cluster from the R shell
• Exposes the RDD API of Spark as distributed lists in R
• Packages and ships variables in the closure to each node
• Use includePackage to include third-party packages on
other nodes
• Currently only suports R lists and vectors. Data frame
support is in the works.
map / lapply
mapPartitions / lapplyPartition
groupByKey
reduceByKey
sampleRDD
collect
cache
textFile
parallelize
broadcast
includePackage

18. How does it work
• The R executable must be installed on each node
• Work is sent to a Spark Executor (Java) on each node
• Some overhead in starting R interpreter – looking at background process as in PySpark
R
Java
Spark
Context
Java
Spark
Context
Java Native Interface (JNI)
using rJava
Local JVM
Local Machine Remote Machine
Remote Machine
Spark
Executor
Spark
Executor
Spark
Executor
Spark
Executor
R
R
tasks
tasks
broadcast vars, packages
broadcast vars, packages
Send R environment of vars used
Uses R save() function to serialize closure

19. Roadmap
• Feature-complete DataFrame API
• MLLib integration
DataFrame Methods
•Filter
– filter(df, df$col1 > 0)
•Sort
– sortDF(df, asc(df$col1), desc(df$col2))
•Join
– join(df1, df2, df1$col1 == df2$col2, "right_outer")
•GroupBy
– groupBy(df, df$col1)
•Agg
– agg(groupDF, sum(groupDF$col2), max(groupDF$col3))

20. Demos

21. Document Similarity Example
• Collection of inaugural speeches of US presidents
• Using Shingles and Jaccard similarity
– A k-shingle for a document is a sequence of k characters that appear in the document
– Example: k = 2; doc = abcab. Set of 2-shingles = {ab, bc, ca}
– Intuitively, documents that are similar will have many shingles in common
– Robust to small changes, e.g. reordering a paragraph only affects the 2k chingles that cross
paragraph boundaries
– Jaccard similarity is intersection / union
k = 6
doc1: “The cat sat”
doc2: “The cat ate”
‘The ca’
‘he cat’
‘e cat ‘
‘ cat s’
‘cat sa’
‘at sat’
‘ cat a’
‘cat at’
‘at ate’
doc1
1
1
1
1
1
1
0
0
0
doc2
1
1
1
0
0
0
1
1
1
characteristic matrix:
too large in real-world problems
therefore use minhashing

22. Machine Learning Example
Digit Classification (MNIST database)
Digit(Classification:(MNIST(
Digits consist of 784 pixel values
Training set: 60,000 images; Test set: 10,000

23. Citations
• http://amplab-extras.github.io/SparkR-pkg/ - main site
• http://ampcamp.berkeley.edu/5/exercises/sparkr.html - hands-on exercises
• http://spark-summit.org/2014/talk/sparkr-interactive-r-programs-at-scale-2 - MNIST example
• https://github.com/kiendang/sparkr-naivebayes-example - how to integrate with MLLib now
• https://github.com/cafreeman/Demo_SparkR - New API

24. 24
Spark Machine Learning
Experiences
Speaker: Zhen He

25. 25
Talk Outline
●
Spark is awesome!
●
Example of spark in action
●
Why spark is good for machine learning?
●
Big Data versus big parameters
●
What I wish you could do in spark
●
Competing scalable machine learning systems
●
Dispel some common myths about spark and
Scala
●
Current spark machine learning projects from our
group
●
Conclusion
●
Demonstration

26. 26
Spark is Awesome!

27. 27
Focus of Talk is on
Performance

28. 28
Example of Spark in
Action
●
Recently did some work with an Australian Government
Agency
●
Taught Mastering Hadoop and Spark course
●
Students loved it
●
Brain dead at the end 
●
Use Spark for Machine Learning
●
Really made excellent use of skills leant in the course for
their project.

29. 29
Summary
●
Initial R solution
●
Never finished
●
Approximate R + C single core solution
●
A couple of days
●
Accurate spark parallel solution
●
18 minutes
●
The model worked very well in initial
trial.

30. 30
Lessons Learnt
●
First reduce complexity of problem
●
Pre and post processing performance is
very important
●
Spark is excellent for both the modeling
and the pre and post processing

31. 31
Why Spark is good for machine learning
compared to MapReduce?
●
Machine learning algorithms are iterative
●
The following happens when using MapReduce
●
A lot of reading and writing to HDFS
●
The following happens when using spark
●
Reading and writing to RAM instead of HDFS
iter. 1iter. 1 iter. 2iter. 2 . . .
HDFS
read
HDFS
write
HDFS
read
HDFS
write
iter. 1iter. 1 iter. 2iter. 2 . . .
Input

32. 32
Make sure you actually cache the data!
val data = MLUtils.loadLibSVMFile(sc, ”datafile.txt")
val LR = new LogisticRegressionWithSGD()
LR.optimizer.setNumIterations(10)
val model = LR.run(data)
The above runs 10 x slower then the code below!
val data = MLUtils.loadLibSVMFile(sc, ”datafile.txt").cache
val LR = new LogisticRegressionWithSGD()
LR.optimizer.setNumIterations(10)
val model = LR.run(data)

33. 33
Big Data versus Big
Parameters
●
Spark is great at doing machine learning on Big Data as
long as the size of the parameters is small.
●
Big parameters is hard for Spark to handle efficiently.

34. 34
What is Big Parameters?
●
Big parameter is caused by high dimensional data
●
Modeling high dimensional data requires a large number
of parameters
● In the example above the parameters are the b0, b1, b2,
… bn

35. 35
Examples of High Dimensional Data
Time series
Text
Image
Speech

36. 36
Varying dimensionality (100 – 10, 000, 000)
●
Logistic regression on 30 cores, 10 iterations
●
Dimensionality shown in [ ]
●
The second number is the number of training instances
●
Total data size is constant
●
As dimensionality increases execution time increases significantly!
Time(secs)
Low dimensional data
High dimensional data

37. 37
Varying number of cores
●
For high dimensional data (10 million)
●
Using more cores actually slows execution!
●
1 core is the optimal!
●
For low dimensional data
●
Execution time decreases with the increase in the number of
cores.
Time(secs)
# cores
10 million dimensional data
# cores
Time(secs)
100 dimensional data

38. 38
Why is big parameters /
high dimensional data so
hard for spark?

39. 39
Why is big parameters hard for spark?
mini-batch
of data
Master node combine separate models
Broadcast combined model

40. 40
Reason 1: Requires all nodes to
synchronize
mini-batch
of data
Master node combine separate models
Broadcast combined model

41. 41
Reason 2: Inefficient memory usage
●
Although all cores working on same set of parameters
they all have their own copy
●
It would be nice if they can all share the same copy
RAM
Core 1 Core 2 Core 3 Core 4
within one node

42. 42
Parameters can be multi-GB is size
Reason 3: High shuffle cost
mini-batch
of data
Master node combine separate models
Broadcast combined model

43. 43
Reason 4: Master node is a bottleneck
mini-batch
of data
Master node
Broadcast combined model

44. 44
What I wish we could do on Spark
●
All machines share the same a single copy of the shared
model in the shared RAM.
●
All cores can update the model asynchronously.
●
No bottleneck master node
RAM
Core 1 Core 2 Core 3 Core 4
RAM
Core 1 Core 2 Core 3 Core 4
Node 1 Node 2
Parameter Server
Async update parameters Async update parameters

45. 45
Such a system already exists
●
Google’s DistBelief system
●
J. Dean, G.S. Corrado, R. Monga, K. Chen, M. Devin, Q.V. Le,
M.Z. Mao, M.A. Ranzato, A. Senior, P. Tucker, K. Yang, A. Y. Ng.
Large Scale Distributed Deep Networks. NIPS, 2012.
●
Closed source
●
Only people in Google can use it
●
Used to train deep learning system with 1 billion parameters on 16,
000 cores

46. 46
Comparison of some key machine
learning algorithms in Spark
●
Linear Models
●
Logistic regression
●
Linear regression
●
Support vector machines
●
Linear kernel
●
Random forest

47. 47
Logistic Regression, Linear Regression,
Linear Support Vector Machines (SVM)
●
Need to repeatedly merge parameters
●
High dimensional data => Big parameters => high communication
costs

48. 48
Random Forest
●
Can be trained very efficiently
●
Each task trains on the data of its own partition
●
Each tree can be trained separately
●
No communication or synchronization needed

49. 49
Competing Scalable Machine
Learning Systems
●
Greenplum
●
Machine learning via SQL
●
Mahout
●
MapReduce based
●
Moving to using spark as underlying engine
●
Vowpal Wabbit
●
Good performance
●
Specialized system
●
Graph Lab
●
Good performance
●
Need to turn everything into graphs

50. 50
Spark Misconceptions

51. 51
Common Misconception: Spark is
only good if data fits in RAM
●
From inception Spark was designed to be a
general execution engine that works both in-
memory and on-disk
●
Almost all operators perform external operations
when data does not fit in RAM.
●
Spark breaks large-scale sort record

52. 52
Results of Large-scale Sort
●
Spark is 3X faster than Hadoop using 10X fewer machines
●
Spark sorts 1PB in 4 hours on 190 machines
●
Compared to 16 hours for Hadoop using 3800 machines
●
All sorting on disk (HDFS) no use of Spark in-memory cache

53. 53
How did they do it?
●
More efficient Shuffle
●
Very efficient scheduling of tasks

54. 54
A little bit of RAM can go a long way
●
Performance degrades gracefully with decreasing RAM size

55. 55
A little bit of RAM can go a
long way
●
Just cache the small parts of the data that you will use again
●
Use filter and projection to reduce the amount that is cached.
●
Spark SQL stores data compressed in columns
●
Fast compression/decompression
●
Column-stores have been proven to be way better than row-
stores for analytics
●
100x better
●
For normal Spark code (non-SparkSQL), you can store data
serialized and compressed in RAM.
●
RAM is cheap now days
●
Aggregate RAM on large cluster can be very large

56. 56
Common Misconceptions
●
I need to learn a new language Scala
●
Scala is easy to learn
●
Can program in Java and Python
●
Distributed Dataframes
●
SparkR
●
I have to rewrite all my code
●
Spark will run MapReduce code unmodified
●
Spark SQL runs HiveQL

57. 57
Some Real Problems
●
Scala allocates a lot of objects
●
High GC overhead
●
The nice functional scala code is much slower than
writing C-styled while loops
●
For example:
●
val y = x.map(x => x * x)
●
is slower than
●
var i = 0
●
while (i < x.length) {
y(i) = x(i) * x(i)
i += 1
}

58. 58
Our Work

59. 59
Spark API Examples
●
Almost no good examples on the Spark API calls.
●
Matthias and I have written examples for over 110 Spark
API calls
●
http://homepage.cs.latrobe.edu.au/zhe/ZhenHeSparkRDDAPIEx
amples.html
●
Great reference
●
Currently over 33, 000 hits
●
Often top Google page returned when searching spark
API calls

60. 60
Deep Learning on Spark
●
Deep learning is currently very popular
●
State-of-the-art performance in area such as
●
Image recognition
●
Speech recognition
●
Natural language processing
●
etc.
●
No good existing distributed open source deep
learning implementation
●
We are implementing the first serious deep
learning implementation on Spark.
●
Implemented in Scala

61. 61
Features of our Open Source Spark
Deep Learning System
●
Stacked auto-encoder
●
Convolutional neural networks
●
RICA
●
Sparse coding
●
Fully connected networks
●
Many many different state-of-the-art optimization tricks
like
●
Dropout, reLU, adagrad, momentum, RMS prop,
etc.

62. 62
Other Current Projects
●
Zendesk
●
Australian institute of sports
●
Precision agriculture
●
More welcome

63. 63
Mastering Big Data Analytics with
Hadoop Course
●
3 day course on Hadoop and Spark
●
Only pre-requisite is Java programming experience
●
More than 35 programming exercises
●
Contents
●
Hadoop and MapReduce
●
Hive
●
Hadoop 2 ecosystem
●
Storm, Yarn, Giraph
●
Spark and Scala
●
Spark on Amazon

64. 64
Recruiting PhD Students
●
We have a lot of real world projects to do.
●
A lot of projects from industry
●
Zendesk
●
Australian Institute of Sports
●
Precision agriculture
●
UXC Professional Solutions
●
More welcome
●
We need to expand our research group.
●
Topics:
●
Text mining
●
Time series mining
●
Reinforcement learning combined with deep learning
●
Video mining
●
etc.

65. 65
Conclusion
●
Spark is best open source software for distributed
machine learning on Big Data
●
Be careful of using spark for high dimensional
data
●
Random forest does not suffer from
performance penalty
●
Spark programming using Scala is great for ease
of use
●
Spark is good even if data does not fit in RAM

66. 66
Demonstration
R versus Spark Fight!

67. 67
Score
R Spark
Round 1 Load + count 56.8s 10.6s
Round 2 Selection 2.5s 0.86s
Round 3 Sample 50% of data 7.1s 0.38s
Round 4 K means clustering (10
centers, 5 iterations)
100.0s 53.0s
● Results are for single core Spark versus Single core R
● Size of data is around 250 MB
● Data has 5 dimensions
● Clustering done on 3 dimensions

68. 68
Questions?
● Name: Zhen He
● Email Address: z.he@latrobe.edu.au