The state of analytics has changed dramatically over the last few years. Hadoop is now commonplace, and the ecosystem has evolved to include new tools such as Spark, Shark, and Drill, that live alongside the old MapReduce-based standards. It can be difficult to keep up with the pace of change, and newcomers are left with a dizzying variety of seemingly similar choices. This is compounded by the number of possible deployment permutations, which can cause all but the most determined to simply stick with the tried and true. In this talk I will introduce you to a powerhouse combination of Cassandra and Spark, which provides a high-speed platform for both real-time and batch analysis.

1. The New
Analytics Toolbox
Going beyond Hadoop
Robbie Strickland
DevNexus 2015

2. whoami
Robbie Strickland
Director, Software Engineering
linkedin.com/in/robbiestrickland
@rs_atl
rostrickland@gmail.com

3. whoami
● Contributor: core Cassandra,
Java driver, Spark driver, Hive
driver, other stuff
● DataStax MVP
● User since 2010 (0.5 release)
● Author, Cassandra High
Availability
● Founder, ATL Cassandra Users

4. Thanks to ...
Helena Edelson
DataStax Engineering
@helenaedelson

5. Weather @ scale
● ~10 billion API requests per day
● 4 AWS regions
● Over 100 million app users
● Lots of data to analyze!

6. Agenda
● Tool landscape
● Spark vs. Hadoop
● Spark overview
● Spark + Cassandra
● A typical Spark application
● Spark SQL
● Getting set up
● Demo
● Spark Streaming
● Task distribution
● Language comparison
● Questions

7. Hadoop works fine, right?

8. Hadoop works fine, right?
● It’s 11 years old (!!)

9. Hadoop works fine, right?
● It’s 11 years old (!!)
● It’s relatively slow and inefficient

10. Hadoop works fine, right?
● It’s 11 years old (!!)
● It’s relatively slow and inefficient
● … because everything gets written to disk

11. Hadoop works fine, right?
● It’s 11 years old (!!)
● It’s relatively slow and inefficient
● … because everything gets written to disk
● Writing mapreduce code sucks

12. Hadoop works fine, right?
● It’s 11 years old (!!)
● It’s relatively slow and inefficient
● … because everything gets written to disk
● Writing mapreduce code sucks
● Lots of code for even simple tasks

13. Hadoop works fine, right?
● It’s 11 years old (!!)
● It’s relatively slow and inefficient
● … because everything gets written to disk
● Writing mapreduce code sucks
● Lots of code for even simple tasks
● Boilerplate

14. Hadoop works fine, right?
● It’s 11 years old (!!)
● It’s relatively slow and inefficient
● … because everything gets written to disk
● Writing mapreduce code sucks
● Lots of code for even simple tasks
● Boilerplate
● Configuration isn’t for the faint of heart

15. Landscape has changed ...
Lots of new tools available:
● Cloudera Impala
● Apache Drill
● Proprietary (Splunk, keen.io, etc.)
● Spark / Spark SQL
● Shark

16. Landscape has changed ...
Lots of new tools available:
● Cloudera Impala
● Apache Drill
● Proprietary (Splunk, keen.io, etc.)
● Spark / Spark SQL
● Shark
MPP queries for Hadoop data

17. Landscape has changed ...
Lots of new tools available:
● Cloudera Impala
● Apache Drill
● Proprietary (Splunk, keen.io, etc.)
● Spark / Spark SQL
● Shark
Varies by vendor

18. Landscape has changed ...
Lots of new tools available:
● Cloudera Impala
● Apache Drill
● Proprietary (Splunk, keen.io, etc.)
● Spark / Spark SQL
● Shark
Generic in-memory analysis

19. Landscape has changed ...
Lots of new tools available:
● Cloudera Impala
● Apache Drill
● Proprietary (Splunk, keen.io, etc.)
● Spark / Spark SQL
● Shark Hive queries on Spark, replaced by Spark SQL

20. Spark wins?

21. Spark wins?
● Can be a Hadoop replacement

22. Spark wins?
● Can be a Hadoop replacement
● Works with any existing InputFormat

23. Spark wins?
● Can be a Hadoop replacement
● Works with any existing InputFormat
● Doesn’t require Hadoop

24. Spark wins?
● Can be a Hadoop replacement
● Works with any existing InputFormat
● Doesn’t require Hadoop
● Supports batch & streaming analysis

25. Spark wins?
● Can be a Hadoop replacement
● Works with any existing InputFormat
● Doesn’t require Hadoop
● Supports batch & streaming analysis
● Functional programming model

26. Spark wins?
● Can be a Hadoop replacement
● Works with any existing InputFormat
● Doesn’t require Hadoop
● Supports batch & streaming analysis
● Functional programming model
● Direct Cassandra integration

27. Spark vs. Hadoop

28. MapReduce:

29. MapReduce:

30. MapReduce:

31. MapReduce:

32. MapReduce:

33. MapReduce:

34. Spark (angels sing!):

35. What is Spark?

36. What is Spark?
● In-memory cluster computing

37. What is Spark?
● In-memory cluster computing
● 10-100x faster than MapReduce

38. What is Spark?
● In-memory cluster computing
● 10-100x faster than MapReduce
● Collection API over large datasets

39. What is Spark?
● In-memory cluster computing
● 10-100x faster than MapReduce
● Collection API over large datasets
● Scala, Python, Java

40. What is Spark?
● In-memory cluster computing
● 10-100x faster than MapReduce
● Collection API over large datasets
● Scala, Python, Java
● Stream processing

41. What is Spark?
● In-memory cluster computing
● 10-100x faster than MapReduce
● Collection API over large datasets
● Scala, Python, Java
● Stream processing
● Supports any existing Hadoop input / output
format

42. What is Spark?
● Native graph processing via GraphX

43. What is Spark?
● Native graph processing via GraphX
● Native machine learning via MLlib

44. What is Spark?
● Native graph processing via GraphX
● Native machine learning via MLlib
● SQL queries via SparkSQL

45. What is Spark?
● Native graph processing via GraphX
● Native machine learning via MLlib
● SQL queries via SparkSQL
● Works out of the box on EMR

46. What is Spark?
● Native graph processing via GraphX
● Native machine learning via MLlib
● SQL queries via SparkSQL
● Works out of the box on EMR
● Easily join datasets from disparate sources

47. Spark Components
Spark Core
Spark
Streaming
real-time
Spark SQL
structured
queries
MLlib
machine
learning
GraphX
graph
processing

48. Deployment Options
3 cluster manager choices:
● Standalone - included with Spark & easy to
set up
● Mesos - generic cluster manager that can
also handle MapReduce
● YARN - Hadoop 2 resource manager

49. Spark Word Count
val file = sc.textFile(“hdfs://…”)
val counts = file.flatMap(line => line.split(“ “))
.map(word => (word, 1))
.reduceByKey(_ + _)
counts.saveAsTextFile(“hdfs://…”)

50. Cassandra

51. Why Cassandra?
It’s fast:
● No locks
● Tunable consistency
● Sequential R/W

52. Why Cassandra?
It scales (linearly):
● Peer-to-peer (decentralized)
● DHT
● Read/write to any node
● Largest cluster = 75,000 nodes!

53. Why Cassandra?
It’s fault tolerant:
● Automatic replication
● Masterless (i.e. no SPOF)
● Failed nodes replaced with ease
● Multi data center

54. Why Cassandra?
It’s perfect for analysis:
● Unstructured & semi-structured data
● Partition aware
● Multi-DC replication
● Sharding is automatic
● Natural time-series support

55. Why Cassandra?
It’s easy to use:
● Familiar CQL syntax
● Light administrative burden
● Simple configuration

56. Spark on Cassandra

57. Spark on Cassandra
● Direct integration via DataStax driver -
cassandra-driver-spark on github

58. Spark on Cassandra
● Direct integration via DataStax driver -
cassandra-driver-spark on github
● No job config cruft

59. Spark on Cassandra
● Direct integration via DataStax driver -
cassandra-driver-spark on github
● No job config cruft
● Supports server-side filters (where clauses)

60. Spark on Cassandra
● Direct integration via DataStax driver -
cassandra-driver-spark on github
● No job config cruft
● Supports server-side filters (where clauses)
● Data locality aware

61. Spark on Cassandra
● Direct integration via DataStax driver -
cassandra-driver-spark on github
● No job config cruft
● Supports server-side filters (where clauses)
● Data locality aware
● Uses HDFS, CassandraFS, or other
distributed FS for checkpointing

62. Spark on Cassandra
Spark Core
Spark
Streaming
real-time
Spark SQL
structured
queries
MLlib
machine
learning
GraphX
graph
processing
Cassandra

63. Cassandra with Hadoop
Cassandra
DataNode
TaskTracker
Cassandra
DataNode
TaskTracker
Cassandra
DataNode
TaskTracker
NameNode
2ndaryNN
JobTracker

64. Cassandra with Spark (using HDFS)
Cassandra
Spark Worker
DataNode
Cassandra
Spark Worker
DataNode
Cassandra
Spark Worker
DataNode
Spark Master
NameNode
2ndaryNN

65. Online analytics
C*
Spark
C*
Spark
C*
Spark
C*
C*C*
Real-time
replication
Operational
data center
Analytics
data center

66. A Typical Spark Application

67. A Typical Spark Application
● SparkContext + SparkConf

68. A Typical Spark Application
● SparkContext + SparkConf
● Data Source to RDD[T]

69. A Typical Spark Application
● SparkContext + SparkConf
● Data Source to RDD[T]
● Transformations/Actions

70. A Typical Spark Application
● SparkContext + SparkConf
● Data Source to RDD[T]
● Transformations/Actions
● Saving/Displaying

71. Resilient Distributed Dataset (RDD)

72. Resilient Distributed Dataset (RDD)
● A distributed collection of items

73. Resilient Distributed Dataset (RDD)
● A distributed collection of items
● Transformations
○ Similar to those found in scala collections
○ Lazily processed

74. Resilient Distributed Dataset (RDD)
● A distributed collection of items
● Transformations
○ Similar to those found in Scala collections
○ Lazily processed
● Can recalculate from any point of failure

75. RDD Transformations vs Actions
Transformations:
Produce new RDDs
Actions:
Require the materialization of the records to produce a
value

76. RDD Transformations/Actions
Transformations:
filter, map, flatMap, collect(λ):RDD[T], distinct, groupBy,
subtract, union, zip, reduceByKey ...
Actions:
collect:Array[T], count, fold, reduce ...

77. Resilient Distributed Dataset (RDD)
val numOfAdults = persons.filter(_.age > 17).count()
Transformation Action

78. Example

79. Spark SQL
● Provides SQL access to structured data
○ Existing RDDs
○ Hive warehouses (uses existing metastore, SerDes
and UDFs)
○ JDBC/ODBC - use existing BI tools to query large
datasets

80. Spark SQL RDD Example

81. Getting set up
● Download Spark 1.2.x
● Download Cassandra 2.1.x
● Add the spark-cassandra-connector to
your project
"com.datastax.spark" % "spark-cassandra-connector_2.10" % "1.2.0-alpha3"

82. Running applications
./bin/spark-submit
--class org.apache.spark.examples.SparkPi
--master local[8]
/path/to/examples.jar

83. Running applications
./bin/spark-submit
--class org.apache.spark.examples.SparkPi
--master spark://192.168.1.1:7077
--executor-memory 20G
--total-executor-cores 100
/path/to/examples.jar

84. Demo

85. Spark Streaming

86. Spark Streaming
Almost any
source
Almost any
destination

87. Spark Streaming
● Creates RDDs from stream source on a
defined interval

88. Spark Streaming
● Creates RDDs from stream source on a
defined interval
● Same ops as “normal” RDDs

89. Spark Streaming
● Creates RDDs from stream source on a
defined interval
● Same ops as “normal” RDDs
● Supports a variety of sources

90. Spark Streaming
● Creates RDDs from stream source on a
defined interval
● Same ops as “normal” RDDs
● Supports a variety of sources
● Exactly once message guarantee

91. Spark Streaming - Use Cases
Applications Sensors Web Mobile
Intrusion detection
Malfunction
detection
Site analytics Network analytics
Fraud detection
Dynamic process
optimization
Recommendations
Location based
advertising
Log processing
Supply chain
planning
Sentiment
analysis
...

92. Spark Streaming
Input stream Batches of
input data
Batches of
processed data

93. Spark Streaming
Micro batch (RDD) Micro batch (RDD) Micro batch (RDD) Micro batch (RDD)
● DStream = continuous sequence of micro batches
● Each batch is an RDD
● Interval is configurable
DStream

94. Spark Streaming

95. The Output

96. Real World Task Distribution

97. Real World Task Distribution
Transformations and Actions:
Similar to Scala but …
Your choice of transformations need be made with task
distribution and memory in mind

98. Real World Task Distribution
How do we do that?
● Partition the data appropriately for number of cores (at
least 2x number of cores)

99. Real World Task Distribution
How do we do that?
● Partition the data appropriately for number of cores (at
least 2x number of cores)
● Filter early and often (Queries, Filters, Distinct...)

100. Real World Task Distribution
How do we do that?
● Partition the data appropriately for number of cores (at
least 2x number of cores)
● Filter early and often (Queries, Filters, Distinct...)
● Use pipelines

101. Real World Task Distribution
How do we do that?
● Partial Aggregation

102. Real World Task Distribution
How do we do that?
● Partial Aggregation
● Create an algorithm to be as simple/efficient as it can
be to appropriately answer the question

103. Real World Task Distribution

104. Real World Task Distribution
Some Common Costly Transformations:
● sorting
● groupByKey
● reduceByKey
● ...

105. Partitioning Example
User event log:
● Time-series events
● Tracks user interactions with system over
time
● Location check-ins, page/module views,
profile changes, ad impressions/clicks, etc.

106. Partitioning Example
CREATE TABLE Event (
user_id text,
timestamp int,
event_type text,
event_data map<text, text>,
PRIMARY KEY (user_id, timestamp, event_type)
);

107. Partitioning Example
CREATE TABLE Event (
user_id text,
timestamp int,
event_type text,
event_data map<text, text>,
PRIMARY KEY (user_id, timestamp, event_type)
);
partition key

108. Partitioning Example
CREATE TABLE Event (
user_id text,
timestamp int,
event_type text,
event_data map<text, text>,
PRIMARY KEY (user_id, timestamp, event_type)
);
clustering columns

109. Partitioning Example
Potential analysis:
● Location graph
● Individual usage habits
● Page/module view counts

110. Partitioning Example
Potential analysis:
● Location graph
● Individual usage habits
● Page/module view counts
Grouped by:
user_id

111. Partitioning Example
Potential analysis:
● Location graph
● Individual usage habits
● Page/module view counts
Grouped by:
user_id
user_id

112. Partitioning Example
Potential analysis:
● Location graph
● Individual usage habits
● Page/module view counts
Grouped by:
user_id
user_id
event_data

113. Partitioning Example
Node 1
rstrickland
jsmith
tjones
Node 2
awilson
ptaylor
gwatson
Node 3
lmiller
mjohnson
scarter

114. Partitioning Example
Node 1
rstrickland
jsmith
tjones
Node 2
awilson
ptaylor
gwatson
Node 3
lmiller
mjohnson
scarter
users.reduceByKey { … }

115. Partitioning Example
Node 1
rstrickland
jsmith
tjones
Node 2
awilson
ptaylor
gwatson
Node 3
lmiller
mjohnson
scarter
users.reduceByKey { … }
No shuffling required!

116. Partitioning Example
Node 1
rstrickland: home,
local, video
jsmith: home, radar
tjones: radar, video
Node 2
awilson: home, local,
radar
ptaylor: home, video
gwatson: local, radar
Node 3
lmiller: home, radar,
video
mjohnson: radar
scarter: home, local,
radar
users.filter(_.eventType == “PageView”)

117. Partitioning Example
Node 1
home: 2
local: 1
radar: 2
video: 2
Node 2
home: 2
local: 2
radar: 2
video: 1
Node 3
home: 2
local: 1
radar: 2
video: 1
users.filter(_.eventType == “PageView”)
.map { e => (e.event_data[“page”], 1) }
Combining is automatic :)

118. Partitioning Example
Node 1
home: 2, 2, 2
Node 2
local: 1, 2, 1
radar: 2, 2, 2
Node 3
video: 2, 1, 1
users.filter(_.eventType == “PageView”)
.map { e => (e.event_data[“pageview”], 1) }
.reduceByKey(_ + _) Requires a shuffle!

119. Java vs. Scala
Should I learn Scala?
… or leverage existing Java skills?

120. Java vs. Scala
● Spark uses a functional paradigm

121. Java vs. Scala
● Spark uses a functional paradigm
● Spark is written in Scala

122. Java vs. Scala
● Spark uses a functional paradigm
● Spark is written in Scala
● Scala > Java 8 > Java 7

123. Java vs. Scala
● Spark uses a functional paradigm
● Spark is written in Scala
● Scala > Java 8 > Java 7
● You will want lambdas!

124. Language Comparison - Scala
text.flatMap { line => line.split(“ “) }
.map(word => (word, 1))
.reduceByKey(_ + _)

125. Language Comparison - Java 8
text.flatMap(line -> Arrays.asList(line.split(“ “)))
.mapToPair(word -> new Tuple2<String, Integer>(word, 1))
.reduceByKey((x, y) -> x + y)

126. Language Comparison - Java 7
JavaRDD<String> words = text.flatMap(
new FlatMapFunction<String, String>() {
public Iterable<String> call(String line) {
return Arrays.asList(line.split(" "));
}
})

127. Language Comparison - Java 7
JavaPairRDD<String, Integer> ones = words.mapToPair(
new PairFunction<String, String, Integer>() {
public Tuple2<String, Integer> call(String w) {
return new Tuple2<String, Integer>(w, 1);
}
});

128. Language Comparison - Java 7
ones.reduceByKey(
new Function2<Integer, Integer, Integer>() {
public Integer call(Integer i1, Integer i2) {
return i1 + i2;
}
});

129. Shameless book plug
https://www.packtpub.com/big-data-and-business-intelligence/cassandra-high-availability

130. We’re hiring!

131. Thank you!
Robbie Strickland
linkedin.com/in/robbiestrickland
@rs_atl
rostrickland@gmail.com