Spark cassandra integration 2016

1. Spark-Cassandra Integration 2016
DuyHai DOAN
Apache Cassandra Evangelist

2. @doanduyhai
Main use-cases
Load data from various
sources
Analytics (join, aggregate, transform, …)
Sanitize, validate, normalize, transform data
Schema migration,
Data conversion

3. @doanduyhai
Data import
3
• Read data from CSV and dump into Cassandra ?
☞ Spark Job to distribute the import !
Load data from various
sources

4. Demo
4

5. @doanduyhai
Data cleaning
5
Sanitize, validate, normalize, transform data
• Bugs in your application ?
• Dirty input data ?
☞ Spark Job to clean it up!

6. Demo
6

7. @doanduyhai
Schema migration
7
• Business requirements change with time ?
• Current data model no longer relevant ?
☞ Spark Job to migrate data !
Schema migration,
Data conversion

8. Demo
8

9. @doanduyhai
Analytics
9
Given existing tables of performers and albums, I want:
① top 10 most common music styles (pop,rock, RnB, …) ?
② performer productivity(albums count) by origin country and by decade ?
☞ Spark Job to compute analytics !
Analytics (join, aggregate, transform, …)

10. Connector Architecture
• Cluster Deployment
• Data Locality
• Failure Handling
• Cross DC/cluster operations

11. @doanduyhai
Cluster Deployment
11
• Stand-alone cluster
C*
SparkM
SparkW
C*
SparkW
C*
SparkW
C*
SparkW
C*
SparkW

12. @doanduyhai
Data Locality – remember token ranges ?
12
A: −x,−
3x
4
⎤
⎦
⎥
⎥
⎤
⎦
⎥
⎥
B: −
3x
4
,−
2x
4
⎤
⎦
⎥
⎥
⎤
⎦
⎥
⎥
C: −
2x
4
,−
x
4
⎤
⎦
⎥
⎥
⎤
⎦
⎥
⎥
D: −
x
4
,0
⎤
⎦
⎥
⎥
⎤
⎦
⎥
⎥
E: 0,
x
4
⎤
⎦
⎥
⎥
⎤
⎦
⎥
⎥
F:
x
4
,
2x
4
⎤
⎦
⎥
⎥
⎤
⎦
⎥
⎥
G:
2x
4
,
3x
4
⎤
⎦
⎥
⎥
⎤
⎦
⎥
⎥
H :
3x
4
,x
⎤
⎦
⎥
⎥
⎤
⎦
⎥
⎥
C*
C*
C*
C*
C* C*
C* C*

13. @doanduyhai
Data Locality – how to
13
Spark partition RDD
Cassandra
tokens ranges
C*
SparkM
SparkW
C*
SparkW
C*
SparkW
C*
SparkW
C*
SparkW

14. @doanduyhai
Data Locality – how to
14
C*
SparkM
SparkW
C*
SparkW
C*
SparkW
C*
SparkW
C*
SparkW

15. @doanduyhai
Perfect data locality scenario
• read localy from Cassandra
• use operations that do not require shuffle in Spark (map, filter, …)
• repartitionbyCassandraReplica()
à to a table having same partition key as original table
• save back into this Cassandra table
Sanitize, validate, normalize, transform data
USE CASE
15

16. @doanduyhai
Failure Handling
16
C*
SparkM
SparkW
C*
SparkW
C*
SparkW
C*
SparkW
C*
SparkW
What if 1 node down ?
What if 1 node overloaded ?

17. @doanduyhai
Failure Handling
17
C*
SparkM
SparkW
C*
SparkW
C*
SparkW
C*
SparkW
C*
SparkW
What if 1 node down ?
What if 1 node overloaded ?
☞ Spark master will re-assign
the job to another worker

18. @doanduyhai
Failure Handling
18
Oh no, my data locality !!!

19. @doanduyhai
Failure Handling
19

20. @doanduyhai
Data Locality Impl
20
abstract'class'RDD[T](…)'{'
' @DeveloperApi'
' def'compute(split:'Partition,'context:'TaskContext):'Iterator[T]'
'
' protected'def'getPartitions:'Array[Partition]'
' '
' protected'def'getPreferredLocations(split:'Partition):'Seq[String]'='Nil''''''''
}'

21. @doanduyhai
CassandraRDD
21
def getPreferredLocations(split: Partition): Cassandra replicas IP address
corresponding to this Spark partition

22. @doanduyhai
Failure Handling
22
If RF > 1 the Spark master choses
the next preferred location, which
is a replica 😎
Tune parameters:
• spark.locality.wait
• spark.locality.wait.process
• spark.locality.wait.node

23. @doanduyhai
Failure Handling
23
If RF > 1 the Spark master choses
the next preferred location, which
is a replica 😎
Tune parameters:
• spark.locality.wait
• spark.locality.wait.process
• spark.locality.wait.node
Only work for fixed
token ranges (vnodes)

24. @doanduyhai
Cross cluster/DC operations
24

25. Tales from the field, SASI index benchmark
• Deployment automation
• Parallel ingestion
• Migrating data
• Spark + Cassandra 3.4 SASI index for topK query

26. @doanduyhai
Deployment Automation
26
Use Ansible to bootstrap a cluster
• role tools (install vim, htop, dstat, fio, jmxterm..)
• role Cassandra. Do not put all nodes as seeds ….
• role Spark (vanilla Spark). Slave on all nodes, master on a random node
DO NOT START ALL CASSANDRA NODES AT THE SAME TIME !!!!
• bootstrap first seeds nodes
• give ≥ 30secs between 2 node bootstrap for token range agreement
• watch -n 5 nodetool status

27. @doanduyhai
Parallel ingestion for SASI index benchmark
27
Hardware specs
• 13 nodes
• 6 cores CPU (HT)
• 4 SSD in RAID 0 😎
• 64 Gb of RAM
Cassandra conf:
• G1GC 32Gb JVM Heap
• compaction throughput in MB = 256
• concurrent compactor = 2

28. @doanduyhai
Parallel ingestion for SASI index benchmark
28

29. @doanduyhai
Parallel ingestion for SASI index benchmark
29
3.2 billions row in 17h
(compaction disabled)
RF = 2
☞ ≈ 8000 ips
I/O idle, high CPU

30. @doanduyhai
Migrating Data
30

31. @doanduyhai
Migrating Data
31

32. @doanduyhai
TopK query
32
Pass 1, for each music provider
• sum albums sales count by title
• take top N, associate weight from descending order (1st = 1000, 2nd = 999 …)
Retrieve all albums from pass 1
• re-sum the sum(sales count) and weight group by title
• order again by sum(sales count) in descending order
• take top N

33. @doanduyhai
TopK query
33
Target data set = 3.2 billions rows
• minimum filter = 1 month (period_end_month = 201404 for ex)
• worst filter = 3 months range
• +8 other dynamic filters (music provider, distribution type …)
☞ SASI indices for filtering
☞ Spark for aggregation

34. @doanduyhai
TopK query results
34
3.2 billions rows in total
• random distribution over 3 years (36 months) à 88 millions rows/month
Filters #rows Duration #rows/sec
3 months 376 947 612 14 mins (840 secs) 448 747
1 month 94 239 127 6.1 mins (366 secs) 257 483
1 month + 1 provider 7 267 983 2.1 mins (126 secs) 57 682
1 month + 1 provider + 1 country 2 737 178 1.5 mins (90 secs) 30 413

35. 35
Q & A
! "

36. 36
@doanduyhai
duy_hai.doan@datastax.com
https://academy.datastax.com/
Thank You