Flash Memory technology, deployed as server-side PCIe or solid state disks (SSDs), is emerging as a critical tool for performance and efficiency in data centers of all scales. This presentation will discuss how the use of Flash impacts Cassandra deployments in terms of configuration, DRAM requirements and performance expectations. Ideas on leveraging C*'s cutting-edge data-center awareness to blend flash and disk storage nodes for cost and workload efficiency will also be shared. Flash media itself will be examined from a physical perspective to understand endurance issues. Data on write amplification under bulk-load and operational workload conditions will be presented to explain the impact to Flash of C*'s Log Structured Merge Tree architecture and the associated compactions. Finally, we will examine strategies to make Cassandra more Flash-aware using both conventional techniques as well as emerging Non-volatile memory (NVM) programming capabilities. Lessons learned from real-world customer deployments will be shared to complete this presentation.

1. Fusion-io Confidential—Copyright © 2013 Fusion-io, Inc. All rights reserved.
Cassandra With No Moving Parts
Matt Kennedy
Cassandra Summit: June 12, 2013

2. Switch your database to flash
now. Or you’re doing it wrong.
Brian Bulkowski, Aerospike Founder and CTO
June 18, 2013 2#Cassandra13
http://highscalability.com/blog/2012/12/10/switch-your-
databases-to-flash-storage-now-or-youre-doing-it.html

3. June 18, 2013 3#Cassandra13
Why?

4. Flash IOPS Drives Server Adoption
June 18, 2013 4
▸ Capacity
▸ IOPS
▸ Cost per IOP
4TB 3TB
150 200,000
$$$$ ¢¢¢¢
#Cassandra13

5. June 18, 2013 5#Cassandra13
What is flash?

6. NAND Flash Memory
June 18, 2013 6
Flash is a persistent memory technology invented by
Dr. Fujio Masuoka at Toshiba in 1980.
Bit
Line
Source
Line Word Line
Control Gate
Float Gate
NPN
#Cassandra13

7. Consumer Volume Drives Economics
June 18, 2013 7#Cassandra13

8. Flash in Servers
June 18, 2013 8#Cassandra13

9. Direct Cut Through Architecture
June 18, 2013 #Cassandra13 9
PCIe
DRAM
Host
CPU
App
OS
LEGACY APPROACH FUSION DIRECT APPROACH
PCIeSAS
DRAM
Data path
Controller
NAND
Host
CPU
RAID
Controller
App
OS
Goal of every I/O operation to move data to/from DRAM and flash.
SC
Super
Capacitors

10. June 18, 2013 10#Cassandra13
How can we use
it in Cassandra?

11. Cassandra I/O - Writes
June 18, 2013 11
http://www.datastax.com/docs/1.2/dml/about_writes
#Cassandra13

12. Cassandra I/O - Reads
June 18, 2013 12
http://www.datastax.com/docs/1.2/dml/about_reads
#Cassandra13
Memory

13. DRAM Dictates Cassandra Scaling
June 18, 2013 13
▸ Key Design Principle:
▸ Working Set < DRAM
#Cassandra13

14. DOLLARS
Cost of DRAM Modules
0
200
400
600
800
1000
1200
1400
1600
4GB 8GB 16GB 32GB
#Cassandra13June 18, 2013 14
$ $$
$$$
$$$$$$

15. When do we scale out?
June 18, 2013 15
▸ A typical server…
CPU Cores: 32 with HT
Memory: 128 GB
…is your working set > 128GB?
#Cassandra13

16. Is there a better way?
June 18, 2013 16
▸ With NoSQL Databases, we tend to scale out for
DRAM
Combined Resources
CPU Cores: 96
Memory: 384 GB
More cores than needed to serve reads and writes.
#Cassandra13

17. Flash Offers A New Architectural
Choice
June 18, 2013 #Cassandra13 17
Milliseconds 10-3 Microseconds 10-6 Nanoseconds 10-9
CPU Cache DRAM
Disk Drives
Server-based Flash

18. Three Deployment Options
June 18, 2013 18
1. All Flash
2. Data Placement (CASSANDRA-2749)
3. Use Logical Data Centers
#Cassandra13

19. Cassandra with All-Flash Storage
June 18, 2013 #Cassandra13 19
Step 1: Mount ioMemory at /var/lib/cassandra/data
Step 2:

20. Data Placement
June 18, 2013 20
▸ https://issues.apache.org/jira/browse/CASSANDRA-2749
• Thanks Marcus!
▸ Takes advantage of filesystem hierarchy
▸ Use mount points to pin Keyspaces or Column
Families to flash:
• /var/lib/cassandra/data/{Keyspace}/{CF}
▸ Use flash for high performance needs, disk for
capacity needs
#Cassandra13

21. Data Centers for Storage Control
June 18, 2013 21
DC1
(Interactive requests)
DC3
(High density replicas)
DC2
(Hadoop MR Jobs)
PERFORMANCE
CAPACITY/NODE
HIGH
MEDIUM
LOW
HIGH
Cassandra cluster
#Cassandra13

22. June 18, 2013 #Cassandra13 22
The Numbers

23. YCSB Testing Setup
June 18, 2013 23#Cassandra13
x4x4
YCSB Load Generator
10GB 16-cores
24GB DRAM
Workloads use uniform
random key selection
instead of Zipfian.
150 million 1KB records, RF=3: ~ 120GB SSTables/node

24. YCSB: Bulk Load (CL=ALL)
June 18, 2013 #Cassandra13 24
YCSBINSERTS
0
10000
20000
30000
40000
50000
60000
70000
10
70
130
190
250
310
370
430
490
550
610
670
730
790
850
910
970
1030
1090
1150
1210
1270
1330
1390
1450
1510
1570
1630
1690
1750
1810
1870
1930
1990
2050
2110
2170
2230
2290
2350
2410
2470
2530
2590
2650
2710
2770
2830
inserts/sec
Avg
Latency:
0.9
ms
95th
Percen?le:
1
ms
99th
Percen?le:
4
ms

25. 95/5 R/W Uniform distribution
June 18, 2013 #Cassandra13 25
MIXEDOPS/SEC
0
10000
20000
30000
40000
50000
60000
70000
80000
10
30
50
70
90
110
130
150
170
190
210
230
250
270
290
310
330
350
370
390
410
430
450
470
490
510
530
550
570
590
610
630
650
670
690
75
threads
200
threads
300
threads
# threads Avg Lat. 95th pctl 99th pctl
75 1.4/0.22 ms 2/0 ms 5/0 ms
200 3.1/0.19 ms 7/0 ms 13/0 ms
300 4.4/2.2 ms 11/0 ms 19/0 ms

26. 50/50 R/W Uniform distribution 10hrs
June 18, 2013 #Cassandra13 26
YCSBMIXEDOPS/SEC
0
10000
20000
30000
40000
50000
60000
70000
10
730
1450
2170
2890
3610
4330
5050
5770
6490
7210
7930
8650
9370
10090
10810
11530
12250
12970
13690
14410
15130
15850
16570
17290
18010
18730
19450
20170
20890
21610
22330
23050
23770
24490
25210
25930
26650
27370
28091
28811
29531
30251
30971
31691
32411
33131
33851
34571
35291
mixed
ops/sec
Update
Latency
Average:
511
µs
95th
Pctl:1
ms
99th
Pctl:
2
ms
Read
Latency
Average:
7.0
ms
95th
Pctl:
18
ms
99th
Pctl:
42
ms

27. Write Amplification
June 18, 2013 27#Cassandra13
Amplification Factor =
Physical Bytes Written
Workload Bytes Written
Workload Write Amp
Leveled
Compaction Load
(250MB tier-0)
0.8-1.2x
24-hour mixed
workloads
1.2-2.1x
Size-tiered w/Major
Compactions (old
skool)
3-15x
Workload Type Amplification
Factor
Bulk Load 14.8
Normal
Operations
(80/20 update/
insert split)
4.2
Cassandra
Compares favorably to HBase

28. Next Step in Flash Evolution
June 18, 2013 28
FLASH AS
MEMORY
NATIVE FLASH
APIs
FLASH AS DISK
#Cassandra13

29. Rethinking Cassandra I/O
June 18, 2013 29
http://www.datastax.com/docs/1.2/dml/about_writes
Flash
#Cassandra13

30. Rethinking Cassandra I/O
June 18, 2013 30#Cassandra13
http://www.datastax.com/docs/1.2/dml/about_writes
Flashtable

31. Accelerating Cassandra With Flash
June 18, 2013 31
+
#Cassandra13
NAND Flash Accelerator

32. Real-World Cassandra on Fusion
June 18, 2013 32#Cassandra13

33. f u s i o n i o . c o m | R E D E F I N E W H A T ’ S P O S S I B L E
THANK YOU
f u s i o n i o . c o m | R E D E F I N E W H A T ’ S P O S S I B L E
THANK YOU

34. Cassandra: ioDrive2 vs 10 disk
RAID-0
June 18, 2013 34#Cassandra13

35. 12-hour mixed read/
write workload
June 18, 2013 Fusion-io Confidential 35
MIXEDWORKLOAD
0
5000
10000
15000
20000
25000
30000
35000
40000
10
880
1750
2620
3490
4360
5230
6100
6970
7840
8710
9580
10450
11320
12190
13060
13930
14800
15670
16540
17410
18280
19150
20020
20890
21760
22630
23500
24370
25240
26110
26980
27850
28720
29590
30460
31331
32201
33071
33941
34811
35681
36551
37421
38291
39161
40031
40901
41771
42641
CL=1
Reads
CL=Q
Reads
CL=Q
Writes
(throMled)

36. 50/50 R/W Uniform distribution
June 18, 2013 #Cassandra13 36
YCSBMIXEDOPS/SEC
0
20000
40000
60000
80000
100000
120000
10
30
50
70
90
110
130
150
170
190
210
230
250
270
290
310
330
350
370
390
410
430
450
470
490
510
530
550
mixed
ops/sec
Update
Latency
Average:
311
µs
95th
Pctl:0
ms
99th
Pctl:
1
ms
Read
Latency
Average:
8.2
ms
95th
Pctl:
20
ms
99th
Pctl:
62
ms