1. Storage on EC2
(& Cassandra)
Tom Wilkie
Cassandra Workshop 8/06/11
Wednesday, 8 June 2011

2. ACHTUNG!
Data only collected over
past 5 days
Didn’t repeat experiments
(that much)
EC2 is a moving target
Wednesday, 8 June 2011

3. Consider: Not considering:
• Ephemeral vs EBS • Cluster Performance
• ... vs Instance Type • Internode latency,
throughput
• ... vs RAID level
• Tuning...
• ... vs # threads
ES ...
• (...vs storage engine) D F A I L UR
A TE EL
C OR R
Wednesday, 8 June 2011

4. m1.large 7.5 GB RAM, 4 CU, 64-bit, ‘High’ IO
m1.xlarge 15 GB RAM, 8 CU, 64-bit, ‘High’ IO
c1.xlarge 7GB RAM, 20 CU, 64-bit, ‘High’ IO
Cassandra 0.7.6, CentOS 5.5, OpenJDK...
Wednesday, 8 June 2011

5. Ephemeral Storage
Wednesday, 8 June 2011

6.
[ih-fem-er-uhl] Show IPA –adjective
1. lasting a very short time; short-lived; transitory:
the ephemeral joys of childhood.
2. lasting but one day: an ephemeral flower. –noun
3. anything short-lived, as certain insects.
Wednesday, 8 June 2011

7. Ephemeral Storage
Seek Performance
8000
7000
6000
7000 IOPs from a disk??
5000
m1.large, ephemeral
Seek / s
4000 m1.xlarge, ephemeral
c1.xlarge, ephemeral
3000
2000
1000
0
1 2 3 4
# Devices http://www.slideshare.net/davegardnerisme/
running-cassandra-on-amazon-ec2
Wednesday, 8 June 2011

8. Ephemeral Storage
Seek Performance
1000
900
800
700
600
m1.large, ephemeral
Seek / s
500 m1.xlarge, ephemeral
c1.xlarge, ephemeral
400
300
200
100
0
1 2 3 4
# Devices
Wednesday, 8 June 2011

9. Ephemeral Throughput
m1.xlarge
500
450
400
350
Write (Raid-0, dd)
Throughput (MB/s)
300
Read (Raid-0, dd)
Write (Random 10MB
250 chunks)
Read (Random 10MB
200 chunks)
150
100
50
0
1 2 3 4
# Devices
Wednesday, 8 June 2011

10. #
# dd if=/dev/zero of=/dev/sdd bs=512k count=20000
...
10485760000 bytes (10 GB) copied, 201.995
seconds, 51.9 MB/s
#
# dd if=/dev/zero of=/dev/sdd bs=512k count=20000
...
10485760000 bytes (10 GB) copied, 80.3673
seconds, 130 MB/s
Wednesday, 8 June 2011

11. • Max 4 devices per instance
• Data goes away when instance is
terminated (or crashes!)
• Suspect there is some sort indirection layer
underneath - thin provisioning / dedupe /
CoW or something
• Linux software RAID sucks
Wednesday, 8 June 2011

12. R ES ...
F AI LU
E LA T ED
CO RR
What happens if a bug in your software
causes all your nodes to crash?
ie say a memory leak causes an
OOM... on all nodes
Wednesday, 8 June 2011

13. EBS
Wednesday, 8 June 2011

14. EBS Seek performance
3000
2500
2000
Seeks / s
m1.large, ebs
1500
m1.large, ebs
c1.xlarge, ebs
1000
500
0
0 5 10 15 20 25 30
# Devices
Wednesday, 8 June 2011

15. EBS Random Reads
m1.xlarge, raid-0
1000
900
800
700 1
2
600 3
4
Total Seek / s
5
500 6
7
400 8
9
10
300
200
100
0
1 2 3 4 5 6 7 8 9 10
# Threads
Wednesday, 8 June 2011

16. EBS Random Reads
m1.xlarge, raid-0
1000
900
800
700
600
Max seek / s
500
400
300
200
100
0
0 1 2 3 4 5 6 7 8 9 10
# Devices
Wednesday, 8 June 2011

17. EBS Random Reads
m1.xlarge, raid-0
450
400
350
Seeks per device per second
300
250 max
min
avg
200
150
100
50
0
1 2 3 4 5 6 7 8 9 10
# Devices
Wednesday, 8 June 2011

18. EBS Throughput
10MB chunks) m1.xlarge
350
300
250 Write (Raid-0, dd)
Write (Raid-0, dd)
Throughput (MB/s)
Write (Raid-0, dd)
200
Read (Raid-0, dd)
Read (Random 10MB
150 chunks)
Read (Random 10MB
chunks)
100
50
0
1 2 3 4 5 6 7 8 9 10
# Devices
Wednesday, 8 June 2011

19. • Limited to ~100 IOPS per device?
• Or just 10ms latency?
• Seems to scale pretty linearly for random IO
• Sequential IO limited by network bandwidth,
independent of # devices
• shared with other network traffic?
• Linux software RAID sucks
Wednesday, 8 June 2011

20. R ES ...
F AI LU
E LA T ED
CO RR
What happens when EBS breaks?
http://storagemojo.com/2011/04/29/amazons-ebs-outage/
http://status.heroku.com/incident/151
Wednesday, 8 June 2011

21. +
II
???
Wednesday, 8 June 2011

22. “Use Elastic Block Storage”
http://stackoverflow.com/questions/4714879/deploy-cassandra-on-ec2
“Raid 0 EBS drives are the way to go”
http://coreyhulen.org/2010/10/03/%EF%BB%BFcassandra-performance-tests-on-ec2/
“we recommend using raid0 ephemeral disks”
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cold-boot-
performance-problems-td5615829.html#a5615889
Wednesday, 8 June 2011

23. http://coreyhulen.org/2010/10/03/%EF%BB%BFcassandra-performance-tests-on-ec2/
Wednesday, 8 June 2011

24. http://coreyhulen.org/2010/10/03/%EF%BB%BFcassandra-performance-tests-on-ec2/
Wednesday, 8 June 2011

25. Insert Rates by Instance Type
35000
30000
25000
20000
Inserts / s
15000
10000
5000
0
e ral e ral e ral ebs ebs ebs
hem hem hem ar ge, ar ge, ar ge,
ep ep , ep 1.l 1.x
l
1.x
l
ge, ge, rge m m c
1 .lar .x lar . xl a
m m1 c1
100 threads, batch mutate size 100, values length 10, 1 column per row, 300 million values
Wednesday, 8 June 2011

26. Wednesday, 8 June 2011

27. Wednesday, 8 June 2011

28. Get Rates by Instance Type
1700
1275
Gets / s
850
425
0
m1.xlarge, ephemeral m1.xlarge, ebs
100 threads, 700 thousand values
Wednesday, 8 June 2011

29. Wednesday, 8 June 2011

30. Range Query Rates by Instance Type
Too slow. No
results
Wednesday, 8 June 2011

31. Wednesday, 8 June 2011

32. TODO
• Repeat experiments
• # threads vs # devices for ephemeral
• Repeat experiments
• Cluster performance - scaling, latency,
throughput etc
• Repeat experiments
• Strategies for mixed EBS and Ephemeral?
• Repeat experiments
Wednesday, 8 June 2011

33. $470
110 million IOs, 360 GB-months, 560 machine hours
Wednesday, 8 June 2011

34. Questions?
http://github.com/acunu
http://bitbucket.org/acunu
http://www.slideshare.net/acunu
Wednesday, 8 June 2011