Touches on what Coursera aims to get out of Cassandra, what goes into a good deployment, and our experience so far transitioning off MySQL.

1. Cassandra @ Coursera
Deploying in AWS
MySQL Transition
Daniel Chia
@DanielJHChia
Software Engineer, Infrastructure

2. Overview
• Why Cassandra
• What goes into a good deployment
• MySQL → Cassandra transition experience

4. 110 partners
!
698 courses
!
8.5 million learners

5. A Coursera Course

8. Your Final Project
This is your chance to apply the course concepts to real-world situations

10. Identity Verified Certificates

11. Technical
• 100% hosted on AWS
• Service-oriented architecture
• Mix of MySQL and Cassandra for persistence

12. What do we care about?

13. We care about…
• Availability
• Scalability
• Operational Ease
• Latency
• (Bonus) Multi-region writes

14. Availability matters

16. EBS Outage (2012)
Master
us-east-1a
Slave
us-east-1c

17. Scalability

18. Scalability

19. Sharded by class
class1
class2
class3
class4
class5
Machine 1
class6
class7
class8
class9
class10
Machine 2
class11
class12
class13
class14
class15
Machine 3

20. New use-case
Uh-oh… doesn’t fit in existing sharding

21. We care about…
• Availability
• Scalability
• Operational Ease
• Performance
• (Bonus) Multi-region

22. Try Cassandra!
So we decided to…

23. Cassandra ≠ [database XYZ]

24. –Albert Einstein
“But if you judge a fish by its ability to climb a tree, it
will live its whole life believing that it is stupid.”

25. Time to deploy Cassandra!
sudo apt-get install dse-full

26. A good deployment
Machine-level
Cluster-level

27. Picking a machine
• Disk
• IOPS… IOPS… IOPS
• Latency
Author: D-Kuru/Wikimedia Commons
Licence: CC-BY-SA-3.0-AT

28. Picking a machine
• CPU
Author: Mark Sze
Licence: CC BY-NC-ND 2.0

29. Picking a machine
• Memory
• Save some for page
cache!
Author: brutalSoCal
Licence: CC BY-NC-ND 2.0

30. On AWS
• Ephemeral disks.
• Please don’t use EBS. Really.
• IOPS usually the problem
• Instance sizes:
• spinning disk: m1.large, m1.xlarge, m2.4xlarge
• ssd: m3.xlarge, c3.2xlarge, i2.*

31. Set up the machine
• Lots of documentation / talks about this
• Recommended reading: Datastax guide [1]
[1] http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html

32. Cluster configuration
A
C B

33. Priam
care and feeding of Cassandra on AWS
https://github.com/Netflix/Priam

34. Cluster Topology
• We use RF=3
• Ring balanced within datacenter
• Nodes alternate racks (or AZs)

35. Cluster Topology (Priam)
• Token assignments stored in a database
• Can takeover token in instance of node failure

36. Cluster Topology (Priam)
• Priam assigns tokens
evenly per region
• Alternates AZs within
region
az1
az3
az2
az1
az2
az3

37. Autoscaling groups
• Recover from lost instance
• We don't use it for scaling with traffic

38. Important: Need one ASG per AZ
east-1a east-1a east-1a
east-1b east-1beast-1b
east-1ceast-1c east-1c
ASG
size: 9

39. Important: Need one ASG per AZ
ASG
size: 9
east-1a east-1a east-1a
east-1b east-1beast-1b
east-1ceast-1c
east-1b

40. Important: Need one ASG per AZ
ASG-1a
size: 3
east-1a east-1a east-1a
east-1b east-1beast-1b
east-1ceast-1c
ASG-1b
size: 3
ASG-1c
size: 3
east-1c

41. Backups
• Data on ephemeral disks
• Guard against application errors
• SSTables immutable -> ship to S3
• Priam does this

42. Restore
• Have to be able use your backup
• Also useful for QA / test
• Priam handles this rather nicely

43. Deployed!
Time to chill?
https://www.flickr.com/photos/spunkinator/2394514059
Creative Commons

44. Monitoring
working / not working doesn’t count.

45. We have our own custom reporter agent for Datadog
There’s pluggable reporter support in 2.0.2 now.

46. JVM GC woes

47. JVM GC woes
All happy now

48. SSTables Read Histogram

49. Questions?
before we carry on

51. Transition takes
time
mindset shift
expertise
(some) risk

52. Our experience
• Pick one feature first
• Mindset shift
• Data modeling consulting
• Libraries / Patterns / Data-as-a-service

53. Pick one feature
• Don’t go all in with
Cassandra with
something important
right away
• Work closely with that
team

54. You probably will make mistakes
Oops!

55. Mindset shift
• Everyone knows SQL
• Not everyone knows Cassandra / NoSQL
• Need to know queries beforehand

56. Enrollment Example
• Learners enroll into a course
• learner (many-to-many) course
• Need to keep track of this membership

57. MySQL Model
CREATE TABLE `courses_learners` (
`id` INT(11) NOT NULL auto_increment,
`course_id` INT(11) NOT NULL,
`learner_id` INT(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `c_l` (`learner_id`, `course_id`),
CONSTRAINT `ref1` FOREIGN KEY (`course_id`)
CONSTRAINT `ref2` FOREIGN KEY (`learner_id`)
)

58. MySQL Model
CREATE TABLE `courses_learners` (
`id` INT(11) NOT NULL auto_increment,
`course_id` INT(11) NOT NULL,
`learner_id` INT(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `c_l` (`learner_id`, `course_id`),
CONSTRAINT `ref1` FOREIGN KEY (`course_id`)
CONSTRAINT `ref2` FOREIGN KEY (`learner_id`)
)

59. MySQL Model
CREATE TABLE `courses_learners` (
`id` INT(11) NOT NULL auto_increment,
`course_id` INT(11) NOT NULL,
`learner_id` INT(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `c_l` (`learner_id`, `course_id`),
CONSTRAINT `ref1` FOREIGN KEY (`course_id`)
CONSTRAINT `ref2` FOREIGN KEY (`learner_id`)
)

60. MySQL Model
CREATE TABLE `courses_learners` (
`id` INT(11) NOT NULL auto_increment,
`course_id` INT(11) NOT NULL,
`learner_id` INT(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `c_l` (`learner_id`, `course_id`),
CONSTRAINT `ref1` FOREIGN KEY (`course_id`)
CONSTRAINT `ref2` FOREIGN KEY (`learner_id`)
)

61. Cassandra Style
CREATE TABLE courses_by_learner (
learner_id uuid,
course_id uuid,
PRIMARY KEY (learner_id, course_id)
)

62. Data modeling consulting
• Build core team proficient at C* data modeling
• Available to consult for trickier use cases

63. Libraries / Patterns
• Abstract away simple (but common) use-cases
• Key-value storage
• Simple time series
• Maybe every developer won’t need deep C*
knowledge?
• More radical: data as a service (e.g. STAASH)
STAASH: https://github.com/Netflix/staash

64. It’s a long road
but we’ll get there…
Author: Carissa Rogers
License: CC BY 2.0

65. Conclusion
• Know Cassandra
• Know what makes a good deployment
• Know that new skills have to be acquired

66. Questions?
We’re hiring!
coursera.org/jobs