Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1vmnhxz. The authors take a deep dive into the history of NoSQL at Amazon.com, from the world of relational databases to the Dynamo days to the world of managed services like DynamoDB. Filmed at qconnewyork.com. Khawaja Shams is the Head of Engineering for NoSQL at Amazon Web Services. He is a passionate distributed systems engineer who loves to code. Swami Sivasubramanian is the Chief Architect and General Manager for NoSQL services at Amazon. Swami owns some of the biggest distributed systems and services at Amazon.

1. NoSQL like There is No Tomorrow
Head of Engineering, NoSQL
Swaminathan Swami
Sivasubramanian
GM, NoSQL
Khawaja
@swami_79 @ksshams

2. Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/aws-nosql-lessons
InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. how can you build your own DynamoDB Scale service?
@swami_79 @ksshams

5. let’s start with a story about a
little company called amazon.com
@swami_79 @ksshams

6. episode 1
once upon a time...
(in 2000)
@swami_79 @ksshams

7. a few thousand miles away...
(seattle)
@swami_79 @ksshams

8. amazon.com - a rapidly growing Internet based
retail business relied on relational databases
@swami_79 @ksshams

9. we had 1000s of independent services
@swami_79 @ksshams

10. each service managed its state in RDBMs
@swami_79 @ksshams

11. RDBMs are actually kind of cool
@swami_79 @ksshams

12. first of all... SQL!!
@swami_79 @ksshams

13. so it is easier to query..
@swami_79 @ksshams

14. easier to learn
@swami_79 @ksshams

15. as versatile as a swiss army knife
key-value access complex queries
analytics transactions
@swami_79 @ksshams

16. RDBMs are too similar to
Swiss Army Knives
@swami_79 @ksshams

17. but sometimes.. swiss army knifes..
can be more than what you bargained for
@swami_79 @ksshams

18. partitioning
easy
re-partitioning
hard..
@swami_79 @ksshams

19. so we bought
bigger boxes...
@swami_79 @ksshams

20. benchmark
new hardware
migrate to new
hardware
Q4 was hard-work at Amazon
repartition
databases
pray
...
@swami_79 @ksshams

21. RDBMs availability challenges..
@swami_79 @ksshams

22. episode 2
then.. (in 2005)
@swami_79 @ksshams

23. amazon dynamo
predecessor to dynamoDB
replicated DHT with consistent hashing
optimistic replication
“sloppy quorum”
anti-entropy mechanism
object versioning
specialist tool :
•limited querying capabilities
•simpler consistency
@swami_79 @ksshams

24. dynamo had many benefits
• higher availability
• we traded it off for eventual consistency
• incremental scalability
• no more repartitioning
• no need to architect apps for peak
• just add boxes
• simpler querying model ==>> predictable performance
@swami_79 @ksshams

25. but dynamo was not perfect...
lacked strong consistency
@swami_79 @ksshams

26. but dynamo was not perfect...
scaling was easier, but...
@swami_79 @ksshams

27. but dynamo was not perfect...
steep learning curve
@swami_79 @ksshams

28. but dynamo was not perfect...
dynamo was a product ... ==>> not a service...
@swami_79 @ksshams

29. episode 3
then.. (in 2012)
@swami_79 @ksshams

30. DynamoDB
• NoSQL database
• fast & predictable performance
• seamless scalability
• easy administration
“Even though we have years of experience with large, complex
NoSQL architectures, we are happy to be finally out of the
business of managing it ourselves.” - Don MacAskill, CEO
@swami_79 @ksshams

31. build services not software!!
@swami_79 @ksshams

32. amazon.com’s experience with services
@swami_79 @ksshams

33. how do you create a successful service?
@swami_79 @ksshams

34. with great services, comes great responsibility
@swami_79 @ksshams

35. DynamoDB Goals and Philosophies
never compromise on durability
scale is our problem
easy to use
scale in rps
consistent and low latencies
@swami_79 @ksshams

36. Architect
Develop
Test
Goals
Customer
Monitor
Deploy
@swami_79 @ksshams

37. Architect
Develop
Test
Goals
Customer
Monitor
Deploy
@swami_79 @ksshams

38. Sacred Tenets in Services
don’t compromise durability for performance
plan for success - plan for scalability
plan for failures - fault -tolerance is key
consistent performance is important
design - think of blast radius
insist on correctness
@swami_79 @ksshams

39. fault tolerance is a lesson best learned offline
@swami_79 @ksshams

40. a simple 2-way replication system of a traditional database…
Writes
Primary Standby
@swami_79 @ksshams

41. S
is
dead,
need
to
trigger
new
P
is
dead,
need
to
promote
myself
P S
replica
P’
@swami_79 @swami_79 @@kskssshhaammss

42. improved Replication: quorum
Replica Replica
Writes
Replica
Quorum: Successful write on a majority
@swami_79 @swami_79 @@kskssshhaammss

43. Not so easy..
Replica B
Replica D
Replica C
Writes from
Replica A client A
New member in the
group
Should I continue to serve reads?
Should I start a new quorum?
Replica E Replica F
Reads and
Writes from
client B
Classic Split Brain Issue in Replicated systems leading to lost writes!

44. Building correct distributed systems is not straight forward..
• How do you handle replica failures?
• How do you ensure there is not a parallel
quorum?
• How do you handle partial failures of replicas?
• How do you handle concurrent failures?
@swami_79 @ksshams

45. correctness is hard, but necessary

46. Formal Methods

47. Formal Methods
to minimize bugs, we must have a precise
description of the design

48. Formal Methods
code is too detailed
design documents and diagrams are vague & imprecise
how would you express partial
failures or concurrency?

49. Formal Methods
law of large numbers is your friend,
until you hit large numbers
so design for scale

50. TLA+ to the rescue?
@swami_79 @ksshams

51. PlusCal
@swami_79 @ksshams

52. formal methods are necessary
but not sufficient..
@swami_79 @ksshams

53. Develop
Architect
Test
customer
Deploy
Monitor
Goals
@swami_79 @ksshams

54. don’t forget to test - no, serious l y ..
@swami_79 @ksshams

55. simulate failures at
unit test level fault injection testing
scale testing
embrace failure and don’t be surprised
datacenter testing
network brown out testing

56. testing is a lifelong journey

57. testing is necessary
but not sufficient..
@swami_79 @ksshams

58. Monitor
Goals
Deploy
Develop
Customer
Architect
Test
@swami_79 @ksshams

59. gamma
simulate real world
one box
does it work?
release cycle
phased deployment
treading lightly
monitor
does it still work?
@swami_79 @ksshams

60. Canaries
@swami_79 @ksshams

61. Alarms
@swami_79 @ksshams

62. Monitor customer behavior
Monitor
Goals
Deploy
Customer
Architect
Test
Develop
@swami_79 @ksshams

63. measuring customer experience is key
don’t be satisfied by average - look at 99 percentile
@swami_79 @ksshams

64. understand the scaling dimensions
@swami_79 @ksshams

65. understand how your service will be abused
@swami_79 @ksshams

66. let’s see these rules in action through a true story
@swami_79 @ksshams

67. we were building distributed systems all over amazon.com
@swami_79 @ksshams

68. we needed a uniform and correct way to do consensus..
@swami_79 @ksshams

69. so we built a paxos lock library service
@swami_79 @ksshams

70. such a service is so much more useful than just leader election..
it became a distributed state store
@swami_79 @ksshams

71. such a service is so much more useful than just leader election..
or a distributed state store
wait wait.. you’re telling me if I poll,
I can detect node failure?
@swami_79 @ksshams

72. we acted quickly - and scaled up our entire fleet with more
nodes
doh!!!!
we slowed consensus...
@swami_79 @ksshams

73. understand the scaling dimensions
& scale them independently...
@swami_79 @ksshams

74. a lock service has 3 components..
Leader Election
Failure Notification
State Store
@swami_79 @ksshams

75. they must be scaled independently..
Leader Election
Failure Notification
State Store
@swami_79 @ksshams

76. they must be scaled independently..
Leader Election
Failure Notification
State Store
@swami_79 @ksshams

77. they must be scaled independently..
Leader Election
Failure Notification
State Store
@swami_79 @ksshams

78. understand
scaling dimensions
observe
how service is used
!
relentlessly
test
scalability over features
monitor
like a hawk
strive
for correctness
@swami_79 @ksshams

79. Thank You!
@swami_79
@kshams
@swami_79 @ksshams

80. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/aws-nosql-
lessons