Learn about the thought and decisions that went into designing and building DynamoDB. We'll talk about its roots and how we can deliver the performance and throughput you enjoy today. We’ll also show you how to model data, maintain maximum throughput, and drive analytics against the data with DynamoDB. Finally, you'll hear from some of our customers on how they've built large-scale applications on DynamoDB and about the lessons they've learned along the way.

1. Under the Covers of Amazon DynamoDB
Matt Wood, Chief Data Scientist

2. Hello

3. Amazon DynamoDB

4. Two decisions + three clicks
= ready for use

5. Level of throughput
Primary keys
Two decisions + three clicks
= ready for use

6. Provisioned
throughput
Amazon DynamoDB

7. Provisioned
throughput
Data patterns
Amazon DynamoDB

8. Provisioned
throughput
Data patterns
Amazon DynamoDB

9. DynamoDB is a managed NoSQL
database service
Store and retrieve any amount of data
Serve any level of request traffic

10. Without the operational burden

11. Consistent, predictable performance
Single digit millisecond latency
Backed on solid-state drives

12. Flexible data model
Key/attribute pairs. No schema required.
Easy to create. Easy to adjust.

13. Seamless scalability
No table size limits. Unlimited storage.
No downtime.

14. Durable
Consistent, disk only writes
Replication across data centers
and availability zones

15. Without the operational burden

16. Without the operational burden
Focus on your app

17. Two decisions + three clicks
= ready for use

18. Provisioned
throughput
Amazon DynamoDB

19. Provisioned throughput
Reserve IOPS for reads and writes
Scale up for down at any time

20. Pay per capacity unit
Priced per hour of provisioned throughput

21. Write throughput
Size of item x writes per second
$0.01 for 10 write units

22. Consistent writes
Atomic increment and decrement
Optimistic concurrency control: conditional writes

23. Transactions
Item level transactions only
Puts, updates and deletes are ACID

24. Strong or eventual consitency
Read throughput

25. Strong or eventual consitency
Read throughput
Provisioned units = size of item x reads per second
$0.01 per hour for 50 units

26. Strong or eventual consistency
Read throughput
Provisioned units = size of item x reads per second
2
$0.01 per hour for 100 units

27. Strong or eventual consitency
Read throughput
Same latency expectations
Mix and match at ‘read time’

28. Provisioned throughput is
managed by DynamoDB

29. Data is partitioned and managed
by DynamoDB

30. Achieving full provisioned throughput
requires a uniform workload

31. The DynamoDB Uniform Workload
DynamoDB divides table data in to multiple partitions
Data is distributed primarily by primary key
Provisioned throughput is divided evenly across partitions

32. The DynamoDB Uniform Workload
To achieve and maintain full provisioned throughput
for a table, spread the workload evenly
across primary keys

33. Non-uniform workloads
Some requests might be throttled,
even at high levels of provisioned throughput

34. Model data for a uniform workload

35. Provisioned
throughput
Data patterns
Amazon DynamoDB

36. Level of throughput
Primary keys
Two decisions + three clicks
= ready for use

37. DynamoDB semantics
Tables, items and attributes

38. date = 2012-05-16-09-00-
id = 100 10 total = 25.00
date = 2012-05-15-15-00-
id = 101 11 total = 35.00
date = 2012-05-16-12-00-
id = 101 10 total = 100.00
date = 2012-03-20-18-23-
id = 102 10 total = 20.00

39. Table
date = 2012-05-16-09-00-
id = 100 10 total = 25.00
date = 2012-05-15-15-00-
id = 101 11 total = 35.00
date = 2012-05-16-12-00-
id = 101 10 total = 100.00
date = 2012-03-20-18-23-
id = 102 10 total = 20.00

40. date = 2012-05-16-09-00-
id = 100 10 total = 25.00
date = 2012-05-15-15-00-
id = 101 11 total = 35.00 Item
date = 2012-05-16-12-00-
id = 101 10 total = 100.00
date = 2012-03-20-18-23-
id = 102 10 total = 20.00

41. date = 2012-05-16-09-00-
id = 100 10 total = 25.00
date = 2012-05-15-15-00-
id = 101 11 total = 35.00 Attribute
date = 2012-05-16-12-00-
id = 101 10 total = 100.00
date = 2012-03-20-18-23-
id = 102 10 total = 20.00

42. Items are indexed by primary key
Single hash keys and composite range keys

43. Hash key
date = 2012-05-16-09-00-
id = 100 10 total = 25.00
date = 2012-05-15-15-00-
id = 101 11 total = 35.00
date = 2012-05-16-12-00-
id = 101 10 total = 100.00
date = 2012-03-20-18-23-
id = 102 10 total = 20.00

44. Range key
date = 2012-05-16-09-00-
id = 100 10 total = 25.00
date = 2012-05-15-15-00-
id = 101 11 total = 35.00
date = 2012-05-16-12-00-
id = 101 10 total = 100.00
date = 2012-03-20-18-23-
id = 102 10 total = 20.00

45. Items are retrieved by primary key

46. Range keys for queries
For example: all items for November

47. Relationships are not hard coded,
but can be modeled

48. Players
user_id = location = joined =
mza Cambridge 2011-07-04
user_id = location = joined =
jeffbarr Seattle 2012-01-20
user_id = location = joined =
werner Worldwide 2011-05-15

49. Players
user_id = location = joined =
mza Cambridge 2011-07-04
user_id = location = joined =
jeffbarr Seattle 2012-01-20
user_id = location = joined =
werner Worldwide 2011-05-15
Scores
user_id = game = score =
mza angry-birds 11,000
user_id = game = score =
mza tetris 1,223,000
user_id = location = score =
werner bejewelled 55,000

50. Players
user_id = location = joined =
mza Cambridge 2011-07-04
user_id = location = joined =
jeffbarr Seattle 2012-01-20
user_id = location = joined =
werner Worldwide 2011-05-15
Scores Leader boards
user_id = game = score = game = score = user_id =
mza angry-birds 11,000 angry-birds 11,000 mza
user_id = game = score = game = score = user_id =
mza tetris 1,223,000 tetris 1,223,000 mza
user_id = location = score = game = score = user_id =
werner bejewelled 55,000 tetris 9,000,000 jeffbarr

51. Players
user_id = location = joined =
mza Cambridge 2011-07-04 Scores by user
user_id = location = joined =
jeffbarr Seattle 2012-01-20
user_id = location = joined =
werner Worldwide 2011-05-15
Scores Leader boards
user_id = game = score = game = score = user_id =
mza angry-birds 11,000 angry-birds 11,000 mza
user_id = game = score = game = score = user_id =
mza tetris 1,223,000 tetris 1,223,000 mza
user_id = location = score = game = score = user_id =
werner bejewelled 55,000 tetris 9,000,000 jeffbarr

52. Players
user_id = location = joined =
mza Cambridge 2011-07-04 High scores by game
user_id = location = joined =
jeffbarr Seattle 2012-01-20
user_id = location = joined =
werner Worldwide 2011-05-15
Scores Leader boards
user_id = game = score = game = score = user_id =
mza angry-birds 11,000 angry-birds 11,000 mza
user_id = game = score = game = score = user_id =
mza tetris 1,223,000 tetris 1,223,000 mza
user_id = location = score = game = score = user_id =
werner bejewelled 55,000 tetris 9,000,000 jeffbarr

53. NoSQL data modeling for maximal
provisioned throughput

54. Distinct values for hash keys
Hash key elements should have a high
number of distinct values

55. Lots of unique user IDs: workload well distributed
user_id = first_name = last_name =
mza Matt Wood
user_id = first_name = last_name =
jeffbarr Jeff Barr
user_id = first_name = last_name =
werner Werner Vogels
user_id = first_name = last_name =
mattfox Matt Fox
... ... ...

56. Limited response codes: workload poorly distributed
status = date =
200 2012-04-01-00-00-01
status = date =
404 2012-04-01-00-00-01
status date =
404 2012-04-01-00-00-01
status = date =
404 2012-04-01-00-00-01

57. Provisioned
throughput
Data patterns
Amazon DynamoDB

58. Provisioned
throughput
Data patterns
Amazon DynamoDB

59. NYT faбrik
AWS re:Invent – November 2012
Andrew Canaday & Michael Laing
New York Times Digital

60. What we’ll cover
 faбrik overview
 Getting more out of DynamoDB with python/boto
– More throughput / provisioned capacity
– Across more endpoints / table
– More reliably and controllably
60

61. Frank McCloud: “He wants more, don't you, Rocco?”
Johnny Rocco: “Yeah. That's it. More. That's right! I want more!”
James Temple: “Will you ever get enough?”
Frank McCloud: “Will you, Rocco?”
Johnny Rocco: “Well, I never have. No, I guess I won't.”
61

62. Takeaways
 Messaging infrastructure is cool (again)
 Old dogs have tricks you can apply
– The Internet is your friend
– BUT: much good computer science was done prior
– HENCE: not so readily findable
 Boto is great – clone and contribute!
62

63. NYT Mission
Enhance society by creating, collecting and distributing high quality
news, information and entertainment
- Distributing: publish / subscribe
- Collecting: gather / analyze
- High Quality: fast, reliable, accurate
63

64. faбrik
 Asynchronous Messaging Framework
 For client devices as well as our apps
 Enabled by:
– Websockets
– Robust message handling software
– Amazon Web Services
 Focusing on simple, common services
64

65. Typical Web Architecture
 Clients interact with front-end via load balancers
 Front-end makes requests to back-end on behalf of client
 Bottlenecks abound
 Information transfer is initiated by client
65

66. Typical Request Flow
Load Balancer
Client
… Front End
… API
… Data

67. Typical Response Flow
Load Balancer
Client
… Front End
… API
… Data

68. faбrik Web Architecture
 Clients interact with the nearest “App Buddy” front-end
 The “App Buddy” is connected to the “Bad Rabbit” backbone
 The “Bad Rabbit” backbone is clustered regionally and federated globally
 NYT content producers connect directly to the backbone
 Information flow is bidirectional and event-driven
68

69. faбrik Information Flow
Client
Client Client
NYT Globally distributed
Internal “faбrik ” layer
69

70. faбrik – basic
App
Message Broker
App
App
70

71. faбrik – basic
Amazon Web Services
App
Message Broker
• EC2
• S3
App
• Identity & Access
App Mgt
• DynamoDB
• Route 53
…
71

72. faбrik – basic++
App Service
Buddy Buddy
Message Broker
“Retail”
Message Broker
“Wholesale”
Service
Buddy
OtherAp
p
72

73. faбrik: Current Implementation
 Open source:
– Erlang/OTP 14B04 – Sockjs (websockets +)
– RabbitMQ 2.8.7/3.pre – Python 2.6/2.7
– Nodejs 8.xx – ZeroMQ
 Automated deployment using CloudFormation
 DynamoDB & S3 for persistence
73

74. faбrik – active/active cluster
Region Wherever
Zone ‘a’ Zone ‘b’
Service Service
Buddy Buddy
‘a’ ‘b’
74

75. faбrik – active/active cluster
Region Wherever
Zone ‘a’ Zone ‘b’
Service Service
Buddy Buddy
‘a’ ‘b’
75

76. 76

77. faбrik
77

78. 78

79. 79

80. So why DynamoDB?
 faбrik services are reliable but stateless (mostly)
 A happy faбrik has short queues (measurable by the way)
 So persist everything as rapidly as possible (enter DynamoDB)
 Plus we want to gather & analyze
– Pulse: Map / Reduce, rapid cycle
– Longitudinal analysis
– Complex Event Processing in parallel (maybe)
Note: the faбrik is asynchronous and facilitates parallelization
80

81. DynamoDB requirements
 Store all messages crossing each ‘virtual host’
Note: think of a ‘virtual host’ as a horizontal band of related, reliable
services/endpoints across zones/instances in a region
 Store log messages for all application and system instances
 Facilitate ‘burst’ loads as well as steady state
 Support gather / analyze for all of the above
 Generational storage: DDB to S3 to Glacier (with some weeding)
 Fairly allocate resources among many competing endpoints
81

82. Conventional wisdom…
Uh oh – we have an unpredictable mix of all these…
82

83. More conventional wisdom…
“In addition to simple retries, we recommend using an exponential backoff algorithm
for better flow control. The concept behind exponential backoff is to use progressively
longer waits between retries for consecutive error responses. For example, up to 50
milliseconds before the first retry, up to 100 milliseconds before the second, up to 2400
milliseconds before third, and so on. However, after a minute, if the request has not
succeeded, the problem might be the request size exceeding your provisioned
throughput, and not the request rate. Set the maximum number of retries to stop around
one minute. If the request is not successful, investigate your provisioned throughput
options.” [i.e. increase provisioned throughput – hmmmm…]
83

84. So…
 We would have to provision for peaks
 Exponential backoff would give us about a 1 minute buffer
 But! The faбrik does buffering and we can monitor queue lengths
 Plus we have asynchronous event scheduling/handling facilities built in…
84

85. First strategy
 With node.js, asynchronously blast all requests at dynamo, reschedule exponentially
based on backpressure
 This worked pretty well!
– * Dynamo would deliver about 3 times stated capacity in bursts
– Nothing got lost
– Converged reasonably onto table capacity
 But…
– Problems exerting backpressure on the faбrik from node.js… hence requests could get scheduled
WAY into el futuro… and WAY out of order
– Competition among endpoints was ‘unfair’ and fostered convergence problems
85

86. Current strategy
 Be smarter, look for similar patterns and tested solutions, plus select tools that give
the right level of control
 Old dog:
– “I remember when TCP was new and throughput was not very high…”
(time passes)
– “The ‘ThroughputExceeded’ backpressure from DynamoDB is sort of like TCP backpressure…”
(more time passes)
– “Perhaps we could leverage that thought by applying the research and practices that have
improved TCP etc. to our use of DynamoDB.”
(time for a nap)
86

87. Current strategy
 Be smarter, look for similar patterns and tested solutions, plus select tools that give
the right level of control
 Token Bucket (circa 1986) for traffic shaping
“…an algorithm used in packet switched computer networks and telecommunications networks to
check that data transmissions conform to defined limits on bandwidth and burstiness.” – Wikipedia
 Additive Increase/Multiplicative Decrease (AIMD)
“…a feedback control algorithm best known for its use in TCP congestion control…combines linear
growth of the congestion window with an exponential reduction when congestion takes place…flows
will eventually converge to use equal amounts of a contended link.” – Wikipedia
 Explicit Congestion Notification (ECN) etc. etc.
87

88. Current strategy
 Be smarter, look for similar patterns and tested solutions, plus select reliable tools
that give the right level of control
 Tools:
– Use python to get a more mature and lower level event-driven interface (pika) to RabbitMQ –
easier to exert backpressure on the message source
– Use boto to get a mature interface to DynamoDB that can be easily ‘tweaked’ to give better
information about backpressure from DynamoDB (ThroughputExceeded exception)
– Use python’s concurrent futures to easily add asynchronous capability to boto, making use of
boto’s connection pooling
88

89. Managed Access to DynamoDB
89

90. Placeholder
 Here we show some code, describe the testing methodology briefly, and show
generated results.
90

91. Provisioned
throughput
Data patterns
Amazon DynamoDB

92. Thank you!
matthew@amazon.com
@mza

93. We are sincerely eager to
hear your FEEDBACK on this
presentation and on re:Invent.
Please fill out an evaluation
form when you have a
chance.