This webinar discusses Amazon DynamoDB, a NoSQL, highly scalable, SSD-based, zero administration database service in the AWS Cloud. We explain how DynamoDB works and also walk through some best practices and tips to get the most out of the service.

1. Introducing
DynamoDB
20th March, 2012
Dr. Matt Wood - matthew@amazon.com

2. Storage
Tools &
Compute
Support
Databases

3. Storage
Databases
Tools &
Compute
Support

4. Databases
Relational “NoSQL”
databases databases

5. Any database on Amazon EC2
MySQL, DB2, Oracle, PostgreSQL...

6. Relational Database Service
Managed MySQL and Oracle databases

7. Rapid High
provisioning availability
Scalable Scalable
storage compute
Relational Database Service
Managed MySQL and Oracle databases

8. High performance databases
Increase throughput
Increase availability
Reduce latency

9. High performance databases
Read replicas
Push-button
scaling Increase throughput
ElastiCache
Increase availability
Reduce latency

10. High performance databases
Increase throughput
Multi-AZ Increase availability
Reduce latency

11. High performance databases
Increase throughput
Increase availability
Reduce latency ElastiCache

12. Rich query semantics
Joins, transactions, query optimisation

13. Problem
Complexity. Performance decreases at scale.

14. Performance
Predictable, consistent
Scale

15. Performance
Predictable, consistent
Degraded performance
with scale
Scale

16. Performance
Predictable, consistent
Degraded performance
with scale
Scale

17. = more problems

18. Data caching
Provisioning!
Data sharding
= more problems
Cluster management
Fault management

19. Undifferentiated
heavy lifting

20. DynamoDB

21. Fully managed
NoSQL database
service

22. Offload admin and
operational burden

23. Extremely fast
performance

24. Seamless scalability

25. Focus on your stuff

27. AGENDA
Getting to know DynamoDB
Gu ided tour of service highlights
Provisioned throughput
Data model
DynamoDB in practice
Analytics with Elastic MapReduce

28. H I G H L I G H T S
Low latency Flexible
Large scale Durable storage
Seamless scaling Zero admin
Predictable performance

29. H I G H L I G H T S
SSD backed
Low latency
Single digit millisecond
< 5 ms reads < 10 ms writes

30. H I G H L I G H T S
Massive scale
No table size limits. Unlimited storage.

31. H I G H L I G H T S
Seamless scale
Live repartitioning. Zero admin.

32. H I G H L I G H T S
Flexible data model
Key/attribute store for evolving models

33. H I G H L I G H T S
Predictable performance
Provisioned throughput

34. H I G H L I G H T S
Durable and available
Consistent, disk-only writes

35. H I G H L I G H T S
Zero administration

36. What is provisioned
throughput?

37. Reserve required IOPS
Per table. Set at creation. Scale via API.

38. Scale at any time
No downtime

39. Pay for throughput

40. Per 1kb item:
$0.01 per hour for every 10 writes/second
$0.01 per hour for every 50 strongly
consistent reads/second

41. Per 1kb item:
$0.28 per million writes
$0.056 per million strongly consistent reads

42. Pay for storage
$1.00 per Gb per month of indexed storage

43. Data model
Flexible. Schema-less.

44. Simple key/value pairs
title => “Introduction to DynamoDB”
date => “20120320”

45. Associative array,
or Hash
[ title => “Introduction to DynamoDB”,
date => “20120320” ]

46. Attributes
[ title => “Introduction to DynamoDB”,
date => “20120320” ]

47. [ title => “Disaster Recovery with AWS”,
date => “20120320”,
format => “webinar”,
presenter => “Jeff Barr” ]
Attributes
[ title => “Introduction to DynamoDB”,
date => “20120320” ]

48. [ title => “Disaster Recovery with AWS”,
date => “20120320”,
format => “webinar”,
presenter => “Jeff Barr” ]
Items
[ title => “Introduction to DynamoDB”,
date => “20120320” ]

49. [ title => “Disaster Recovery with AWS”,
date => “20120328”,
format => “webinar”,
presenter => “Jeff Barr” ]
Table
[ title => “Introduction to DynamoDB”,
date => “20120320” ]

50. Table

51. Item
“ImageID” = “1”
“Date” =
“20100915”
“Title” = “flower”
“Tags” = “flower”,
“jasmine”, “white”

52. “ImageID” = “1” “ImageID” =”2” “ImageID” =”3”
“Date” = “Date” = “Date” =
“20100915” “20100916” “20100917”
“Title” = “flower” “Title” = “ferrari” “Title” = “coffee”
“Tags” = “flower”, “Tags” = “car”, “Tags” = “drink”,
“jasmine”, “white” “italian” “delicious”

53. “ImageID” = “1” Primary or hash key
“Date” = “20100915”
“Title” = “flower”
“Tags” = “flower”,
“jasmine”, “white”

54. “ImageID” = “1” Primary or hash key
“Date” = “20100915” Composite or range key
“Title” = “flower”
“Tags” = “flower”,
“jasmine”, “white”

55. “ImageID” = “1” Primary or hash key
“Date” = “20100915” Composite or range key
“Title” = “flower”
“Tags” = “flower”, Sets of strings
“jasmine”, “white”
or numbers

56. Best practice
Well balanced, fine grained hash keys.
Customer, order, item, etc. rather than store_id.

57. Simple API
Only 12 operations.

58. Consistency
Writes are always consistent.
Reads are consistent or eventually consistent.

59. Durability
Writes occur to disk, not memory.
Writes are acknowledged once they have been
made in two physical data centres.

60. Availability
Region specific (not AZ)
Continuously replicated across multiple AZs

61. Let’s take a look!
Building a simple DynamoDB powered web application

62. Threaded discussions
NP-Complete.me
Book reviews for programmers
Page view counts Tagging

63. np-complete.me

64. np-complete.me

65. np-complete.me/asin

66. np-complete.me/discuss

67. np-complete.me/discuss

68. Book
Thread
Thread
Thread
Reply
Reply

69. Book table
Book metadata
Hash key
asin => 0980576830

70. Book table
Book metadata
asin => 0980576830
title => “Host Your Website on the Cloud”
pages => “364”
list-price => “£31.49”

71. Book table
Book metadata, page views
asin => 0980576830
title => “Host Your Website on the Cloud”
pages => “364”
list-price => “£31.49”
views => 145

72. Book table
Book metadata, page views, book tags
asin => 0980576830
title => “Host Your Website on the Cloud”
pages => “364”
list-price => “£31.49”
views => 145
tags => [“php”, “aws”]

74. Thread table
Conversation thread
Hash key
Range key asin => 0980576830
subject => “Very informative”

75. Thread table
Conversation thread
Hash key
Range key asin => 0980576830
subject => “Very informative”
content => “This is a first class book...”
name => “Matt Wood”

76. Reply table
Conversation replies
Hash key
id => 0980576830:very-informative
Range key datetime => “20120320”

77. Reply table
Conversation replies
Hash key
id => 0980576830:very-informative
Range key datetime => “20120320”
reply => “I agree!”
name => “Werner Vogels”

79. DynamoDB tables
Books Threads Replies
(asin) (asin, subject) (id, datetime)

95. DynamoDB tables
Books Threads Replies
(asin) (asin, subject) (id, datetime)

96. Book Logical model
(asin)
Thread
(asin, subject)
Thread
(asin, subject)
Thread
(asin, subject)
Reply
(id, datetime)
Reply
(id, datetime)

97. Conditional writes
Client #1
DynamoDB asin => 1934356
pages => 384
Client #2
Time

98. Conditional writes
asin => 1934356 asin => 1934356
Client #1 pages => 384 pages => 502
DynamoDB asin => 1934356 asin => 1934356
pages => 384 pages => 502
asin => 1934356
Client #2 pages => 384
Time

99. Conditional writes
asin => 1934356 asin => 1934356
Client #1 pages => 384 pages => 502
DynamoDB asin => 1934356 asin => 1934356 ?
pages => 384 pages => 502
asin => 1934356 asin => 1934356
Client #2 pages => 384 pages => 450
Time

100. Conditional writes
asin => 1934356 asin => 1934356
Client #1 pages => 384 pages => 502
Failed condition
DynamoDB asin => 1934356 asin => 1934356
pages => 384 pages => 502
asin => 1934356 asin => 1934356
Client #2 pages => 384 pages => 450
Time

101. Atomic increment/decrement
asin => 0980576830
views => 145
tables[‘books’].items[‘0980576830’].attributes.add(:views => 1)
asin => 0980576830
views => 146

102. Tagging: many to many
Book
(asin, tags = [“php”, “aws”])
Query by key, retrieve tag collection
Add tags conditionally
No secondary indexes
Retrieve all books by tag

103. Tagging: many to many
Book
(asin, tags = [“php”, “aws”])
Tag
(tag, asin =
[“1449393683”, “0596515812”])
Query by book, retrieve tag collection
Query by tag, retrieve book collection

104. Autoscaling via SNS
$Res = $DDB->describe_table(array('TableName' => 'books'));
$Read = (int) $Res->body->Table->ProvisionedThroughput->ReadCapacityUnits;
$Write = (int) $Res->body->Table->ProvisionedThroughput->WriteCapacityUnits;
$Read *= 2;
$Write *= 2;
$PT = array('ReadCapacityUnits' => (string) $Read,
'WriteCapacityUnits' => (string) $Write);
$Res = $ddb->update_table(array('TableName' => 'books',
'ProvisionedThroughPut' => $PT));

105. Autoscaling via SNS
$Res = $DDB->describe_table(array('TableName' => 'books'));
$Read = (int) $Res->body->Table->ProvisionedThroughput->ReadCapacityUnits;
$Write = (int) $Res->body->Table->ProvisionedThroughput->WriteCapacityUnits;
$Read *= 2;
$Write *= 2;
$PT = array('ReadCapacityUnits' => (string) $Read,
'WriteCapacityUnits' => (string) $Write);
$Res = $ddb->update_table(array('TableName' => 'books',
'ProvisionedThroughPut' => $PT));

106. Autoscaling via SNS
$Res = $DDB->describe_table(array('TableName' => 'books'));
$Read = (int) $Res->body->Table->ProvisionedThroughput->ReadCapacityUnits;
$Write = (int) $Res->body->Table->ProvisionedThroughput->WriteCapacityUnits;
$Read *= 2;
$Write *= 2;
$PT = array('ReadCapacityUnits' => (string) $Read,
'WriteCapacityUnits' => (string) $Write);
$Res = $ddb->update_table(array('TableName' => 'books',
'ProvisionedThroughPut' => $PT));

107. Autoscaling via SNS
$Res = $DDB->describe_table(array('TableName' => 'books'));
$Read = (int) $Res->body->Table->ProvisionedThroughput->ReadCapacityUnits;
$Write = (int) $Res->body->Table->ProvisionedThroughput->WriteCapacityUnits;
$Read *= 2;
$Write *= 2;
$PT = array('ReadCapacityUnits' => (string) $Read,
'WriteCapacityUnits' => (string) $Write);
$Res = $ddb->update_table(array('TableName' => 'books',
'ProvisionedThroughPut' => $PT));

108. Autoscaling via SNS
$Res = $DDB->describe_table(array('TableName' => 'books'));
$Read = (int) $Res->body->Table->ProvisionedThroughput->ReadCapacityUnits;
$Write = (int) $Res->body->Table->ProvisionedThroughput->WriteCapacityUnits;
$Read *= 2;
$Write *= 2;
$PT = array('ReadCapacityUnits' => (string) $Read,
'WriteCapacityUnits' => (string) $Write);
$Res = $ddb->update_table(array('TableName' => 'books',
'ProvisionedThroughPut' => $PT));

109. Considerations
Limited index and query model
Throughput is provisioned in 1K operations
Maximum 64K item size
Backup and restore via Elastic MapReduce

110. Elastic MapReduce
Built for data. Designed for humans.

111. Collection Computation Collaboration

112. Collection Computation Collaboration
DynamoDB Elastic MapReduce
Amazon S3 Amazon EC2

113. DynamoDB
Data

114. DynamoDB
Data
Code Elastic
MapReduce

115. DynamoDB
Data
Code Elastic Name
MapReduce node

116. DynamoDB
Data
Code Elastic Name
MapReduce node
Elastic
cluster

117. DynamoDB
Data
Code Elastic Name
MapReduce node
HDFS
Elastic
cluster

118. DynamoDB
Data
Code Elastic Name Output
MapReduce node S3
HDFS
Elastic
cluster

119. DynamoDB
Data
Output
S3

120. Export to S3
CREATE EXTERNAL TABLE orders_s3_new_export ( order_id
string, customer_id string, order_date int, total
double )
PARTITIONED BY (year string, month string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION 's3://export_bucket';
INSERT OVERWRITE TABLE
orders_s3_new_export
PARTITION (year='2012', month='01')
SELECT * from orders_ddb_2012_01;

121. Live data in DynamoDB
SELECT customer_id, sum(total) spend, count(*)
order_count
FROM orders_ddb_2012_01
WHERE order_date >= unix_timestamp('2012-01-01', 'yyyy-
MM-dd')
AND order_date < unix_timestamp('2012-01-08', 'yyyy-MM-
dd')
GROUP BY customer_id
ORDER BY spend desc
LIMIT 5;

122. Live and archive data

123. AGENDA
Getting to know DynamoDB
Gu ided tour of service highlights
Provisioned throughput
Data model
DynamoDB in practice
Analytics with Elastic MapReduce
Slides available shortly.

124. DynamoDB free tier
5 writes/second
10 consistent reads/second
100Mb storage

125. Developer Guide
aws.amazon.com/documentation/dynamodb

126. Drop us a line!
aws.amazon.com/contact-us

127. Thank you!

128. Q&A
matthew@amazon.com

129. SimpleDB
Zero maintenance, NoSQL datastore

130. Flexible queries
10Gb / 1 billion No native
attributes per tabel data sharding
SimpleDB
Zero maintenance, NoSQL datastore