Sam Harley, Solutions Architect, MongoDB

1. #MDBlocal
ETL for Pros: Getting Data Into
MongoDB
Sam Harley, Senior Solutions Architect

2. #MDBlocal
Agenda
● Discuss the MongoDB document model
● Discuss how data can be migrated to MongoDB
● Discuss important MongoDB schema design considerations
● Talk through common approaches to MongoDB schema migration
● Outline how threading and batching can be leveraged for ETL purposes

3. #MDBlocal
Introduction
● At some point, most applications need to batch-load large amounts of data
○ Billions of rows
○ Huge initial load
○ Daily updates (CDC)
● This is typically facilitated using an ETL platform

4. #MDBlocal
Schema Design Principles

5. #MDBlocal
The Document Model
{
first_name: ‘Paul’,
surname: ‘Miller’,
city: ‘London’,
location: {
type : ‘Point’,
coordinates : [45.123,47.232]
},
cars: [
{ model: ‘Bentley’,
year: 1973,
value: 100000, … },
{ model: ‘Rolls Royce’,
year: 1965,
value: 330000, … }
]
}
MongoDB
RDBMS

6. #MDBlocal
Documents are Rich Data Structures
{
first_name: ‘Paul’,
surname: ‘Miller’,
cell: 447557505611,
city: ‘London’,
location: {
type : ‘Point’,
coordinates : [45.123,47.232]
},
Profession: [‘banking’, ‘finance’, ‘trader’],
cars: [
{ model: ‘Bentley’,
year: 1973,
value: 100000, … },
{ model: ‘Rolls Royce’,
year: 1965,
value: 330000, … }
]
}
Fields can contain an array
of sub-documents
Typed field values
Fields can contain arrays
Number
Fields

7. #MDBlocal
Two Big ‘Rules’
● Consider how the data will be used
● Data that works together lives together

8. #MDBlocal
Example > Guitar Collectors
ERD
Relational DB MongoDB
Table → Collection
Column → Field
Row → Document
Terminology

9. #MDBlocal
Example > Guitar Collectors
Relational
Consider your consumers:
how will this data be used?

10. #MDBlocal
Example > Guitar Collectors
MongoDB - Embedding

11. #MDBlocal
Example > Guitar Collectors
Typical queries from a consumer of this data might be:
● “What guitars does Aimee Doe own?”
● “Show me all the people who own a Gibson
Les Paul of any age”
● “How many people within 100 miles of Seattle
own guitars made before 1970?”
Consider how the data will be used
X

12. #MDBlocal
Example > Guitar Collectors
Consider how the data will be used
Typical queries from a consumer of this data might be:
● “What guitars does Aimee Doe own?”
● “Show me all the people who own a Gibson
Les Paul of any age”
● “How many people within 100 miles of Seattle
own guitars made before 1970?”

13. #MDBlocal
• Ab Initio
• Talend
• Pentaho
• Informatica
How Do I Migrate Data?

14. #MDBlocal
WYOC (Write Your Own Code)
More challenging, but you have got
ultimate control
How Do I Migrate Data?

15. #MDBlocal
How do I do it efficiently?
Image: Julian Lim

16. #MDBlocal
An Example

17. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED

18. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED

19. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED

20. #MDBlocal
{
"first_name" : "James",
"last_name" : "Bond",
"address" : "Nassau, Bahamas, US",
"items" : [
{ "qty": 1, "description" : "Aston Martin", "price" : 120000 },
{ "qty": 1, "description" : "Dinner Jacket", "price" : 4000 },
{ "qty": 3, "description" : "Champagne Veuve-Cliquot", "price": 200 }
],
"tracking" : [
{ "timestamp" : "1985-04-30 09:48:00", "status": "ORDERED" }
]
}

21. #MDBlocal
{
"first_name" : "James",
"last_name" : "Bond",
"address" : "Nassau, Bahamas, US",
"items" : [
{ "qty": 1, "description" : "Aston Martin", "price" : 120000 },
{ "qty": 1, "description" : "Dinner Jacket", "price" : 4000 },
{ "qty": 3, "description" : "Champagne Veuve-Cliquot", "price": 200 }
],
"tracking" : [
{ "timestamp" : "1985-04-30 09:48:00", "status": "ORDERED" }
]
}

22. #MDBlocal
{
"first_name" : "James",
"last_name" : "Bond",
"address" : "Nassau, Bahamas, US",
"items" : [
{ "qty": 1, "description" : "Aston Martin", "price" : 120000 },
{ "qty": 1, "description" : "Dinner Jacket", "price" : 4000 },
{ "qty": 3, "description" : "Champagne Veuve-Cliquot", "price": 200 }
],
"tracking" : [
{ "timestamp" : "1985-04-30 09:48:00", "status": "ORDERED" }
]
}

23. #MDBlocal
{
"first_name" : "James",
"last_name" : "Bond",
"address" : "Nassau, Bahamas, US",
"items" : [
{ "qty": 1, "description" : "Aston Martin", "price" : 120000 },
{ "qty": 1, "description" : "Dinner Jacket", "price" : 4000 },
{ "qty": 3, "description" : "Champagne Veuve-Cliquot", "price": 200 }
],
"tracking" : [
{ "timestamp" : "1985-04-30 09:48:00", "status": "ORDERED" }
]
}

24. #MDBlocal
Three Common Approaches

25. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED

26. #MDBlocal
Approach #1 – Nested Queries
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id
doc.items.push (y)
for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id
doc.tracking.push (y)
mongodb.insert (doc)

27. #MDBlocal
Approach #1 – Nested Queries
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id
doc.items.push (y)
for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id
doc.tracking.push (y)
mongodb.insert (doc)

28. #MDBlocal
Approach #1 – Nested Queries
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id
doc.items.push (y)
for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id
doc.tracking.push (y)
mongodb.insert (doc)

29. #MDBlocal
Approach #1 – Nested Queries
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id
doc.items.push (y)
for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id
doc.tracking.push (y)
mongodb.insert (doc)

30. #MDBlocal
Approach #1 – Nested Queries
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id
doc.items.push (y)
for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id
doc.tracking.push (y)
mongodb.insert (doc)

31. #MDBlocal
Approach #1 – Nested Queries
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id
doc.items.push (y)
for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id
doc.tracking.push (y)
mongodb.insert (doc)

32. #MDBlocal
Fan-In & Fan-Out
Number of Database Operations per MongoDB Document
ETL Job
1 per order
+ 2
1

33. #MDBlocal
Benchmark Results
14.5
0
5
10
15
20
Time (min)
Nested Queries
• 1 million orders
• 10 million line items
• 3 million tracking states
• MySQL (local) to MongoDB (local)
• Python

34. #MDBlocal
Approach #2 – Build Documents In DB
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
mongodb.insert (doc)
for y in SELECT * FROM ITEMS
mongodb.update ({"_id" : y.order_id},
{"$push" : {"items" : y}})
for z in SELECT * FROM TRACKING
mongodb.update ({"_id" : z.order_id},
{"$push" : {"tracking" : z}})

35. #MDBlocal
Approach #2 – Build Documents In DB
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
mongodb.insert (doc)
for y in SELECT * FROM ITEMS
mongodb.update ({"_id" : y.order_id},
{"$push" : {"items" : y}})
for z in SELECT * FROM TRACKING
mongodb.update ({"_id" : z.order_id},
{"$push" : {"tracking" : z}})

36. #MDBlocal
Approach #2 – Build Documents In DB
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
mongodb.insert (doc)
for y in SELECT * FROM ITEMS
mongodb.update ({"_id" : y.order_id},
{"$push" : {"items" : y}})
for z in SELECT * FROM TRACKING
mongodb.update ({"_id" : z.order_id},
{"$push" : {"tracking" : z}})

37. #MDBlocal
Approach #2 – Build Documents In DB
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
mongodb.insert (doc)
for y in SELECT * FROM ITEMS
mongodb.update ({"_id" : y.order_id},
{"$push" : {"items" : y}})
for z in SELECT * FROM TRACKING
mongodb.update ({"_id" : z.order_id},
{"$push" : {"tracking" : z}})

38. #MDBlocal
Approach #2 – Build Documents In DB
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
mongodb.insert (doc)
for y in SELECT * FROM ITEMS
mongodb.update ({"_id" : y.order_id},
{"$push" : {"items" : y}})
for z in SELECT * FROM TRACKING
mongodb.update ({"_id" : z.order_id},
{"$push" : {"tracking" : z}})

39. #MDBlocal
Approach #2 – Build Documents In DB
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
mongodb.insert (doc)
for y in SELECT * FROM ITEMS
mongodb.update ({"_id" : y.order_id},
{"$push" : {"items" : y}})
for z in SELECT * FROM TRACKING
mongodb.update ({"_id" : z.order_id},
{"$push" : {"tracking" : z}})

40. #MDBlocal
Fan-In & Fan-Out
Number of Database Operations per MongoDB Document
ETL Job3 per order 1 + i + t
i = # of assoc. line items
t = # of assoc. tracking rows

41. #MDBlocal
Benchmark Results
14.5
95.9
0
20
40
60
80
100
120
Time (min)
Nested Queries Build in DB

42. #MDBlocal
Approach #3 – Load It All Into Memory
db_items = SELECT * FROM ITEMS
db_tracking = SELECT * FROM TRACKING
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
doc.items.pushAll (db_items.getAll(x.order_id))
doc.tracking.pushAll (db_tracking.getAll(x.order_id))
mongodb.insert (doc)

43. #MDBlocal
Approach #3 – Load It All Into Memory
db_items = SELECT * FROM ITEMS
db_tracking = SELECT * FROM TRACKING
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
doc.items.pushAll (db_items.getAll(x.order_id))
doc.tracking.pushAll (db_tracking.getAll(x.order_id))
mongodb.insert (doc)

44. #MDBlocal
Approach #3 – Load It All Into Memory
db_items = SELECT * FROM ITEMS
db_tracking = SELECT * FROM TRACKING
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
doc.items.pushAll (db_items.getAll(x.order_id))
doc.tracking.pushAll (db_tracking.getAll(x.order_id))
mongodb.insert (doc)

45. #MDBlocal
Approach #3 – Load It All Into Memory
db_items = SELECT * FROM ITEMS
db_tracking = SELECT * FROM TRACKING
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
doc.items.pushAll (db_items.getAll(x.order_id))
doc.tracking.pushAll (db_tracking.getAll(x.order_id))
mongodb.insert (doc)

46. #MDBlocal
Approach #3 – Load It All Into Memory
db_items = SELECT * FROM ITEMS
db_tracking = SELECT * FROM TRACKING
for x in SELECT * FROM ORDERS
doc = { "first_name" : x.first_name,
"last_name" : x.last_name,
"address" : x.address,
"items" : [], "tracking" : [] }
doc.items.pushAll (db_items.getAll(x.order_id))
doc.tracking.pushAll (db_tracking.getAll(x.order_id))
mongodb.insert (doc)

47. #MDBlocal
Fan-In & Fan-Out
Number of Database Operations per MongoDB Document
ETL Job3 per order 1

48. #MDBlocal
Benchmark Results
14.5
95.9
8.5
0
20
40
60
80
100
120
Time (min)
Nested Queries Build in DB Lookup from Memory

49. #MDBlocal
Co-Iteration

50. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED

51. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED

52. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED
{
"first_name" : "James",
"last_name" : "Bond",
"address" : "Nassau, Bahamas, US"
}

53. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED
{
"first_name" : "James",
"last_name" : "Bond",
"address" : "Nassau, Bahamas, US",
"items" : [
{ ..., "description" : "Aston Martin", ... }
]
}

54. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED
{
"first_name" : "James",
"last_name" : "Bond",
"address" : "Nassau, Bahamas, US",
"items" : [
{ ..., "description" : "Aston Martin", ... },
{ ..., "description" : "Dinner Jacket", ... }
]
}

55. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED
{
"first_name" : "James",
"last_name" : "Bond",
"address" : "Nassau, Bahamas, US",
"items" : [
{ ..., "description" : "Aston Martin", ... },
{ ..., "description" : "Dinner Jacket", ... },
{ ..., "description" : "Champagne...", ... }
]
}

56. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED
{
"first_name" : "James",
"last_name" : "Bond",
"address" : "Nassau, Bahamas, US",
"items" : [
{ ..., "description" : "Aston Martin", ... },
{ ..., "description" : "Dinner Jacket", ... },
{ ..., "description" : "Champagne...", ... }
]
}

57. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED
{
"first_name" : "James",
"last_name" : "Bond",
"address" : "Nassau, Bahamas, US",
"items" : [
{ ..., "description" : "Aston Martin", ... },
{ ..., "description" : "Dinner Jacket", ... },
{ ..., "description" : "Champagne...", ... }
],
"tracking" : [
{ ... "1985-04-30 09:48:00", ... "ORDERED" }
]
}

58. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED
{
"first_name" : "James",
"last_name" : "Bond",
"address" : "Nassau, Bahamas, US",
"items" : [
{ ..., "description" : "Aston Martin", ... },
{ ..., "description" : "Dinner Jacket", ... },
{ ..., "description" : "Champagne...", ... }
],
"tracking" : [
{ ... "1985-04-30 09:48:00", ... "ORDERED" }
]
}

59. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED

60. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED
{
"first_name" : "Ernst",
"last_name" : "Blofeldt",
"address" : "Caracas, Venezuela"
}

61. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED
{
"first_name" : "Ernst",
"last_name" : "Blofeldt",
"address" : "Caracas, Venezuela",
"items" : [
{ ..., "description" : "Cat Food", ... }
]
}

62. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED
{
"first_name" : "Ernst",
"last_name" : "Blofeldt",
"address" : "Caracas, Venezuela",
"items" : [
{ ..., "description" : "Cat Food", ... },
{ ..., "description" : "Launch Pad", ... }
]
}

63. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED
{
"first_name" : "Ernst",
"last_name" : "Blofeldt",
"address" : "Caracas, Venezuela",
"items" : [
{ ..., "description" : "Cat Food", ... },
{ ..., "description" : "Launch Pad", ... }
],
"tracking" : [
{ ... "1985-04-23 01:30:22", ... "ORDERED" }
]
}

64. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED
{
"first_name" : "Ernst",
"last_name" : "Blofeldt",
"address" : "Caracas, Venezuela",
"items" : [
{ ..., "description" : "Cat Food", ... },
{ ..., "description" : "Launch Pad", ... }
],
"tracking" : [
{ ... "1985-04-23 01:30:22", ... "ORDERED" },
{ ... "1985-04-25 08:30:00", ... "SHIPPED" }
]
}

65. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED
{
"first_name" : "Ernst",
"last_name" : "Blofeldt",
"address" : "Caracas, Venezuela",
"items" : [
{ ..., "description" : "Cat Food", ... },
{ ..., "description" : "Launch Pad", ... }
],
"tracking" : [
{ ... "1985-04-23 01:30:22", ... "ORDERED" },
{ ... "1985-04-25 08:30:00", ... "SHIPPED" },
{ ... "1985-05-14 21:37:00", .. "DELIVERED" }
]
}

66. #MDBlocal
ORDER
S
TRACKING
ITEM
S
ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS
1 James Bond Nassau, Bahamas, US
2 Ernst Blofeldt Caracas, Venezuela
ID ORDER_ID QTY DESCRIPTION PRICE
1 1 1 Aston Martin 120,000
2 1 1 Dinner Jacket 4,000
3 1 3 Champagne Veuve-Cliquot 200
4 2 100 Cat Food 1
5 2 1 Launch Pad 1,000,000
ORDER_ID TIMESTAMP STATUS
1 1985-04-30 09:48:00 ORDERED
2 1985-04-23 01:30:22 ORDERED
2 1985-04-25 08:30:00 SHIPPED
2 1985-05-14 21:37:00 DELIVERED
Done!

67. #MDBlocal
Fan-In & Fan-Out
Number of Database Operations per MongoDB Document
ETL Job3 per order 1

68. #MDBlocal
Benchmark Results
14.5
95.9
8.5 8.1
0
20
40
60
80
100
120
Time (min)
Nested Queries Build in DB Lookup from Memory Co-Iteration

69. #MDBlocal
Oh, And One More Thing …

70. #MDBlocal
Threading & Batching
Batch
Size
Threads
Throughput

71. #MDBlocal
Fan-In & Fan-Out
Number of Database Operations per MongoDB Document
ETL Job3 per order
1 for every
1000 orders

72. #MDBlocal
Benchmark Results
14.5 9.1
95.9
36.2
8.5 48.1 3.9
0
20
40
60
80
100
120
Simple Batch = 1000
Nested Queries Build in DB Lookup from Memory Co-Iteration

73. #MDBlocal
Summary
● Remember the two big rules regarding MongoDB schema design
○ Consider how the data will be used
○ Data that works together lives together
● Consider the common approaches to MongoDB schema migration
● Prototype, prototype, prototype
● Don’t forget to utilise batching and threading

74. #MDBlocal
What Now?
university.mongodb.comAsk The Experts
Sydney MongoDB
User Group

75. #MDBlocal
THANK YOU
FOR JOINING!