MongoDB is easy to download and run locally but requires some thought and further understanding when deploying to production. At scale, schema design, indexes and query patterns really matter. So does data structure on disk, sharding, replication and data centre awareness. This talk will examine these factors in the context of analytics, and more generally, to help you optimise MongoDB for any scale. Presented at MongoDB Days London 2013 by David Mytton.

1. Optimising for performance, scale, analytics

2. ~3333 write ops/s 0.07 - 0.05 ms response

3. David Mytton
Woop Japan!

5. MongoDB at Server Density

6. MongoDB at Server Density
•27 nodes

7. MongoDB at Server Density
•27 nodes
• June 2009 - +3yrs

8. MongoDB at Server Density
•27 nodes
• June 2009 - +3yrs
•MySQL -> MongoDB

9. MongoDB at Server Density
•27 nodes
• June 2009 - +3yrs
•MySQL -> MongoDB
•17TB data per month

10. MongoDB at Server Density
Queues
Primary
data store
Time series

11. Why?

12. Why?
• Replication

13. Why?
• Replication
• Official drivers

14. Why?
• Replication
• Official drivers
• Easy deployment

15. Why?
• Replication
• Official drivers
• Easy deployment
• Fast out of the box (sort of)

16. Fast out of the box?
Photo: dannychoo.com

17. ~3333 write ops/s 0.07 - 0.05 ms response

18. Fast out of the box?
• Softlayer cloud (1 core, 8GB)
Photo: dannychoo.com

19. Fast out of the box?
• Softlayer cloud (1 core, 8GB)
• Local instance storage
Photo: dannychoo.com

20. Fast out of the box?
• Softlayer cloud (1 core, 8GB)
• Local instance storage
• Ubuntu 10/12.04 LTS
Photo: dannychoo.com

22. Fast out of the box?
• Journaling
Photo: dannychoo.com

23. Fast out of the box?
• Journaling
• Replication
Photo: dannychoo.com

24. Fast out of the box?
Picture is unrelated! Mmm, ice cream.

25. Fast out of the box?
• Fast network
Picture is unrelated! Mmm, ice cream.

26. Fast out of the box?
• Fast network
• Working set in RAM
Picture is unrelated! Mmm, ice cream.

27. mongos> db.metrics_20120508_15_1m.stats()
{
"sharded" : true,
"flags" : 1,
"ns" : "metrics.metrics_20120508_15_1m",
"count" : 2752934,
"numExtents" : 46,
"size" : 746837640,
"storageSize" : 823717888,
"totalIndexSize" : 517581680,
"indexSizes" : {
"_id_" : 130358144,
"a_1_i_1" : 155711920,
"a_1_i_1_m_1_t_1" : 231511616
},
"avgObjSize" : 271.2878841265355,
"nindexes" : 3,
"nchunks" : 61,

28. "size" : 746837640,
"totalIndexSize" : 517581680
Indexes = 493MB Data = 712MB

29. "size" : 746837640,
"totalIndexSize" : 517581680
Indexes = 493MB Data = 712MB

30. "size" : 746837640,
"totalIndexSize" : 517581680
Indexes = 493MB Data = 712MB
Total = 1205MB

31. Where should it go?
Should it be in
What?
memory?
Indexes Always
Data If you can
http://www.flickr.com/photos/comedynose/4388430444/

32. How you’ll know
1) Slow queries
Thu Oct 14 17:01:11 [conn7410] update sd.apiLog
query: { c: "android/setDeviceToken", a: 1466, u:
"blah", ua: "Server Density Android" } 51926ms
www.flickr.com/photos/tonivc/2283676770/

33. How you’ll know
2) Timeouts
cursor timed out (20000 ms)

34. How you’ll know
3) Disk i/o spikes
www.flickr.com/photos/daddo83/3406962115/

35. Fast out of the box?
• Fast network
• Working set in RAM
Picture is unrelated! Mmm, ice cream.

36. Fast out of the box?
• Fast network
• Working set in RAM
• Fast disks (optional)
Picture is unrelated! Mmm, ice cream.

39. Growing documents is bad

40. Updates in place
db.my_collection.update( { _id : ... },
{ $inc : { y : 2 } } )

41. BSON

42. BSON

43. Growing documents is bad
> db.coll.stats()
{
"ns" : "...", ...,
"paddingFactor" : 1, ...,
"ok" : 1
}

44. Scaling writes
• Global DB lock

45. Scaling writes
• Global DB lock
• Concurrency

46. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

47. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

48. http://bit.ly/mongolock

49. Fast out of the box?
• Fast network
• Working set in RAM
• Fast disks (optional)
• Sharding (optional)
Picture is unrelated! Mmm, ice cream.

50. Scaling writes
• Global DB lock
• Concurrency
• Sharding

51. Scaling writes

53. Scaling writes
• Collection location

54. Scaling writes
• Collection location
• Pre-split / moveChunk

55. Scaling writes
• Collection location
• Pre-split / moveChunk
• Hashing (v2.4)

56. Failover
•Replica sets

57. Failover
•Replica sets
•Master/slave

58. Failover
•Replica sets
•Master/slave
•Min 3 nodes

59. Failover
•Replica sets
•Master/slave
•Min 3 nodes
•Automatic failover

60. rs.status()
{
! "_id" : 1,
! "name" : "rs3b:27018",
! "health" : 1,
! "state" : 2,
! "stateStr" : "SECONDARY",
! "uptime" : 1886098,
! "optime" : {
! ! "t" : 1291252178000,
! ! "i" : 13
! },
! "optimeDate" : ISODate("2010-12-02T01:09:38Z"),
"lastHeartbeat" : ISODate("2010-12-02T01:09:38Z")
},
www.ex-astris-scientia.org/inconsistencies/ent_vs_tng.htm (yes it’s a replicator from Star Trek)

61. rs.status()
1) myState
Value Meaning
0 Starting up (phase 1)
1 Primary
2 Secondary
3 Recovering
4 Fatal error
5 Starting up (phase 2)
6 Unknown state
7 Arbiter
8 Down

62. rs.status()
2) Optime
"optimeDate" : ISODate("2010-12-02T01:09:38Z")
www.flickr.com/photos/robbie73/4244846566/

63. rs.status()
3) Heartbeat
"lastHeartbeat" : ISODate("2010-12-02T01:09:38Z")
www.flickr.com/photos/drawblindfaith/3400981091/

64. Scaling reads

65. Scaling reads
•Replica slaves

66. Scaling reads
•Replica slaves
•Consistency

67. Scaling reads
•Replica slaves
•Consistency
•w flag / tags

68. WriteConcern
Changed Nov 27 2012

69. WriteConcern
• Safe by default
>>> from pymongo import MongoClient
>>> connection = MongoClient()

70. WriteConcern
• Safe by default
>>> from pymongo import MongoClient
>>> connection = MongoClient(w=int/str)
Value Meaning
0 Unsafe
1 Primary
2 Primary + x1 secondary
3 Primary + x2 secondaries

71. Tags
{
_id : "someSet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}
]
settings : {
getLastErrorModes : {
veryImportant : {"dc" : 3},
sortOfImportant : {"dc" : 2}
}
}
}
> db.foo.insert({x:1})
> db.runCommand({getLastError : 1, w : "veryImportant"})

72. Tags
{
_id : "someSet",
(A or B) + (C or D) + E
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}
]
settings : {
getLastErrorModes : {
veryImportant : {"dc" : 3},
sortOfImportant : {"dc" : 2}
}
}
}
> db.foo.insert({x:1})
> db.runCommand({getLastError : 1, w : "veryImportant"})

73. Tags
{
_id : "someSet",
(A + C) or (D + E) ...
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}
]
settings : {
getLastErrorModes : {
veryImportant : {"dc" : 3},
sortOfImportant : {"dc" : 2}
}
}
}
> db.foo.insert({x:1})
> db.runCommand({getLastError : 1, w : "sortOfImportant"})

74. WriteConcern
• Safe by default
•J

75. WriteConcern
• Safe by default
•J
• fsync

76. Bottlenecks
•EC2

77. Bottlenecks
•EC2
•Local storage

78. Bottlenecks
•EC2
•Local storage
•EBS: RAID10 4-8 volumes

79. Bottlenecks
•EC2
•Local storage
•EBS: RAID10 4-8 volumes
•i/o: rand but not sequential

80. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs

81. http://bit.ly/
ec2mongodb

82. Bottlenecks
•CPU
•Index building

83. Tips: rand()
•_id

84. Tips: rand()
•_id
•Field names

85. Tips: rand()
•_id
•Field names
•Covered indexes

86. Tips: rand()
•_id
•Field names
•Covered indexes
•Collections / databases

87. mongostat

88. mongostat
Locks/Queues

89. mongostat
Diagnostics

90. Current operations
db.currentOp();
{
! ! ! "opid" : "shard1:299939199",
! ! ! "active" : true,
! ! ! "lockType" : "write",
! ! ! "waitingForLock" : false,
! ! ! "secs_running" : 15419,
! ! ! "op" : "remove",
! ! ! "ns" : "sd.metrics",
! ! ! "query" : {
! ! ! ! "accId" : 1391,
! ! ! ! "tA" : {
! ! ! ! ! "$lte" : ISODate("2010-11-24T19:53:00Z")
! ! ! ! }
! ! ! },
! ! ! "client" : "10.121.12.228:44426",
! ! ! "desc" : "conn"
! ! },
www.flickr.com/photos/jeffhester/2784666811/

91. Monitoring tools
Run yourself
Ganglia

92. Monitoring tools
Server Density

93. Fast out of the box?
• Fast network
• Working set in RAM
Picture is unrelated! Mmm, ice cream.

94. Fast out of the box?
• Fast network
• Working set in RAM
• bit.ly/benchrun
Picture is unrelated! Mmm, ice cream.

95. www.serverdensity.com/mdb
Woop Japan!

96. David Mytton
@davidmytton
david@serverdensity.com
www.serverdensity.com
Woop Japan!