Asya is back, and so is Sherlock Holmes and his techniques to gather and analyze data from your poorly performing MongoDB clusters. In this advanced talk we take a deep look at all the diagnostic data that lives inside MongoDB - how to interrogate and interpret it to help you solve those frustrating performance bottlenecks that we all face occasionally.

2. Asya Kamsky,
Lead Product Manager
MongoDB
Diagnostics and Debugging 3.4

3. Asya Kamsky,
Lead Product Manager
MongoDB
Diagnostics and Debugging 3.4

6. “
I thought of my old teacher Joe Bell, ... of his eerie
trick of spotting details. If he were a detective he
would surely reduce this ... business to something
nearer an exact science.
—Arthur Conan Doyle

7. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red
x still appears, you may have to delete the image and then insert it again.

8. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red
x still appears, you may have to delete the image and then insert it again.

9. Understanding The Patient

10. #MDBW16
Understanding The Patient
Understand the system
Knowledge
Monitor trends over time
Trends
Record all metrics "at rest"
Baseline

13. Gathering Data
“Data! Data! Data!
I can't make bricks without clay.”
-- Sherlock Holmes,
The Adventure of the Copper Beeches

15. “There is nothing like first-hand evidence.”
-- Sherlock Holmes, A Study in Scarlet

16. Available Tools

17. #MDBW16
Available Tools: Say "AH"

18. #MDBW16
Available Tools: Say "AH"

19. #MDBW16
Available Tools: Say "AH"

20. #MDBW16
Available Tools: Say "AH"
> db.isMaster( )
> rs.conf( )
> rs.status( )
> sh.status( )
> db.version( )
> db.serverCmdLineOpts( )

21. #MDBW16
Available Tools: Say "AH"

22. #MDBW16
> show dbs
> show collections
> db.getCollectionNames().forEach(function(c) {
printjson(db.getCollection(c).getIndexes());
});
Available Tools: Say "AH"

23. #MDBW16
Available Tools: Say "AH"

24. #MDBW16
Available Tools: Compass

25. #MDBW16
Available Tools: OS
% free
% blockdev --report
% dmesg
% ulimit -a
% ifconfig, ip <...>, iptables
% iostat
% netstat
% top; htop
% perf
% iperf3

26. #MDBW16
Available Tools: OS for MongoDB
% mongostat
% mongotop

27. mongod logs

28. 2014-09-01T15:50:03.184-0004 [conn208] query
data.activity query: { $query: { _updated_at: { $gt: new
Date(1396459946346) }, _to: "_UserA" } }, $orderby:
{ _updated_at: -1 } } planSummary: IXSCAN { _to: 1,
_created_at: -1 } ntoreturn:100 ntoskip:0 nscanned:19692
nscannedObjects:19692 numYields:170 locks(micros) r:
283957 nreturned:65 reslen:24939 342ms
2014-09-01T15:50:03.184-0004 [conn208] query
data.activity query: { $query: { _updated_at: { $gt: new
Date(1396459946346) }, _to: "_UserA" } }, $orderby:
{ _updated_at: -1 } } planSummary: IXSCAN { _to: 1,
_created_at: -1 } ntoreturn:100 ntoskip:0 nscanned:19692
nscannedObjects:19692 numYields:170 locks(micros) r:
283957 nreturned:65 reslen:24939 342ms
mongod logs

29. 2015-05-26T23:33:32.774-0500 I
2015-05-28T12:37:11.440-0500 I
2015-05-28T12:38:35.839-0500 I
D, I, W, E, F
3.0 mongod logs

30. 2015-05-26T23:33:32.774-0500 I COMMAND
2015-05-28T12:37:11.440-0500 I NETWORK
2015-05-28T12:38:35.839-0500 I QUERY
3.0
COMMAND,NETWORK,QUERY,REPL,ACCESS,INDEX,JOURNAL,SHARDING,WRITE,-
mongod logs

31. 2015-05-26T23:33:32.774-0500 I COMMAND [conn45] command admin.$cmd command:
listDatabases { listDatabases: 1.0 } ntoskip:0 keyUpdates:0 writeConflicts:0
numYields:0 reslen:393 locks:{ Global: { acquireCount: { r: 12 } },
Database: { acquireCount: { r: 6 } } } 321ms
2015-05-28T12:37:11.440-0500 I NETWORK [initandlisten] connection accepted
from 127.0.0.1:48625 #183 (21 connections now open)
2015-05-28T12:38:35.839-0500 I QUERY [conn183] getmore
tableau.flights201406 query: { origin_city_market_id: 31703.0 } cursorid:
61957110347 ntoreturn:0 cursorExhausted:1 keyUpdates:0 writeConflicts:0
numYields:176 nreturned:22579 reslen:1332181 locks:{ Global: { acquireCount:
{ r: 354 } }, Database: { acquireCount: { r: 177 } }, Collection:
{ acquireCount: { r: 177 } } } 114ms
3.0 mongod logs

32. 3.2 mongod logs

33. 2016-06-23T10:34:41.559-0700 I COMMAND [conn26] command test.c
command: find { find: "c", filter: { a: { $elemMatch: { v: { $gte: 3.0, $lt:
4.0 } } } } } planSummary: IXSCAN { a.v: 1.0 }
keysExamined:2 docsExamined:1 fromMultiPlanner:1 cursorExhausted:1
keyUpdates:0 writeConflicts:0 numYields:1 nreturned:1 reslen:187
locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r:
2 } }, Collection: { acquireCount: { r: 2 } } } protocol:op_command 110ms
3.2 mongod logs

34. 2016-06-23T10:34:41.559-0700 I COMMAND [conn26] command test.c
command: find { find: "c", filter: { a: { $elemMatch: { v: { $gte: 3.0, $lt:
4.0 } } } } } planSummary: IXSCAN { a.v: 1.0 }
keysExamined:2 docsExamined:1 fromMultiPlanner:1 cursorExhausted:1
keyUpdates:0 writeConflicts:0 numYields:1 nreturned:1 reslen:187
locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r:
2 } }, Collection: { acquireCount: { r: 2 } } } protocol:op_command 110ms
3.2 mongod logs

35. 3.4 mongod logs

36. 3.4 mongod logs
2016-06-01T15:30:04.373-0700 I COMMAND [conn99] command socialite.following
command: aggregate { aggregate: "following", pipeline: [ { $match: { _f: "45705" } },
{ $group: { _id: null, followees: { $addToSet: "$_t" } } }, { $lookup: { from: "following",
localField: "followees", foreignField: "_f", as: "fofollowees" } }, { $project: { fofs: { $setUnion:
[ "$followees", "$fofollowees._t" ] } } } ] } planSummary: IXSCAN { _f: 1, _t: 1 } keysExamined:1
docsExamined:0 numYields:11 reslen:316214 locks:{ Global: { acquireCount: { r: 3800 } },
Database: { acquireCount: { r: 1900 } }, Collection: { acquireCount: { r: 1900 } } }
protocol:op_query 218ms

37. 3.4 mongod logs
2016-06-25T23:38:27.346-0500 I WRITE [conn128] update ycsb.usertable query:
{ _id: "user7074965863272626663" } planSummary: IDHACK update: { $set: { field1: BinData(0, 2
} } keysExamined:1 docsExamined:1 nMatched:1 nModified:1 numYields:1 locks:{ Global: {
acquireCount: { r: 3, w: 3 } }, Database: { acquireCount: { w: 3 } }, Collection: { acquireCount:
{ w: 2 } }, Metadata: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 11ms

38. > db.getLogComponents()

39. > db.getLogComponents()
{ "verbosity" : 1,
"accessControl" : { "verbosity" : -1},
"command" : { "verbosity" : -1},
"control" : { "verbosity" : -1},
"geo" : { "verbosity" : -1},
"index" : { "verbosity" : -1},
"network" : { "verbosity" : -1},
"query" : { "verbosity" : -1},
"replication" : { "verbosity" : -1},
"sharding" : { "verbosity" : -1},
"storage" : { "verbosity" : -1,
"journal" : { "verbosity" : -1}},
"write" : { "verbosity" : -1}
}

40. > db.getLogComponents()
{ "verbosity" : 1,
"accessControl" : { "verbosity" : -1},
"command" : { "verbosity" : -1},
"control" : { "verbosity" : -1},
"executor" : { "verbosity" : -1},
"geo" : { "verbosity" : -1},
"index" : { "verbosity" : -1},
"network" : { "verbosity" : -1},
"query" : { "verbosity" : -1},
"replication" : { "verbosity" : -1},
"sharding" : { "verbosity" : -1},
"storage" : { "verbosity" : -1,
"journal" : { "verbosity" : -1}},
"write" : { "verbosity" : -1},
"ftdc" : { "verbosity" : -1}
}

41. > db.setLogLevel( logLevel, component )
> db.setLogLevel( 1, "sharding" )
> db.setLogLevel( 2, "query" )

42. "It is of the highest importance ... to be able to recognize,
out of a number of facts, which are incidental and which vital. "
Sherlock Holmes, The Reigate Puzzle

45. https://github.com/rueckstiess/mtools
byThomasRückstieß

46. https://github.com/rueckstiess/mtools
byThomasRückstieß

47. #MDBW16
mplotqueries
$ mplotqueries --help
usage: mplotqueries [OPTIONS] [logfile [logfile ...]]
A script to plot various information from logfiles. ...

48. #MDBW16
mplotqueries
$ mplotqueries --help
usage: mplotqueries [OPTIONS] [logfile [logfile ...]]
A script to plot various information from logfiles. ...
optional arguments:
--type {nscanned/n,rsstate,connchurn,durline,histogram,range,scatter,event}
type of plot (default=scatter with --yaxis duration)

49. #MDBW16
mplotqueries
$ mplotqueries --help
usage: mplotqueries [OPTIONS] [logfile [logfile ...]]
A script to plot various information from logfiles. ...
optional arguments:
--type {nscanned/n,rsstate,connchurn,durline,histogram,range,scatter,event}
type of plot (default=scatter with --yaxis duration)
--group GROUP specify value to group on.
All basic plot types can group on
'namespace', 'operation', 'thread', 'pattern' ...

51. "... what is out of the common is usually a guide
rather than a hindrance."
— Sherlock Holmes, A Study in Scarlet

52. mplotqueries
% mplotqueries firstmongo.log --type nscanned/n

53. % mplotqueries firstmongo.log --type nscanned/n
2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary:
COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72
reslen:32707 30129ms
mplotqueries

54. % mplotqueries firstmongo.log --type nscanned/n
2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary:
COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72
reslen:32707 30129ms
mplotqueries

55. % mplotqueries firstmongo.log --type nscanned/n
2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary:
COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72
reslen:32707 30129ms
mplotqueries

56. % mplotqueries firstmongo.log --type nscanned/n
2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary:
COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72
reslen:32707 30129ms
2014-06-17T17:13:47.607 [conn1569990] query db2.coll query: { time: { $lt: "2014-06-17 17:14:05", $gte: "2014-06-17 17:05:05" }, status: 8 }
planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:2057564 nscannedObjects:2057564 keyUpdates:0 numYields:5008 locks(micros) r:11557172
nreturned:56 reslen:18745 13086ms
mplotqueries

57. % mplotqueries firstmongo.log --type nscanned/n
2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary:
COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72
reslen:32707 30129ms
2014-06-17T17:13:47.607 [conn1569990] query db2.coll query: { time: { $lt: "2014-06-17 17:14:05", $gte: "2014-06-17 17:05:05" }, status: 8 }
planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:2057564 nscannedObjects:2057564 keyUpdates:0 numYields:5008 locks(micros) r:11557172
nreturned:56 reslen:18745 13086ms
mplotqueries

58. % mplotqueries firstmongo.log --type nscanned/n
2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary:
COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72
reslen:32707 30129ms
2014-06-17T17:13:47.607 [conn1569990] query db2.coll query: { time: { $lt: "2014-06-17 17:14:05", $gte: "2014-06-17 17:05:05" }, status: 8 }
planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:2057564 nscannedObjects:2057564 keyUpdates:0 numYields:5008 locks(micros) r:11557172
nreturned:56 reslen:18745 13086ms
mplotqueries

60. % mplotqueries updates?.log

61. % mplotqueries updates?.log

62. % mplotqueries updates?.log

63. % mplotqueries updates?.log

64. % mplotqueries updates?.log

65. "Eliminate all other factors, and the
one which remains must be the truth."
Sherlock Holmes -The Sign of Four

66. #MDBW16
Available Tools
% mongostat

67. #MDBW16
mongostat

68. #MDBW16
--discover
mongostat

69. #MDBW16
mongostat
insert query update delete getmore command % dirty % used flushes vsize res faults qr|qw ar|aw netIn netOut conn ReplSetName role ts

70. #MDBW16
Trends

71. #MDBW16
Available Tools

72. #MDBW16
Available Tools: in the Cloud

73. #MDBW16
Available Tools: in the Cloud

74. #MDBW16
Available Tools: in the Cloud

75. #MDBW16
Available Tools: in the Cloud

76. #MDBW16
Available Tools: in the Cloud

77. #MDBW16
Available Tools: in the Cloud

78. #MDBW16
Available Tools: in the Cloud

79. #MDBW16
Available Tools: in the Cloud

80. #MDBW16
Available Tools: in the Cloud

83. % mplotqueries –type connchurn

95. #MDBW16
db.currentOp()

96. #MDBW16
db.currentOp()
> db.currentOp({secs_running:{$gt:10}})
{ "desc" : "conn3482",
"threadId" : "140512575297280",
"connectionId" : 3482,
"client_s" : "10.0.149.179:55295",
"active" : true,
"opid" : "SocialiteAtlas-shard-2:55547103",
"secs_running" : 13,
"microsecs_running" : NumberLong(13483812),
"op" : "query",
"ns" : "db1.coll1",
"numYields" : 48815,
"query" : {
"field1" : 5,

97. #MDBW16
db.currentOp()
> db.currentOp({desc:/^conn/,secs_running:{$gt:0}}).inprog.forEach(function(op) {
print(op.opid, op.op, op.ns, op.secs_running);
});
SocialiteAtlas-shard-0:68479014 query socialite.content 3
SocialiteAtlas-shard-0:68479730 query socialite.content 2
SocialiteAtlas-shard-0:68480174 query socialite.content 1
SocialiteAtlas-shard-0:68479234 query socialite.content 2
SocialiteAtlas-shard-0:68480136 query socialite.content 1
SocialiteAtlas-shard-0:68479529 query socialite.content 2
SocialiteAtlas-shard-0:68480182 query socialite.content 1
SocialiteAtlas-shard-0:68480261 query socialite.content 1
>
db.killOp ( opid )

99. #MDBW16

100. Asya Kamsky,
Lead Product Manager
MongoDB
Diagnostics and Debugging 3.4

102. Title Slide Option 1
Click here to add speaker name and title

103. Title Slide Option 2
Click here to add speaker name
and title

104. This is divider slide option 2

105. #MDBW16
Click to add title: keep it to 56 characters w/spaces
This is a typical content slide with full width body.
• First level bullet list
• Second level bullet list
• Third level bullet list

106. #MDBW16
Two content
Click to add text.
• First level bullet list
• Second level bullet list
• Third level bullet list
Click to add text.
• First level bullet list
• Second level bullet list
• Third level bullet list

107. #MDBW16
Left content
Click to add text.
• First level bullet list
• Second level bullet list

108. #MDBW16
Left content
Click to add text.
• First level bullet list
• Second level bullet list

109. #MDBW16
Pie Chart
64%
25%
11%
1st Qtr 2nd Qtr 3rd Qtr
1st Quarter
Lorem ipsum dolor sit amet, onsectetur adipiscing
elit. Praesent sodales odio sit amet odio tristique .
2nd Quarter
Lorem ipsum dolor sit amet, onsectetur adipiscing
elit. Praesent sodales odio sit amet odio tristique .
3rd Quarter
Lorem ipsum dolor sit amet, onsectetur adipiscing
elit. Praesent sodales odio sit amet odio tristique .

110. #MDBW16
Bar Graph
0
1
2
3
4
5
6
Category 1 Category 2 Category 3 Category 4
Chart Title
Series 1 Series 2 Series 3

111. #MDBW16
Column Header 1 Column Header 2 Column Header 3 Column Header 4 Column Header 5
Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet
Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet
Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet
Table

112. #MDBW16
Title only

113. #MDBW16
Coding Example – Light Background
// Retrieve
var MongoClient = require('mongodb').MongoClient;
// Connect to the db
MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {
if(err) { return console.dir(err); }
db.collection('test', function(err, collection) {});
db.collection('test', {w:1}, function(err, collection) {});
db.createCollection('test', function(err, collection) {});
db.createCollection('test', {w:1}, function(err, collection) {});
});

114. #MDBW16
Coding Example – Dark Background
// Retrieve
var MongoClient = require('mongodb').MongoClient;
// Connect to the db
MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {
if(err) { return console.dir(err); }
db.collection('test', function(err, collection) {});
db.collection('test', {w:1}, function(err, collection) {});
db.createCollection('test', function(err, collection) {});
db.createCollection('test', {w:1}, function(err, collection) {});
});

118. #MDBW16
List slide – can also be used for agenda
Lorem ipsum dolor sit amet,
onsectetur adipiscing elit. Praesent
sodales odio sit amet odio tristique
sit elit.
Lorem Ipsum
01 Lorem ipsum dolor sit amet,
onsectetur adipiscing elit. Praesent
sodales odio sit amet odio tristique
sit elit.
Lorem Ipsum
03Lorem ipsum dolor sit amet,
onsectetur adipiscing elit. Praesent
sodales odio sit amet odio tristique
sit elit.
Lorem Ipsum
02
Lorem ipsum dolor sit amet,
onsectetur adipiscing elit. Praesent
sodales odio sit amet odio tristique
sit elit.
Lorem Ipsum
05 Lorem ipsum dolor sit amet,
onsectetur adipiscing elit. Praesent
sodales odio sit amet odio tristique
sit elit.
Lorem Ipsum
06Lorem ipsum dolor sit amet,
onsectetur adipiscing elit. Praesent
sodales odio sit amet odio tristique
sit elit.
Lorem Ipsum
05

119. #MDBW16
Columns and icons with copy (option 1)
Lorem ipsum dolor sit
amet, onsectetur
adipiscing elit.
Praesent sodales odio
sit amet odio tristique.
Linked
Lorem ipsum dolor sit
amet, onsectetur
adipiscing elit.
Praesent sodales odio
sit amet odio tristique.
Planning
Lorem ipsum dolor sit
amet, onsectetur
adipiscing elit.
Praesent sodales odio
sit amet odio tristique.
Writing
Lorem ipsum dolor sit
amet, onsectetur
adipiscing elit.
Praesent sodales odio
sit amet odio tristique.
Research

120. #MDBW16
Columns and icons with copy (option 2)
Lorem ipsum dolor sit
amet, onsectetur
adipiscing elit.
Praesent sodales odio
sit amet odio tristique .
Linked
Lorem ipsum dolor sit
amet, onsectetur
adipiscing elit.
Praesent sodales odio
sit amet odio tristique .
Planning
Lorem ipsum dolor sit
amet, onsectetur
adipiscing elit.
Praesent sodales odio
sit amet odio tristique .
Writing
Lorem ipsum dolor sit
amet, onsectetur
adipiscing elit.
Praesent sodales odio
sit amet odio tristique .
Research

121. #MDBW16
Timeline or progress
2013
Lorem ipsum dolor sit amet,
onsectetur adipiscing elit.
Praesent sodales odio sit
amet odio tristique sit elit.
Lorem ipsum dolor sit amet,
onsectetur adipiscing elit.
Praesent sodales odio sit
amet odio tristique sit elit.
2014
2015
Lorem ipsum dolor sit amet,
onsectetur adipiscing elit.
Praesent sodales odio sit
amet odio tristique sit elit.
Lorem ipsum dolor sit amet,
onsectetur adipiscing elit.
Praesent sodales odio sit
amet odio tristique sit elit.
2016

122. “
Quote sample. Lorem ipsum dolor sit amet,
onsectetur adipiscing elit amet sodales. Praesent
sodales odio sit amet odio tristique. Lorem ipsum
dolor sit amet, onsectetur adipiscing elit. Praesent
sodales odio sit amet odio tristique. Lorem ipsum
dolor sit amet, onsectetur adipiscing elit.”