Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Deciphering Explain Output
Charlie Swanson
Charlie Swanson
Software Engineer - Query Team
Goals Of This Talk
Understand how
MongoDB answers
queries
Knowledge
Figure out what's
going on
Debugging
Learn some tricks...
Non-goals of This Talk
Picking the best
indexes for your
needs
Index Selection
It's mostly extends
naturally
Sharded
Clust...
Overview
Make explain() exciting
Motivation
01 Different Verbosities
Diagnostics
Query
03Why Do You Care?
What is explain(...
Overview
Make explain() exciting
Motivation
01 Different Verbosities
Diagnostics
Query
03Why Do You Care?
What is explain(...
Monitoring the Spread
Our Task
MongoDB
created by Christopher Holm-Hansen from Noun Project
Overview
Make explain() exciting
Motivation
01 Different Verbosities
Diagnostics
Query
03Why Do You Care?
What is explain(...
01: What Is explain()?
Example Explain Output
> db.foo.find({"nFollowers": {$gte:
1000}}).explain()
{
"queryPlanner" : {
"winningPlan" : {
"stage...
Why Do You Care?
Your Application MongoDB
created by Mike Ashley from Noun Project
Why Do You Care?
Your Application MongoDB
Documents
Please!
Why Do You Care?
Your Application MongoDB
Hmm… Let
me think
about
that…
Why Do You Care?
Your Application MongoDB
Ah! Here
are your
results!
Why Do You Care?
Your Application MongoDB
What took
you so
long?!
Overview
Make explain() exciting
Motivation
01 Different Verbosities
Diagnostics
Query
03Why Do You Care?
What is explain(...
Overview of Query Explain
• Query Plans
• explain() Output Format
• Different Verbosities
Query Plans
//	
  db.tweets.ensureIndex({hashtags:	
  1});	
  
db.tweets.find({hashtags:	
  "#MDBW16"})	
  
	
  	
  	
  .s...
Query Plans
db.tweets.find({hashtags:	
  "#MDBW16"})	
  
	
  	
  	
  .sort({nFavorites:	
  -­‐1});
FETCH
INDEX SCAN
{hasht...
Query Plans
db.tweets.find({hashtags:	
  "#MDBW16"})	
  
	
  	
  	
  .sort({nFavorites:	
  -­‐1});
FETCH
INDEX SCAN
{hasht...
Query Plans
db.tweets.find({hashtags:	
  "#MDBW16"})	
  
	
  	
  	
  .sort({nFavorites:	
  -­‐1});
FETCH
INDEX SCAN
{hasht...
Query Plans
db.tweets.find({$or:	
  [	
  
	
  	
  	
  	
  {hashtag:	
  {"#MongoDB"}},	
  
	
  	
  	
  	
  {username:	
  {$...
Explain Output
>	
  db.collection.find(…).explain()	
  
{	
  
	
   "queryPlanner"	
  :	
  {	
  
	
   	
   …	
  
	
   	
   ...
Explain Output
>	
  db.collection.find(…).explain()	
  
{queryPlanner:	
  {	
  
	
  	
  	
  winningPlan:	
  {	
  
	
  	
  ...
Explain Output
>	
  db.collection.find(…).explain()	
  
{	
  
…	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  stage:	
  ...
Explain Output
>	
  db.collection.find(…).explain()	
  
{	
  
	
   "queryPlanner"	
  :	
  {	
  
	
   	
   …	
  
	
   	
   ...
>	
  db.collection.find(…).explain()	
  
{"queryPlanner"	
  :	
  {	
  
	
   "winningPlan"	
  :	
  {	
  
	
   	
   {"stage"...
Explain Output
>	
  db.collection.find(…).explain()	
  
{	
  
	
   "queryPlanner"	
  :	
  {	
  
	
   	
   …	
  
	
   	
   ...
Applying This Information
Is your query using the index you expect?
Is your query using the index you expect?
FETCH
SORT
✓ COLLECTION
SORT
✗
INDEX SCAN
keyPattern: {nFollowers:
Is your query using an index to provide the sort?
Is your query using an index to provide the sort?
FETCH
INDEX SCAN
✓ ✗FETCH
INDEX SCAN
SORT
Is your query using an index to provide the sort?
✓SORT_MERGE
INDEX SCAN
FETCH
INDEX SCAN
Is your query using an index to provide the
projection?
PROJECTI
INDEX SCAN
✓ ✗
PROJECTI
INDEX SCAN
FETCH
Is your query using an index to provide the
projection?
Is your query using an index to provide the
projection?
//	
  With	
  index	
  
//	
  {nFollowers:	
  1,	
  username:	
  1...
Some Questions We Can Answer
• Is your query using the index you expect?
• Is your query using an index to provide the sor...
Some Questions We Can't Yet Answer
• How selective is your index?
• What is the most expensive part of your plan?
• Why wa...
Query Explain Modes
• "queryPlanner" (default)
• "executionStats"
• "allPlansExecution"
Query Explain Modes
• "queryPlanner" (default)
>	
  db.collection.find(…).explain()	
  
{"queryPlanner"	
  :	
  {	
  
	
  ...
• "queryPlanner" (default)
• "executionStats"
• "allPlansExecution"
Query Explain Modes
✓
• "executionStats"
Query Explain Modes
>	
  db.collection.find(…).explain()	
  
{"queryPlanner"	
  :	
  {	
  
	
   "winnin...
• "executionStats"
Query Explain Modes
created by Mike Ashley from Noun Project
created by Creative Stall from Noun Project
Explain Mode: "executionStats"
>	
  db.tweets.find(…).explain("executionStats")	
  
{	
  
	
   "queryPlanner"	
  :	
  {	
 ...
Explain Mode: "executionStats"
>	
  db.tweets.find(…).explain("executionStats")	
  
{	
  
	
  	
  "queryPlanner"	
  :	
  {...
Explain Mode: "executionStats"
>	
  db.tweets.find(…).explain("executionStats")	
  
{
…,
"executionStats" : {
// Top-level...
Explain Mode: "executionStats"
db.tweets.find(…).explain("executionStats")
{
"executionStats" : {
// Top-level stats.
"exe...
db.tweets.find(…).explain("executionStats")
{
"executionStats" : {
// Top-level stats.
"executionStages" : {
"stage" : "SO...
Execution Stats: works, advanced, etc.
• These are all PlanStages
• SortStage
• FetchStage
• IndexScanStage FETCH
SORT
IND...
Execution Stats: works, advanced, etc.
• These are all PlanStages
• Each PlanStage implements
work()
FETCH
SORT
INDEX SCAN...
Execution Stats: works, advanced, etc.
• These are all PlanStages
• Each PlanStage implements
work()	
  
• work() returns ...
Execution Stats: works, advanced, etc.
FETCH
SORT
INDEX SCAN
keyPattern:
work()
Execution Stats: works, advanced, etc.
FETCH
SORT
INDEX SCAN
keyPattern:
work()
work()
Execution Stats: works, advanced, etc.
FETCH
SORT
INDEX SCAN
keyPattern:
work()
work()
work()
Execution Stats: works, advanced, etc.
FETCH
SORT
INDEX SCAN
keyPattern:
work()
work()
ADVANCED ID
Execution Stats: works, advanced, etc.
FETCH
SORT
INDEX SCAN
keyPattern:
work()
ADVANCED
ADVANCED {…}
Execution Stats: works, advanced, etc.
FETCH
SORT
INDEX SCAN
keyPattern:
work()
ADVANCED
ADVANCEDNEED_TIME
Explain Mode: "executionStats"
db.tweets.find(…).explain("executionStats")
{
"executionStats" : {
// Top-level stats.
"exe...
Explain Mode: "executionStats"
>	
  db.tweets.find(…).explain("executionStats")	
  
{"executionStats":	
  {	
  
	
   "exec...
• "queryPlanner" (default)
• "executionStats"
• "allPlansExecution"
Query Explain Modes
✓
✓
Applying This Information
How selective is your index?
How selective is your index?
db.tweets.explain("executionStats").find({	
  
	
  	
  createdDate:	
  {$gte:	
  <today>},	
 ...
How selective is your index?
db.tweets.explain("executionStats").find({	
  
	
  	
  createdDate:	
  {$gte:	
  <today>},	
 ...
How selective is your index?
db.tweets.explain("executionStats").find({	
  
	
  	
  createdDate:	
  {$gte:	
  <today>},	
 ...
How selective is your index?
db.tweets.explain("executionStats").find({	
  
	
  	
  createdDate:	
  {$gte:	
  <today>},	
 ...
What's the most expensive part of your plan?
What's the most expensive part of your plan?
db.tweets.explain("executionStats").find({	
  
	
  	
  createdDate:	
  {$gte:...
What's the most expensive part of your plan?
db.tweets.explain("executionStats").find(…)	
  
FETCH
executionTimeMillisEsti...
What's the most expensive part of your plan?
db.tweets.explain("executionStats").find(…)	
  
FETCH
works: 2705
advanced: 3...
Some Questions We Can Answer
• Is your query using the index you expect?
• Is your query using an index to provide the sor...
Some Questions We Can't Yet Answer
• Why was your winning plan chosen?
!
Why was your winning plan chosen?
db.tweets.explain("executionStats").find({	
  
	
  	
  createdDate:	
  {$gte:	
  <today>...
• "queryPlanner" (default)
• "executionStats"
• "allPlansExecution"
Query Explain Modes
✓
✓ 😃
Query Explain Mode: "allPlansExecution"
>	
  db.collection.find(…).explain()	
  
{"queryPlanner"	
  :	
  {	
  
	
   "winni...
Query Explain Mode: "allPlansExecution"
• MultiPlanStage::pickBestPlan()
Query Planning
…
MultiPlanStage
work()
work()
work()
• MultiPlanStage::pickBestPlan()
Query Planning
…
MultiPlanStage
ADVANCED
NEED_TIME
ADVANCED
• MultiPlanStage::pickBestPlan()
Query Planning
…
MultiPlanStage
Advances: 78 22 50
• MultiPlanStage::pickBestPlan()
Query Planning
…
MultiPlanStage
Advances: 78 22 50
Explain Mode: "allPlansExecution"
>	
  db.tweets.find(…).explain("allPlansExecution")	
  
{	
  
	
   "queryPlanner"	
  :	
...
Explain Mode: "allPlansExecution"
>	
  db.collection.find(…).explain()	
  
{"queryPlanner"	
  :	
  {	
  
	
   "winningPlan...
Explain Mode: "allPlansExecution"
db.tweets.explain("allPlansExecution").find({	
  
	
  	
  createdDate:	
  {$gte:	
  <tod...
• "queryPlanner" (default)
• "executionStats"
• "allPlansExecution"
Query Explain Modes
✓
✓
✓
One Final Warning
explain() output can get VERY
large
Compass Shoutout
Compass Shoutout
Compass Shoutout
Query Summary
• Query Plans
• Explain Format
• Different Verbosities
Different Verbosities
Diagnostics
Query
03
Overview
Make explain() exciting
Motivation
01 Why Do You Care?
What is explain...
Aggregation: Overview
• Aggregation Pipelines
• Explain Output
• Tips and Tricks
Aggregation Pipelines
• Like PlanStages, input of one stage feeds into the next
• db.tweets.aggregate([

	
  {$match:	
  {...
db.tweets.aggregate([

	
  {$match:	
  {hashtag:	
  "#MDBW16}},

	
  {$group:	
  {

	
  	
  	
  _id:	
  "$author",

	
  	
...
db.tweets.aggregate([

	
  {$match:	
  {hashtag:	
  "#MDBW16}},

	
  {$group:	
  {

	
  	
  	
  _id:	
  "$author",

	
  	
...
Aggregation Pipelines
• Aggregation delegates document selection to the query system
This is just a query!
MATCH GROUP{…}{...
Aggregation Pipelines
• Aggregation delegates document selection to the query system
GROU {…}CURSOR
Aggregation Pipelines
• Aggregation delegates document selection to the query system
GROU {…}CURSOR
Aggregation: Overview
• Aggregation Pipelines
• Explain Output
• Tips and Tricks
✓
Explain Output
• db.tweets.explain().aggregate([{$match:	
  {…}},	
  {$group:	
  {…}}])	
  
{	
  
	
   "stages"	
  :	
  [	...
Explain Output
• db.tweets.explain().aggregate([{$match:	
  {…}},	
  {$group:	
  {…}}])	
  
{	
  
	
   "stages"	
  :	
  [	...
Explain Output
{	
  
	
  	
  "$cursor"	
  :	
  {	
  
	
  	
  	
  	
  	
  "query"	
  :	
  {	
  /*	
  Same	
  as	
  $match	
...
Explain Output
{	
  
	
  	
  "$cursor"	
  :	
  {	
  
	
  	
  	
  	
  	
  "query"	
  :	
  {…},	
  
	
   	
  	
  "queryPlann...
Explain Output
• db.tweets.explain().aggregate([{$match:	
  {…}},	
  {$group:	
  {…}}])	
  
{	
  
	
   "stages"	
  :	
  [	...
Explain Output
db.tweets.explain().aggregate([	
  
	
  	
  	
  {$match:	
  {…}},	
  
	
  	
  	
  {$group:	
  {…}},	
  
	
 ...
Explain Output
Aggregation: Overview
• Aggregation Pipelines
• Explain Output
• Tips and Tricks
✓
✓
Tips & Tricks: Overview
1. Making the most of your indexes
2. Pipeline Optimizations
Tips & Tricks: Overview
1. Making the most of your indexes
2. Pipeline Optimizations
Disclaimer: All discussions assume Mo...
Aggregation & Indexes
1. Index Provided Sorts
2. Index Provided Projections
Index Provided Sorts
//	
  Compute	
  the	
  most	
  favorited	
  #MDBW16	
  tweets.	
  
db.tweets.explain().aggregate([	
...
Index Provided Sorts
db.tweets.explain().aggregate([	
  
	
  	
  	
  {$match:	
  {hashtag:	
  "#MDBW16"},

	
  	
  	
  {$s...
Index Provided Sorts
db.tweets.explain().aggregate([	
  
	
  	
  	
  {$match:	
  {hashtag:	
  "#MDBW16"},

	
  	
  	
  {$s...
Index Provided Sorts
db.tweets.explain().aggregate([	
  
	
  	
  	
  {$match:	
  {hashtag:	
  "#MDBW16"},

	
  	
  	
  {$s...
Index Provided Projections
//	
  Compute	
  the	
  users	
  who	
  tweeted	
  about	
  #MDBW16	
  most	
  often.	
  
db.tw...
Index Provided Projections
db.tweets.explain().aggregate([	
  
	
  	
  	
  {$match:	
  {hashtag:	
  "#MDBW16"},	
  
	
  	
...
Index Provided Projections
db.tweets.explain().aggregate([	
  
	
  	
  	
  {$match:	
  {hashtag:	
  "#MDBW16"},	
  
	
  	
...
Index Provided Projections
db.tweets.explain().aggregate([	
  
	
  	
  	
  {$match:	
  {hashtag:	
  "#MDBW16"},	
  
	
  	
...
Tips & Tricks: Overview
1. Making the most of your indexes
2. Pipeline Optimizations
✓
Other Pipeline Optimizations
//	
  Compute	
  the	
  top-­‐10	
  users	
  who	
  tweeted	
  about	
  #MDBW16	
  most	
  of...
Other Pipeline Optimizations
db.tweets.explain().aggregate([	
  
	
  	
  	
  {$match:	
  {hashtag:	
  "#MDBW16"},	
  
	
  ...
Other Pipeline Optimizations
db.tweets.explain().aggregate([	
  
	
  	
  	
  {$match:	
  {hashtag:	
  "#MDBW16"},	
  
	
  ...
Aggregation: Summary
• Explain output includes query section
• Sometimes the query is doing a lot of your work
• We don't ...
Summary
Understand how
MongoDB answers
queries
Knowledge
Figure out what's
going on
Debugging
Learn some tricks to
optimiz...
MongoDB World 2016: Deciphering .explain() Output
MongoDB World 2016: Deciphering .explain() Output
MongoDB World 2016: Deciphering .explain() Output
MongoDB World 2016: Deciphering .explain() Output
Prochain SlideShare
Chargement dans…5
×

MongoDB World 2016: Deciphering .explain() Output

1 392 vues

Publié le

Presented by Charlie Swanson, Software Engineer, MongoDB

Publié dans : Technologie

MongoDB World 2016: Deciphering .explain() Output

  1. 1. Deciphering Explain Output Charlie Swanson
  2. 2. Charlie Swanson Software Engineer - Query Team
  3. 3. Goals Of This Talk Understand how MongoDB answers queries Knowledge Figure out what's going on Debugging Learn some tricks to optimize your queries & aggregations Best Practices created by Francielly Constantin Senra from Noun Project created by Ramon Sandino from Noun Project
  4. 4. Non-goals of This Talk Picking the best indexes for your needs Index Selection It's mostly extends naturally Sharded ClustersSorry, we don't have time… Fixing Problems created by ChangHoon Baek from Noun Project created by David Marioni from Noun Project created by factor[e] design initiative from Noun Project
  5. 5. Overview Make explain() exciting Motivation 01 Different Verbosities Diagnostics Query 03Why Do You Care? What is explain()? 02 Similarities & Differences Aggregation 04
  6. 6. Overview Make explain() exciting Motivation 01 Different Verbosities Diagnostics Query 03Why Do You Care? What is explain()? 02 Similarities & Differences Aggregation 04
  7. 7. Monitoring the Spread
  8. 8. Our Task MongoDB created by Christopher Holm-Hansen from Noun Project
  9. 9. Overview Make explain() exciting Motivation 01 Different Verbosities Diagnostics Query 03Why Do You Care? What is explain()? 02 Similarities & Differences Aggregation 04
  10. 10. 01: What Is explain()?
  11. 11. Example Explain Output > db.foo.find({"nFollowers": {$gte: 1000}}).explain() { "queryPlanner" : { "winningPlan" : { "stage" : "FETCH", "inputStage" : { "stage" : "IXSCAN", "keyPattern" : {"nFollowers" : 1}, } }, "rejectedPlans" : [ ] } }
  12. 12. Why Do You Care? Your Application MongoDB created by Mike Ashley from Noun Project
  13. 13. Why Do You Care? Your Application MongoDB Documents Please!
  14. 14. Why Do You Care? Your Application MongoDB Hmm… Let me think about that…
  15. 15. Why Do You Care? Your Application MongoDB Ah! Here are your results!
  16. 16. Why Do You Care? Your Application MongoDB What took you so long?!
  17. 17. Overview Make explain() exciting Motivation 01 Different Verbosities Diagnostics Query 03Why Do You Care? What is explain()? 02 Similarities & Differences Aggregation 04
  18. 18. Overview of Query Explain • Query Plans • explain() Output Format • Different Verbosities
  19. 19. Query Plans //  db.tweets.ensureIndex({hashtags:  1});   db.tweets.find({hashtags:  "#MDBW16"})        .sort({nFavorites:  -­‐1}); FETCH INDEX SCAN (hashtags) SORT
  20. 20. Query Plans db.tweets.find({hashtags:  "#MDBW16"})        .sort({nFavorites:  -­‐1}); FETCH INDEX SCAN {hashtags: 1} SORT Returns the IDs of tweets with a hashtag of "MDBW16"
  21. 21. Query Plans db.tweets.find({hashtags:  "#MDBW16"})        .sort({nFavorites:  -­‐1}); FETCH INDEX SCAN {hashtags: 1} SORT Converts from an ID to a full BSON object
  22. 22. Query Plans db.tweets.find({hashtags:  "#MDBW16"})        .sort({nFavorites:  -­‐1}); FETCH INDEX SCAN {hashtags: 1} SORT Sorts the BSON objects by the field "nFavorites"
  23. 23. Query Plans db.tweets.find({$or:  [          {hashtag:  {"#MongoDB"}},          {username:  {$in:  [              "@MongoDB",              "@MongoDBEng",                "@MongoDBcareers"          ]}}   ]}); INDEX SCAN {hashtags: 1} OR FETCH INDEX SCAN {username: 1}
  24. 24. Explain Output >  db.collection.find(…).explain()   {     "queryPlanner"  :  {       …       "winningPlan"  :  {…},       "rejectedPlans"  :  […]     },     …   }
  25. 25. Explain Output >  db.collection.find(…).explain()   {queryPlanner:  {        winningPlan:  {            stage:  "SORT",            inputStage:  {                stage:  "FETCH",                    inputStage:  {                        stage:  "IXSCAN"                    }                }            }        }   }} FETCH INDEX SCAN {hashtags: 1} SORT
  26. 26. Explain Output >  db.collection.find(…).explain()   {   …                        stage:  "IXSCAN"                        keyPattern:  {hashtags:  1},                        indexBounds:  {                 a:  [  "[4.0,  inf.0]"  ]
                      },                        …  //  Other  index  scan                              //  specific  stats.   …   }} FETCH INDEX SCAN keyPattern: { hashtags: 1 } indexBounds: […] … SORT
  27. 27. Explain Output >  db.collection.find(…).explain()   {     "queryPlanner"  :  {       …       "winningPlan"  :  {                //  Encodes  selected  plan.          },       "rejectedPlans"  :  […]     },     …   } FETCH INDEX SORT
  28. 28. >  db.collection.find(…).explain()   {"queryPlanner"  :  {     "winningPlan"  :  {       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                           },     "rejectedPlans"  :  [       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                …     ]   }} Explain Output FETCH INDEX SORT COLLECTI ON SCAN SORT
  29. 29. Explain Output >  db.collection.find(…).explain()   {     "queryPlanner"  :  {       …       "winningPlan"  :  {                //  Encodes  selected  plan.          },       "rejectedPlans"  :  [                //  Encodes  rejected  plan(s).            ]     },     …   } FETCH INDEX SORT COLLECTI ON SCAN SORT
  30. 30. Applying This Information
  31. 31. Is your query using the index you expect?
  32. 32. Is your query using the index you expect? FETCH SORT ✓ COLLECTION SORT ✗ INDEX SCAN keyPattern: {nFollowers:
  33. 33. Is your query using an index to provide the sort?
  34. 34. Is your query using an index to provide the sort? FETCH INDEX SCAN ✓ ✗FETCH INDEX SCAN SORT
  35. 35. Is your query using an index to provide the sort? ✓SORT_MERGE INDEX SCAN FETCH INDEX SCAN
  36. 36. Is your query using an index to provide the projection?
  37. 37. PROJECTI INDEX SCAN ✓ ✗ PROJECTI INDEX SCAN FETCH Is your query using an index to provide the projection?
  38. 38. Is your query using an index to provide the projection? //  With  index   //  {nFollowers:  1,  username:  1}.   db.users.find(          {nFollowers:  {$gte:  1000}},          {_id:  0,  nFollowers:  1,  username:  1}   ) ✓ PROJECTI INDEX SCAN keyPattern: { nFollowers: 1, username: 1 }
  39. 39. Some Questions We Can Answer • Is your query using the index you expect? • Is your query using an index to provide the sort? • Is your query using an index to provide the projection? !
  40. 40. Some Questions We Can't Yet Answer • How selective is your index? • What is the most expensive part of your plan? • Why was your winning plan chosen? !
  41. 41. Query Explain Modes • "queryPlanner" (default) • "executionStats" • "allPlansExecution"
  42. 42. Query Explain Modes • "queryPlanner" (default) >  db.collection.find(…).explain()   {"queryPlanner"  :  {     "winningPlan"  :  {       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                           },     "rejectedPlans"  :  [       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                …     ]   }}
  43. 43. • "queryPlanner" (default) • "executionStats" • "allPlansExecution" Query Explain Modes ✓
  44. 44. • "executionStats" Query Explain Modes >  db.collection.find(…).explain()   {"queryPlanner"  :  {     "winningPlan"  :  {       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                           },     "rejectedPlans"  :  [       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                …     ]   }}
  45. 45. • "executionStats" Query Explain Modes created by Mike Ashley from Noun Project created by Creative Stall from Noun Project
  46. 46. Explain Mode: "executionStats" >  db.tweets.find(…).explain("executionStats")   {     "queryPlanner"  :  {       …       "winningPlan"  :  {…},       "rejectedPlans"  :  […]     },      "executionStats":  {    //  New!       …,       "executionStages":  {…}     }     …   }
  47. 47. Explain Mode: "executionStats" >  db.tweets.find(…).explain("executionStats")   {      "queryPlanner"  :  {  /*  Same  as  before.  */  },      "executionStats":  {       //  Top-­‐level  stats.       "executionStages":  {              stage:  "SORT",              //  Sort  stats.              inputStage:  {                  //  etc,  etc.
            }          }      }     …   } FETCH SORT INDEX SCAN keyPattern:
  48. 48. Explain Mode: "executionStats" >  db.tweets.find(…).explain("executionStats")   { …, "executionStats" : { // Top-level stats. "nReturned" : 390000, "executionTimeMillis" : 4431, "totalKeysExamined" : 390000, "totalDocsExamined" : 390000, "executionStages" : {…} }, } FETCH SORT INDEX SCAN keyPattern:
  49. 49. Explain Mode: "executionStats" db.tweets.find(…).explain("executionStats") { "executionStats" : { // Top-level stats. "executionStages" : { "stage" : "SORT", "nReturned" : 390000, "executionTimeMillisEstimate" : 2030, … "sortPattern" : { "nFollowers" : 1 }, "memUsage" : 20280000, "memLimit" : 33554432, "inputStage" : {…} } } } FETCH SORT INDEX SCAN keyPattern:
  50. 50. db.tweets.find(…).explain("executionStats") { "executionStats" : { // Top-level stats. "executionStages" : { "stage" : "SORT", "nReturned" : 390000, "executionTimeMillisEstimate" : 2030, "works" : 780003, "advanced" : 390000, "needTime" : 390002, "isEOF" : 1, "sortPattern" : { "b" : 1 }, … "inputStage" : {…} } } } ? Explain Mode: "executionStats" FETCH SORT INDEX SCAN keyPattern:
  51. 51. Execution Stats: works, advanced, etc. • These are all PlanStages • SortStage • FetchStage • IndexScanStage FETCH SORT INDEX SCAN keyPattern:
  52. 52. Execution Stats: works, advanced, etc. • These are all PlanStages • Each PlanStage implements work() FETCH SORT INDEX SCAN keyPattern:
  53. 53. Execution Stats: works, advanced, etc. • These are all PlanStages • Each PlanStage implements work()   • work() returns one of:   • ADVANCED   • NEED_TIME   • IS_EOF FETCH SORT INDEX SCAN keyPattern:
  54. 54. Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work()
  55. 55. Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work() work()
  56. 56. Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work() work() work()
  57. 57. Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work() work() ADVANCED ID
  58. 58. Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work() ADVANCED ADVANCED {…}
  59. 59. Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work() ADVANCED ADVANCEDNEED_TIME
  60. 60. Explain Mode: "executionStats" db.tweets.find(…).explain("executionStats") { "executionStats" : { // Top-level stats. "executionStages" : { "stage" : "SORT", "nReturned" : 390000, "executionTimeMillisEstimate" : 2030, "works" : 780003, "advanced" : 390000, "needTime" : 390002, "isEOF" : 1, "sortPattern" : { "b" : 1 }, … "inputStage" : {…} } } } FETCH SORT INDEX SCAN keyPattern:
  61. 61. Explain Mode: "executionStats" >  db.tweets.find(…).explain("executionStats")   {"executionStats":  {     "executionStages":  {              stage:  "SORT",              //  Sort  stats,  includes  "works",  "advanced",  …              inputStage:  {                  stage:  "FETCH",                  //  Fetch  stats,  includes  "works",  "advanced",  …                  inputStage:  {
                    //  etc,  etc.                  }
            }          }      }      …   } FETCH … SORT … INDEX SCAN …
  62. 62. • "queryPlanner" (default) • "executionStats" • "allPlansExecution" Query Explain Modes ✓ ✓
  63. 63. Applying This Information
  64. 64. How selective is your index?
  65. 65. How selective is your index? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   {      "executionStats"  :  {   "nReturned" : 314, "totalKeysExamined" : 2704, // < 8% matched … } FETCH INDEX SCAN keyPattern: {createdDate:
  66. 66. How selective is your index? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   12:02 12:03 12:04 12:06 … INDEX SCAN FETCH INDEX SCAN keyPattern: {createdDate:
  67. 67. How selective is your index? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   12:02 {createdDate:  12:02,      favorites:  [        "@MongoDB",        "@taylorswift"   ]} 12:03 12:04 12:06 … FETCH FETCH filter: { favorites: "@eliothorowitz" INDEX SCAN keyPattern: {createdDate:
  68. 68. How selective is your index? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   12:03 12:02 12:04 12:06 … {createdDate:  12:03,      favorites:  [        "@eliothorowitz",        "@taylorswift"   ]} FETCH FETCH filter: { favorites: "@eliothorowitz" INDEX SCAN keyPattern: {createdDate:
  69. 69. What's the most expensive part of your plan?
  70. 70. What's the most expensive part of your plan? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   FETCH filter: { favorites: "@eliothorowitz" INDEX SCAN keyPattern: {createdDate:
  71. 71. What's the most expensive part of your plan? db.tweets.explain("executionStats").find(…)   FETCH executionTimeMillisEstimate: 431 INDEX SCAN executionTimeMillisEstimate:
  72. 72. What's the most expensive part of your plan? db.tweets.explain("executionStats").find(…)   FETCH works: 2705 advanced: 314 needTime: 2391 //  314/2705  ≈  8% INDEX SCAN
  73. 73. Some Questions We Can Answer • Is your query using the index you expect? • Is your query using an index to provide the sort? • Is your query using an index to provide the projection? • How selective is your index? • Which part of your plan is the most expensive? !
  74. 74. Some Questions We Can't Yet Answer • Why was your winning plan chosen? !
  75. 75. Why was your winning plan chosen? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   • We had an index on {favorites: 1}, would that have been faster? !
  76. 76. • "queryPlanner" (default) • "executionStats" • "allPlansExecution" Query Explain Modes ✓ ✓ 😃
  77. 77. Query Explain Mode: "allPlansExecution" >  db.collection.find(…).explain()   {"queryPlanner"  :  {     "winningPlan"  :  {       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                           },     "rejectedPlans"  :  [       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                …     ]   }} ❓
  78. 78. Query Explain Mode: "allPlansExecution"
  79. 79. • MultiPlanStage::pickBestPlan() Query Planning … MultiPlanStage work() work() work()
  80. 80. • MultiPlanStage::pickBestPlan() Query Planning … MultiPlanStage ADVANCED NEED_TIME ADVANCED
  81. 81. • MultiPlanStage::pickBestPlan() Query Planning … MultiPlanStage Advances: 78 22 50
  82. 82. • MultiPlanStage::pickBestPlan() Query Planning … MultiPlanStage Advances: 78 22 50
  83. 83. Explain Mode: "allPlansExecution" >  db.tweets.find(…).explain("allPlansExecution")   {     "queryPlanner"  :  {       …       "winningPlan"  :  {…},       "rejectedPlans"  :  […]     },      "executionStats":  {       …,       "executionStages":  {…},            "allPlansExecution":  […]    //  New!     }     …   }
  84. 84. Explain Mode: "allPlansExecution" >  db.collection.find(…).explain()   {"queryPlanner"  :  {     "winningPlan"  :  {…}                           },     "rejectedPlans"  :  [       {…},            …     ],        "executionStats":  {
          "executionStages":  {…}            "allPlansExecution":  [
              {…},                {…},                …            ]        }   }}
  85. 85. Explain Mode: "allPlansExecution" db.tweets.explain("allPlansExecution").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   {      "executionStats":  {            "allPlansExecution":  [                    {nReturned:  34,                      executionStages:  {  /*  Index  Scan  on  "favorites"  */  }                    },                    {nReturned:  101,                      executionStages:  {  /*  Index  Scan  on  "createdDate"  */  }                    }            ]     }     …   }
  86. 86. • "queryPlanner" (default) • "executionStats" • "allPlansExecution" Query Explain Modes ✓ ✓ ✓
  87. 87. One Final Warning explain() output can get VERY large
  88. 88. Compass Shoutout
  89. 89. Compass Shoutout
  90. 90. Compass Shoutout
  91. 91. Query Summary • Query Plans • Explain Format • Different Verbosities
  92. 92. Different Verbosities Diagnostics Query 03 Overview Make explain() exciting Motivation 01 Why Do You Care? What is explain()? 02 Similarities & Differences Aggregation 04
  93. 93. Aggregation: Overview • Aggregation Pipelines • Explain Output • Tips and Tricks
  94. 94. Aggregation Pipelines • Like PlanStages, input of one stage feeds into the next • db.tweets.aggregate([
  {$match:  {hashtag:  "#MDBW16}},
  {$group:  {
      _id:  "$author",
      nTweets:  {$sum:  1}
  }}
 ]) MATCH GROUP{…}{…}{…} {…}
  95. 95. db.tweets.aggregate([
  {$match:  {hashtag:  "#MDBW16}},
  {$group:  {
      _id:  "$author",
      nTweets:  {$sum:  1}
  }}
 ]) Collection Scan?!? Are you kidding me?? Aggregation Pipelines MATCH GROUP{…}{…}{…} {…}
  96. 96. db.tweets.aggregate([
  {$match:  {hashtag:  "#MDBW16}},
  {$group:  {
      _id:  "$author",
      nTweets:  {$sum:  1}
  }}
 ]) Collection Scan?!? Are you kidding me?? Aggregation Pipelines MATCH GROUP{…}{…}{…} {…}
  97. 97. Aggregation Pipelines • Aggregation delegates document selection to the query system This is just a query! MATCH GROUP{…}{…}{…} {…}
  98. 98. Aggregation Pipelines • Aggregation delegates document selection to the query system GROU {…}CURSOR
  99. 99. Aggregation Pipelines • Aggregation delegates document selection to the query system GROU {…}CURSOR
  100. 100. Aggregation: Overview • Aggregation Pipelines • Explain Output • Tips and Tricks ✓
  101. 101. Explain Output • db.tweets.explain().aggregate([{$match:  {…}},  {$group:  {…}}])   {     "stages"  :  [       {"$cursor"  :  {…}},       {"$group"  :  {…}}     ],   }   GROCURSO
  102. 102. Explain Output • db.tweets.explain().aggregate([{$match:  {…}},  {$group:  {…}}])   {     "stages"  :  [       {"$cursor"  :  {…}},       {"$group"  :  {…}}     ],   }   GROCURSO
  103. 103. Explain Output {      "$cursor"  :  {            "query"  :  {  /*  Same  as  $match  */  },        "queryPlanner"  :  {…}      }   }   GROCURSO
  104. 104. Explain Output {      "$cursor"  :  {            "query"  :  {…},        "queryPlanner"  :  {              "winningPlan":  {…},              "rejectedPlans":  […]          }      }   }   GROCURSO
  105. 105. Explain Output • db.tweets.explain().aggregate([{$match:  {…}},  {$group:  {…}}])   {     "stages"  :  [       {"$cursor"  :  {query:  {…},  queryPlanner:  {…}}},       {"$group"  :  {/*  Same  as  $group  specification  */}}     ],   }   GROCURSO
  106. 106. Explain Output db.tweets.explain().aggregate([        {$match:  {…}},        {$group:  {…}},        {$sort:  {…}},        {$project:  {…}},   ])   {     "stages"  :  [       {"$cursor"  :  {query:  {…},  queryPlanner:  {…}}},       {"$group"  :  {/*  Same  as  $group  specification  */}}       {"$sort"  :  {/*  Same  as  $sort  specification  */}}       {"$project"  :  {/*  Same  as  $project  specification  */}}     ],   }   GROCURSO PROJECSOR
  107. 107. Explain Output
  108. 108. Aggregation: Overview • Aggregation Pipelines • Explain Output • Tips and Tricks ✓ ✓
  109. 109. Tips & Tricks: Overview 1. Making the most of your indexes 2. Pipeline Optimizations
  110. 110. Tips & Tricks: Overview 1. Making the most of your indexes 2. Pipeline Optimizations Disclaimer: All discussions assume MongoDB 3.2
  111. 111. Aggregation & Indexes 1. Index Provided Sorts 2. Index Provided Projections
  112. 112. Index Provided Sorts //  Compute  the  most  favorited  #MDBW16  tweets.   db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},
      {$sort:  {nFavorites:  -­‐1}},   ])  
  113. 113. Index Provided Sorts db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},
      {$sort:  {nFavorites:  -­‐1}},   ])   {     "stages"  :  [       {"$cursor"  :  {…}},     ],   }   CURSO
  114. 114. Index Provided Sorts db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},
      {$sort:  {nFavorites:  -­‐1}},   ])   {     "stages"  :  [        {"$cursor"  :  {                  "query"  :  {hashtag:  "#MDBW16"},                  "sort"  :  {nFavorites:  -­‐1},                  "queryPlanner"  :  {                      "winningPlan":  {  /*  Index  Scan  on  nFavorites  */  }                  }
          }     ],   }   CURSO
  115. 115. Index Provided Sorts db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},
      {$sort:  {nFavorites:  -­‐1}},   ])   {     "stages"  :  [       {"$cursor"  :  {                  "query"  :  {hashtag:  "#MDBW16"},                  //  No  "sort"  field.                  "queryPlanner"  :  {          "winningPlan":  {  /*  No  index  on  nFavorites  */  },      }   },   {$sort:  {nFavorites:  -­‐1}}     ],   }   CURSO SORT
  116. 116. Index Provided Projections //  Compute  the  users  who  tweeted  about  #MDBW16  most  often.   db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},   ])  
  117. 117. Index Provided Projections db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},   ])   {     "stages"  :  [       {"$cursor"  :  {…}},       {"$group"  :  {…}},       {"$sort"  :  {…}},     ],   }   CURSO GROUP SORT
  118. 118. Index Provided Projections db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},   ])   {     "stages"  :  [       "$cursor"  :  {                "query"  :  {hashtag:  "#MDBW16"},                "fields"  :  {_id:  0,  username:  1},                "queryPlanner"  :  {          "winningPlan":  {  /*  Index  Scan  on  {hashtag:  1,  username:  1}  */  },   },
 {$group:  {…}},   {$sort:  {…}}     ],   } CURS GROU SORT
  119. 119. Index Provided Projections db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},   ])   {     "stages"  :  [       "$cursor"  :  {                "query"  :  {hashtag:  "#MDBW16"},                //  No  "fields"  entry                "queryPlanner"  :  {          "winningPlan":  {  /*  Index  Scan  on  {hashtag:  1}  */  },   },
 {$group:  {…}},   {$sort:  {…}}     ],   } CURS GROU SORT
  120. 120. Tips & Tricks: Overview 1. Making the most of your indexes 2. Pipeline Optimizations ✓
  121. 121. Other Pipeline Optimizations //  Compute  the  top-­‐10  users  who  tweeted  about  #MDBW16  most  often.   db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},        {$limit:  10}   ])  
  122. 122. Other Pipeline Optimizations db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},        {$limit:  10}   ])   {     "stages"  :  [       {"$cursor"  :  {…}},       {"$group"  :  {…}},       {"$sort"  :  {sortKey:  {count:  -­‐1},  limit:  NumberLong(10)}},     ],   }   CURS GROU SORT*
  123. 123. Other Pipeline Optimizations db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},        {$project:  {…}},        {$limit:  10}   ])   {     "stages"  :  [            …            {"$sort"  :  {sortKey:  {count:  -­‐1},  limit:  NumberLong(10)}},            {$project:  {…}}     ],   }   CURS GROU SOR PROJE
  124. 124. Aggregation: Summary • Explain output includes query section • Sometimes the query is doing a lot of your work • We don't always execute the stages in the order you specified
  125. 125. Summary Understand how MongoDB answers queries Knowledge Figure out what's going on Debugging Learn some tricks to optimize your queries & aggregations Best Practices created by Francielly Constantin Senra from Noun Project created by Ramon Sandino from Noun Project

×