MongoDB Chunks – Distribution, Splitting, and Merging This presentation will discuss how the distribution of chunks can create scenarios where you may need to manually move, split, or merge chunks in your sharded cluster. Scenarios requiring these actions can exist with both optimal and sub-optimal shard keys. This can be especially true when a new change has been introduced to the workload. It may cause hot-spotting, unbalanced storage, or empty chunks.

1. MongoDB Chunks
Jason Terpko
MongoDB Chunks – Distribution, Splitting, and Merging
NoSQL DBA, Rackspace/ObjectRocket
www.linkedin.com/in/jterpko, jason.terpko@rackspace.com

2. My Story
• Started out in relational databases in public education then financial
services
• Next came online media distribution combined with a paywall
• For analytics, started working with columnar databases and engines with
compression
• Made the switch to NoSQL at ObjectRocket by Rackspace

3. Overview
• MongoDB and Sharding
• What is a chunk in MongoDB?
• Chunk Distribution and Scaling MongoDB
• Use Cases For Splitting
• Use Cases For Merging
• Reconsidering Your Shard Key

4. Documents
• Common MongoDB BSON Types
• ObjectId
• String
• Integer
• Double
• NumberLong
• Date
• Boolean
• Null
• Array
{
"_id" : ObjectId("570680d891442f6efaff2005"),
"n" : "John Doe",
"a" : 45,
"h" : 5.9,
"w" : NumberLong(165),
"u" : ISODate("2016-01-07T15:46:32.085Z"),
"ac" : true,
"nn" : null,
"z" : [
"10013",
"10018"
]
}

5. MongoDB Sharded Cluster
s1 s2

6. What is a Chunk?
config.chunks
{
"_id" : "mydb.mycoll-uuid_"00005cf6-1217-4414-935b-
cf1bde09cc77"",
"lastmod" : Timestamp(1, 5),
"lastmodEpoch" : ObjectId("570733145f2bf94777a62155"),
"ns" : "mydb.mycoll",
"min" : {
"uuid" : "00005cf6-1217-4414-935b-cf1bde09cc77"
},
"max" : {
"uuid" : "7fe55637-74c0-4e51-8eed-ab6b411d2b6e"
},
"shard" : ”s1"
}

7. Use Cases For Splitting (Case 1)
s1 s2
80% 20%
Unbalanced
Operations

8. Use Cases For Splitting (Case 1)
Profiler Example :
db.system.profile.aggregate([
{$match: { $and: [ {op:"update"}, {ns : "mydb.mycoll"} ] }}, {$group: { "_id":"$query.uuid",
count:{$sum:1}}},
{$sort: {"count": -1}}, {$limit : 5 } ]);
{
"result" : [
{
"_id" : "00005cf6-1217-4414-935b-
cf1bde09cc77",
"count" : 28672
},
..........
],
"ok" : 1
}

9. Use Cases For Splitting (Case 1)
{
"_id" : "mydb.mycoll-uuid_"00005cf6-1217-4414-935b-cf1bde09cc77"",
"lastmod" : Timestamp(1, 5),
"lastmodEpoch" : ObjectId("570733145f2bf94777a62155"),
"ns" : "mydb.mycoll",
"min" : {
"uuid" : "00005cf6-1217-4414-935b-cf1bde09cc77"
},
"max" : {
"uuid" : "7fe55637-74c0-4e51-8eed-ab6b411d2b6e"
},
"shard" : ”s1"
}
{ “uuid” : “7800e10e-273d-4186-bca7-9e3f6647d33a”, …
{ “uuid” : “7801a55e-e326-4cdb-a5fb-a5d133016f13” , …
{ “uuid” : “7801c3d5-8506-4715-934b-ad66904ae01e” , …
{ “uuid” : “7801ff57-50b7-4505-9557-06058cc8ff80” , …
sh.splitAt(
)
Unbalanced
Operations

10. Moving the Chunks
db.runCommand({
"moveChunk" : "mydb.mycoll",
"bounds" : [
{
"uuid" : "7fe55742-7879-44bf-9a00-462a0284c982"
},
{
"uuid" : "ffffe5bc-d04d-4fcf-b8e2-283fa5998079"
}
],
"to" : "s2",
"_secondaryThrottle" : true
});
Example Command:

11. Use Cases For Splitting (Case 2)
s1 s2
s1 s2
Unbalanced
Resources

12. Use Cases For Splitting (Case 2)
Unbalanced Resources – Jumbo
Chunks
• Requirements
• Larger than db.settings.find({"_id" :
"chunksize"})
• Or contains 250,000 or more documents
• Causes
• Importing of data
• Config metadata lock
• Compound Shard Key
• Identifying
• sh.status(true)
• moveChunk()
• count() or Aggregation Pipeline
• ObjectRocket Utility : ChunkHunter.py*
• Rectifying
• sh.splitAt() and sh.splitFind()
• ObjectRocket Utilities : ChunkManager.py*
• Data reduction
• Re-sharding the collection
*https://github.com/objectrocket/Utils

13. Splitting : Identifying and Rectifying
Aggregation Example:
ObjectRocket Utils :
ChunkManager.py ChunkHunter.py ChunkSplitter.pyCn
t
exit
db.mycoll.aggregate([{$group: {"_id": {"uuid": "$uuid"}, "count": {$sum: 1}}},{$sort: {"count": -1}},{$lim
db.adminCommand({ dataSize: "mydb.mycoll", keyPattern: { "uuid" : 1 },
min: { "uuid" : "7fe55637-74c0-4e51-8eed-ab6b411d2b6e" },
max: { "uuid" : "7fe55742-7879-44bf-9a00-462a0284c982" } })
Data Size:

14. Use Cases For Splitting (Case 3)
s1 s2
100%
New and Small
Collections

15. Use Cases For Splitting (Case 3)
New and Small Collections
• Considerations
– Workload
– Shard Key
– Shard Count
– Current Collection Size
– Expected Collection Size
• Command
db.runCommand( { shardCollection: "mydb.mycoll", key: { ”uuid": "hashed" }, numInitialChunks :
1024 } )
• Calculating The Value
(size_in_mb / chunk_size) * 2

16. Use Cases For Merging
s1 s2
80% 20%
s1 s2
Unbalanced
Operations
Unbalanced
Resources
Empty Chunks
{ "uuid" : "00005cf6-....." } -->> { "uuid" : "7fe55637-....." } on :
s1
100K Docs
@ 45MB
{ "uuid" : "7fe55637-....." } -->> { "uuid" : "7fe55742-....." } on :
s2
0 Docs
@ 0MB

17. Use Cases For Merging (Deletes)
Unexpected remove()
Operations
insert()
& split
insert()
& split
insert() -> split ->
remove()
insert() -> split ->
remove()
s1 s2 s3 s4

18. Merging Process
How we have resolved this with JavaScript in past:
1. Check Balancer State
2. Read in empty chunks from ChunkHunter.py results
collection
3. Locate adjacent chunk
4. If empty and adjacent chunks reside on the same shard,
merge
5. Else move chunk to the shard with the adjacent chunk, then
merge
What have we learned from the current process?

19. Use Cases For Merging (TTL)
TTL : db.mycoll.ensureIndex( { "ca": 1 }, { expireAfterSeconds:
5227200 } )
splitVector
& split
moveChunk()
s1 s2 s3 s4

20. Chunk Size and Splits
How frequently are chunks being split?
• Global Change
• split & splitVector
• Decreasing the size
• Increased the size

21. Review Your Shard Key
How frequently does this occur?
• Is this a re-occurring problem?
• What impact does it have to your business?
• Re-analyzing your structure, workload, and access patterns
• What method will you use to re-shard a sharded collection?

22. ObjectRocket by Rackspace

23. Questions?
• MongoDB
• Sharding
• Chunks
• Splitting Chunks
• Merging Chunks
• ObjectRocket
• Rackspace