5. Agenda
• Shard keys
– Desired properties
– Evaluating shard key choices
• Hashed shard keys
– Why and how to use hashed shard keys
– Limitations
• Tag-aware sharding
– How it works
– Use case examples
7. What is a shard key?
• Incorporates one or more fields
• Used to partition your collection
• Must be indexed and exist in every document
• Definition and values are immutable
• Used to route requests to shards
25. {
_id: ObjectId(),
user: 123,
time: Date(),
subject: "…",
recipients: [],
body: "…",
attachments: []
}
Example: email storage
Most common scenario, can
be applied to 90% of cases
Each document can be up to
16MB
Each user may have GBs of
storage
Most common query: get
user emails sorted by time
Indexes on {_id}, {user, time},
{recipients}
36. Why is this relevant?
• Documents may not already have a suitable
value
• Hashing allows us to utilize an existing field
• More efficient index storage
– At the expense of locality
38. minKey 0 0 maxKey
Hashed shard keys avoids a hot
shard
39. Under the hood
• Create a hashed index for use with sharding
• Contains first 64 bits of a field’s md5 hash
• Considers BSON type and value
• Represented as NumberLong in the JS shell
40. // hash on 1 as an integer
> db.runCommand({ _hashBSONElement: 1 })
{
"key" : 1,
"seed" : 0,
"out" : NumberLong("5902408780260971510"),
"ok" : 1
}
// hash on "1" as a string
> db.runCommand({ _hashBSONElement: "1" })
{
"key" : "1",
"seed" : 0,
"out" : NumberLong("-2448670538483119681"),
"ok" : 1
}
Hashing BSON elements
41. Using hashed indexes
• Create index:
– db.collection.ensureIndex({ field : "hashed" })
• Options:
– seed: specify a hash seed to use (default: 0)
– hashVersion: currently supports only version 0 (md5)
42. Using hashed shard keys
• Enable sharding on collection:
– sh.shardCollection("test.collection", { field: "hashed" })
• Options:
– numInitialChunks: chunks to create (default: 2 per
shard)
43. // enabling sharding on test database
mongos> sh.enableSharding("test")
{ "ok" : 1 }
// shard by hashed _id field
mongos> sh.shardCollection("test.hash", { _id: "hashed" })
{ "collectionsharded" : "test.hash", "ok" : 1 }
Sharding on hashed ObjectId
46. Hashed keys are great for equality
queries
• Equality queries routed to a specific shard
• Will make use of the hashed index
• Most efficient query possible
47. mongos> db.hash.find({ x: 1 }).explain()
{
"cursor" : "BtreeCursor x_hashed",
"n" : 1,
"nscanned" : 1,
"nscannedObjects" : 1,
"numQueries" : 1,
"numShards" : 1,
"indexBounds" : {
"x" : [
[
NumberLong("5902408780260971510"),
NumberLong("5902408780260971510")
]
]
},
"millis" : 0
}
Explain plan of an equality query
48. But not so good for range queries
• Range queries will be scatter/gather
• Cannot utilize a hashed index
– Supplemental, ordered index may be used at the shard
level
• Inefficient query pattern
49. mongos> db.hash.find({ x: { $gt: 1, $lt: 99 }}).explain()
{
"cursor" : "BasicCursor",
"n" : 97,
"nscanned" : 1000,
"nscannedObjects" : 1000,
"numQueries" : 2,
"numShards" : 2,
"millis" : 3
}
Explain plan of a range query
50. Other limitations of hashed indexes
• Cannot be used in compound or unique indexes
• No support for multi-key indexes (i.e. array
values)
• Incompatible with tag aware sharding
– Tags would be assigned hashed values, not the original
key
• Will not overcome keys with poor cardinality
– Floating point numbers are truncated before hashing
51. Summary
• There are multiple approaches for sharding
• Hashed shard keys give great distribution
• Hashed shard keys are good for equality queries
• Pick a shard key that best suits your application
56. Tag aware sharding
• Associate shard key ranges with specific shards
• Shards may have multiple tags, and vice versa
• Dictates behavior of the balancer process
• No relation to replica set member tags
57. // tag a shard
mongos> sh.addShardTag("shard0001", "APAC")
// shard by country code and user ID
mongos> sh.shardCollection("test.tas", { c: 1, uid: 1 })
{ "collectionsharded" : "test.tas", "ok" : 1 }
// tag a shard key range
mongos> sh.addTagRange("test.tas",
... { c: "aus", uid: MinKey },
... { c: "aut", uid: MaxKey },
... "APAC"
... )
Configuring tag aware sharding
58. Use cases for tag aware sharding
• Operational and/or location-based separation
• Legal requirements for data storage
• Reducing latency of geographical requests
• Cost of overseas network bandwidth
• Controlling collection distribution
– http://www.kchodorow.com/blog/2012/07/25/controlling-collection-distribution/
60. Other changes in 2.4
• Make secondaryThrottle the default
– https://jira.mongodb.org/browse/SERVER-7779
• Faster migration of empty chunks
– https://jira.mongodb.org/browse/SERVER-3602
• Specify chunk by bounds for moveChunk
– https://jira.mongodb.org/browse/SERVER-7674
• Read preferences for commands
– https://jira.mongodb.org/browse/SERVER-7423
Remind everyone what a sharded cluster is. We will be talking about shard key considerations, hashed shard keys, and tag aware sharding.
Make this point: *If you need to. Not everyone needs to shard!*
Shard key: standard stuffHashed shard keys: useful for some applications that need uniform write distribution but don’t have suitable fields available to use as a shard key.Tag aware sharding: how to influence the balancer
There is also a case of sorting on the shard key, which entails multiple targeted queries in order.
Routing targeted requests scales better than scattered requests.
Entails additional processing on mongos to sort all results from the shards. When possible, sorting on the shard key is preferable, as it entails multiple targeted queries executed in order.
Cardinality – Can your data be broken down enough? How many documents share the same shard key?Write distribution – How well are writes and data balanced across all shards?Query Isolation - Query targeting to a specific shard, vs. scatter/gatherReliability – Impact on the application if a shard outage occurs (ideally, good replica set design can avoid this)Index locality – How hot the index on each shard will be, and how distributed the most used indexes are across our cluster.A good shard key can:Optimize routingMinimize (unnecessary) trafficAllow best scaling
The visuals for index locality and shard usage go hand in hand.For sharding, we want to have reads and writes distributed across the cluster to make efficient use of hardware. But for indexes, the inverse is true. We’d prefer not to access all parts of an index if we can avoid it. Just like disk access, we’d prefer to read/write sequentially, and only to a portion. This avoids having the entire index “hot”.
This show good index or disk locality on a single shard.
This also illustrates the inverse relationship between index usage on a shard and write distribution on the shard cluster. In this case, one shard is receiving all of the writes, and the index is being used for right-balanced insertions (incremental data).
While hashed shard keys offer poor index locality, that may be ok for some architectures if uniform write distribution is a higher priority. Also, if the indexes can fit in memory, random access of the index may not be a concern.
Compound shard key, utilizes an existing index we already need.For queries on a user’s inbox ordered by time, if that user is on multiple shards it may be an example of ordered, targeted queries.
Hashed shard keys are not for everyone, but they will generally be a better option for folks that don’t have suitable shard keys in their schema, or those that were manually hashing their shard keys. Since hashed values are 8 bytes (64-bit portion of the md5), the index will also be tinier than a typical, ordered ObjectId index.
_hashBSONElement requires --enableTestCommands to work
Note: the “2 per shard” default is a special computed value. If any value is specified for numInitialChunks, it will be divided by the total number of shards to determine how many chunks to create on each shard. The shardCollection helper method doesn’t seem to support this option, so users will have to use the raw command to specify this option.
Illustrates default pre-splitting behavior. Each shard for this new collection has two chunks from the get-go.
Uses the hashed index
A supplemental, ordered index is a regular, non-hashed index that starts with the shard key. Although mongos will have to scatter the query across all shards, the shards themselves will be able to use the ordered index on the shard key field instead of a BasicCursor. The take-away is that a hashed index is simply not usable for range queries; thankfully, MongoDB allows an ordered index to co-exist for the same field.
Emphasize the last point, since users cannot change their shard key after the fact. Additionally, if the application is new and users don’t fully understand their query patterns, they cannot make an informed decision on what would make a suitable shard key.
An application has a global user base and we’d like to optimize read/write latency for those users. This will also reduce overall network traffic.We’re not discussing replica set distribution for disaster recover; that is a separate topic.
Having all of our writeable databases/shards in a single region is a bottleneck.
Ideally, we’d like to have writeable shards in each region, to service users (and application servers) in that region.
Many-to-many relationship between shards and tags.
Configuring the range may be unintuitive for a fixed string value. The lower bound is inclusive and the upper bound is exclusive. For this example, we have to concoct a meaningless country code for the upper bound, because it is the next logical value above “aus”.APAC stands for “Asia Pacific” (for Australia in this case).
For controlling collection distribution, Kristina suggested creating tag ranges for the entire shard key range and assigning it to a particular shard. Although we technically enabled sharding for such collections, all of their data will reside on a single shard. This technique is used to get around the default behavior to place non-sharded collections on the primary shard for a database.