Over the past several months, Solr has reached a critical milestone of being able to elastically scale-out to handle indexes reaching into the hundreds of millions of documents. At Dachis Group, we've scaled our largest Solr 4 index to nearly 900M documents and growing. As our index grows, so does our need to manage this growth. In practice, it's common for indexes to continue to grow as organizations acquire new data. Over time, even the best designed Solr cluster will reach a point where individual shards are too large to maintain query performance. In this Webinar, you'll learn about new features in Solr to help manage large-scale clusters. Specifically, we'll cover data partitioning and shard splitting. Partitioning helps you organize subsets of data based on data contained in your documents, such as a date or customer ID. We'll see how to use custom hashing to route documents to specific shards during indexing. Shard splitting allows you to split a large shard into 2 smaller shards to increase parallelism during query execution. Attendees will come away from this presentation with a real-world use case that proves Solr 4 is elastically scalable, stable, and is production ready.

1. Confidential and Proprietary
© 2013 LucidWorks
1
Scaling Through Partitioning and Shard Splitting
in Solr 4
Bigger, Better, Faster, Stronger, Safer
• Agenda
- Next set of problems
- Shard splitting
- Data partitioning using custom hashing

2. Confidential and Proprietary
© 2013 LucidWorks2
About me
• Independent consultant specializing in search, machine
learning, and big data analytics
• Co-author Solr In Action from Manning Publishers, with Trey
Grainger
- Use code 12mp45 for 42% off your MEAP
• Previously, lead architect, developer, and dev-ops engineer
for large-scale Solr cloud implementation at Dachis Group
• Coming soon! Big Data Jumpstart: 2-day intensive hands-on
workshop covering Hadoop (Hive, Pig, HDFS), Solr, and
Storm

3. Confidential and Proprietary
© 2013 LucidWorks3
A nice problem to have …
• Solr cloud can scale!
- 18 shards / ~900M docs
- Indexing rate of 6-8k docs / sec
- Millions of updates, deletes, and new docs per day
• Some lessons learned
- Eventually, your shards will outgrow the hardware they are
hosted on
- Search is addictive to organizations, expect broader usage and
greater query complexity
- Beware of the urge to do online analytics with Solr

4. Confidential and Proprietary
© 2013 LucidWorks4
Sharding review
• Split large index into multiple “shards” containing
unique slices of the index
- Documents assigned to one and only one shard using a hash of
the unique ID field
- Hash function designed to distribute documents evenly across N
shards (default: MurmurHash3 32-bit)
- Every shard must have at least one active host
• Benefits
- Distribute cost of indexing across multiple nodes
- Parallelize complex query computations such as sorting and
facets across multiple nodes
- Smaller index == less JVM heap == less GC headaches
- Smaller index == More of it fits in OS cache (MMapDirectory)

5. Confidential and Proprietary
© 2013 LucidWorks5
Distributed indexing
View of cluster state from Zk
Shard 1
Leader
Node 1 Node 2
Shard 2
Leader
Shard 2
Replica
Shard 1
Replica
Zookeeper
CloudSolrServer
“smart client”
Hash on docID
1
2
3
tlogtlog
Get URLs of current leaders?
4
5
2 shards with 1 replica each
shard1 range:
80000000-ffffffff
shard2 range:
0-7fffffff

6. Confidential and Proprietary
© 2013 LucidWorks6
Document routing
• Composite (default)
- numShards specific when collection is bootstrapped
- Each shard has a hash range (32-bit)
• Implicit
- numShards is unknown when collection is bootstrapped
- No “range” for a shard; indexing client is responsible for sending
documents to the correct shard
- Feels a little old school to me ;-)

7. Confidential and Proprietary
© 2013 LucidWorks7
Distributed Search
Send query request to any node
Two-stage process
1. Query controller sends query to all
shards and merges results
One host per shard must be online or queries
fail
2. Query controller sends 2nd query to
all shards with documents in the
merged result set to get requested
fields
Solr client applications built for 3.x do
not need to change (our query code still
uses SolrJ 3.6)
Limitations
JOINs / Grouping need custom hashing
View of cluster state from Zk
Shard 1
Leader
Node 1 Node 2
Shard 2
Leader
Shard 2
Replica
Shard 1
Replica
Zookeeper
CloudSolrServer
1
3
q=*:*
Get URLs of all live nodes
4
2
Query controller
Or just a load balancer works too
get fields

8. Confidential and Proprietary
© 2013 LucidWorks8
Shard splitting: overview
• Bit of history, before shard splitting ...
- Re-index to create more shards (difficult at
scale), or
- Over shard (inefficient and costly)
- Better to scale-up organically
• Shard splitting (SOLR-3755)
- Split an existing shard into 2 sub-shards
» May need to double your node count
» Custom hashing may create hotspots in your
cluster

9. Confidential and Proprietary
© 2013 LucidWorks9
Shard splitting: when to split?
• Can you purge some docs?
• No cut and dry answer, but you
might want to split shards when,
- Avg. query performance begins to slow down
» (hint: this means you have to keep track of this)
- Indexing throughput degrades
- Out Of Memory errors when querying
» And you’ve done as much cache, GC, and query tuning as possible
• Not necessarily have to add more nodes
- May just want to split shards to add more parallelism

10. Confidential and Proprietary
© 2013 LucidWorks10
Shard splitting mechanics (in Solr 4.3.1)
• Split existing shard into 2 sub-shards
- SPLITSHARD action in collections API
http://localhost:8983/solr/admin/collections?action=SPLITSHARD&collection=logmill&
shard=shard1
- The actual “splitting” occurs using low-level Lucene API, see
org.apache.solr.update.SolrIndexSplitter
• Hard commit after the split completes (fixed in 4.4)
• Unload the “core” of the original shard (on all nodes)
http://localhost:8983/solr/admin/cores?action=UNLOAD&core=logmill&deleteIndex=tr
ue
• Migrate one of the splits to another host (optional)

11. Confidential and Proprietary
© 2013 LucidWorks11
Interesting questions about the process
• What happens to the replicas when
a shard is split?
• What happens to update requests being sent to
the original shard leader during the split operation?
• The original shard remains in the cluster – doesn’t
that create duplicate docs in queries?
New sub-shards are replicated automatically
using Solr replication
Updates are sent to the correct sub-split during “construction”
No, the original shard enters the “inactive” state and no longer
participates in distributed queries for the collection

12. Confidential and Proprietary
© 2013 LucidWorks12
Split shard in action: cluster with 2 shards
Graph view before the split
Graph view after the split
Graph view after unloading original shard

13. Confidential and Proprietary
© 2013 LucidWorks
Shard 1_1
Leader
Node 1 Node 2
Shard 2
Leader
Shard 2
Replica
Shard 1_1
Replica
shard1_0 range:
80000000-bfffffff
shard2 range:
0-7fffffff
Shard 1_0
Leader
Shard 1_0
Replica
shard1_1 range:
c0000000-ffffffff
13
Before and After shard splitting
13
Shard 1
Leader
Node 1 Node 2
Shard 2
Leader
Shard 2
Replica
Shard 1
Replica
shard1 range:
80000000-ffffffff
shard2 range:
0-7fffffff
Before
After

14. Confidential and Proprietary
© 2013 LucidWorks14
Shard splitting: Limitations
• Both splits end up on same node … but easy to
migrate to another node
- Assuming you have replication, you can unload the core of one
of the new sub-shards, making the replica the leader, and then
bring up another replica for that shard on another node.
- Nice to have ability to specify the disk location of the new sub-
shard indexes (splitting 50GB using 1 disk can take a while)
• No control where the replicas end up
- Possible future enhancement
• Not a collection-wide rebalancing operation
- you can’t grow your cluster from 16 nodes to 24 nodes and end
up with an even distribution of documents per shard

15. Confidential and Proprietary
© 2013 LucidWorks15
On to data partitioning …

16. Confidential and Proprietary
© 2013 LucidWorks16
Data Partitioning: Sports Aggregator
• Collection containing all things
sports related: blogs, tweets, news,
photos, etc.
- Bulk of queries for one sport at a time
- Sports have a seasonality aspect to them
• Use custom hashing to route documents to specific shards
based on the sport
• If you only need docs about “baseball”, can query the
“baseball” shard(s)
- Allows you to do JOINs and Grouping as if you are not distributed
- Replicate specific shards based on query volume to that shard

17. Confidential and Proprietary
© 2013 LucidWorks17
Cluster state
/clusterstate.json
shows the hash range
and document router

18. Confidential and Proprietary
© 2013 LucidWorks18
Custom hashing: Indexing
18
View of cluster state from Zk
Shard 1
Leader
Node 1 Node 2
Shard 2
Leader
Shard 2
Replica
Shard 1
Replica
Zookeeper
CloudSolrServer
“smart client”
1
2
3
tlogtlog
Get URLs of current leaders?
4
5
shard1 range:
80000000-ffffffff
shard2 range:
0-7fffffff
{
"id" : "football!2",
"sport_s" : "football",
"type_s" : "post",
"lang_s" : "en",
...
},
Hash:
shardKey!docID

19. Confidential and Proprietary
© 2013 LucidWorks19
Custom hashing: Query side
19
Shard 1
Leader
Node 1 Node 2
Shard 2
Leader
Shard 2
Replica
Shard 1
Replica
1
3
q=*:*&
shard.keys=golf!
4
2
Query controller
get fields

20. Confidential and Proprietary
© 2013 LucidWorks20
Custom hashing key points
• Co-locate documents having a common property in the
same shard
- e.g. golf!10 and golf!22 will be in the same shard
• Scale-up the replicas for specific shards to address
high query volume – e.g. Golf in summer
• Not as much control over the distribution of keys
- golf, baseball, and tennis all in shard1 in my example
• Can split unbalanced shards when using custom
hashing

21. Confidential and Proprietary
© 2013 LucidWorks21
What’s next?
• Improvements of splitting
feature coming 4.4
• Client-side routing
- Smart client will decide the best leader to send a document to
- SOLR-4816
• Re-balance collection after adding N more nodes
- SOLR-5025
• Splitting optimizations
- Control the path where sub-shards create their index (similar to
path when doing a snapshot backup)

22. Confidential and Proprietary
© 2013 LucidWorks22
Thank you for attending!
• Keeping in touch
- Solr mailing list: solr-user@lucene.apache.org
- Solr In Action book: http://www.manning.com/grainger/
- Twitter: @thelabdude
- Email: thelabdude@gmail.com
- LinkedIn: linkedin.com/in/thelabdude
• Questions?