Scaling Massive ElasticSearch Clusters

Scaling Massive ElasticSearch
Clusters
Rafał Kuć – Sematext International
@kucrafal @sematext sematext.com

Who Am I
• „Solr 3.1 Cookbook” author
• Sematext software engineer
• Solr.pl co-founder
• Father and husband :-)
Copyright 2012 Sematext Int’l. All rights reserved

What Will I Talk About ?
• ElasticSearch scaling
• Indexing thousands of documents per second
• Performing queries in tens of milliseconds
• Controling shard and replica placement
• Handling multilingual content
• Performance testing
• Cluster monitoring

The Challenge
• More than 50 millions of documents a day
• Real time search
• Less than 200ms average query latency
• Throughput of at least 1000 QPS
• Multilingual indexing
• Multilingual querying

Why ElasticSearch ?
• Written with NRT and cloud support in mind
• Uses Lucene and all its goodness
• Distributed indexing with document
distribution control out of the box
• Easy index, shard and replicas creation on live
cluster

Index Design
• Several indices (at least one index for each day
of data)
• Indices divided into multiple shards
• Multiple replicas of a single shard
• Real-time, synchronous replication
• Near-real-time index refresh (1 to 30 seconds)

Shard Deployment Problems
• Multiple shards per node
• Replicas on the same nodes as shards
• Not evenly distributed shards and replicas
• Some nodes being hot, while others are cold

Default Shard Deployment
ElasticSearch Cluster
Node 1 Node 2
Node 3
Shard 1 Shard 2 Shard 3 Replica 1
Replica 2
Replica 3

What Can We Do With Shards Then ?
• Contol shard placement with node tags:
– index.routing.allocation.include.tag
– index.routing.allocation.exclude.tag
• Control shard placement with nodes IP
addresses:
– cluster.routing.allocation.include._ip
– cluster.routing.allocation.exclude._ip
• Specified on index or cluster level
• Can be changed on live cluster !

Shard Allocation Examples
• Cluster level:
curl -XPUT localhost:9200/_cluster/settings -d '{
"persistent" : {
"cluster.routing.allocation.exclude._ip" : "192.168.2.1"
}
}'
• Index level:
curl -XPUT localhost:9200/sematext/ -d '{
"index.routing.allocation.include.tag" : "nodeOne,nodeTwo"
}'

Number of Shards Per Node
• Allows one to specify number of shards per
node
• Specified on index level
• Can be changed on live indices
• Example:
curl -XPUT localhost:9200/sematext -d '{
"index.routing.allocation.total_shards_per_node" : 2
}'

Controlled Shard Deployment
Node 1 Node 2
Node 3
Shard 1
Shard 2
Shard 3 Replica 1Replica 2
Replica 3

Does Routing Matters ?
• Controls target shard for each document
• Defaults to hash of a document identifier
• Can be specified explicitly (routing parameter) or
as a field value (a bit less performant)
• Can take any value
• Example:
curl -XPUT localhost:9200/sematext/test/1?routing=1234 -d '{
"title" : "Test routing document"
}'

Indexing the Data
Node 1 Node 2
Node 3
Shard
1
Shard
2
Shard
3
Replica
1
Replica
2
Replica
3
Indexing application

How We Indexed Data
Node 1 Node 2
Shard 1 Shard 2
Node 3
Indexing application

Nodes Without Data
• Nodes used only to route data and queries to
other nodes in the cluster
• Such nodes don’t suffer from I/O waits (of
course Data Nodes don’t suffer from I/O waits
all the time)
• Not default ElasticSearch behavior
• Setup by setting node.data to false

Multilingual Indexing
• Detection of document's language before
sending it for indexing
• With, e.g. Sematext LangID or Apache Tika
• Set known language analyzers in configuration
or mappings
• Set analyzer during indexing (_analyzer field)

Multilingual Indexing Example
curl -XPUT localhost:9200/sematext/test/10 -d '{
"title" : "Test document",
"langId" : "english"
}'
{
"test" : {
"_analyzer" : { "path" : "langId" },
"properties" : {
"id" : { "type" : "long", "store" : "yes", "precision_step" : "0" },
"title" : { "type" : "string", "store" : "yes", "index" : "analyzed" },
"langId" : { "type" : "string", "store" : "yes", "index" : "not_analyzed" }
}
}
}

Multilingual Queries
• Identify language of query before its execution
(can be problematic)
• Query analyzer can be specified per query
(analyzer parameter):
curl -XGET
localhost:9200/sematext/_search?q=let+AND+me&analyzer=english

Query Performance Factors – Lucene
level
• Refresh interval
– Defaults to 1 second
– Can be specified on cluster or index level
– curl -XPUT localhost:9200/_settings -d '{ "index" : {
"refresh_interval" : "600s" } }'
• Merge factor
– Defaults to 10
– Can be specified on cluster or index level
– curl -XPUT localhost:9200/_settings -d '{ "index" : {
"merge.policy.merge_factor" : 30 } }'

Let’s Talk About Routing Once Again
• Routes a query to a particular shard
• Speeds up queries depending on number of
shards for a given index
• Have to be specified manualy with routing
parameter during query
• routing parameter can take any value:
curl -XGET
'localhost:9200/sematext/_search?q=test&routing=2012-02-16'

Querying ElasticSearch – No Routing
Shard 1 Shard 2 Shard 3 Shard 4
ElasticSearch Index
Application

ElasticSearch Index
Application
Querying ElasticSearch – With Routing

Performance Numbers
Queries without routing (200 shards, 1 replica)
#threads Avg response time Throughput 90% line Median CPU Utilization
1 3169ms 19,0/min 5214ms 2692ms 95 – 99%
Queries with routing (200 shards, 1 replica)
#threads Avg response time Throughput 90% line Median CPU Utilization
10 196ms 50,6/sec 642ms 29ms 25 – 40%
20 218ms 91,2/sec 718ms 11ms 10 – 15%

Scaling Query Throughput – What Else ?
• Increasing the number of shards for data
distribution
• Increasing the number of replicas
• Using routing
• Avoid always hitting the same node and
hotspotting it

FieldCache and OutOfMemory
• ElasticSearch default setup doesn’t limit field
data cache size

FieldCache – What We Can do With It ?
• Keep its default type and set:
– Maximum size (index.cache.field.max_size)
– Expiration time (index.cache.field.expire)
• Change its type:
– soft (index.cache.field.type)
• Change your data:
– Make your fields less precise (ie: dates)
– If you sort or facet on fields think if you can reduce
fields granularity
• Buy more servers :-)

FieldCache After Changes

Additional Problems We Encountered
• Rebalancing after full cluster restarts
– cluster.routing.allocation.disable_allocation
– cluster.routing.allocation.disable_replica_allocation
• Long startup and initialization
• Faceting with strings vs faceting on numbers on
high cardinality fields

JVM Optimization
• Remember to leave enough memory to OS for
cache
• Make GC frequent ans short vs. rare and long
– -XX:+UseParNewGC
– -XX:+UseConcMarkSweepGC
– -XX:+CMSParallelRemarkEnabled
• -XX:+AlwaysPreTouch (for short performance
tests)

Performance Testing
• Data
– How much data do I need ?
– Choosing the right queries
• Make changes
– One change at a time
– Understand the impact of the change
• Monitor your cluster (jstat, dstat/vmstat,
SPM)
• Analyze your results

ElasticSearch Cluster Monitoring
• Cluster health
• Indexing statistics
• Query rate
• JVM memory and garbage collector work
• Cache usage
• Node memory and CPU usage

Cluster Health
Node restart

Indexing Statistics

Query Rate

JVM Memory and GC

Cache Usage

CPU and Memory

Summary
• Controlling shard and replica placement
• Indexing and querying multilingual data
• How to use sharding and routing and not to
tear your hair out
• How to test your cluster performance to find
bottle-necks
• How to monitor your cluster and find
problems right away

We Are Hiring !
• Dig Search ?
• Dig Analytics ?
• Dig Big Data ?
• Dig Performance ?
• Dig working with and in open – source ?
• We’re hiring world – wide !
http://sematext.com/about/jobs.html

How to Reach Us
• Rafał Kuć
– Twitter: @kucrafal
– E-mail: rafal.kuc@sematext.com
• Sematext
– Twitter: @sematext
– Website: http://sematext.com
• Graphs used in the presentation are from:
– SPM for ElasticSearch (http://sematext.com/spm)

Scaling Massive ElasticSearch Clusters

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (19)

En vedette

En vedette (20)

Similaire à Scaling Massive ElasticSearch Clusters

Similaire à Scaling Massive ElasticSearch Clusters (20)

Plus de Sematext Group, Inc.

Plus de Sematext Group, Inc. (20)

Dernier

Dernier (20)

Scaling Massive ElasticSearch Clusters