In the past few years I have helped a lot of customers optimising their elastic cluster. With each version elasticsearch has more options to track performance of your nodes and recently profiling your queries was added. In this talk I am going to discuss the steps you have to take when starting with elasticsearch. The choices you have to make for the size of your cluster, the amount of indexes, amount of shards, choosing the right mappings, and creating better queries. After the setup I'll continue showing how to monitor your cluster and profile your queries.

1. MAKING YOUR ELASTIC CLUSTER PERFORM
Created by @jettroCoenradie

2. WHY USE ELASTICSEARCH

7. START WITH ELASTICSEARCH

10.
curl 'localhost:9200?pretty'
{
"name" : "Tatterdemalion",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.3.1",
"build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39",
"build_timestamp" : "2016-04-04T12:25:05Z",
"build_snapshot" : false,
"lucene_version" : "5.5.0"
},
"tagline" : "You Know, for Search"
}

11.
curl -XPOST 'localhost:9200/conferences/conference/1?pretty' -d '
{
"name": "Codemotion Amsterdam",
"location": "Kromhouthal"
}'

12. THAT WAS EASY!

14. DESIGN YOUR CLUSTER
How to install and configure?
How many nodes?
What hardware?

15. INSTALLATION
Just download, unzip and run
Use package manager: yum, apt
Use ansible, chef or puppet

16. CONFIGURATION
/etc/defaults/elasticsearch
/etc/elasticsearch/elasticsearch.yml

17. /ETC/DEFAULTS/ELASTICSEARCH
# Heap size defaults to 256m min, 1g max
# Set ES_HEAP_SIZE to 50% of available RAM, but no more than 31g
ES_HEAP_SIZE=2g

18. /ETC/ELASTICSEARCH/ELASTICSEARCH.YML
cluster.name: playground
node.name: node-1
discovery.zen.ping.unicast.hosts: ["node-1", "node-2", "node-3"]
discovery.zen.minimum_master_nodes: 2
path.repo: /opt/es_snapshots/
script.inline: true

19. How many nodes do I need?

20. Development / non-critical

21. Small production

22. Large production

23. What hardware do I need?

24. HARDWARE
Prefer cores over clock speed
Choose between 8-64Gb
Prefer SSD

25. DESIGN YOUR INDICES
How many shards?
How many replicas?
Time based indices?

26. What does an index look like?

28. How many shards do I need?
Amount of docs or terms
Indexing speeds
Not bigger than than 50Gb
Why not a lot of shards?
Start small and test

29. How many replicas do I need?

30. Should I use Types?

31. Should I use Aliases?

32. Working with time based indices?
Option to change shards per time period
Use index templates

33. DESIGN YOUR MAPPING
Do I need a mapping?
What do analyzers do?
Do I need an analyzer?

34. The default uses dynamic type mapping
Make your mapping explicit: Date, Geo_point, long
disable dynamic type mapping
PUT /_settings
{
"index.mapper.dynamic":false
}

35. A mapping is persistent, can only add new things.

36. Use multi field mapping: name
PUT /conferences
{
"mappings": {
"conference": {
"properties": {
"name": {
"type": "string",
"analyzer": "standard",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
}
}}}}

37. An analyzer creates terms out of data
Has three components:
Character filter - replace & with and
Tokenizer - on whitespace, regexp, ngrams
Filters - ascii folding, language specific, lowercase, stop words

38. CHOOSE THE RIGHT ANALYZER FOR THE JOB
Custom using tokenizer and filters combinations
Use the multi field approach for special analyzers.
Do not analyze if you don't need it.

39. INDEXING DOCUMENTS
How to improve indexing performance?

40. What happens when we index a document?

47. TIPS
DISABLE OR DECREASE REFRESH RATE
curl -XGET 'http://localhost:9200/meetups/_settings' -d '{
"index" : {
"refresh_interval" : "-1"
} }'

48. INDEX WITHOUT REPLICAS
curl -XGET 'http://localhost:9200/meetups/_settings' -d '{
"index" : {
"number_of_replicas" : 0
} }'

49. USE BULK
Bulk request should be between 5-15Mb max
Round robin requests over nodes

50. QUERYING DOCUMENTS
How to make queries faster?

51. curl -XGET 'http://localhost:9200/_search'

57. curl -XGET 'http://localhost:9200/meetups/_search?
q=venue.city:amsterdam%20AND%20description:elasticsearch
&pretty'

58. curl -XGET "http://localhost:9200/meetups/_search" -d'
{
"query": {
"bool": {
"must": [
{
"match": {
"venue.city": "amsterdam"
}
},
{
"match": {
"description": "elasticsearch"
}
}
]
}}}'

59. How to make a query Faster?

60. curl -XGET "http://localhost:9200/meetups/_search" -d'
{
"query": {
"bool": {
"must": [
{
"match": {
"description": "elasticsearch"
}
}
],
"filter": {
"term": {
"venue.city.raw": "Amsterdam"
}
}
}}}'

61. Query context Filter context
How well does it match? Does it match?
Calculates score true/false
Not-cacheable Cacheable

62. Use filter context if you do not need a score
Don't ask for hits if you do not use them
Request only the fields that you need

63. curl -XGET "http://localhost:9200/meetups/_search" -d'
{
"_source": {
"include": ["venue.*", "group.name", "name"]
},
"query": {
"simple_query_string": {
"query": "elastic OR elasticsearch"
}
}
}'

64. Profile api to learn about the performance
"profile": true

65. ANALYTICS FROM DOCUMENTS
Why use not_analyzed fields?

66. Aggregations, maybe the reason why elasticsearch became so popular

67. curl -XGET "http://localhost:9200/meetups/_search" -d'
{
"size": 0,
"aggs": {
"byCity": {
"terms": {
"field": "venue.city.raw",
"size": 10
}
}
}
}'

68. "buckets": [
{
"key": "Amsterdam",
"doc_count": 8
},
{
"key": "Ede",
"doc_count": 1
},
{
"key": "Leidschendam",
"doc_count": 1
},
{
"key": "Rotterdam",
"doc_count": 1
}
]

70. Inverted index not suitable for aggregations

71. DOC_VALUES
Stored on disk during indexing
All fields except analyzed strings

72. FIELDDATA
For analyzed strings
Stored in the heap

73. MONITORING THE CLUSTER
How can I see what elastic is doing?
What numbers are important?

74. GET /_cluster/health
{
"cluster_name": "playground",
"status": "yellow",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"active_primary_shards": 55,
"active_shards": 55,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 16,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 77.46478873239437
}

75. GET /_cluster/health?level=indices
"indices": {
"conferences": {
"status": "yellow",
"number_of_shards": 5,
"number_of_replicas": 1,
"active_primary_shards": 5,
"active_shards": 5,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 5
}
}

76. indices - field_data, filter_cache
os - cpu, memory, load
process - file descriptors, cpu, memory
jvm - memory, garbage collection
thread_pool - threads, rejected
fs - disk space
GET /_nodes/stats?human

77. _CAT API
GET /_cat/health?v
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks ma
1462882362 14:12:42 playground yellow 1 1 56 56 0 0
GET /_cat/indices?v
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open gridshore-logs-2016.01.19 5 0 1007 0 1.2mb
green open .kibana 1 0 100 1 100.2kb
yellow open topbeat-2016.05.04 5 1 170264 0 44.4mb
green open meetups-20160509113909 1 0 11 0 67.3kb
GET /_cat/fielddata?v
id host ip node total
aqj9L-DPR86J8CgYitcHsA 127.0.0.1 127.0.0.1 node-JC 0b

79. CLUSTER LOGS
How to configure what is logged?

80. LOGGING
Can be changed dynamically
PUT /_cluster/settings
{
"transient" : {
"logger.discovery" : "DEBUG"
}
}

81. SLOWLOG
PUT /meetups/_settings
{
"index.search.slowlog.threshold.query.warn" : "10s",
"index.search.slowlog.threshold.fetch.debug": "500ms",
"index.indexing.slowlog.threshold.index.info": "5s"
}
PUT /_cluster/settings
{
"transient" : {
"logger.index.search.slowlog" : "DEBUG",
"logger.index.indexing.slowlog" : "WARN"
}
}

82. [2016-05-11 16:25:02,105][DEBUG][index.search.slowlog.query]
[meetups-20160509113909]took[518.5micros],took_millis[0],
types[], stats[], search_type[QUERY_AND_FETCH], total_shards[1]
, source[{"size":0,"aggs":{"byCity":{"terms":{"field":
"venue.city.raw","size":10}}}}], extra_source[],

83. QUESTIONS?
Twitter: @jettroCoenradie
Github: https://github.com/jettro
Blog: https://amsterdam.luminis.eu/news/
Licence: http://creativecommons.org/licenses/by-nc-sa/3.0/