Contenu connexe


Deep Dive Into Elasticsearch

  1. Deep Dive Into Elasticsearch Kunal Kapoor Software Consultant Knoldus Software LLP
  2. AGENDA ● What is Elasticsearch ● Getting Started ● Key Terminologies ● CRUD Operations ● Understanding the physical layout ● What happens when you index a document ● How to make an inverted index mutable ● How per-segment search works ● How a delete operation works ● Segment Merging
  3. What is Elasticsearch ● Search engine based on Lucene. ● Provides near real-time search ● Distributed ● Fault Tolerant ● Notable users: – Facebook – Github – CERN – LinkedIn
  4. Getting Started ● Download the elasticsearch distribution from ● To start the elasticsearch server run the following command from within the extracted directory – ./bin/elasticsearch ● Once the server or node is created you can check the health of your cluster by running – curl 'localhost:9200/_cat/health?v'
  5. Key Terminologies ● Node - A node is a single server that is part of your cluster, stores your data, and participates in the cluster’s indexing and search capabilities ● Cluster - A cluster is a collection of one or more nodes (servers) that together holds your entire data and provides indexing and search capabilities across all nodes. ● Index - An index is a collection of documents that have somewhat similar characteristics.
  6. Key Terminologies ● Type - A type is a logical category/partition of your index. It is defined for documents that have a set of common fields. ● Shard – A shard is basically an lucene index. Contains the documents and various data structures that help in searching.
  7. CRUD Operations ● Indexing a document – curl -XPOST 'localhost:9200/test/test/1?pretty' -d '{"text":"Hello World"}' ● Updating a document – curl -XPOST 'localhost:9200/test/test/1?pretty' -d '{"text":"Hello"}' ● Delete a document – curl -XDELETE 'localhost:9200/test/test/1?pretty' ● Search a document – curl -XPOST 'localhost:9200/test/_search?pretty' -d '{"query": { "match": {"text": "hello" }}}'
  8. Understanding the physical layout
  9. SHARDS (Lucene indices)
  11. Inverted Index ● Data structure storing a mapping, from content such as words or numbers, to its locations in a database file, or a set of documents. ● Provides full-text search ● Consists of 2 parts – Sorted Dictionary – Postings ● Immutable
  12. What happens when you index a document? ● The node that receives the request becomes the controller for that request. ● That node determines the shard in which the document should reside on the basis of the documents Id. – shard = hash(document_id) % number_of_primary_shards ● The request is then forwarded to the appropriate node which contains the shard. ● The node forwards the request to the appropriate shard.
  13. What happens when you index a document? ● The shard performs analysis on the document and creates the appropriate inverted index which is helpful for searching. ● The request is then sent to the replica shards. ● The documents are analyzed by the standard analyzer by default. ● It split the documents on white-space and lowercases the documents. ● The documents are then ready to be inserted in the inverted index
  14. What happens when you index a document? { “text”:”Elasticsearch is an awesome search engine” } { “text:”Elasticsearch is not a database” }
  15. elasticsearch, is, an, awesome, search, engine, elasticsearch,not, a, database
  16. Terms Document1 Document2 elasticsearch ✓ ✓ is ✓ ✓ an ✓ - awesome ✓ - search ✓ - engine ✓ - not - ✓ a - ✓ database - ✓
  17. How to make an inverted index mutable? ● Earlier, the whole inverted index would be rewritten to disk with the changes. ● Very costly approach ● Lucene introduced the concept of per-segment search. ● Now a Lucene index would mean a collection of segments plus a commit point. ● A commit point is a file that contains the list of segments that are ready for search.
  18. How per-segment search works? ● New documents are collected in an in-memory buffer. ● Every so often, the buffer is commited (refresh) – A new supplementary segment with a commit point is written to file-system cache. – The transaction log is updated with the request for a full commit later. ● The buffer is cleared and the segment is made available for search.
  19. In-memory Buffer COMMIT POINT Transaction Log
  20. How a delete operation works? ● Every shard in the Elasticsearch node maintains a .del file along with the commit point. ● .del file lists which documents in which segments have been deleted. ● When a delete request is encountered, the appropriate document is marked as deleted. ● The deleted document will still match the search query but will be filtered later on. ● Later the document is purged from the file-system while segment merging.
  21. Segment Merging ● Each segment consumes file handles, memory, and CPU cycles. ● The more the number of segments, the slower the search will be. ● Elasticsearch solves this problem by merging segments in the background. ● Small segments are merged into bigger segments. ● This is the moment when those old deleted documents are purged from the file-system.
  22. Segment Merging ● This is how the merge process works:- – The merge process selects a few segments of similar size and merges them into a new bigger segment in the background. – The new segment is flushed to disk. – A new commit point is written that includes the new segment and excludes the old, smaller segments. – The new segment is opened for search. – The old segments are deleted.
  23. References ● The Definitive Guide by Clinton Gormley and Zachary Tong ●
  24. Questions?
  25. References ● The Definitive Guide by Clinton Gormley and Zachary Tong ●

Notes de l'éditeur

  1. 1. It allows you to explore your data at a speed. 2. Distributed-- 2 or more servers can be started at different locations and the data will be stored on both of them. 3. fault- Es-index is distributed across different nodes As the nodes are distributed across different zones, it is unlikely that a failure affects both the servers at the same time
  2. Es is configured to use multicast out of the box.In multicast it sends UDP pings across the local network to discover the nodes. Other nodes receive this ping and respond thus creating a cluster. Not good for production as it can also discover unwanted nodes if cluster name is same Thus unicast is used which accepts IpAddress of nodes that you want to add in the cluster
  3. A node has multiple shards within themAn Es index can span across multiple nodes through shards. A shard is the lowest level worker that contains the data that is inserted in the index.
  4. A lucene index or a shard contains various segments that are like mini indicesThese indices contain the datastructures required by elasticsearch to provide near-real time search.