Introduction to Elasticsearch How to store big data in Elasticsearch Elasticsearch capabilities

1. Elasticsearch Get started
Presenter: Ismail Anjrini
Date: 31 Aug - 2022

2. About Me
More than 15 years experience
Chief Architect

3. Middle-east Elasticsearch Community
https://www.meetup.com/elastic-saudi-arabia/
700+ Members
https://t.me/ElasticArabi
500+ Members
https://www.youtube.com/c/IsmailAnjrini
280+ Videos

4. Session Preparation
Download Elasticsearch 8.3.3
Download Kibana 8.3.3
run elasticsearch or elasticsearch.bat
Copy elastic user password
Copy Kibana token
run kibana or kibana.bat
Open Kibana URL and use Kibana token
Done

5. ELASTICSEARCH IS A CAREER PATH

6. Elastic Stack (ELK)

7. Elasticsearch

8. Kibana

9. Logstash

10. Beats

11. Lucene
Apache Lucene is an open source project available for free
Lucene is a Java library
Elasticsearch is built over Lucene and provides a JSON based REST API to refer to Lucene features
Elasticsearch provides a distributed system on top of Lucene

12. Elasticsearch - Concepts

13. Cluster
Node 1 Node 2 Node 3 Node 1 Node 2 Node 3
Cluster 1 Cluster 2

14. Node types
master data ingest
Cluster
master
data
data_warm
transform
data_hot
data_cold
remote_cluster_client
data_frozen
ingest
data_content
ml

15. Index
RDBMS Database Table Columns/Rows
Elasticsearch ? Index Document
NoSQL

16. Index (cont.)
NAME TYPE NULL?
ID INT NO
NAME VARCHAR(20) NO
ID Name
1 ISMAIL
2 MOHAMMAD

17. Document
A document is a basic unit of information that can be indexed

18. Shards
Each shard is in itself a fully-functional and independent "index" that can be hosted on any node in the cluster
index
shard 1 shard 2 shard 3

19. Replicas
index
shard 1 shard 2 shard 3
replica 2-1
replica 1-1 replica 3-1

20. Building Search Based Application

21. e-Commerce
Search Results

22. e-Commerce (cont.)
Search Results

25. e-Commerce (cont.)

26. e-Commerce (cont.)
CDC SI RabbitMQ Client
NEST

27. e-Commerce (cont.)
Categories
ID
Name
Products
ID
Name
Price
LastUpdateDatetTime
ProductsCategories
ID
ProductID
CategoryID

28. e-Commerce (cont.)

29. e-Commerce (cont.)

30. e-Commerce (cont.)

31. e-Commerce (cont.)

32. Storing Big Data in Elasticsearch

33. Big Data - Business
Ingestion Performance Query Performance BOTH

34. Big Data - Index Architecture
index
shard 1 shard 2 shard 3

35. Big Data - Ingestion
Ingest
Data Data

36. Big Data - Ingestion (cont.)
Ingest
Data Data
Logstash

37. Big Data - Query
Ingest
Data Data
Coordinating

38. Big Data - Multi Cluster
Node 1 Node 2 Node 3 Node 1 Node 2 Node 3
Cluster 1 Cluster 2

39. Big Data - Features
Rollup jobs Summarize and store historical data in a smaller index for future analysis
Transforms Use transforms to pivot existing Elasticsearch indices into summarized entity-centric
indices or to create an indexed view of the latest documents for fast access
ILM Makes it easier to manage indices in hot-warm-cold architectures, which are common
when you’re working with time series data such as logs and metrics
Data streams A data stream lets you store append-only time series data across multiple indices while
giving you a single named resource for requests

40. Elasticsearch - Search Capabilities

41. Search Capabilities
Autocomplete
Aggregations
Highlighting
Paginate search results
Geospatial search
“Did you mean”?
Sort search results
Elasticsearch SQL
Scripting
Text analysis
Pinned Query

42. Autocomplete
Completion Suggester
completion type
Category Context
Geo location Context

43. Aggregations
An aggregation summarizes your data as metrics, statistics, or other analytics
Metric Aggregations that calculate metrics, such as a sum or average, from field values
Bucket Aggregations that group documents into buckets, also called bins, based on field values, ranges,
or other criteria
Pipeline Aggregations that take input from other aggregations instead of documents or fields

44. Highlighting
Highlighters enable you to get highlighted snippets from one or more fields in your search results so you
can show users where the query matches are

45. Paginate search results
By default, searches return the top 10 matching hits
The from parameter defines the number of hits to skip, defaulting to 0
The size parameter is the maximum number of hits to return
Search after

46. Geospatial search
Elasticsearch supports two types of geo data: geo_point fields which support lat/lon pairs, and geo_shape
fields, which support points, lines, circles, polygons, multi-polygons, etc

47. “Did you mean”?
Suggesters
multi_match with bool_prefix
match_phrase_prefix

48. Sort search results
Allows you to add one or more sorts on specific fields
The sort is defined on a per field level, with special field name for _score to sort by score, and _doc to
sort by index order

49. Elasticsearch SQL
Elasticsearch SQL aims to provide a powerful yet lightweight SQL interface to Elasticsearch
Elasticsearch SQL is built from the ground up for Elasticsearch
No need for additional hardware, processes, runtimes or libraries to query Elasticsearch
Elasticsearch’s SQL jdbc driver is a rich, fully featured JDBC driver for Elasticsearch
Elasticsearch SQL ODBC Driver is a 3.80 compliant ODBC driver for Elasticsearch

50. Scripting
With scripting, you can evaluate custom expressions in Elasticsearch
you can use a script to return a computed value as a field or evaluate a custom score for a query
painless
expression
mustache

51. Text analysis
Text analysis enables Elasticsearch to perform full-text search, where the search returns all relevant
results rather than just exact matches
Tokenization Breaking a text down into smaller chunks, called tokens. In most cases, these tokens
are individual words
This allows you to match tokens that are not exactly the same as the search terms, but
similar enough to still be relevant
Normalization

52. Pinned Query
Promotes selected documents to rank higher than those matching a given query
This feature is typically used to guide searchers to curated documents that are promoted over and above
any "organic" matches for a search
The promoted or "pinned" documents are identified using the document IDs stored in the _id field