2. ElasticSearch - Know Your Data
7 de Abril, 2016
Nuno Ochoa
Elastic Evangelist@Polarising
3. 3
ElasticSearch - Know Your Data
Elastic Stack
Polarisign 2014 3
Kibana
ElasticSearch
Logstash Beats
Plugins
Data Visualization
Store, Index,
Search, Analyse
Data Ingestion
4. 4
ElasticSearch - Know Your Data
ElasticSearch – Store, Index, Search and Analyze
Polarisign 2014 4
Distributed and Scalable
﹥ Design to scale-out: improved resiliency
﹥ High Availability
﹥ Multi tenant
Search and Analytics
﹥ Real-time (near)
﹥ Full text search
﹥ Multi language support
﹥ Aggregations
﹥ Geospacial Info support
Developing Key Features
﹥ Schemaless (structure and unstructured data)
﹥ Document oriented (JSON)
﹥ RESTful API
﹥ Client Libraries (Java, .Net, PHP, etc)
﹥ Build on top of Apache Lucene
5. 5
ElasticSearch - Know Your Data
ElasticSearch – Scale-Out and High-Availability
Polarisign 2014 5
Elastic Cluster
Node 1
Shard 1 Shard 2
Shard 3 Shard 4
Elastic Cluster
Node 1
Shard 1 Shard 2
Shard 3 Shard 4
Node 2
Shard 1 Shard 2
Shard 3 Shard 4
Add one node
• 1 index
• Nr of shards: 4
• Nr of Replicas: 1
6. 6
ElasticSearch - Know Your Data
Logstash – Data Ingest
Polarisign 2014 6
Collect, Enrich and Transport Data
﹥ Centralize data processing
﹥ Normalize/Format distinct data types
﹥ Easily extensible
﹥ 200+ plugins
Data Pipeline
Input Filter(s) Output
• elasticsearch
• file
• syslog
• rabbitmq
• csv
• geoip
• elasticsearch
• mutate
• file
• mongodb
• elasticsearch
• rabbitmq
7. 7
ElasticSearch - Know Your Data
Beats – Data Ingest
Polarisign 2014 7
Collect, Parse and Ship Data
﹥ Lightweight data shippers
﹥ Forward host-based operational metrics
﹥ Single purpose
﹥ Libbeat, framework to build new Beats
Operational Data Examples
﹥ Wire Data => Packetbeat (multiple decoders available, like HTTP, MySql)
﹥ System Stats => Topbeat
﹥ Logs => Filebeat, Winlogbeat
8. 8
ElasticSearch - Know Your Data
Kibana – Data Visualization
Polarisign 2014 8
Search and Analytics
﹥ Web based interface for data visualization stored in ES
﹥ Explore and analyze patterns in data
﹥ Leverage the power of Elasticsearch analytics capabilities
Visualization
﹥ Supports multiple types, charts, maps, histograms
﹥ Share and embed dashboards into operational dashaboards
﹥ Supports custom visualizations and applications
﹥ Plugins for cluster management and administration
9. 9
ElasticSearch - Know Your Data
Elastic – Use Cases
Polarisign 2014 9
Search
﹥ Full text search (fast)
﹥ Fuzzy text search
﹥ Geospacial
Analytics
﹥ Explore your data
﹥ Ask complex queries about your data
﹥ Leveraged by ES aggregations feature
﹥ Kibana!
Logging
﹥ Logstash or Beats => ElasticSearch => Kibana
﹥ Centralize log store, analytics and visualization
10. 10
ElasticSearch - Know Your Data
Elastic – Aggregations
Polarisign 2014 10
﹥ Summarize our data vs looking for particular documents
Sample Questions
﹥ Most popular items?
﹥ Average shopping value for each day?
﹥ Which stores sells more?
﹥ Sky is the limit …
﹥ Quick and near real-time, just like search
﹥ Powerful for reports and dashboards => No need for long run jobs
﹥ Can be combined with search/filter queries:
﹥ Most popular items? => Most popular Electronic items?
11. 11
ElasticSearch - Know Your Data
Elastic – Aggregations
Polarisign 2014 11
Concepts
﹥ Buckets: Collections of documents that meet a criterion
﹥ Metrics: Statistics calculated on the documents in a bucket
﹥ An aggregation is a combination of one or more buckets and zero or more metrics
SQL
SELECT COUNT(product)
FROM SALE
GROUP BY product
GET sales/_search
{
"size": 0,
"aggs": {
"products": {
"terms": {
"field": “product",
"size": 100
}
}
}
}
Elastic
12. 12
ElasticSearch - Know Your Data
Elastic – Aggregations
Polarisign 2014 12
Concepts
﹥ Buckets: Collections of documents that meet a criterion
﹥ Metrics: Statistics calculated on the documents in a bucket
﹥ An aggregation is a combination of one or more buckets and zero or more metrics
SQL
SELECT COUNT(product)
FROM SALE
GROUP BY product
GET sales/_search
{
"size": 0,
"aggs": {
"products": {
"terms": {
"field": “product",
"size": 100
}
}
}
}
Elastic
13. 13
ElasticSearch - Know Your Data
Elastic – Aggregations
Polarisign 2014 13
Types
﹥ Bucketing: builds buckets according to a criteria
﹥ Histogram, Date Histogram, Geo Distance, Terms
﹥ Metrics: compute metrics over a set of documents
﹥ Avg, Stats, Sum, Top Hits
﹥ Pipeline: aggregate the output of other aggregations and their associated metrics
Sub-Aggregation
﹥ Bucketing aggregations can have sub-aggregations
﹥ No hard-limit of nested aggregation
﹥ Sub-aggregations will execute within the context of parent buckets
35. 35
Ability to customize colors, text,
numbers, labels, layouts, skins,
and visualizations.
1 All-new Visualization tools for
Graph and Time Series data.
2 Strong integration with Security,
Monitoring, and the rest of the
Elastic Stack
3
37. 37
What to talk about
• space efficiency up 20% => more real estate
• export all the things!
• plugin all the things!
• kibana app generator
• status page
Kibana
Source: Gray Arial 10pt