The place of Elasticsearch in Big Data landscape.

1. (Elastic)search in Big Data
Radu Gheorghe
@radu0gheorghe @sematext

2. What is “search in Big Data”? Challenges?
Some solutions?
How does Elasticsearch do it?
Agenda

3. Search Expectations

4. headphones for iPhone 4, iPhone 5, iPhone 6 and iPhone 7
iPhone 5
iPhone 4
Relevancy...
iphone

5. iphone
iphone 5
Institute of Public Health
...and autocomplete...
iph

6. No results found for “iphnoe”
iPhone 5
iPhone 4
… and fuzziness...
iphnoe

7. Did you mean “iPhone”?
iPhone 5
iPhone 4
...and corrections...
iphnoe
shows results
anyway

8. iPhone 5
iPhone 4
iPhone 3
Galaxy S4
...and similar terms...
iphone

9. iPhone 5
iPhone 4
...and don’t forget the statistics!
iphone
☑ iOS
☐ other
☑ <100RON
☐ 100-200RON
☐ >200RON

10. Wait. Fancy search == Big Data?

11. Fancy stuff isn’t free
iphone
☑ iOS
☐ other
☑ <100RON
☐ 100-200RON
☐ >200RON
N requests for
autocomplete
Did you mean...
iPhone 5
iPhone 4
iPhone 3
Galaxy S4
1 request for
each of the stats
1 request for
synonyms, 1
for exact
matches, etc
1 request for
corrections

12. Distributed search. When one server
doesn’t cut it

13. Log Search
web_server01
database01
backend01
search engine
10:01 - webapp - DB connect error
10:00 - DB - I/O error
error

14. Log Analytics
unique IPs: 7584
iPhone 5
iPhone 4
Galaxy S4
best sellers
Romania: 200
France: 150
Hungary: 120
users per country
revenue per day

15. Distributed search solutions
Elasticsearch
Solr
Others: SenseiDB, Sphinx…
SaaS: CloudSearch, Logsene...
built on top of
Lucene

16. Document-oriented
Lucene awesome: index & store data,
relevancy, fuzzy, suggesters...
...all wrapped up in JSON over HTTP
Elasticsearch

17. Aggregations
revenue per day
unique IPs: 7584

18. Aggregations
revenue per day
unique IPs: 7584
Romania: 200
France: 150
Hungary: 120
unique IPs per country

19. Aggregations
revenue per day
Romania: 200
France: 150
Hungary: 120
unique IPs per country
unique IPs per country per day
Romania
unique IPs: 7584

20. Node 1

21. Node 1

22. Node 1 Node 2

23. Node 1 Node 2

24. Node 1 Node 2 Node 3

25. Node 1 Node 2 Node 3

26. Node 1 Node 2 Node 3

27. Node 1 Node 2

28. Node 1 Node 2

29. Big Data distributed
search
search and real-
time analytics

30. Big Data distributed
search
search and real-
time analytics
more search
features

31. Big Data distributed
search
search and real-
time analytics
more search
features
clients
usage
(logs)

32. Thank you!
radu.gheorghe@sematext.com
@radu0gheorghe @sematext
Big Data distributed
search
search and real-
time analytics
more search
features
clients
usage
(logs)