Anatoliy Sokolenko, Software Engineer at Grid Dynamics

1. Apache
Lucene/Solr
Internals

2. About me
Java and all around
Principal Software Engineer
at Grid Dynamics
Kharkiv
asokolenko@griddynamics.com

3. Apache
Lucene/Solr
Internals

4. 4 nodes
✕ 12GB disk space
June 2013 database 14.630.209 records
Indexing took 5 hours
in 100 threads
1000 batch
Lucene.net
VM
16 CPU cores
16 GB memory

5. lightweight
performant
search
library

7. Data Model
• document oriented
• flat
• store
• index

8. Data Model
• document oriented
• flat
• store
• index
score:1
tag:java
type:answer
Document
boost = 1.1
docID = 23

9. Showcase

10. Basic Flow
Lucene
Index
Index Writer Index Searcher
Analyzer Index Reader

11. Basic Flow
score:1
tag:java
type:answer
Document
boost = 1.1
Lucene
Index
Index Writer Index Searcher
addDocument
Analyzer Index Reader

12. Basic Flow
score:1
tag:java
type:answer
Document
boost = 1.1
Lucene
Index
Index Writer Index Searcher
addDocument
query
tag:java
Analyzer Index Reader

13. Basic Flow
score:1
tag:java
type:answer
Document
boost = 1.1
Lucene
Index
Index Writer Index Searcher
addDocument
query
tag:java
score:1
tag:java
type:answer
Document
boost = 1.1
search
Analyzer Index Reader

14. Lucene Index Structure

15. Index

16. Index
Segment
A

17. Index
Segment
A
Segment
B Segment
C
Segment
D

19. score:0
score:1
score:5
...
tag:java
tag:mysql
tag:css
...
type:answer
type:question
3
4
2
2
3
4
3
2
Term Infos

20. score:0
score:1
score:5
...
tag:java
tag:mysql
tag:css
...
type:answer
type:question
3 +1 +2
10 +3 +1
4 +11
5 +2
6 +52 +1
1 +30 +27
3 +7 +1
5 +2
3
+7
+2
4
2
2
3
4
3
2
1 1 1
1 1 1
1 1
1 1
1 1 1
1
1 1 1 1
1 3 5
2 1
Term Infos Term Frequencies

21. score:0
...
tag:mysql
...
score:0
score:1
score:5
...
tag:java
tag:mysql
tag:css
...
type:answer
type:questiontype:question
3 +1 +2
10 +3 +1
4 +11
5 +2
6 +52 +1
1 +30 +27
3 +7 +1
5 +2
3
+7
+2
4
2
2
3
4
3
2
1 1 1
1 1 1
1 1
1 1
1 1 1
1
1 1 1 1
1 3 5
2 1
Term InfosTerm Info Index Term Frequencies
3
3
2

22. Showcase

23. scalable
enterprise search
server

24. Request
Handlers
Data Import
Handler
SolrCloud
solrconfig.xml
schema.xml

25. SolrCloud

26. Join Cluster

27. Shard 3Shard 2Shard 1
Join Cluster

28. Indexing
Shard 1 Shard 2 Shard 3

29. Indexing
Shard 1 Shard 2 Shard 3

30. Indexing
Shard 1 Shard 2 Shard 3

31. Indexing
Shard 1 Shard 2 Shard 3

32. Indexing
Shard 1 Shard 2 Shard 3

33. Indexing
Shard 1 Shard 2 Shard 3

34. Query
Shard 1 Shard 2 Shard 3

35. Query
Shard 1 Shard 2 Shard 3 query
tag:java

36. Query
Shard 1 Shard 2 Shard 3 query
tag:java

37. Query
Shard 1 Shard 2 Shard 3 query
tag:java

38. Query
Shard 1 Shard 2 Shard 3 query
tag:java

39. Failure
Shard 1 Shard 2 Shard 3

40. Failure
Shard 1 Shard 2 Shard 3

41. Failure
Shard 1 Shard 2 Shard 3

42. CAP Model
C
A
P
SolrCloud
Solr

43. Showcase

44. Faceted Navigation

45. Showcase

46. Algorithm
tag:java
tag:mysql
tag:css
5 +2
6 +52 +1
1 +30 +27 +2
7 31 58 59Query Result
Index

47. Algorithm
tag:java
tag:mysql
tag:css
5 +2
6 +52 +1
1 +30 +27 +2
7 31 58 59Query Result
Index
5 7
6 58 59
1 31 58 60

48. Algorithm
tag:java
tag:mysql
tag:css
5 +2
6 +52 +1
1 +30 +27 +2
7 31 58 59Query Result
Index Facet
1
2
2
5 7
6 58 59
1 31 58 60

49. Showcase

50. Text Analysis

51. Analyzer
Tokenizer
Filter
Char filter

52. Analyzer
Tokenizer
Filter
Char filter
Index time

53. Analyzer
<strong>There are no pointers in
Java!</strong>
Tokenizer
Filter
Char filter
Index time

54. Analyzer
<strong>There are no pointers in
Java!</strong>
Tokenizer
Filter
There are no pointers in Java!
Char filter
Index time

55. Analyzer
<strong>There are no pointers in
Java!</strong>
Tokenizer
There
are
no
pointers
in
Java
Filter
There are no pointers in Java!
Char filter
Index time

56. Analyzer
<strong>There are no pointers in
Java!</strong>
Tokenizer
There
are
no
pointers
in
Java
Filter
There are no pointers in Java!
Char filter
Index time
?
?
?
pointer
?
java

57. Analyzer
<strong>There are no pointers in
Java!</strong>
Tokenizer
There
are
no
pointers
in
Java
Filter
There are no pointers in Java!
Char filter
Index time Query time
?
?
?
pointer
?
java

58. Analyzer
<strong>There are no pointers in
Java!</strong>
Tokenizer
There
are
no
pointers
in
Java
Filter
There are no pointers in Java!
Char filter
pointers in Java
Index time Query time
?
?
?
pointer
?
java

59. Analyzer
<strong>There are no pointers in
Java!</strong>
Tokenizer
There
are
no
pointers
in
Java
Filter
There are no pointers in Java!
Char filter
pointers in Java
Index time Query time
pointers in Java
?
?
?
pointer
?
java

60. Analyzer
<strong>There are no pointers in
Java!</strong>
Tokenizer
There
are
no
pointers
in
Java
Filter
There are no pointers in Java!
Char filter
pointers in Java
Index time Query time
pointers in Java
pointers Java
in
?
?
?
pointer
?
java

61. Analyzer
<strong>There are no pointers in
Java!</strong>
Tokenizer
There
are
no
pointers
in
Java
Filter
There are no pointers in Java!
Char filter
pointers in Java
Index time Query time
pointers in Java
pointers Java
in
?
?
?
pointer
?
java
pointer java
?

62. Showcase

63. Spell Suggestions

64. Levenshtein Distance
html
htmm
Levenshtein
distance = 1
hlmz
html
Levenshtein
distance = 2
tag:php
tag:jquery
tag:json
tag:java
tag:c#
tag:apache
tag:osx
tag:html

65. Levenshtein Automaton
html
Levenshtein
distance = 1
Ht
t
m
m
l
l
t
m
l
H t
t
m
m
l l
m
l
tt
m
H
l
m

66. Showcase

67. Solr is...
• enterprise level search engine
• vertically scalable
• horizontaly scalable, but...
• tunable
• poorly documentation
• with active community

68. • http://blog.mikemccandless.com
• http://lucene.apache.org/core/4_3_1/
index.html
• Introduction to Information Retrieval
http://nlp.stanford.edu/IR-book/
• http://wiki.apache.org/solr/
• https://cwiki.apache.org/confluence/display/
solr/Apache+Solr+Reference+Guide
References

69. Q&A
http://twitter.com/AnatolSokolenko
http://flip.it/whFqy asokolenko@griddynamics.com

70. The End