intro to full text search solution, Apache Solr

1. Full Text Search with
Apache Solr
Pittaya Sroilong
pittaya@gmail.com

2. Who am I?

4. Solr?

6. Not her!

7. But a search server

8. based on Lucene

9. Lucene?

10. Full-text search
library

11. 100% java
:-(

12. Solr is based on
Lucene

13. XML/HTTP, JSON
interface

14. Open Source

15. Shield us from using
Java
:-)

16. Who use Solr/Lucene?

17. Who use Solr/Lucene?

18. What is our problem?

19. How do we
implement this?

20. SELECT * FROM post WHERE
topic LIKE ‘%aoi%’ OR author
LIKE ‘%aoi%’ ORDER BY id DESC

21. SELECT * FROM post WHERE
(topic LIKE ‘%aoi%’ OR author
LIKE ‘%aoi%’)
OR
(topic LIKE ‘%miyabi%’ OR
author LIKE ‘%miyabi%’)
ORDER BY id DESC

22. Full table scan
=
Performance killer

23. No search scoring

24. RDBMS isn’t designed
to do this

25. Use the right tool!

26. Indexer
Update index
Query
Solr Web App
Lucene
Result

27. 1

28. De ne schema.xml

29. <field name=quot;idquot; type=quot;stringquot;
indexed=quot;truequot; stored=quot;truequot; />
<field name=quot;fullnamequot; type=quot;stringquot;
indexed=quot;truequot; stored=quot;truequot; />
<field name=quot;positionquot; type=quot;stringquot;
indexed=quot;truequot; stored=quot;truequot; />
<field name=quot;tagquot; type=quot;stringiquot;
indexed=quot;truequot; stored=quot;truequot;
multiValued=quot;truequot; />

30. 2

31. Deploy on any J2EE
container

32. Tomcat, Jetty, etc.

33. 3

34. Index documents

35. Document format
<add><doc>
<field name=”id”>555</field>
<field name=”fullname”>Kaka</field>
<field name=”position”>Midfielder</field>
<field name=”tag”>AC Milan</field>
<field name=”tag”>Brazil</field>
</doc></add>

36. Post to Solr
http://<host>/solr/update

37. Any language that can
do HTTP POST

38. PHP, Perl, Python

39. cURL

40. Commit
<commit />

41. 4

42. Search

43. Query from
http://<host>/solr/select

44. Use Solr query syntax

45. http://<host>/solr/select?
q=tag:madrid&start=0&rows
=2& =fullname,position,tag

46. Response in XML or
JSON (con gurable)

47. <response>
<result numFound=”46” start=”0”>
<doc>
<str name=”fullname”>Sergio Ramos</str>
<str name=”position”>Defender</str>
<str name=”tag”>Real Madrid</str>
<str name=”tag”>Spain</str>
</doc>
<doc>
<str name=”fullname”>Diego Forlan</str>
<str name=”position”>Striker</str>
<str name=”tag”>Atletico Madrid</str>
<str name=”tag”>Uruguay</str>
</doc>
</result>
</response>

48. &wt=json

49. {
“result”: { “numFound”: 46, “start”: 0,
“docs” : [
{ “fullname”: “Sergio Ramos”,
“position”: “Defender”,
“tag”: [“Real Madrid”, “Spain”] },
{ “fullname”: “Diego Forlan”,
“position”: “Striker”,
“tag”: [“Atletico Madrid”, “Uruguay”] }
]
}
}

50. Query examples

51. • David Pizzarro
• Equiv: David OR Pizzarro
• Default operator is
“OR” (con gurable)
• Result: David Villa, David
Pizzarro, Claudio Pizzarro,
David Seaman

52. • +David +tag:Roma
• Equiv: David AND tag:Roma
• Result: David Pizzarro

53. • +David +position:(Striker OR
Mid elder)
• Result: David Villa, David
Pizzarro

54. Updating

55. Post new document to
http://<host>/solr/update

56. Deleting

57. <delete>
<id>345</id>
</delete>

58. <delete>
<query>tag:Brazil</query>
</delete>

59. <delete>
<query>*:*</query>
</delete>

60. Thai support

61. fwdder.com

62. Sharing forward mails

65. Use customized eld
in schema.xml

66. <fieldType name=quot;html_thquot; class=quot;solr.TextFieldquot;
positionIncrementGap=quot;100quot;>
<analyzer type=quot;indexquot;>
<tokenizer
class=quot;solr.HTMLStripStandardTokenizerFactoryquot;/>
<filter class=quot;solr.ThaiWordFilterFactoryquot; />
<filter class=quot;solr.StopFilterFactoryquot;
ignoreCase=quot;truequot; words=quot;stopwords.txtquot;/>
<filter class=quot;solr.LowerCaseFilterFactoryquot;/>
<filter class=quot;solr.EnglishPorterFilterFactoryquot;
protected=quot;protwords.txtquot;/>
<filter
class=quot;solr.RemoveDuplicatesTokenFilterFactoryquot;/>
</analyzer>
</fieldType>

67. <field name=quot;idquot; type=quot;stringquot;
indexed=quot;truequot; stored=quot;truequot; />
<field name=quot;titlequot; type=quot;html_thquot;
indexed=quot;truequot; stored=quot;truequot; />
<field name=quot;detailquot; type=quot;html_thquot;
indexed=quot;truequot; stored=quot;truequot; />
<field name=quot;tagquot; type=quot;stringiquot;
indexed=quot;truequot; stored=quot;truequot;
multiValued=quot;truequot; />
<field name=quot;useridquot; type=quot;integerquot;
indexed=quot;falsequot; stored=quot;truequot; />

68. Index analyzer

69. Debugging

70. &debugQuery=on

71. Further readings
• http://lucene.apache.org/solr/
• http://wiki.apache.org/solr
• http://www.xml.com/pub/a/2006/08/09/
solr-indexing-xml-with-lucene-
andrest.html
• http://lucene.apache.org/java/docs/
scoring.html

72. Q&A