Ravi provides an overview of building a world-class directory and search solution. He demonstrates setting up a Solr collection using sample Congressional data and exploring various search features like facets, substrings, and fuzzy matching. Ravi emphasizes iterating on data modeling, features, and the user interface to build the best solution.

1. Copyright © President & Fellows of Harvard College
Build Your Own World-Class Directory
Search From Α to Ω
Ravi Mynampaty

2. About Ravi
A hustler making a living by pretending to know more about
Enterprise Search than he actually does...

4. “I can live on a good
compliment two
weeks with nothing
else to eat...”
@RaviMynampaty

5. Why the heck should I listen to Ravi?

7. Agenda
Why?
What’s the fuss about?
What features?
What data?
How?

8. Where are we going?
Search Index
Model &
Structure
Raw data
Prototype
UI

9. My goal… (how many iterations?)

10. Icebreaker

11. Why?

16. What’s the big deal?

17. Records
vs.
Documents

18. vs.
Person
family_name
first_name
phone
email
...
Document
Title: ...
Description: ..
Content: ……..
……
…….
……
…….
……
…….
……
…….
……
…….
……
…….
……
…….
……

19. Content: ……..
……
…….
……
…….
……
…….
……
…….
……
…….
……
…….
……
…….
……
…….
……
…….
……
…….
……
Content: ……..
……
…….
……
…….
……
…….
……
…….
……
…….
……
…….
……
…….
……
…….
……
…….
……
…….
vs.
Person
family_name
first_name
phone
email
...
Document
Title: ...
Description: ..
Content: ……..
……
…….
……
…….
……
…….
……
…….
……
…….
……
…….
……
…….
……

20. Nicknames

21. Predictable
• Elizabeth
• Beth, Bess, Betty, Liz
• Richard
• Rich, Dick
• David
• Dave

22. Simple Substrings
Srinivas → “Srini”
Mohammad → “Mo”

23. Somewhat Predictable
Yakub → “Jacob”
Yusuf → “Joseph”
Xian → “Sean”

24. Unpredictable
Hanuman → “Hank”
Madhav → “Mike”
Babu → “Bob”
Wongsu → “Richard”
Herman → “Dutch”

25. Abbreviations & Acronyms

26. Department Names
Information Technology
ITG
Info Tech.
HBS IT

27. Job Titles
CEO
PM
VP
...

28. Educational Degrees
PhD
JD
...
ALM → “Master of Liberal Arts”
(magistri in artibus liberalibus studiorum prolatorum)

29. Substrings

30. Experiment

31. I’d like you to meet...

32. Garoppolo

33. What was that guy’s name?

35. What did you search for?

36. One more...

37. Roethlisberger

38. What was that guy’s name?

40. What did you search for?

41. My prediction...

42. G…. & R...

43. Jimmy Garoppolo

44. Ben Roethlisberger

45. Exercise!

46. Wishlist: How should search work?
Search mechanisms
What should be searchable?
How should users be able to search?
Query interface: What should be supported?
Results interface: What should be displayed?
etc.

47. Wishlist discussion

48. Features
Search by:
Name, Department, Email, Job title, Phone number
Nicknames, Aliases
Substrings
Scoped search, Sort options
Faceting/filtering options
Spelling suggestions, Autocomplete, Devices, Voice search

49. Hands-on!

50. Install Sublime Text
https://www.sublimetext.com/3

51. Let’s create some data

53. Solr XML
<add>
<doc>
<field name="id">1813-05-05</field>
<field name="LastName">Kierkegaard</field>
<field name="FirstName">Søren</field>
</doc>
<doc>
<field name="id">1966-12-14</field>
<field name="LastName">Thorning-Schmidt</field>
<field name="FirstName">Helle</field>
</doc>
</add>

54. That was just for practice

55. Our Dataset: Members of US Congress
Need to create XML for 400+ people records

56. Install JDK 1.8
http://tinyurl.com/ie17java
(set JAVA_HOME env variable)
java -version
javac -version
echo $JAVA_HOME // *nix
echo %JAVA_HOME% // windows

57. Install Fusion
https://lucidworks.com/
Download + Unzip
Run it!
Open cmd prompt
cd ...fusion-3.0.0fusion3.0.0bin

58. Run it: “fusion.cmd start”
C:..Desktopfusion-3.0.0fusion3.0.0bin>fusion.cmd start
Starting zookeeper..
Successfully started zookeeper on port 9983 (process ID 144
Starting solr..............
Successfully started solr on port 8983 (process ID 19564)
Starting api............................
Successfully started api on port 8765 (process ID 12568)
Starting connectors..........................
Successfully started connectors on port 8984 (process ID 18
Starting ui.............
Successfully started ui on port 8764 (process ID 14096)

59. Admin UI: http://localhost:8764/
1. Create password
Follow along with me:
1. Quickstart
2. Create a new collection (call it “Test1”)
3. Select a dataset: “Revolution Session Data”
4. Try some searches
5. Add faceted search

60. Break

61. 1st Matrix

64. 1st Matrix: matrix1.xml
<doc>
<field name="PersonId">Gabbard, Tulsi</field>
<field name="LastName">Gabbard</field>
<field name="FirstName">Tulsi</field>
<field name="State">Hawaii</field>
<field name="District">2nd District</field>
<field name="Room">1433 LHOB</field>
<field name="Phone">202-225-4906</field>
<field name="Party">Democratic</field>
<field name="Committee">Armed Services</field>
<field name="Email">Tulsi.Gabbard@mail.house.gov</field>
</doc>

65. Matrix XML
http://tinyurl.com/ie17matrix

66. Create Solr collection for US Congress
http://localhost:8764/
Devops → New → Collection Name “house” → Save Collection
Configure Fields
Create Datasource → Add → Filesystem → SolrXML → Datasource ID
Path: set path to XML file on disk: C:cygwin64homermynampatyhousematrix1.xml
Start Crawl → (Wait for finish) → Job History → (Observe success/fail)

67. Let’s search!
Query Workbench
Format Results → Documents
- One Primary Field (which one do you think?)
- One Secondary
- One Other

68. _s vs. _t
String:
Preserves entirely: no tokenizing, preserve case
text:
Tokenizes, stopwords, lowercase

69. Query Workbench
Try some searches: are they generally working?
Sort: what sort makes sense for people search?

70. Scoped Search
<field_name>:<search_string>
e.g.,
State_s:Hawaii

71. Booleans
AND, "+", OR, NOT and "-"
e.g.,
LastName_s:Smith AND Party_s:Republican

72. Fuzzy Matching
e.g.,
Castor~
Castor~0.8

73. Synonyms

75. No one uses scoped / Boolean
:-(

76. Trick them!

77. Facets
What facets make sense for people search? Add some.

78. 2nd Matrix

79. 2nd Matrix: matrix2.xml
<doc>
<field name="PersonId">Gabbard, Tulsi</field>
<field name="LastInitial">G</field>
<field name="LastName">Gabbard</field>
<field name="FirstName">Tulsi</field>
<field name="Nickname">POTUS2024</field>
<field name="State">Hawaii</field>
<field name="District">2nd District</field>
<field name="Room">1433 LHOB</field>
<field name="Phone">202-225-4906</field>
<field name="Party">Democratic</field>
<field name="Committee">Armed Services</field>
<field name="Email">Tulsi.Gabbard@mail.house.gov</field>
</doc>

80. Recrawl Solr collection for US Congress
http://localhost:8764/
Devops → Collection Name → Datasource → Clear Datasource
Path: set path to XML file on disk: C:cygwin64homermynampatyhousematrix2.xml
Start Crawl → (Wait for finish) → Job History → (Observe success/fail)
Search using Query Workbench

81. Substrings
Did they work?

82. 3rd Matrix

83. 3rd Matrix: matrix3.xml
Gabbard
- Gabbard
- gabbar
- gabba
- gabb
- gab
- ga
- g
Substrings
via
N-grams

84. Recrawl Solr collection for US Congress
http://localhost:8764/
Devops → Collection Name → Datasource → Clear Datasource
Path: set path to XML file on disk: C:cygwin64homermynampatyhousematrix3.xml
Start Crawl → (Wait for finish) → Job History → (Observe success/fail)
Search using Query Workbench

85. Where are we?
Search Index
Model &
Structure
Raw data
Prototype
UI

86. Next steps for you
Search Index
Model &
Structure
End-user
UI
Raw data
Prototype
UI

87. Thank you!
Questions?
searchguy@hbs.edu
@RaviMynampaty
linkedin.com/in/mynampaty
facebook.com/ravi.mynampaty