This document discusses when Solr is a good tool to use versus other options. It provides an overview of Solr in the big data ecosystem and the concept of polyglot persistence, where different data stores are used for different needs. Common use cases for Solr like search-based recommendations and log analysis are described. A checklist is presented for determining if Solr is a good fit based on factors like data volume, query characteristics, throughput needs, and data type. The document concludes by listing some red flags where Solr may not be suitable, such as if strong consistency, transactions, or graphs are needed requirements.

2. USE CASE DIAGNOSIS: WHEN IS SOLR
REALLY THE BEST TOOL?
Michael Hausenblas
Twitter: @mhausenblas
Chief Data Engineer EMEA, MapR Technologies

3. Agenda
•
•
•
•
•
Solr in the Big Data ecosystem
Polyglot Persistence
Common (Big Data) use cases
A checklist
When not to use Solr …

4. processing
storage
Apache Pig
Apache Zookeeper

5. Polyglot Persistence

6. $ ls -al
$ tail –f some.log
$ nc localhost 80
awk 'BEGIN { FS = "," }
/2013-[[:digit:]]+-[[:digit:]]+/ { print $3 }’
sample.csv
tool box
one-size-fits-all

7. Polyglot Persistence—Backdrop
•
Michael Stonebraker and Ugur Çetintemel—2005
"One Size Fits All": An Idea Whose Time Has Come and Gone
•
Martin Fowler—2011
Polyglot Persistence1
•
Eric Brewer—2012
Ricon Keynote—Advancing Distributed Systems2
1) http://martinfowler.com/bliki/PolyglotPersistence.html
2) https://speakerdeck.com/eric_brewer/ricon-2012-keynote

8. Polyglot Persistence—Key Points
•
Use different datastores for different needs
•
Can apply within an application or cross-enterprise
•
Encapsulating data access yields loosely coupled components
•
Find sweet spot between dev/op complexity and flexibility

9. Common (Big Data)
use cases

10. Where are we coming from?
•
•
•
•
•
Keyword search
Spellcheck & autosuggest
Ranking
Faceted search
Spatial search

11. Use case:
search-based
recommendation

12. Search-based recommendation (credit card issuer)
•
Given
– customer purchase history
– merchant designations
– merchant special offers
•
Goal
– Improve existing recommender system
– Throughput important

13. Analyze with MapReduce
complete
history
Co-­‐occurrence
(Mahout)
Item
meta-­‐data
SolR
SolR
Solr
Indexer
Indexer
indexing
Index
shards

14. Deploy with search system
user
history
Web
>er
Item
meta-­‐data
SolR
SolR
Solr
Indexer
Indexer
search
Index
shards

15. Use case:
log analysis

16. Log analysis
•
Given
– Receive 200,000+ log lines per second
•
Goal
– Want to do multi-field search
– Want to search on log lines with <30 second delay before search

17. Data Ingestion and Indexing
incoming
data
Ka@a
SolR
SolR
Text
Indexer
Indexer
analysis
Solr
indexer
Real-­‐>me
Raw
documents
Older
index
shards
Live
index
shard
>me-­‐sharded
Solr
indexes

18. Search
Query
Solr
search
Web
>er
SolR
SolR
Solr
Indexer
Indexer
search
Raw
documents
Older
index
shards
Live
index
shard

19. A checklist

20. Question you may want to ask …
•
What is the volume of your data* (few GB? up to PB?)
•
How are your query characteristics?
– full scans
– look-ups
– multiple passes over large parts
– continuous queries
•
What’s (more) important: throughput or latency?
*)
Note:
as
long
as
Moore's
law
s>ll
holds,
these
figures
obviously
change
on
a
yearly
if
not
monthly
basis.

21. Key qualifiers
•
Want exploratory interface rather than aggregates in a dashboard
•
Data are sparse symbol sets like words or recommendation indicators
•
Small-ish return sets are OK, especially if facets are good enough
•
Near-real-time is good enough

22. When not to use Solr …

23. Red Flags
•
You need strong consistency?
•
JOINS, anyone?
•
•
•
reme
mber
:
one
fit
all
size
d
—too
Want (complex) transactions?
l
belt
oes
n
appr
ot
oach!
OLTP, streaming (but: near-real-time)
Graphs?

24. Let’s stay in touch …
•
Twitter:
@mhausenblas
@MapR
MapR
Nordics
MapR
UK
MapR
HQ
San
Jose,
US
MapR
DACH
MapR
Japan
MapR
SE
&
Benelux
MapR
Hyderbad
•
We’re hiring!
MapR
Korea