Open up your user base to the data! Contrary to programming and SQL, almost everybody knows how to search. This talk describes through an interactive demo based on open source Hue how users can graphically search their data in Hadoop. The underlying technical details of the application and its interaction with Apache Solr will be clarified. The session will detail how to get started with data indexing in just a few clicks as well as explore several data analysis scenarios with the latest Solr Analytics Facets and Spark Streaming. Through a Web browser, attendees will be shown how to explore and visualize data for quick answers. The search dashboard in Hue, with its draggable charts and dynamic interface, lets any non-technical user look for documents or patterns. Attendees of this talk will learn how to get started with interactive search visualization in their Solr cluster.

1. INTERACTIVELY
SEARCH AND
VISUALIZE YOUR DATA
WITH SOLR AND SPARK
Romain Rigaux

2. GOALS
Build
a
Web
app
Quickly
explore
data
…
with
Solr
make
Solr
/
Hadoop
easier
to
use
+

3. ARCHITECTURE
“Just
a
view”
on
top
of
the
standard
Solr
API
REST

4. HISTORY
V1 USER

5. HISTORY
V1 ADMIN

6. ARCHITECTURE
NEXT!
Lot
of
learning,
UX
Boost
needed
Simple,
don’t
know
it
is
Solr

7. HISTORY
V2 USER

8. HISTORY
V2 ADMIN

9. HISTORY
V2 BETTER UX

10. ARCHITECTURE
/select
/admin/collections
/get
/luke...
/add_widget
/zoom_in
/select_facet
/select_range...
REST AJAX
Templates
+
JS
Model
www….

11. ARCHITECTURE
UI FOR FACETS
Query
Collection
Layout All
the
2D
positioning
(cell
ids),
visual,
drag&drop
Dashboard,
fields,
template,
widgets
(ids)
Search
terms,
selected
facets
(q,
fqs)

12. ADDING A WIDGET
LIFECYCLE
Load
the
initial
page
Edit
mode
and
Drag&Drop
/solr/zookeeper/clusterstate.json
/solr/admin/luke…
/get_collection

13. ADDING A WIDGET
LIFECYCLE
/solr/select?stats=true /new_facet
Select
the
field
Guess
ranges
(number
or
dates)
Rounding
(number
or
dates)

14. ADDING A WIDGET
LIFECYCLE
Query
part
1
Query
Part
2
Augment
Solr
response
facet.range={!ex=bytes}bytes&f.bytes.facet.range.start=0&f.bytes.facet.range.end=9000000&
f.bytes.facet.range.gap=900000&f.bytes.facet.mincount=0&f.bytes.facet.limit=10
q=Chrome&fq={!tag=bytes}bytes:[900000+TO+1800000]
{
'facet_counts':{
'facet_ranges':{
'bytes':{
'start':10000,
'counts':[
'900000',
3423,
'1800000',
339,
...
]
}
}
}
{
...,
'normalized_facets':[
{
'extraSeries':[
],
'label':'bytes',
'field':'bytes',
'counts':[
{
'from’:'900000',
'to':'1800000',
'selected':True,
'value':3423,
'field’:'bytes',
'exclude':False
}
], ...
}
}
}

15. JSON TO WIDGET
{
"field":"rate_code",
"counts":[
{
"count":97797,
"exclude":true,
"selected":false,
"value":"1",
"cat":"rate_code"
} ...
{
"field":"medallion",
"counts":[
{
"count":159,
"exclude":true,
"selected":false,
"value":"6CA28FC49A4C49A9A96",
"cat":"medallion"
} ….
{
"extraSeries":[
],
"label":"trip_time_in_secs",
"field":"trip_time_in_secs",
"counts":[
{
"from":"0",
"to":"10",
"selected":false,
"value":527,
"field":"trip_time_in_secs",
"exclude":true
} ...
{
"field":"passenger_count",
"counts":[
{
"count":74766,
"exclude":true,
"selected":false,
"value":"1",
"cat":"passenger_count"
} ...

16. REPEAT
UNTIL…

17. GAME CHANGER!
Possibilihes
5.1
/
5.2
Analyhc
Facets

18. FACET
FUNCTIONS
Count
Sum
Avg
Percentile
Max
...
Count(id)
Sum(bytes)
Avg(mul(price,
quantity))
Percentile(salary,
50,
90)
Max(temperature)
...

19. FACET
FUNCTIONS

20. SUB “NESTED”
FACETS
top_os
{
type:
term,
field:
os,
limit:
5
}
top_os
{
type:
term,
field:
os,
limit:
5,
facet
:
{
by_country:
{
type:
term,
field:
country
}
}
}

21. FUNCTION + NESTED =
ANALYTICS states
{
type:
term,
field:
state,
facet
:
{
by_month
:
{
type:
range,
field:
time,
start:
“TODAY-­‐6MONTHS”,
end:
“TODAY”,
gap:
“MONTH”,
facet
:
{
avg_sal:
“avg(salary)”
}
}
}
}
states
{
type:
term,
field:
state,
facet
:
{
avg_sal:
“avg(salary)”
}
}

22. OPERATIONS ON
BUCKETS OF DATA
Counts
→
Functions

23. OPERATIONS ON
BUCKETS OF DATA
Nested
→
nD
functions

24. SEARCH AS ONLY
APP IN HUE
gethue.com/solr-­‐search-­‐ui-­‐only/

25. • Spark
in
your
browser
• Notebooks
• New
REST
Server
SPARK
INDEXING
WHAT

26. • Open
source
REST
for
Spark
Shell
• Runs
locally
or
inside
YARN
• Spark
Scala,
PySpark
and
jar/py
submission
SPARK
INDEXING
WHAT
hpps://github.com/cloudera/hue/tree/master/apps/spark/java

27. LIVY ARCH
YARN LOCAL
Livy
Server
Livy
REPL
Spark
Contexts
Spark
Worker
Livy
Server
YARN
Master
YARN
Node
Livy
REPL
Spark
Context
/
PySpark
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
1
2
3
4

28. SPARK STREAMING
Real
hme!
Spark
Solr

29. • Python
• Scala
• Charts
NOTEBOOKS / SHELL
WHAT

30. DEMO
TIME
• Analyze
Bay
area
bike
share
• Visualize
one
year
of
data
• Know
your
users,
predict
behavior

31. MISSED
SOMETHING?
demo.gethue.com

32. • Full
Analyhcs
• Easier
indexing
• Geo
• Export/Share
results
• Solr
Joins,
Solr
SQL
• Spark,
SQL...
integrahon,
Hue
4
WHAT’S NEXT
NEW FEATURES

33. TWITTER
@gethue
USER GROUP
hue-­‐user@
WEBSITE
hpp://gethue.com
LEARN
hpp://learn.gethue.com
THANKS!