Harness the power of Spark and Solr in Hue: Big Data Amsterdam v.2.0

1. SPARK AND SOLR
IN HUE
Enrico Berti
INDEX AND VISUALIZE DATA IN A FEW CLICKS!

2. GOAL
OF HUE
WEB INTERFACE FOR ANALYZING DATA
WITH APACHE HADOOP
SIMPLIFY AND INTEGRATE
FREE AND OPEN SOURCE
—> OPEN UP BIG DATA

3. VIEW FROM
30K FEET
Hadoop Web Server
You, your colleagues and even that
friend that uses IE9 ;)

4. OPEN SOURCE
~5230 COMMITS
65 CONTRIBUTORS
1137 STARS
443 FORKS
github.com/cloudera/hue

5. TREND: GROWTH
gethue.com

6. WHERE TO PUT HUE? IN ONE MACHINE

7. WHERE TO PUT HUE? OUTSIDE THE CLUSTER

8. WHERE TO PUT HUE? INSIDE THE CLUSTER

9. FULL SUITE OF APPS

10. LIST OF GROUPS AND PERMISSIONS
A
permission
can:
- allow
access
to
one
app
(e.g.
Hive
Editor)
- modify
data
from
the
app
(e.g
drop
Hive
Tables
or
edit
cells
in
HBase
Browser)
CONFIGURE APPS
AND PERMISSIONS
A
list
of
permissions

11. PERMISSIONS IN ACTION
User
‘test’
belonging
to
the
group
‘hiveonly’
that
has
just
the
‘hive’
permissions
CONFIGURE APPS
AND PERMISSIONS

12. • Impala,
Hive
integraGon
• InteracGve
SQL
editor
• IntegraGon
with
MapReduce,
Metastore,
HDFS
SQL
WHAT

13. • AuthorizaGon
for
SQL,
HDFS
URI
• Who
can
see
and
do
what
on
the
database/tables
• Edit
security
role/privileges
• Next:
Solr
SENTRY APP
WHAT

14. • Solr
&
Cloud
integraGon
• Custom
interacGve
dashboards
• Drag
&
drop
widgets
(charts,
Gmeline…)
SEARCH
WHAT

15. • Open
source
REST
for
Spark
Shell
• Runs
locally
or
inside
YARN
• Spark
Scala,
PySpark
and
jar/py
submission
LIVY SERVER
WHAT
hZps://github.com/cloudera/hue/tree/master/apps/spark/java

16. LIVY ARCH
YARN LOCAL
Livy
Server
Livy
REPL
Spark
Contexts
Spark
Worker
Livy
Server
YARN
Master
YARN
Node
Livy
REPL
Spark
Context
/
PySpark
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
1
2
3
4

17. LIVY ARCH
Livy
Server+ =

18. • Scala
• Python
• Java
• SQL
• Text
SPARK NOTEBOOKS
WHAT

19. DEMO
TIME

20. TWITTER
@gethue
USER GROUP
hue-­‐user@
WEBSITE
hZp://gethue.com
LEARN
hZp://learn.gethue.com
THANKS!