This document describes building a REST job server for interactive Spark as a service using Livy. It discusses the history and challenges of running Spark jobs in Hue, introduces Livy as a Spark server, and details its local and YARN-cluster modes as well as session creation, execution flows, and interpreter support for Scala, Python, R and more. Magic commands are also covered for JSON, table, plotting and other output formats.

1. BUILDING A REST JOB SERVER
FOR INTERACTIVE SPARK
AS A SERVICE
Romain Rigaux - Cloudera
Erick Tryzelaar - Cloudera

2. WHY?

5. NOTEBOOKS
EASY ACCESS FROM ANYWHERE
SHARE SPARK CONTEXTS AND RDDs
BUILD APPS
SPARK MAGIC
…
WHY SPARK
AS A SERVICE?

6. MARRIED WITH FULL HADOOP ECOSYSTEM
WHY SPARK
IN HUE?

7. HISTORY
V1: OOZIE
• It works
• Code snippet
THE GOOD
• Submit through Oozie
• Shell ac:on
• Very Slow
• Batch
THE BAD
workflow.xml
snippet.py
stdout

8. HISTORY
V2: SPARK IGNITER
• It works beAer
THE GOOD
• Compiler Jar
• Batch only, no shell
• No Python, R
• Security
• Single point of failure
THE BAD Compile
Implement
Upload
json output
Batch
Scala
jar
Ooyala

9. HISTORY
V3: NOTEBOOK
• Like spark-submit / spark shells
• Scala / Python / R shells
• Jar / Python batch Jobs
• Notebook UI
• YARN
THE GOOD
• Beta?
THE BAD
Livy
code snippet batch

10. GENERAL ARCHITECTURE
Spark
Spark
Spark
Livy YARN
!"
# $

11. Livy
Spark
Spark
Spark
YARN
API
!"
# $
GENERAL ARCHITECTURE

12. LIVY SPARK SERVER

13. LIVY
SPARK SERVER
•REST Web server in Scala for Spark submissions
•Interac:ve Shell Sessions or Batch Jobs
•Backends: Scala, Java, Python, R
•No dependency on Hue
•Open Source: hAps://github.com/cloudera/
hue/tree/master/apps/spark/java
•Read about it: hAp://gethue.com/spark/

14. ARCHITECTURE
• Standard web service: wrapper around spark-submit / Spark shells
• YARN mode, Spark drivers run inside the cluster (supports crashes)
• No need to inherit any interface or compile code
• Extended to work with additional backends

15. LIVY WEB SERVER
ARCHITECTURE
LOCAL “DEV” MODE YARN MODE

16. LOCAL
MODE
Livy Server
Scalatra
Session Manager
Session
Spark
ContextSpark
Client
Spark
Client
Spark
Interpreter

17. LOCAL
MODE
Livy Server
Scalatra
Session Manager
Session
Spark
Client
Spark
Client
Spark
Context
Spark
Interpreter

18. LOCAL
MODE
Spark
Client
1
Livy Server
Scalatra
Session Manager
Session
Spark
Client
Spark
Context
Spark
Interpreter

19. LOCAL
MODE
Spark
Client
1
2
Livy Server
Scalatra
Session Manager
Session
Spark
Client
Spark
Context
Spark
Interpreter

20. LOCAL
MODE
Spark
Client
Spark
Interpreter
1
2
Livy Server
Scalatra
Session Manager
Session
Spark
Client
Spark
Context
3

21. LOCAL
MODE
Spark
Client
1
2
Livy Server
Scalatra
Session Manager
Session
Spark
Client
Spark
Context
3
4 Spark
Interpreter

22. LOCAL
MODE
Spark
Client
1
2
Livy Server
Scalatra
Session Manager
Session
Spark
Client
Spark
Context
3
4
5
Spark
Interpreter

23. YARN-CLUSTER
MODE
PRODUCTION SCALABLE

24. YARN
Master
Spark
Client
YARN
Node
Spark
Context
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
Livy Server
Scalatra
Session Manager
Session
YARN-CLUSTER
MODE
Spark
Interpreter

25. Livy Server
YARN
Master
Scalatra
Spark
Client
Session Manager
Session
YARN
Node
Spark
Context
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
1
YARN-CLUSTER
MODE
Spark
Interpreter

26. YARN
Master
Spark
Client
YARN
Node
Spark
Context
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
1
2
Livy Server
Scalatra
Session Manager
Session
YARN-CLUSTER
MODE
Spark
Interpreter

27. YARN
Master
Spark
Client
YARN
Node
Spark
Context
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
1
2
3
Livy Server
Scalatra
Session Manager
Session
YARN-CLUSTER
MODE
Spark
Interpreter

28. YARN
Master
Spark
Client
YARN
Node
Spark
Context
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
1
2
3
4
Livy Server
Scalatra
Session Manager
Session
YARN-CLUSTER
MODE
Spark
Interpreter

29. YARN
Master
Spark
Client
YARN
Node
Spark
Context
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
1
2
3
4
5
Livy Server
Scalatra
Session Manager
Session
YARN-CLUSTER
MODE
Spark
Interpreter

30. YARN
Master
Spark
Client
YARN
Node
Spark
Context
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
1
2
3
4
5
6
Livy Server
Scalatra
Session Manager
Session
YARN-CLUSTER
MODE
Spark
Interpreter

31. YARN
Master
Spark
Client
YARN
Node
Spark
Context
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
1 7
2
3
4
5
6
Livy Server
Scalatra
Session Manager
Session
YARN-CLUSTER
MODE
Spark
Interpreter

32. SESSION CREATION AND EXECUTION
% curl -XPOST localhost:8998/sessions
-d '{"kind": "spark"}'
{
"id": 0,
"kind": "spark",
"log": [...],
"state": "idle"
}
% curl -XPOST localhost:8998/sessions/0/statements -d '{"code": "1+1"}'
{
"id": 0,
"output": {
"data": { "text/plain": "res0: Int = 2" },
"execution_count": 0,
"status": "ok"
},
"state": "available"
}

33. Jar
Py
Scala
Python
R
Livy
Spark
Spark
Spark
YARN
/batches
/sessions
BATCH OR INTERACTIVE

34. SHELL OR BATCH?
YARN
Master
Spark
Client
YARN
Node
Spark
Interpreter
Spark
Context
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
Livy Server
Scalatra
Session Manager
Session

35. SHELL
YARN
Master
Spark
Client
YARN
Node
pyspark
Spark
Context
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
Livy Server
Scalatra
Session Manager
Session

36. BATCH
YARN
Master
Spark
Client
YARN
Node
spark-
submit
Spark
Context
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
Livy Server
Scalatra
Session Manager
Session

37. LIVY INTERPRETERSScala, Python, R…

38. REMEMBER?
YARN
Master
Spark Client
YARN
Node
Spark
Context
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
Livy Server
Scalatra
Session Manager
Session
Spark
Interpreter

39. INTERPRETERS
• Pipe stdin/stdout to a running shell
• Execute the code / send to Spark
workers
• Perform magic opera:ons
• One interpreter per language
• “Swappable” with other kernels
(python, spark..)
Interpreter
> println(1 + 1)
2
println(1 + 1)
2

40. Livy Server
INTERPRETER FLOW
Interpreter

41. Livy Server
> 1 + 1
Interpreter
INTERPRETER FLOW

42. Livy Server
{“code”: “1+1”}
> 1 + 1
Interpreter
INTERPRETER FLOW

43. Livy Server Interpreter
1+1
{“code”: “1+1”}
> 1 + 1
INTERPRETER FLOW

44. Livy Server Interpreter
1+1
{“code”: “1+1”}
> 1 + 1
Magic
INTERPRETER FLOW

45. Livy Server
2
Interpreter
1+1
{“code”: “1+1”}
> 1 + 1
Magic
INTERPRETER FLOW

46. {
“data”: {
“application/json”: “2”
}
}
Livy Server
2
Interpreter
1+1
{“code”: “1+1”}
> 1 + 1
Magic
INTERPRETER FLOW

47. {
“data”: {
“application/json”: “2”
}
}
Livy Server
2
Interpreter
1+1
{“code”: “1+1”}
> 1 + 1
2 Magic
INTERPRETER FLOW

48. INTERPRETER FLOW CHART
Receive lines
Split into
Chunks
Send output
to server
Send error to
server
Success
Execute ChunkMagic!
Chunks
le[?
Magic
chunk?
No
Yes
NoYes
Example of parsing

49. INTERPRETER MAGIC
• table
• json
• plotting
• ...

50. NO MAGIC
> 1 + 1
Interpreter
1+1
sparkIMain.interpret(“1+1”)
{
"id": 0,
"output": {
"application/json": 2
}
}

51. [('', 506610), ('the', 23407), ('I', 19540)... ]
JSON MAGIC
> counts
sparkIMain.valueOfTerm(“counts”)
.toJson()
Interpreter
val lines = sc.textFile("shakespeare.txt");
val counts = lines.
flatMap(line => line.split(" ")).
map(word => (word, 1)).
reduceByKey(_ + _).
sortBy(-_._2).
map { case (w, c) =>
Map("word" -> w, "count" -> c)
}
%json counts

52. JSON MAGIC
> counts
sparkIMain.valueOfTerm(“counts”)
.toJson()
Interpreter
{
"id": 0,
"output": {
"application/json": [
{ "count": 506610, "word": "" },
{ "count": 23407, "word": "the" },
{ "count": 19540, "word": "I" },
...
]
...
}
val lines = sc.textFile("shakespeare.txt");
val counts = lines.
flatMap(line => line.split(" ")).
map(word => (word, 1)).
reduceByKey(_ + _).
sortBy(-_._2).
map { case (w, c) =>
Map("word" -> w, "count" -> c)
}
%json counts

53. [('', 506610), ('the', 23407), ('I', 19540)... ]
TABLE MAGIC
> counts
Interpreter
val lines = sc.textFile("shakespeare.txt");
val counts = lines.
flatMap(line => line.split(" ")).
map(word => (word, 1)).
reduceByKey(_ + _).
sortBy(-_._2).
map { case (w, c) =>
Map("word" -> w, "count" -> c)
}
%table counts
sparkIMain.valueOfTerm(“counts”)
.guessHeaders().toList()

54. TABLE MAGIC
> counts
sparkIMain.valueOfTerm(“counts”)
.guessHeaders().toList()
Interpreter
val lines = sc.textFile("shakespeare.txt");
val counts = lines.
flatMap(line => line.split(" ")).
map(word => (word, 1)).
reduceByKey(_ + _).
sortBy(-_._2).
map { case (w, c) =>
Map("word" -> w, "count" -> c)
}
%table counts
"application/vnd.livy.table.v1+json": {
"headers": [
{ "name": "count", "type": "BIGINT_TYPE" },
{ "name": "name", "type": "STRING_TYPE" }
],
"data": [
[ 23407, "the" ],
[ 19540, "I" ],
[ 18358, "and" ],
...
]
}

55. PLOT MAGIC
>
sparkIMain.interpret(“png(‘/tmp/
plot.png’) barplot dev.off()”)
Interpreter
...
barplot(sorted_data
$count,names.arg=sorted_data$value,
main="Resource hits", las=2,
col=colfunc(nrow(sorted_data)),
ylim=c(0,300))

56. PLOT MAGIC
>
sparkIMain.interpret(“png(‘/tmp/
plot.png’) barplot dev.off()”)
Interpreter
...
barplot(sorted_data
$count,names.arg=sorted_data$value,
main="Resource hits", las=2,
col=colfunc(nrow(sorted_data)),
ylim=c(0,300))

57. PLOT MAGIC
> png(‘/tmp/..’)
> barplot
> dev.off()
sparkIMain.interpret(“png(‘/tmp/
plot.png’) barplot dev.off()”)
Interpreter
...
barplot(sorted_data
$count,names.arg=sorted_data$value,
main="Resource hits", las=2,
col=colfunc(nrow(sorted_data)),
ylim=c(0,300))

58. PLOT MAGIC
> png(‘/tmp/..’)
> barplot
> dev.off()
sparkIMain.interpret(“png(‘/tmp/
plot.png’) barplot dev.off()”)
File(’/tmp/plot.png’).read().toBase64()
Interpreter
...
barplot(sorted_data
$count,names.arg=sorted_data$value,
main="Resource hits", las=2,
col=colfunc(nrow(sorted_data)),
ylim=c(0,300))

59. PLOT MAGIC
> png(‘/tmp/..’)
> barplot
> dev.off()
sparkIMain.interpret(“png(‘/tmp/
plot.png’) barplot dev.off()”)
File(’/tmp/plot.png’).read().toBase64()
Interpreter
...
barplot(sorted_data
$count,names.arg=sorted_data$value,
main="Resource hits", las=2,
col=colfunc(nrow(sorted_data)),
ylim=c(0,300))
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEU
...
}
...
}

60. • Pluggable Backends
• Livy's Spark Backends
– Scala
– pyspark
– R
• IPython / Jupyter support coming soon
PLUGGABLE INTERPRETERS

61. • Re-using it
• Generic Framework
for Interpreters
• 51 Kernels
JUPYTER BACKEND

62. SPARK AS A SERVICE

63. REMEMBER AGAIN?
YARN
Master
Spark Client
YARN
Node
Spark
Context
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
Livy Server
Scalatra
Session Manager
Session
Spark
Interpreter

64. MULTI USERS
YARN
Node
Spark
Context
Livy Server
Scalatra
Session Manager
Session
Spark
Interpreter YARN
Node
Spark
Context
Spark
Interpreter
YARN
Node
Spark
Context
Spark
Interpreter
Spark
Client
Spark
Client
Spark
Client

65. SHARED CONTEXTS?
YARN
Node
Spark
Context
Livy Server
Scalatra
Session Manager
Session
Spark
Interpreter
Spark
Client
Spark
Client
Spark
Client

66. SHARED RDD?
YARN
Node
Spark
Context
Livy Server
Scalatra
Session Manager
Session
Spark
Interpreter
Spark
Client
Spark
Client
Spark
Client
RDD

67. SHARED RDDS?
YARN
Node
Spark
Context
Livy Server
Scalatra
Session Manager
Session
Spark
Interpreter
Spark
Client
Spark
Client
Spark
Client
RDD
RDD
RDD

68. YARN
Node
Spark
Context
Livy Server
Scalatra
Session Manager
Session
Spark
Interpreter
Spark
Client
Spark
Client
Spark
Client
RDD
RDD
RDD
SECURE IT?

69. YARN
Node
Spark
Context
Livy Server
Scalatra
Session Manager
Session
Spark
Interpreter
Spark
Client
Spark
Client
Spark
Client
RDD
RDD
RDD
SECURE IT?

70. Livy Server
Spark
Spark
Client
Spark
Client
Spark
Client
SPARK AS SERVICE
Spark

71. SHARING RDDS

72. PySpark shell
RDD
Shell
Python
Shell

73. PySpark shell
RDD
Shell
Python
Shell

74. PySpark shell
RDD
Shell
Python
Shell
r = sc.parallelize([])
srdd = ShareableRdd(r)

75. PySpark shell
RDD
{'ak': 'Alaska'}
{'ca': 'California'}
Shell
Python
Shell
r = sc.parallelize([])
srdd = ShareableRdd(r)

76. PySpark shell
RDD
{'ak': 'Alaska'}
{'ca': 'California'}
Shell
Python
Shell
curl -XPOST /sessions/0/statement {
'code': srdd.get('ak')
}
r = sc.parallelize([])
srdd = ShareableRdd(r)

77. PySpark shell
RDD
{'ak': 'Alaska'}
{'ca': 'California'}
Shell
Python
Shell
states = SharedRdd('host/sessions/0', 'srdd')
states.get('ak')
r = sc.parallelize([])
srdd = ShareableRdd(r)
curl -XPOST /sessions/0/statement {
'code': srdd.get('ak')
}

78. DEMO
TIME
https://github.com/romainr/hadoop-tutorials-examples/tree/master/notebook/shared_rdd

79. • SSL Support
• Persistent Sessions
• Kerberos
SECURITY

80. SPARK MAGIC
•From Microsop
•Python magics for working with remote Spark
clusters
•Open Source: hAps://github.com/jupyter-
incubator/sparkmagic

81. FUTURE
•Move to ext repo?
•Security
•iPython/Jupyter backends and file format
•Shared named RDD / contexts?
•Share data
•Spark specific, language generic, both?
•Leverage Hue 4
https://issues.cloudera.org/browse/HUE-2990

82. • Open Source: hAps://github.com/cloudera/
hue/tree/master/apps/spark/java
• Read about it: hAp://gethue.com/spark/
•Scala, Java, Python, R
•Type Introspec:on for Visualiza:on
•YARN-cluster or local modes
•Code snippets / compiled
•REST API
•Pluggable backends
•Magic keywords
•Failure resilient
•Security
LIVY’S
CHEAT SHEET

83. BEDANKT!
TWITTER
@gethue
USER GROUP
hue-user@
WEBSITE
hAp://gethue.com
LEARN
hAp://learn.gethue.com