Building a REST Job Server for interactive Spark as a service by Romain Rigaux and Erick Tryzelaar
•
26 j'aime•10,415 vues
Building a REST Job Server for interactive Spark as a service by Romain Rigaux and Erick Tryzelaar
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Building a REST Job Server for interactive Spark as a service by Romain Rigaux and Erick Tryzelaar
Similaire à Building a REST Job Server for interactive Spark as a service by Romain Rigaux and Erick Tryzelaar (20)
Plus de Spark Summit
Plus de Spark Summit (20)
Dernier
Dernier (20)
Building a REST Job Server for interactive Spark as a service by Romain Rigaux and Erick Tryzelaar
- 1. BUILDING A REST JOB SERVER FOR INTERACTIVE SPARK AS A SERVICE Romain Rigaux - Cloudera Erick Tryzelaar - Cloudera
- 2. WHY?
- 5. NOTEBOOKS EASY ACCESS FROM ANYWHERE SHARE SPARK CONTEXTS AND RDDs BUILD APPS SPARK MAGIC … WHY SPARK AS A SERVICE?
- 7. HISTORY V1: OOZIE • It works • Code snippet THE GOOD • Submit through Oozie • Shell ac:on • Very Slow • Batch THE BAD workflow.xml snippet.py stdout
- 8. HISTORY V2: SPARK IGNITER • It works beAer THE GOOD • Compiler Jar • Batch only, no shell • No Python, R • Security • Single point of failure THE BAD Compile Implement Upload json output Batch Scala jar Ooyala
- 9. HISTORY V3: NOTEBOOK • Like spark-submit / spark shells • Scala / Python / R shells • Jar / Python batch Jobs • Notebook UI • YARN THE GOOD • Beta? THE BAD Livy code snippet batch
- 14. ARCHITECTURE • Standard web service: wrapper around spark-submit / Spark shells • YARN mode, Spark drivers run inside the cluster (supports crashes) • No need to inherit any interface or compile code • Extended to work with additional backends
- 15. LIVY WEB SERVER ARCHITECTURE LOCAL “DEV” MODE YARN MODE
- 32. SESSION CREATION AND EXECUTION % curl -XPOST localhost:8998/sessions -d '{"kind": "spark"}' { "id": 0, "kind": "spark", "log": [...], "state": "idle" } % curl -XPOST localhost:8998/sessions/0/statements -d '{"code": "1+1"}' { "id": 0, "output": { "data": { "text/plain": "res0: Int = 2" }, "execution_count": 0, "status": "ok" }, "state": "available" }
- 39. INTERPRETERS • Pipe stdin/stdout to a running shell • Execute the code / send to Spark workers • Perform magic opera:ons • One interpreter per language • “Swappable” with other kernels (python, spark..) Interpreter > println(1 + 1) 2 println(1 + 1) 2
- 49. INTERPRETER MAGIC • table • json • plotting • ...
- 60. • Pluggable Backends • Livy's Spark Backends – Scala – pyspark – R • IPython / Jupyter support coming soon PLUGGABLE INTERPRETERS
- 61. • Re-using it • Generic Framework for Interpreters • 51 Kernels JUPYTER BACKEND
- 62. SPARK AS A SERVICE
- 71. SHARING RDDS
- 79. • SSL Support • Persistent Sessions • Kerberos SECURITY