Apache Spark 2.0 introduced Structured Streaming which allows users to continually and incrementally update your view of the world as new data arrives while still using the same familiar Spark SQL abstractions. Michael Armbrust from Databricks talks about the progress made since the release of Spark 2.0 on robustness, latency, expressiveness and observability, using examples of production end-to-end continuous applications. Speaker: Michael Armbrust Video: http://go.databricks.com/videos/spark-summit-east-2017/using-structured-streaming-apache-spark This talk was originally presented at Spark Summit East 2017.

1. Insights Without Tradeoffs
Using Structured Streaming
Michael Armbrust - @michaelarmbrust
Spark Summit East 2017

2. Parallelism
Split up the problem
to harness many machines
for computation
Complexity
Handle cross-machine
communication and failures

3. DeveloperProductivity
Quickly and concisely express
common computations
Efficiency
Hand-tune code to minimize
overheads and process the
most data per cycle
SQL

4. Throughput
Process large historical
repositories quickly
Latency
Up-to-date answers
as new data arrives
Structured Streaming

5. Production Use Cases

6. Streaming at
Collect logs and metrics from a variety of
sources to ensure the security, availability
and performance of our cloud platform.

7. Engineer Office Hours
MY HOURS
• Spark Core
• R
• Data Science
• ML
• GraphFrames, Deep Learning
• Databricks
• Spark SQL
• StructuredStreaming
• Spark SQL
• StructuredStreaming
• Databricks
TODAY 4:30 – 5:15
OTHER ENGINEERS
TODAY 1:45 – 5:15 THURS 10:30 – 2:30
@ Databricks Booth

8. Thank you!
All code available at databricks.com/blog
and @michaelarmbrust