The document discusses creating a fast data pipeline using Apache Spark's Structured Streaming and Spark Streaming. It presents a sensor anomaly detection pipeline that uses Structured Streaming for data exploration, preparation, and anomaly detection, and Spark Streaming for online model creation and training. It compares the execution and abstraction models of Structured Streaming and Spark Streaming, and demonstrates how to build the sensor anomaly detection pipeline using Kafka sources and sinks with SQL operations, event time windows, and watermarks.

1. Hands-on with Apache Spark:
Creating a Fast Data Pipeline
with Structured Streaming and
Spark Streaming
Gerard Maas
Senior SW Engineer, Lightbend, Inc.

2. Gerard Maas
Señor SW Engineer
@maasg
https://github.com/maasg
https://www.linkedin.com/
in/gerardmaas/
https://stackoverflow.com
/users/764040/maasg

3. @maasg
Agenda
Creating a Fast Data Pipeline with
Structured Streaming and
Spark Streaming

4. @maasg
Data Pipelines
Data Pipelines

5. @maasg
Data Pipelines
Data Pipelines
Data
Kafka
Processor

6. @maasg
Data Pipelines
● Create Composable Streaming Applications
● Using the Best Tool for the Job
● Generating a Network Effect

7. @maasg
Agenda
Creating a Fast Data Pipeline with
Structured Streaming and
Spark Streaming

8. @maasg
Apache Spark Core
Spark SQL
SparkMLLib
SparkStreaming
Structured
Streaming
Datasets/Frames
GraphFrames
Data Sources

9. @maasg
Apache Spark Core
Spark SQL
SparkMLLib
SparkStreaming
StructuredStreaming
Datasets/Frames
GraphFrames
Data Sources
RDD
DStream
Query

10. @maasg
Structured Streaming
val ctx= new StreamingContext(conf,Seconds(1))
val lines = ssc.socketTextStream ("localhost" , 9999)
val words = lines.flatMap(_.split(" "))
val pairs = words.map(word => (word, 1))
val wordCounts = pairs.reduceByKey(_ + _)
wordCounts.print()
ctx.start()
https://spark.apache.org/docs/latest/streaming-programming-guide.html
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
val lines = spark.readStream
.format("socket")
.option("host", "localhost")
.option("port", 9999)
.load()
val words= lines.as[String].flatMap(_.split(" "))
val wordCounts = words.groupBy("value").count()
val query = wordCounts.writeStream
.outputMode("complete")
.format("console")
.start()

11. @maasg
Time
Execution
Abstraction
Structured Streaming Spark Streaming
Abstract
(Processing Time, Event Time)
Fixed to microbatch
Streaming Interval
Fixed Micro batch, Best Effort MB,
Continuous (NRT)
Fixed Micro batch
DataFrames/Dataset DStream, RDD
Access to the scheduler

12. @maasg
Agenda
Hands On with Spark:
Creating a Fast Data Pipeline with
Structured Streaming and
Spark Streaming

13. @maasg
Sensor
Data
Multiplexer
Data Exploration
[Structured Streaming]
Sensor Anomaly Detection Pipeline
Data Preparation
[Structured Streaming]
Online Model
Creation +
Training
[Spark Streaming]
Anomaly Detection
[Structured Streaming]

14. @maasg
Sensor
Data
Multiplexer
Structured
StreamingLocal Process
Sensor Anomaly Detection
Structured
Streaming

15. @maasg
Sensor
Data
Multiplexer
Structured
StreamingLocal Process
Sensor Anomaly Detection
Structured
Streaming

16. @maasg
Live

17. @maasg
Sensor
Data
Multiplexer
Data Exploration
[Structured Streaming]
Sensor Anomaly Detection Pipeline
Data Preparation
[Structured Streaming]
Online Model
Creation +
Training
[Spark Streaming]
Anomaly Detection
[Structured Streaming]
Kafka Source
Memory Sink
SQL Operations

18. @maasg
Sensor
Data
Multiplexer
Data Exploration
[Structured Streaming]
Sensor Anomaly Detection Pipeline
Data Preparation
[Structured Streaming]
Online Model
Creation +
Training
[Spark Streaming]
Anomaly Detection
[Structured Streaming]
Kafka Source
Memory Sink
SQL Operations
Event Time
Windows
Watermark
Kafka Sink

19. @maasg
Sensor
Data
Multiplexer
Data Exploration
[Structured Streaming]
Sensor Anomaly Detection Pipeline
Data Preparation
[Structured Streaming]
Online Model
Creation +
Training
[Spark Streaming]
Anomaly Detection
[Structured Streaming]
Kafka Source
Memory Sink
SQL Operations
Event Time
Windows
Watermark
Kafka Sink
RDD Programming
Local vs Distributed
Use Spark SQL
Kafka Source + Sink

20. @maasg
Sensor
Data
Multiplexer
Data Exploration
[Structured Streaming]
Sensor Anomaly Detection Pipeline
Data Preparation
[Structured Streaming]
Online Model
Creation +
Training
[Spark Streaming]
Anomaly Detection
[Structured Streaming]
Kafka Source
Memory Sink
SQL Operations
Event Time
Windows
Watermark
Kafka Sink
RDD Programming
Local vs Distributed
Use Spark SQL
Kafka Source + Sink
Controlled Streams:
Spark Streaming +
Structured Streaming

21. @maasg
Time
Execution
Abstraction
Structured Streaming Spark Streaming
Abstract
(Processing Time, Event Time)
Fixed to microbatch
Streaming Interval
Fixed Micro batch, Best Effort MB,
Continuous (NRT)
Fixed Micro batch
DataFrames/Dataset DStream, RDD
Access to the scheduler

22. Resources
Notebooks used today:
https://github.com/maasg/spark-notebooks/tree/master/streaming-anomaly-detection
Pipelines:
https://www.reactivesummit.org/2018/schedule/taking-the-pain-out-of-deploying-streaming-applic
ations
Structured Streaming + Spark Streaming:
https://www.reactivesummit.org/2018/schedule/processing-fast-data-with-apache-spark-the-tale-of
-two-streaming-apis
Fast Data:
https://www.lightbend.com/products/fast-data-platform

23. Gerard Maas
Señor SW Engineer
@maasg
https://github.com/maasg
https://www.linkedin.com/
in/gerardmaas/
https://stackoverflow.com
/users/764040/maasg

24. Your turn for...
Questions?

25. Thank
You!