Aljoscha Krettek offers an overview of the modern stream processing space, details the challenges posed by stateful and event-time-aware stream processing, and shares core archetypes ("application blueprints”) for stream processing drawn from real-world use cases with Apache Flink. Topics include: * Aggregating IoT event data, in which event-time-aware processing, handling of late data, and state are important * Data enrichment, in which a stream of real-time events is “enriched” with data from a slowly changing database of supplemental data points * Dynamic stream processing, in which a stream of control messages and dynamically updated user logic is used to process a stream of events for use cases such as alerting and fraud detection

1. Aljoscha Krettek, Software Engineer
Stream Processing for the
Practitioner: Blueprints for Common
Stream Processing Use Cases with
Apache Flink®

2. © 2018 data Artisans2
What is stream processing and
why‘s it useful?

3. © 2018 data Artisans3
Use Case: “Suspicious Behaviour“ Detection
Some service or
app
• Dropbox, Google Suite, Box
• Sharing, accessing, and modifying
data produces events that we
can/want to analyse
• Banking
• We monitor all transactions, know
the user data

4. © 2018 data Artisans4
Use Case: “Suspicious Behaviour“ Detection
Some service or
app
• Spits out alerts when suspicious stuf
is happening
• “More than 10 failed login
attempts”
• ”Sharing more than 100 files
within 1 Hour”
• “ImpossibleTravel”/”Magic
carpet travel”
• “Continuously increasing
withdrawal amounts”
“Magic” rules
engine

5. © 2018 data Artisans5
Use Case: “Suspicious Behaviour“ Detection
• What if my workload becomes way too high?  I’ll use a distributed batch
processor. (Hadoop MapReduce, Spark, and the like)
• This runs nightly? Isn’t the latency very high then? I don’t want to wait a day
for my alerts 😳
Ok, let me just use a single machine for this!
I need alerts in real time  Apache Flink® is a real-time, distributed,
stateful, and fault-tolerant stream processor

6. © 2018 data Artisans6
Detour: Thinking in Flows
TransformFilter
Data
Ingress
Apply
Rules
Persist to
database
Send
Alerts
source
sink
State
user operations/user code
flow of data / events

7. © 2018 data Artisans7
Why flows and operations?
• Operations can be composed and are reusable
• Makes it easy to think about the flow of data
• Allows a system (Flink) to take these operations and execute them on
different machines
• A system can execute the same operation multiple times on several
machines to deal with high workloads

8. © 2018 data Artisans8
Apache Flink® distributed stateful stream processor
Machine 1
Machine 2
Flink has nice APIs for writing these!
Note how we have the
“expensive” stateful
operation twice

9. © 2018 data Artisans9
Apache Flink® distributed stateful stream processor
• Questions that a good stream processing systems needs to have answers
for:
‒What happens when machines fail or when user operations fail?
‒What happens if I need to move my stateful operations/flows/jobs?
‒What happens if I need to change the schema of the state that
operations keep?
‒How can I update framework code while keeping my program state?
‒Same for user code?

10. © 2018 data Artisans10
Common stream processing
blueprints

11. © 2018 data Artisans11
Blueprint: Aggregation of timestamped data
• Use cases
‒Give me the number of tweet impressions per tweet for every
hour/day/…
‒Calculate the average temperature over 10 minute intervals for each
sensor in my warehouse
‒Aggregate user interaction data for my website to display on my internal
dashboards

12. © 2018 data Artisans12
Blueprint: Aggregation of timestamped data
windowed
aggregation
source sink
state: contents of all the in-flight windows

13. © 2018 data Artisans13
Blueprint: Aggregation of timestamped data
• Do I want to window by event-time or processing time?
• If using event-time, how do I know when my window is “done”?
• What happens if data arrives out of order with respect to their timestamp?
• If using event-time, when is data considered late?
• What should happen with late data?
Some things to look out for.

14. © 2018 data Artisans14
Blueprint: Aggregation of timestamped data
• Windowing API
• Timestamp assigners/watermark extractors for defining event-time and
defining “readiness”
• Allowed lateness for defining when data is late
• Side output of late data as a special flow path
Flink features to look at.

15. © 2018 data Artisans15
Blueprint: Aggregation of timestamped data
windowed
aggregation
kinesis
write to
Elastic
alert real
humanslate data
allowed lateness: 10 min
extract timestamps/watermarks
side output

16. © 2018 data Artisans16
Blueprint: Enriching data with “side input”
• Use cases
‒Enrich user events with known user data
‒Add geolocation information to geotagged events

17. © 2018 data Artisans17
Blueprint: Enriching data with “side input”
filter enrich

18. © 2018 data Artisans18
Blueprint: Enriching data with “side input”
filter enrich
Naïve approach
synchronous access to
external data store for
every element

19. © 2018 data Artisans19
Blueprint: Enriching data with “side input”
filter enrich
Slightly better approach
asynchronous access to
external data store for
every element

20. © 2018 data Artisans20
Blueprint: Enriching data with “side input”
filter enrich
“Next-level” approach
keep the enrichment
data in Flink state itselfchangelog
input

21. © 2018 data Artisans21
Blueprint: Enriching data with “side input”
• Regular user functions/operations
• Async I/O operation for more efficient data store accesses
• Two-input operations and stateful operations
Flink features to look at.

22. © 2018 data Artisans22
Blueprint: Dynamic processing
• Use cases
‒Update of processing rules via DSL, think dynamic fraud-detection
rules/policies
‒Live-update of machine learning models

23. © 2018 data Artisans23
Blueprint: Dynamic processing
pre-
processing
dynamic
processing
rules
input
broadcast
stream
broadcast
state

24. © 2018 data Artisans24
Blueprint: Dynamic processing
• ProcessFunction
• Broadcast streams and broadcast state
Flink features to look at.

25. © 2018 data Artisans25
Closing

26. © 2018 data Artisans26
Learnings
• When you want immediate results you probably need a stream processor
• Getting state, fault-tolerance, and event-time right is tough, check what
your stream processor has as answers for the though questions
• Start thinking in terms of data flows and reusable operations that you can
mix and match
• Flink has your use cases covered

28. Thank you!
aljoscha@apache.org
@dataArtisans
@ApacheFlink
We are hiring!
data-artisans.com/careers

29. © 2018 data Artisans29
About Data Artisans
Original creators of
Apache Flink®
Open Source Apache Flink
+ dA Application Manager

30. © 2018 data Artisans30
dA platform
data-artisans.com/download

31. © 2018 data Artisans31
Powered by Apache Flink

32. © 2018 data Artisans32
Download the free book
info.data-artisans.com/book