This document discusses adding iterative processing capabilities to stream processing systems like Apache Flink. It proposes programming model extensions that would allow iterative computations like PageRank to be expressed over continuous data streams. This would involve new abstractions like structured loops that introduce scoping and nesting to iterate over streaming windows in a distributed fashion similar to how event-time watermarks propagate progress. Challenges addressed include handling cyclic data flows and integrating state management.
Flink Forward Berlin 2018: Paris Carbone - "Stream Loops on Flink: Reinventing the wheel for the streaming era"
1. Stream Loops on Flink
Reinventing the wheel for the streaming era
Paris Carbone
@FF2018-Berlin
Systems Researcher@KTH/SICS <parisc@kth.se>
Committer@Flink <senorcarbone@apache.org>
1
3. Has Streaming Subsumed Batch?
short answer: NO
long answer: not YET
• Staged Processing
• Allows Bulk/Delta Iterative Computation
multi-pass
Flink
DataSet
• Continuous Processing
• Allows correct Acyclic Computation
Flink
DataStream
single-pass
?
3
4. Why Iterations Matter
• Iterations are fundamental building blocks for:
• Graph Analysis (PageRank, Conn.Comp., SSSP etc.)
• Machine Learning (Gradient Descent, PCA etc.)
• Transactions (e.g., optimistic concurrency control)
Iterative Processing Primitives are currently not present
in todays’ production-grade stream processing systems.
4
5. Current Challenges
• Data is born and evolves continuously as a data stream
(e.g., user interactions, sensor events, server logs)
5
6. Current Challenges
Day 1 Day 2 Day 3 Day 4
• To run iterative analysis users have manually do the…
• bucketing
• laundry (schedule jobs)
• cleaning (integration)
6
7. Current Challenges
• To run iterative analysis users have manually do the…
• bucketing
• laundry (iterative job scheduling)
• cleaning (integration)
Day 1 Day 2 Day 3 Day 4
7
8. Current Challenges
• To run iterative analysis users have manually do the…
• bucketing
• laundry (iterative job scheduling)
• sorting (model integration)
Day 1 Day 2 Day 3 Day 4
8
9. Current Challenges
• To run iterative analysis users have manually do the…
• bucketing
• laundry (iterative job scheduling)
• sorting (model integration)
Day 1 Day 2 Day 3 Day 4
9
10. A Partially Solved Problem
• Most challenges mentioned are solved and automated for
single-pass aggregation via stream windowing.
How would a distributed iterative
computation look like as a special
type of window aggregation?
window
computation
windows are a natural fit for
declaring iterative computation
10
11. Anatomy of Iterative Computation
Termination (fixpoint/fixed)
Loop Step Function (computation)
Final Result-Solution
Solution (finite state)
11
21. Enough about what
• We need to examine “how” to implement iterations
executed in parallel and out-of-order
using event-time progress
window
computation
21
22. How Event-Time Progress Works
T
t
progress metric
record
• Every record carries a timestamp t
• t ≥ T
• T can only increment (monotonicity)
T: indication of a low
event-time boundstream
task
22
23. How Event-Time Progress Works
• low watermarks propagate
progress metrics along the
computational graph
• progress metric = minimum
watermark across channels
• works correctly as long as
monotonicity is maintained
23
24. Intuition
Iteration Superstep progress bears similarities to Event-Time progress
1. Can be used to run computation for out-of-order streams.
2. Need for a decentralised mechanism (no coordinator/scheduler).
3. Only relevant to respective parts of the graph (i.e., sources cannot
make use of sink progress)
24
25. Watermarks and
Unstructured Loops
t2 t4t1
W=n
… …t3
• Low watermarking is a very powerful mechanism to measure and
propagate progress of a single progress metric.
• However it breaks when we
have arbitrary cycles
(current Flink iterations)
old watermark
25
which superstep?
which cycle?
27. Structured Loops
IN OUT
structured
loop
global scope
• Structure loops are low level stream graph primitives.
• They can be seen as ‘Iterative Operators’
• Introduce the notion of “structured programming” for data streaming.
• Each structured loop has a scope.
• Each scope operates on its own progress metric.
• Global scope operates on event-time progress (no changes made).
R
27
28. Structured Loops Composition
R
IN OUT
loop scope
global scope
R’
nested loop scope
28
• Encourages compositionally
(arbitrary nesting)
29. Using Progress Timestamps
• Nests progress timestamps
• (e.g., last superstep pT per window Pctx)
• How do we guarantee monotonic progress metrics?
29
30. Structured Loop Embeddings
R
IN OUT
LIN LOUT
LH LT
nest unnest
incr
[32] [0,32] [32]
[0,32]
[1,32]
[1,32]
[31]
term
[ø,32]
User-Defined Loop DAG
[ø,32][ø,32]
• Initializes Scope’s Progress Metric for
both records and watermarks (nest)
• Triggers final output of an iterative operator
• Removes inner progress metric (unnest)
• Increments Progress Metric (incr)
• Forwards Records and Watermarks
• Evaluates and Disseminates Termination (ø)
30
32. Further Challenges
• Cyclic Flow Control
• avoid deadlocks
• encourage iteration completion
• State Management Integration
• one more level of indirection (mapvalue per key)
what keeps us busy
32
33. Cyclic Flow Control
R
IN OUT
LIN LOUT
LH
LT
union
entry
step
fin
window
split
Managed State
flow
control
R
IN
• (Consumed Buffers < Threshold): Round-Robin between IN and R
• (Consumed Buffers >= Threshold): Exclusive Ingestion of R
33
35. Research Team
and Sponsors
• Marius Melzer (TUDresden)
• Asterios Katsifodimos (TUDelft)
• Vasiliki Kalavri (ETH)
• Seif Haridi (KTH)
• Pramod Bhatotia (Un.Edinburgh)
35
36. References
• Scalable and Reliable Data Stream Processing - Paris Carbone (2018)
• Inflight Progress Tracking for Stream Processing Systems with Cyclic
Dataflows - Marius Melzer (2017)
• Naiad: a timely dataflow system - Murray, McSherry et al. (2013)
• The dataflow model: a practical approach to balancing correctness, latency,
and cost in massive-scale, unbounded, out-of-order data processing - Tyler
Akidau et al. (2015)
• Out-of-order processing: a new architecture for high-
performance stream systems - Li, Maier et al. (2008)
• Flexible time management in data stream systems - Srivastava, Widom
(2004)
36
37. Stream Loops on Flink
Reinventing the wheel for the streaming era
Paris Carbone
@FF2018-Berlin
Systems Researcher@KTH/SICS <parisc@kth.se>
Committer@Flink <senorcarbone@apache.org>
37