When Streaming Needs Batch With Konstantin Knauf | Current 2022 A streaming application is started once and then continuously ingests endless, fairly steady streams of events. That's as far as the theory goes. Unfortunately, reality is more complicated. Over time your application's ability to process large historical data sets robustly, efficiently and correctly will be critical: - for exploratory data analysis during development - for bootstrapping the initial state of an application - for back-filling following an outage or bugfix - for keeping up with bursty input streams These scenarios call for batch processing techniques. Apache Flink is as streaming-first as it gets. Yet over the last releases, the community has invested significant resources into unifying stream- and batch processing on all layers of the stack: scheduler to APIs. In this talk, I'll introduce Apache Flink's approach to unified stream and batch processing and discuss - by example - how these scenarios can already be addressed today and what might be possible in the future.

1. When Streaming
Needs Batch
-
Flink's Journey
Towards a
Unified Engine
Konstantin Knauf
–
@snntrable
–
Current 22

2. About Me
2
● Co-Founder @ Immerok
● Apache Flink Committer & Member of the PMC
● Formerly Head of Product & Field Engineer @ Ververica
Visit us at
booth S14.

3. Steady State - Streaming Heaven
3

4. 4

5. Backfilling
following an outage

6. Backfilling
or after an introducing a bug

7. 7
State Bootstrapping

8. Catch Up

9. Bursty Streams

10. Agenda
Motivation CHECK
Setting the Stage
Catching Up & Handling Bursty Streams
Backfilling
Bootstrapping
Takeaways

11. Setting the Stage
11

12. Stream Processing Batch Processing
Bounded Data CHECK CHECK
Unbounded Data CHECK
Some Terminology
Nature
of
Data
Nature of Processing

13. A Typical Streaming Job
Apache Kafka* in/out
* or any other replayable queue like
Apache Pulsar/AWS Kinesis/…

14. A Typical Streaming Job
Flink Pipeline
with multiple operators &
shuffles

15. A Typical Streaming Job
incl. significant state

16. A Typical Streaming Job

17. A Typical Streaming Job

18. A Typical Streaming Job

19. A Typical Streaming Job

20. Catching Up &
Bursty Streams
20

21. Scenario
There is a large backlog of data and you want to catch up to real-time again. For example,
happens when upstream producers send data in big chunks.
Wishlist
● process backlog quickly
● process backlog robustly with existing resources
Scenario
1

22. Backlog -> Backpressure
When under backpressure, what do you want?
● Scale up
● Or catch up steadily

23. Backlog -> Backpressure
When under backpressure, what do you want?
● Scale up
● Or catch up steadily

24. Scaling Up under Backpressure
Adaptive Scheduler and Reactive Mode (Flink 1.13)
Task Manager
Task
Slot
Task
Slot

25. Scaling Up under Backpressure
Adaptive Scheduler and Reactive Mode (Flink 1.13)
Task Manager
Task
Slot
Task
Slot
Task Manager
Task
Slot
Task
Slot

26. Scaling Up under Backpressure
Adaptive Scheduler and Reactive Mode (Flink 1.13)
Task Manager
Task
Slot
Task
Slot
Task Manager
Task
Slot
Task
Slot
● Job automatically scales up when provided with additional resources
● No additional Savepoint needed for rescaling*
* Caveat: This might still take quite some time during restore. Flink 1.16 brings some more improvements in that regard.

27. Backlog -> Backpressure
When under backpressure, what do you want?
● Scale up
● Or catch up steadily

28. Robustness under Backpressure
Watermark Alignment (Flink 1.15)
high throughput topic
low throughput topic

29. Robustness under Backpressure
Watermark Alignment (Flink 1.15)
advances watermark
~ with processing
time
advances watermark
~ with processing
time

30. Robustness under Backpressure
Watermark Alignment (Flink 1.15)
advances watermark
slowly
advances watermark
very quickly

31. Robustness under Backpressure
Watermark Alignment (Flink 1.15)
join state grows
during processing a
backlog

32. Robustness under Backpressure
Watermark Alignment (Flink 1.15)
advances watermark
slowly
advances watermark very
quickly

33. Robustness under Backpressure
Unaligned Checkpoints (Flink 1.11-1.13) & Buffer Debloating (Flink 1.14)
checkpoint
barrier n
6
x
5
4
3
2
1
f
e
c
d
b
a
h
g
e
f
d
c
y
9
8
7
6
5
4
a y
b
1
2
3
input buffer
aligning
begin aligning
operator operator

34. Robustness under Backpressure
Unaligned Checkpoints (Flink 1.11-1.13) & Buffer Debloating (Flink 1.14)
Under backpressure & at scale checkpoint alignment can take hours leading to checkpoint timeouts and job
failures.
● Buffer debloating dramatically reduces the amount of in-flight data
● Unaligned checkpoints allows barriers to overtake in-flight data
checkpoint
barrier n
6
x
5
4
3
2
1
f
e
c
d
b
a
h
g
e
f
d
c
y
9
8
7
6
5
4
a y
b
1
2
3
input buffer
aligning
begin aligning
operator operator

35. Scenario
There is a large backlog of data and you want to catch up to real-time again. For example, happens
when upstream producers send data in big chunks.
Wishlist
● process backlog quickly CHECK (Adaptive Scheduler)
● process backlog robustly with existing resources CHECK (Buffer Debloating, Watermark Alignment, Unaligned Checkpoints)
Scenario

36. Backfilling
36

37. Scenario
You want to reprocess a fixed amount of (recent) historical data to correct a bug or outage.
Wishlist
● Code-reuse for backfilling
● Same semantics and complete & correct results
● Resource efficient
Scenario
[1] https://www.youtube.com/watch?v=4qSlsYogALo&t=668s
1

38. 38
- The Apache Flink Community
Batch is a Special Case of Streaming

39. DataStream API
with Streaming Execution

40. DataStream API
with Streaming Execution

41. DataStream API
with Streaming Execution
All the elasticity and robustness improvements for processing under backpressure apply to here, too.

42. Scenario
We want to reprocess a fixed amount of (recent) historical data to correct a bug or outage.
Wishlist
● Code-reuse for backfilling CHECK
● Same semantics and complete & correct results CHECK
● Resource efficient
Evaluation
Stream Execution Mode

43. Batch Execution Mode
Implementation Timeline
● Apache Flink 1.12
○ Initial Release
○ Unified Sink API (beta)
● Apache Flink 1.13
○ Support for Python DataStream API
● Apache Flink 1.14
○ Batch Execution Mode for mixed DataStream/Table API programs
○ Unified Sink API stable
○ Unified Source API stable
● Apache Flink 1.15
○ Most Sources/Sinks migrated to unified interfaces
○ Adaptive Batch Scheduler

44. DataStream API
with Batch Execution

45. DataStream API
with Batch Execution

46. DataStream API
with Batch Execution

47. Batch Execution Performance

48. Why is Batch Processing faster?

49. It all boils down to completeness & latency.
Stream Processing Batch Processing
Data is incomplete Data is complete
Latency SLAs No Latency SLAs

50. Batch Execution vs Stream Execution

51. Batch Execution vs Stream Execution
Data Exchange Mode

52. Batch Execution vs Stream Execution
Data Exchange Mode

53. Batch Execution vs Stream Execution
Data Exchange Mode

54. Batch Execution vs Stream Execution
Data Exchange Mode

55. Batch Execution vs Stream Execution
Data Exchange Mode

56. Batch Execution vs Stream Execution
Fault Tolerance
Stream Processing

57. Batch Execution vs Stream Execution
Fault Tolerance
Object Store (S3, GCS, HDFS, …)
Periodic Snapshots
Stream Processing
Checkpointing

58. Batch Execution vs Stream Execution
Fault Tolerance
Object Store (S3, GCS, HDFS, …)
Periodic Snapshots
Stream Processing
Checkpointing
Batch Processing

59. Batch Execution vs Stream Execution
Fault Tolerance
Object Store (S3, GCS, HDFS, …)
Periodic Snapshots
Stream Processing
Checkpointing
Batch Processing

60. Batch Execution vs Stream Execution
Fault Tolerance
Object Store (S3, GCS, HDFS, …)
Periodic Snapshots
Stream Processing
Checkpointing
Batch Processing

61. Batch Execution vs Stream Execution
Fault Tolerance
Object Store (S3, GCS, HDFS, …)
Periodic Snapshots
Stream Processing
Checkpointing
Batch Processing
Backtracking
Local Disk or
External Shuffle Service

62. Batch Execution vs Stream Execution
Processing Order and State Backends
0
0
Stream Processing

63. Batch Execution vs Stream Execution
Processing Order and State
0
1
Stream Processing

64. Batch Execution vs Stream Execution
Processing Order and State
1
1
Stream Processing

65. Batch Execution vs Stream Execution
Processing Order and State
1
2
Stream Processing

66. Batch Execution vs Stream Execution
Processing Order and State
2
2
Stream Processing

67. Batch Execution vs Stream Execution
Processing Order and State
Stream Processing
Keys are processed simultaneously.
2
2

68. Batch Execution vs Stream Execution
Processing Order and State
2
2
Batch Processing
Stream Processing
Keys are processed simultaneously.

69. Batch Execution vs Stream Execution
Processing Order and State
2
5
Stream Processing
Keys are processed simultaneously.
Batch Processing

70. Batch Execution vs Stream Execution
Processing Order and State
2
2
5
Stream Processing
Keys are processed simultaneously.
Batch Processing

71. Batch Execution vs Stream Execution
Processing Order and State
2
2
6
5
Stream Processing
Keys are processed simultaneously.
Batch Processing

72. Batch Execution vs Stream Execution
Processing Order and State
2
2
5
6
Stream Processing
Keys are processed simultaneously.
Batch Processing

73. Batch Execution vs Stream Execution
Processing Order and State
2
2
5
6
Stream Processing
Keys are processed simultaneously.
Batch Processing
Keys are processed one after another.

74. Batch Execution vs Stream Execution
Time
● Does Processing Time make sense when processing historical data?
○ Not really.
○ All processing time timers fire at the end of the input.

75. Batch Execution vs Stream Execution
Time
● Does Processing Time make sense when processing historical data?
○ Not really.
○ All processing time timers fire at the end of the input.
● Does historical data arrive out-of-order?
○ No, as it is complete we can sort it by timestamp if needed.

76. Batch Execution vs Stream Execution
Time
● Does Processing Time make sense when processing historical data?
○ Not really.
○ All processing time timers fire at the end of the input.
● Does historical data arrive out-of-order?
○ No, as it is complete we can sort it by timestamp if needed.
● Do watermarks make sense in batch processing?
○ No, we don’t need them. There is no trade off between latency and completeness.
○ Watermark jumps from -∞ to +∞. All event time timers fire at the end of the input.

77. Batch Execution vs Stream Execution
Summary
Stream Processing Batch Processing
Data Exchange Mode Pipelined Blocking
Fault Tolerance Checkpointing Backtracking
Processing Order All keys simultaneously Keys one by one
Time
● Event processed out-of-order
● Event and Processing Time
● Watermarks
● Events processed by event time for each key
● Only Eventtime
● No Watermarks

78. Scenario
We want to reprocess a fixed amount of (recent) historical data to correct a bug or outage.
Wishlist
● Code-reuse for backfilling CHECK
● Same semantics and complete & correct results CHECK
● Resource efficient CHECK (Potential Caveat: Resource Consumption? See Uber Talk.)
Evaluation
Batch Execution Mode

79. Bootstrapping
79

80. Scenario
We want to process historical data (weeks, months, year) to build up the applications state
before switching the application to real-time data.
Wishlist
● Code-reuse for bootstrapping
● Different data source for bootstrapping
● Resource efficient
Scenario
[1] https://www.youtube.com/watch?v=BTWntKy_MJs
[2] https://www.youtube.com/watch?v=JQyfXEQqKeg
[3] https://www.youtube.com/watch?v=JKndMiXphzw
1
2
3

81. Bootstrapping with the Hybrid Source
Hybrid Source automates switching of sources from historical data to real-time
data within a single streaming Job.
S3
Kafka
hours retention
years retention
S3

82. Bootstrapping with the Hybrid Source
Hybrid Source automates switching of sources from historical data to real-time
data within a single streaming Job.
S3
Kafka
hours retention
years retention
Unbounded Source
Bounded Source
S3

83. Bootstrapping with the Hybrid Source
Hybrid Source automates switching of sources from historical data to real-time
data within a single streaming Job.
S3
Kafka
hours retention
years retention
S3
All the elasticity and robustness improvements for processing under backpressure apply to here, too.

84. Bootstrapping with the Hybrid Source
Hybrid Source automates switching of sources from historical data to real-time
data within a single streaming Job.
S3
Kafka
hours retention
years retention
End of Input Reached
S3

85. Bootstrapping with the Hybrid Source
Hybrid Source automates switching of sources from historical data to real-time
data within a single streaming Job.
S3
Kafka
hours retention
years retention
S3

86. Bootstrapping with the Hybrid Source

87. Bootstrapping with the Hybrid Source

88. Scenario
We want to process historical data (weeks, months, year) to build up the applications state
before switching the application to real-time data.
Wishlist
● Code-reuse for bootstrapping 🗸
● Different data source for bootstrapping 🗸
● Resource efficient
Evaluation
Bootstrapping with Hybrid Source

89. Bootstrapping with Batch Execution
/dev/null
Bootstrapping Job
With Batch Execution
Separate Data Source Discarding Sink
S3

90. Bootstrapping with Batch Execution
Savepoint
/dev/null
Bootstrapping Job
With Batch Execution
Separate Data Source Discarding Sink
produces a final Savepoint
S3

91. Bootstrapping with Batch Execution
Savepoint
/dev/null
Bootstrapping Job
With Batch Execution
Separate Data Source Discarding Sink
Real-Time Job
With Stream Execution
produces a final Savepoint takes a final Savepoint as initial state
S3
Pre-Release!

92. Demo

93. Demo

94. Final Savepoints for Batch Jobs
Next Steps
1. Still some limitations & open questions to address in prototype
2. Publish FLIP & discuss with the Community
3. We are optimistic about Flink 1.17.

95. Scenario
We want to process historical data (weeks, months, year) to build up the applications state
before switching the application to real-time data.
Wishlist
● Code-reuse for bootstrapping CHECK
● Different data source for bootstrapping CHECK
● Resource efficient CHECK
Evaluation
Bootstrapping with Batch Execution

96. Takeaways
96

97. Takeaways
● Just because you are streaming, doesn’t mean you can always avoid processing lots of
data at once

98. Takeaways
● Just because you are streaming, doesn’t mean you can always avoid processing lots of
data at once
● Batch processing techniques are usually more resource efficient for this.

99. Takeaways
● Just because you are streaming, doesn’t mean you can always avoid processing lots of
data at once
● Batch processing techniques are usually more resource efficient for this.
● Apache Flink has done a lot recently to make sure those two processing modes work well
together in real-world applications.

100. Takeaways
● Just because you are streaming, doesn’t mean you doesn’t mean you can always avoid
processing lots of data at once
● Batch processing techniques are usually more resource efficient for this.
● Apache Flink has done a lot recently to make sure those two processing modes work well
together in real-world applications.
● Final Savepoints for Batch Jobs is the last mile for Batch Execution in DataStream API.

101. Thanks
Konstantin Knauf
@snntrable
konstantin@immerok.io
CDC Stream Processing with
Apache Flink
Timo Walther, 2pm, Ballroom G