Flink Forward San Francisco 2022. This talk will take you on the long journey of Apache Flink into the cloud-native era. It started all the way from where Hadoop and YARN were the standard way of deploying and operating data applications. We're going to deep dive into the cloud-native set of principles and how they map to the Apache Flink internals and recent improvements. We'll cover fast checkpointing, fault tolerance, resource elasticity, minimal infrastructure dependencies, industry-standard tooling, ease of deployment and declarative APIs. After this talk you'll get a broader understanding of the operational requirements for a modern streaming application and where the current limits are. by David Moravek

1. Apache Flink in
the Cloud-Native
Era
David Moravek
-
@davidmoravek
-
Flink Forward
22

2. ● Co-Founder and Cloud Engineer @ immerok
● Apache Flink & Apache Beam Committer
● I worked with Big Data before it was cool
● I <3 Running
About me
2
Ex…

3. Clusters are dead, long live applications!

4. Fault Tolerance
Stateless Deployment
Resource Elasticity
Automation
Cloud-Native Principles

5. 5
07/06 2020
Flink 1.11.x
(Unaligned Checkpoints)
Flink 1.12.x
12/10 2020
(Kubernetes based HA)
05/03 2021
Flink 1.13.x
(Reactive Scaling Mode)
Flink 1.15.x
TODAY
(Generalized Incremental
Checkpoints, OpenAPI)
Flink 1.14.x
09/29 2021
(Buffer Debloating, Bounded
Checkpoints)
Flink Kubernetes Operator

6. Fault Tolerance

7. Checkpoint n
7
Stateful
Operator
Checkpoint n+1
“Active
Working Set”
Checkpoint barrier n+1
Checkpoint n
Snapshot
in-flight data to process
-> “barrier alignment”

8. Delay by in-flight data
Unaligned checkpoints
“not blocking on in-flight data”
Buffer debloating
“having less in-flight data”
Delay by state snapshot
Incremental checkpoints
“doing smaller snapshots”
Checkpoint while firing timers

9. Checkpoint n
9
Stateful
Operator
Checkpoint n+1
“Active
Working Set”
Checkpoint barrier n+1
Checkpoint n
Snapshot
Fast-track the
checkpoint barrier
Checkpoint n+1
Snapshot

10. Delay by in-flight data
Unaligned checkpoints
“not blocking on in-flight data”
Buffer debloating
“having less in-flight data”
Delay by state snapshot
Incremental checkpoints
“doing smaller snapshots”
Checkpoint while firing timers

11. Checkpoint n
11
Stateful
Operator
“Active
Working Set”
Fixed size buffer
X X
X
X
Buffer 1s worth of
data
⏱

12. Delay by in-flight data
Unaligned checkpoints
“not blocking on in-flight data”
Buffer debloating
“having less in-flight data”
Delay by state snapshot
Incremental checkpoints
“doing smaller snapshots”
Checkpoint while firing timers

13. “Active Working Set” Checkpoint n Checkpoint n+1

14. Delay by in-flight data
Unaligned checkpoints
“not blocking on in-flight data”
Buffer debloating
“having less in-flight data”
Delay by state snapshot
Incremental checkpoints
“doing smaller snapshots”
Checkpoint while firing timers
Yet to be solved.
Stay tuned!

15. Stateless Deployment
… of a stateful computation 🤯

16. 16
Task Manager
Task
Slot
Task
Slot
K8s ConfigMap
Leader Election,
Checkpointing Metadata
Object Store (S3, GCS, HDFS, …)
Asynchronous State
Persistence Pod
Task Manager
Task
Slot
Task
Slot
Pod
Task Manager
Task
Slot
Task
Slot
Pod
Task Manager
Task
Slot
Task
Slot
Pod

17. Resource Elasticity

18. ●General Dynamics
●RocksDB Recap
●Limitations

19. 19
./bin/flink run
--fromSavepoint <path>
--parallelism <newParallelism>
<jar-file> <arguments>
Rinse, Repeat …
./bin/flink stop
--savepointPath <path>
<jobId>
./bin/flink run
--parallelism <parallelism>
<jar-file> <arguments>
⏱ ⏱
./bin/flink stop
--savepointPath <path>
--type native
<jobId>
./bin/flink run
--fromSavepoint <path>
--parallelism <newParallelism>
--restoreMode claim
<jar-file> <arguments>
./bin/flink run
--parallelism <parallelism>
<jar-file> <arguments>

20. 20
$ echo “execution-mode: reactive” >> conf/flink-
conf.yaml
Task Manager
Task
Slot
Task
Slot
Pod
Task Manager
Task
Slot
Task
Slot
Pod
Yo, I have 2 slots!
Great, give them to JM #0
ResourceManager
JobMaster #0
min: 1
max: +inf
Declare
requirements
Task
Slot
Task
Slot
Task
Slot
Task
Slot
Task
Slot
Task
Slot
Here, take my
slots
Start executing!
Here, take my
slots
Task
Slot
Task
Slot
Rescale using latest
checkpoint

21. 21
Task Manager
Task
Slot
Task
Slot
Pod
Task Manager
Task
Slot
Task
Slot
Pod
Task Manager
Task
Slot
Task
Slot
Pod

22. KeyGroup {0,1,2} KeyGroup {3,4,5}
KeyGroup {0,1} KeyGroup {2,3} KeyGroup {4,5}
max-parallelism = 6
current-parallelism = 3
desired-parallelism = 2

23. ●General Dynamics
●RocksDB Recap
●Limitations

24. “Active Working Set”
L0
L1
L2
LN
L0 Compaction

25. Incremental
Checkpoints #2

26. “Active Working Set”
L0
L1
L2
LN
Checkpoint n
(subset)
Checkpoint n+2
Checkpoint n+1
(subset)

27. ●General Dynamics
●RocksDB Recap
●Limitations

28. KeyGroup {0,1} KeyGroup {2,3}
KeyGroup {0,1,2}
Scan & Put
(KeyGroup 2)

29. Automation & Tooling

30. ●Declarative APIs
●OpenAPI Spec

31. ●Declarative APIs
●OpenAPI Spec

32. $ export SPEC_URI="..."
$ go install github.com/deepmap/oapi-codegen/cmd/oapi-codegen@latest
$ curl "${SPEC_URI}" 2> /dev/null | sed '/discriminator:/,+1d' > dispatcher.yml
$ oapi-codegen -old-config-style
-package openapi
-generate client,types
dispatcher.yml
https://nightlies.apache.org/flink/flink-docs-master/generated/rest_v1_dispatcher.yml

33. client, _ := openapi.NewClientWithResponses("http://localhost:8081")
response, _ := client.GetJobDetailsWithResponse(context.Background(), jobId)
if response.StatusCode() == 200 {
details := response.JSON200
fmt.Printf("%s: %s",
pointer.StringDeref(details.Name, "unknown"),
pointer.StringDeref((*string)(details.State), "unknown"))
}
https://nightlies.apache.org/flink/flink-docs-master/generated/rest_v1_dispatcher.yml

34. Fault Tolerance
Stateless Deployment
Resource Elasticity
Automation
Fully Managed Service
for Apache Flink

35. Get ready to ROK!!!
David Moravek
@davidmoravek
david@immerok.com