Container technology experiences an ever increasing adoption throughout many industries. Not only does this technology make your applications portable across different machines and operating systems, it also allows to scale applications in a matter of seconds. Moreover, it significantly simplifies and speeds up deployments which decreases development and operation costs. Consequently, more and more Flink deployments run in containerized environments which poses new challenges for Flink. In this talk, we will take a look at Flink's current and future container support which will make it a first class citizen of the container world. First of all, we will explain how the new reactive execution mode will solve the problem of seamless application scaling and how it blends in with any environment. Complementary to the reactive mode, the active execution mode demonstrates its strengths when it comes to changing workloads such as batch jobs. Last but not least, we will take a look beyond Flink's own nose and investigate how Flink can be used together with Kubernetes operators or data Artisans' Application Manager. We will conclude the talk with a short demo of Flink's native Kubernetes support and giving an outlook on future developments in the container realm.

1. © 2019 Ververica
Till Rohrmann, Engineering Lead
@stsffap
Future of Apache Flink Deployments: Containers,
Kubernetes and More

2. © 2019 Ververica2
Advancements of APIs & Supported Use Cases
Batch Processing
process static and
historic data
Data Stream
Processing
realtime results
from data streams
Event-driven
Applications
data-driven actions
and services
• Deployment & operations features did not keep up with APIs

3. © 2019 Ververica3
Flink as a Library
• Deploying Flink applications should be as easy as starting a process
• Application-oriented instead of cluster-oriented
• Start multiple processes which form distributed execution
• Removing the cluster out of the equation
P1
P2 P3 P4
New process

4. © 2019 Ververica4
Resource Management & Autoscaling
• Multiple short lived stages
• Different resource requirements per
stage
• Efficient execution requires control
over resources
• Flink allocates actively resources
• Long running, continuously processing
operators
• Constrained by external systems and SLAs
• External system with holistic view over
resources better suited for scaling decisions
• Flink reacts to available resources
Real-time &
data-driven
Applications
Batch Analytics
Continuous
Processing

5. © 2019 Ververica5
Reactive vs. Active Execution Mode
• Active mode
– Flink is aware of underlying cluster framework
– Flink allocate resources
– E.g. existing YARN and Mesos integration
• Reactive mode
– Flink is oblivious to its runtime environment
– External system allocates and releases resources
– Flink scales with respect to available resources
– Relevant for environments: Kubernetes, Docker, as a library

6. © 2019 Ververica6
Auto Scaling
• Latency
• Throughput
• Connector signals
• Available resources
Scale up

7. © 2019 Ververica7
Containerized Deployments
• Popular to deploy Flink in containerized environments
• Why Flink on containers?
– Self-contained runtime environment that includes application code, libraries, dependencies, and
configuration files
– Ease of operations by clean separation of concerns
– Offers dynamic resource allocation
• Reactive mode suits containerized deployments well
• Demand for active mode to run batch workloads

8. © 2019 Ververica8
Containerized Deployments (cont.)
• De-facto industry standard for container orchestration
• Resource-oriented with declarative configuration
– You tell K8s the desired state, and a background process asynchronously makes it happen
– “3 replicas of this container should be kept running”
– “a load balancer should exist, listening on port 443, backed by containers with this label”
• Core resource types:
– Pod: a group of one or more containers running on a node
– Deployment: set of n identical pods
Brief Introduction to Kubernetes

9. © 2019 Ververica
Active & Reactive Mode

10. © 2019 Ververica10
Revamped Distributed Architecture (FLIP-6)
– Better separation of concerns
– RESTful API for client-cluster communication
– Full resource elasticity
Dispatcher
Client
ResourceManager ClusterManager
JobManager TaskManager
1. Submit job
2. Start job
3. Request slots
4. Allocate resources
5. Start TaskManager
6. Execute job

11. © 2019 Ververica11
Active K8s Integration
K8s deployment
controller
Client
TaskManager
JobManager
K8sResMngr
ApplicationMaster
TaskManager
(3) Submit job
(1) Submit AM deployment
(4) Start JM
(5) Request slots
(7) Start TM pod
Dispatcher

12. © 2019 Ververica12
Reactive Mode
• Relies on external system to start/release TaskManagers
– Kubernetes Horizontal Pod Autoscaler
– GCP Autoscaling
– AWS Auto Scaling Group
• Re-scale job if resources are added/removed
• By definition works with all cluster managers
Flink cluster
JM TM
TM
ASG
Start new TM
if CPU% >
threshold
Register &
offer slots
Event rate over time

13. © 2019 Ververica
Does Any of This Work?
• Demo: Active K8s Integration & Reactive Container Mode

14. © 2019 Ververica14
Current State
• Recommendations on how to deploy job and session cluster on K8s
– Docker image build script
– K8s resource configs
• StandaloneJobClusterEntryPoint to start a (containerized) job cluster
• Work in progress:
– Active K8s integration
• https://issues.apache.org/jira/browse/FLINK-9953
• Kudos to Chunhui-Shi
• git clone -b native-k8s git@github.com:chunhui-shi/flink.git
– Reactive Container Mode with automatic rescaling
• https://issues.apache.org/jira/browse/FLINK-10407

15. © 2019 Ververica15
TL; DL
• Reactive container mode
– Support applications as first class citizen
– Ease of operations
• Active K8s integration coming soon
• Join the community!
– Subscribe to mailing lists
– Participate in Flink development
– Design the future

16. © 2019 Ververica
Thank you! Questions?

17. © 2019 Ververica
www.ververica.com @VervericaDatatill@ververica.com