Contenu connexe Similaire à Deploying Flink on Kubernetes - David Anderson (20) Deploying Flink on Kubernetes - David Anderson1. © 2019 Ververica
David Anderson | @alpinegizmo | Training Coordinator
Getting Started with
Apache Flink® on Kubernetes
2. 2 © 2019 Ververica
About Ververica
Creators of
Apache Flink®
Real Time
Stream Processing
for the Enterprise
3. 3 © 2019 Ververica
Outline
1. Introduction
2. Detailed Example
3. Debugging Tips
4. Future Plans
4. 4 © 2019 Ververica
Why Containers?
• Containers provide isolation at low cost
– Require fewer resources than VMs
– Smaller, boot faster
• Simpler to manage
– Each container does one thing
– Consistent packaging
• Enables flexible and dynamic resource allocation
– Scalable
– Composable
5. 5 © 2019 Ververica
Container Orchestration with Kubernetes
• Declarative configuration:
– You tell K8s the desired state, and a background process makes it happen
• 3 replicas of this container should be kept running
• A load balancer should exist, listening on port 443, backed by container with this label
• Core resource types:
– Pod: a group of one or more containers
– Job: keeps pod(s) running until finished
– Deployment: keeps n pods running indefinitely
– Service: a REST object backed by a set of pods
– Persistent Volume Claim: storage whose lifetime is not coupled to any of the pods
6. 6 © 2019 Ververica
Vision: Flink as a Library
• Makes deployments simpler
– Focus is on deploying/running an application
– You build one, complete job-specific Docker image that includes:
• Your application code
• Flink libraries
• Other dependencies
• Configuration files
7. 7 © 2019 Ververica
Flink’s Runtime Building Blocks
• Cluster framework-specific
• Manages available TaskManagers
• Acquires / releases resources
ResourceManager
TaskManagerJobManager
• Registers with ResourceManager
• Provides “task slots”
• Assigned tasks by one or more JobManagers
• One per job
• Schedules job in terms of "task slots"
• Monitors task execution
• Coordinates checkpointing
Dispatcher
• Touch-point for job submissions
• Spawns JobManagers
8. 8 © 2019 Ververica
Flink’s Runtime Building Blocks
• Cluster framework-specific
• Manages available TaskManagers
• Acquires / releases resources
ResourceManager
TaskManagerJobManager
• Registers with ResourceManager
• Provides “task slots”
• Assigned tasks by JobManager(s)
• One per job
• Schedules job in terms of "task slots"
• Monitors task execution
• Coordinates checkpointing
Dispatcher
• Touch-point for job submissions
• Spawns JobManagers
9. 9 © 2019 Ververica
Runtime Building Blocks (on Yarn)
ResourceManager
(3) Request slots
TaskManager
JobManager
(4) Start TaskManager
(5) Register
(7) Deploy Tasks
Dispatcher
App/Client
(1) Submit Job
(2) Start JobManager
(6) Offer slots
10. 10 © 2019 Ververica
But we’re not quite there yet with K8s
11. 11 © 2019 Ververica
Flink on K8s: current status
• Still using the legacy standalone resource manager
• Deployment establishes a static execution environment
• You will have a k8s manifest that effectively says
– there should be n taskmanagers that look like this
Flink is not aware of Kubernetes
12. 12 © 2019 Ververica
Master Container
ResourceManager
JobManager
Mini Dispatcher
(2) Run & Start
Worker Container
TaskManager
Worker Container
TaskManager
Worker Container
TaskManager
(3) Register
(4) Deploy Tasks
(0) One image is built that can be either a Master or Worker
(1) Container framework starts Master & Worker Containers
Flink job cluster on K8s
13. 13 © 2019 Ververica
2. EXAMPLE
https://github.com/alpinegizmo/flink-containers-example
14. 14 © 2019 Ververica
Very Simple Streaming Job
https://github.com/alpinegizmo/flink-containers-example
data generator RichFlatMap print
# events per user
keyBy
16. 16 © 2019 Ververica
Desired Runtime Landscape for K8s
17. 17 © 2019 Ververica
Steps
1. Build the docker image
2. Set up job cluster (k8s job) &
task managers (k8s deployment)
3. Set up job cluster service
4. Add minio for checkpoints
18. 18 © 2019 Ververica
1: Build a docker image
ADD $flink_dist $FLINK_INSTALL_PATH
ADD $job_jar $FLINK_INSTALL_PATH/job.jar
. . .
COPY docker/flink/flink-conf.yaml $FLINK_HOME/conf
COPY docker/flink/log4j-console.properties $FLINK_HOME/conf
COPY docker/flink/docker-entrypoint.sh /
. . .
ENTRYPOINT ["/docker-entrypoint.sh"]
Dockerfile
19. 19 © 2019 Ververica
. . .
JOB_CLUSTER="job-cluster"
TASK_MANAGER="task-manager"
CMD="$1"
shift;
if [ "${CMD}" == "${JOB_CLUSTER}" -o "${CMD}" == "${TASK_MANAGER}" ]; then
if [ "${CMD}" == "${TASK_MANAGER}" ]; then
exec $FLINK_HOME/bin/taskmanager.sh start-foreground "$@"
else
exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
fi
fi
exec "$@"
docker-entrypoint.sh
20. 20 © 2019 Ververica
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: flink-task-manager
spec:
replicas: ${FLINK_NUM_OF_TASKMANAGERS}
template:
metadata:
labels:
app: flink
component: task-manager
spec:
containers:
- name: flink-task-manager
image: ${FLINK_IMAGE_NAME}
imagePullPolicy: Never
args: ["task-manager",
"-Djobmanager.rpc.address=flink-job-cluster"]
task-manager-deployment.yaml.template
apiVersion: batch/v1
kind: Job
metadata:
name: flink-job-cluster
spec:
template:
metadata:
labels:
app: flink
component: job-cluster
spec:
restartPolicy: OnFailure
containers:
- name: flink-job-cluster
image: ${FLINK_IMAGE_NAME}
imagePullPolicy: Never
args: ["job-cluster",
"-Djobmanager.rpc.address=flink-job-cluster",
"-Dblob.server.port=6124",
"-Dqueryable-state.server.ports=6125"]
ports:
- containerPort: 6123
name: rpc
- containerPort: 6124
name: blob
- containerPort: 6125
name: query
- containerPort: 8081
name: ui
job-cluster-job.yaml.template
2: K8s manifests
21. 21 © 2019 Ververica
task-manager-deployment.yaml.template
apiVersion: batch/v1
kind: Job
metadata:
name: flink-job-cluster
spec:
template:
metadata:
labels:
app: flink
component: job-cluster
spec:
restartPolicy: OnFailure
containers:
- name: flink-job-cluster
image: ${FLINK_IMAGE_NAME}
imagePullPolicy: Never
args: ["job-cluster",
"-Djobmanager.rpc.address=flink-job-cluster",
"-Dblob.server.port=6124",
"-Dqueryable-state.server.ports=6125"]
ports:
- containerPort: 6123
name: rpc
- containerPort: 6124
name: blob
- containerPort: 6125
name: query
- containerPort: 8081
name: ui
job-cluster-job.yaml.template
2: K8s manifests
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: flink-task-manager
spec:
replicas: ${FLINK_NUM_OF_TASKMANAGERS}
template:
metadata:
labels:
app: flink
component: task-manager
spec:
containers:
- name: flink-task-manager
image: ${FLINK_IMAGE_NAME}
imagePullPolicy: Never
args: ["task-manager",
"-Djobmanager.rpc.address=flink-job-cluster"]
22. 22 © 2019 Ververica
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: flink-task-manager
spec:
replicas: ${FLINK_NUM_OF_TASKMANAGERS}
template:
metadata:
labels:
app: flink
component: task-manager
spec:
containers:
- name: flink-task-manager
image: ${FLINK_IMAGE_NAME}
imagePullPolicy: Never
args: ["task-manager",
"-Djobmanager.rpc.address=flink-job-cluster"]
task-manager-deployment.yaml.template
apiVersion: batch/v1
kind: Job
metadata:
name: flink-job-cluster
spec:
template:
metadata:
labels:
app: flink
component: job-cluster
spec:
restartPolicy: OnFailure
containers:
- name: flink-job-cluster
image: ${FLINK_IMAGE_NAME}
imagePullPolicy: Never
args: ["job-cluster",
"-Djobmanager.rpc.address=flink-job-cluster",
"-Dblob.server.port=6124",
"-Dqueryable-state.server.ports=6125"]
ports:
- containerPort: 6123
name: rpc
- containerPort: 6124
name: blob
- containerPort: 6125
name: query
- containerPort: 8081
name: ui
job-cluster-job.yaml.template
2: K8s manifests
24. 24 © 2019 Ververica
apiVersion: v1
kind: Service
metadata:
name: flink-job-cluster
labels:
app: flink
component: job-cluster
spec:
ports:
- name: rpc
port: 6123
- name: blob
port: 6124
- name: query
port: 6125
nodePort: 30025
- name: ui
port: 8081
nodePort: 30081
type: NodePort
selector:
app: flink
component: job-cluster
3: Expose job cluster as a service
job-cluster-service.yaml
internal ports
external ports
26. 26 © 2019 Ververica
4: Setup minio for checkpoints & savepoints
• S3-compatible storage service
• Apache License v2.0
• Lightweight, easy to setup
27. 27 © 2019 Ververica
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: minio-pv-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
minio-standalone-pvc.yaml
28. 28 © 2019 Ververica
minio-standalone-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: minio
spec:
strategy:
type: Recreate
template:
metadata:
labels:
app: minio
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: minio-pv-claim
containers:
- name: minio
volumeMounts:
- name: data
mountPath: "/data"
image: minio/minio:RELEASE.2019-03-13T21-59-47Z
args:
- server
- /data
env:
- name: MINIO_ACCESS_KEY
value: "minio"
- name: MINIO_SECRET_KEY
value: "minio123"
ports:
- containerPort: 9000
livenessProbe:
httpGet:
path: /minio/health/live
port: 9000
initialDelaySeconds: 120
periodSeconds: 20
29. 29 © 2019 Ververica
apiVersion: v1
kind: Service
metadata:
name: minio-service
spec:
type: NodePort
ports:
- port: 9000
nodePort: 30090
selector:
app: minio
s3.path-style-access: true
s3.endpoint: http://minio-service:9000
minio-standalone-service.yaml
flink-conf.yaml
30. 30 © 2019 Ververica
/bin/sh -c "
sleep 10;
/usr/bin/mc config host add myminio http://minio-service:9000 minio minio123;
/usr/bin/mc mb myminio/state;
exit 0;
"
minio setup job
state.checkpoints.dir: s3://state/checkpoints
state.savepoints.dir: s3://state/savepoints
s3.access-key: minio
s3.secret-key: minio123
flink-conf.yaml
31. 31 © 2019 Ververica
A Note on Bucket Addresses
• Two ways to specify buckets:
– virtual-hosted style: state.minio-service:9000
– path-style: minio-service:9000/state
• It’s easier to get path-style addresses working, by either using
– s3.path-style-access: true (requires flink 1.8+)
or by
– specifying the endpoint with its IP address, rather than hostname
34. 34 © 2019 Ververica
Rescaling
$ kubectl scale deployment -l component=task-manager --replicas=2
deployment.extensions "flink-task-manager" scaled
$ flink modify 00000000000000000000000000000000 -p 8 -m localhost:30081
Modify job 00000000000000000000000000000000.
Rescaled job 00000000000000000000000000000000. Its new parallelism is 8.
36. 36 © 2019 Ververica
. . .
JOB_CLUSTER="job-cluster"
TASK_MANAGER="task-manager"
if [ "${CMD}" == "${JOB_CLUSTER}" -o "${CMD}" == "${TASK_MANAGER}" ]; then
echo "Starting the ${CMD}"
echo "config file: " && grep '^[^n#]' $FLINK_HOME/conf/flink-conf.yaml
if [ "${CMD}" == "${TASK_MANAGER}" ]; then
exec $FLINK_HOME/bin/taskmanager.sh start-foreground "$@"
else
exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
fi
fi
exec "$@"
docker-entrypoint.sh
37. 37 © 2019 Ververica
Starting the job-cluster
config file:
jobmanager.rpc.address: localhost
jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
taskmanager.heap.size: 1024m
taskmanager.numberOfTaskSlots: 4
parallelism.default: 1
high-availability: zookeeper
high-availability.jobmanager.port: 6123
high-availability.storageDir: s3://highavailability/storage
high-availability.zookeeper.quorum: zoo1:2181
state.backend: filesystem
state.checkpoints.dir: s3://state/checkpoints
state.savepoints.dir: s3://state/savepoints
rest.port: 8081
zookeeper.sasl.disable: true
s3.access-key: minio
s3.secret-key: minio123
s3.path-style-access: true
s3.endpoint: http://minio-service:9000
logs
40. 40 © 2019 Ververica
Tighter Integration with K8s
• Active mode
– Flink is aware of the cluster manager that it is running on,
and interacts with it
– Examples exist, e.g., FLIP-6 YARN
• Reactive mode
– Flink is oblivious to its environment
– Flink may react to resources changes by scaling job
41. 41 © 2019 Ververica
Active k8s Integration
K8s deployment
controller
Client
TaskManager
JobManager
K8sResourceManager
ApplicationMaster
TaskManager
(3) Submit job
(1) Submit AM deployment
(2) Start AM
pod
(4) Start JM
(5) Request slots
(6) Submit TM
deployment
(7) Start TM pod
(8) Register(9) Request slots
(10) Offer slots
42. 42 © 2019 Ververica
FLINK-9953: Active Kubernetes integration
The ResourceManager can talk to Kubernetes to launch new pods
43. 43 © 2019 Ververica
Reactive Container Mode
• Relies on external system to start/release
TaskManagers, e.g.,
– Kubernetes Horizontal Pod Autoscaler
– GCP Autoscaling
– AWS Auto Scaling Group
• Re-scale job as resources are
added/removed (take savepoint and resume
job with new parallelism automatically)
• By definition works with all cluster managers
Flink cluster
JM TM TM
ASG
Start new TM if
CPU% > threshold
Monitor metrics, e.g, CPU%
Register
& offer
slots
Event rate over time
44. 44 © 2019 Ververica
FLINK-10407: Reactive container mode
Re-scale job as resources are added/removed
45. 45 © 2019 Ververica
Summary
• Flink currently supports job and session clusters on K8s
• Example
– https://github.com/alpinegizmo/flink-containers-example
• Active K8s integration is in progress
• Reactive container mode has been designed/planned
• Call to action:
– Umbrella tickets: FLINK-9953, FLINK-10407
– Join discussions on dev@flink.apache.org
48. 48 © 2019 Ververica
www.ververica.com @VervericaDatadavid@ververica.com