7. Problem: Setting up a Kubernetes cluster is hard
Today:
Use kube-up.sh (and hope you don’t have to customize)
Compile from HEAD and manually address security
Use a third-party tool (some of which are great!)
Simplified Setup
10. Solution: kubeadm!
Simplified Setup
master.myco.com# apt-get install -y kubelet kubeadm kubectl kubernetes-cni
master.myco.com# kubeadm init
Kubernetes master initialized successfully!
You can now join any number of nodes by running the following command:
kubeadm join --token 48b69e.b61e2d0dd5c 10.140.0.3
11. Solution: kubeadm!
Simplified Setup
master.myco.com# apt-get install -y kubelet kubeadm kubectl kubernetes-cni
master.myco.com# kubeadm init
Kubernetes master initialized successfully!
You can now join any number of nodes by running the following command:
kubeadm join --token 48b69e.b61e2d0dd5c 10.140.0.3
node-01.myco.com# apt-get install -y kubelet kubeadm kubectl kubernetes-cni
node-01.myco.com# kubeadm join --token 48b69e.b61e2d0dd5c 10.140.0.3
12. Solution: kubeadm!
Simplified Setup
master.myco.com# apt-get install -y kubelet kubeadm kubectl kubernetes-cni
master.myco.com# kubeadm init
Kubernetes master initialized successfully!
You can now join any number of nodes by running the following command:
kubeadm join --token 48b69e.b61e2d0dd5c 10.140.0.3
node-01.myco.com# apt-get install -y kubelet kubeadm kubectl kubernetes-cni
node-01.myco.com# kubeadm join --token 48b69e.b61e2d0dd5c 10.140.0.3
Node join complete.
13. Solution: kubeadm!
Simplified Setup
master.myco.com# apt-get install -y kubelet kubeadm kubectl kubernetes-cni
master.myco.com# kubeadm init
Kubernetes master initialized successfully!
You can now join any number of nodes by running the following command:
kubeadm join --token 48b69e.b61e2d0dd5c 10.140.0.3
node-01.myco.com# apt-get install -y kubelet kubeadm kubectl kubernetes-cni
node-01.myco.com# kubeadm join --token 48b69e.b61e2d0dd5c 10.140.0.3
Node join complete.
master.myco.com# kubectl apply -f https://git.io/weave-kube
14. Solution: kubeadm!
Simplified Setup
master.myco.com# apt-get install -y kubelet kubeadm kubectl kubernetes-cni
master.myco.com# kubeadm init
Kubernetes master initialized successfully!
You can now join any number of nodes by running the following command:
kubeadm join --token 48b69e.b61e2d0dd5c 10.140.0.3
node-01.myco.com# apt-get install -y kubelet kubeadm kubectl kubernetes-cni
node-01.myco.com# kubeadm join --token 48b69e.b61e2d0dd5c 10.140.0.3
Node join complete.
master.myco.com# kubectl apply -f https://git.io/weave-kube
Network setup complete.
15. Problem: Using multiple-clusters is hard
Today:
Clusters as multiple independent silos
Use Kubernetes federation from scratch
Simplified Setup: Federation Edition
25. Sophisticated Scheduling
Problem: Deploying and managing workloads on large,
heterogenous clusters is hard
Today:
Liberal use of labels (and keeping your team in sync)
Manual tooling
Didn’t you use Kubernetes to avoid this?
58. Sophisticated Scheduling: Taints/Toleration
Node 1
(Premium)
Node 2
(Premium)
Kubernetes Cluster
Node 3
(Regular)
I will fail to schedule
even though there’s a
spot for me.
Premium
Pod
Regular
Pod
Premium
Pod
SCENARIO: Reserved instances
74. Node 1
Pod
Kubernetes Cluster
Node 2 Node 3
Sophisticated Scheduling: Taints/Toleration
API
Server
Taint the
node
SCENARIO: Hardware failing (but not failed)
75. Node 1
Pod
Kubernetes Cluster
Node 2 Node 3
Sophisticated Scheduling: Taints/Toleration
API
Server
Taint the
node
SCENARIO: Hardware failing (but not failed)
76. Node 1
Pod
Kubernetes Cluster
Node 2 Node 3
Sophisticated Scheduling: Taints/Toleration
API
Server
SCENARIO: Hardware failing (but not failed)
Taint the
node
77. Node 1
Pod
Kubernetes Cluster
Node 2 Node 3
Sophisticated Scheduling: Taints/Toleration
Schedule new
pod and kill the
old one
API
Server
SCENARIO: Hardware failing (but not failed)
78. Node 1
Pod
Kubernetes Cluster
Node 2 Node 3
Sophisticated Scheduling: Taints/Toleration
Schedule new
pod and kill the
old one
New
Pod
API
Server
SCENARIO: Hardware failing (but not failed)
79. Node 1
Pod
Kubernetes Cluster
Node 2 Node 3
Sophisticated Scheduling: Taints/Toleration
Schedule new
pod and kill the
old one
New
Pod
API
Server
SCENARIO: Hardware failing (but not failed)
81. Sophisticated Scheduling: Forgiveness
Node 1 Node 2
Kubernetes Cluster
Node 3
It’s been 1m
since I heard
from Node 1
Pod
(t=5m)
Pod
(t=30m)
API
Server
SCENARIO: Supporting network failure
82. Sophisticated Scheduling: Forgiveness
Node 1 Node 2
Kubernetes Cluster
Node 3
It’s been 2m
since I heard
from Node 1
Pod
(t=5m)
Pod
(t=30m)
API
Server
SCENARIO: Supporting network failure
83. Sophisticated Scheduling: Forgiveness
Node 1 Node 2
Kubernetes Cluster
Node 3
It’s been 3m
since I heard
from Node 1
Pod
(t=5m)
Pod
(t=30m)
API
Server
SCENARIO: Supporting network failure
84. Sophisticated Scheduling: Forgiveness
Node 1 Node 2
Kubernetes Cluster
Node 3
It’s been 4m
since I heard
from Node 1
Pod
(t=5m)
Pod
(t=30m)
API
Server
SCENARIO: Supporting network failure
85. Sophisticated Scheduling: Forgiveness
Node 1 Node 2
Kubernetes Cluster
Node 3
It’s been 5m
since I heard
from Node 1
Pod
(t=5m)
Pod
(t=30m)
API
Server
SCENARIO: Supporting network failure
87. Sophisticated Scheduling: Forgiveness
Node 1 Node 2
Kubernetes Cluster
Node 3
Treat pod as
dead & schedule
new 5m Pod
Pod
(t=30m)
API
Server
Pod
(t=5m)
SCENARIO: Supporting network failure
88. Sophisticated Scheduling: Forgiveness
Node 1 Node 2
Kubernetes Cluster
Node 3
Treat pod as
dead & schedule
new 5m Pod
Pod
(t=30m)
API
Server
Pod
(t=5m)
SCENARIO: Supporting network failure
89. Sophisticated Scheduling: Forgiveness
Node 1 Node 2
Kubernetes Cluster
Node 3
Treat pod as
dead & schedule
new 5m Pod
Pod
(t=30m)
API
Server
Pod
(t=5m)
Pod
(t=5m)
SCENARIO: Supporting network failure
90. Sophisticated Scheduling: Forgiveness
Node 1 Node 2
Kubernetes Cluster
Node 3
Treat pod as
dead & schedule
new 5m Pod
Pod
(t=30m)
API
Server
Pod
(t=5m)
Pod
(t=5m)
SCENARIO: Supporting network failure
91. Sophisticated Scheduling: Forgiveness
Node 1 Node 2
Kubernetes Cluster
Node 3
It’s been 30m
since I heard
from Node 1
Pod
(t=30m)
API
Server
Pod
(t=5m)
Pod
(t=5m)
SCENARIO: Supporting network failure
93. Sophisticated Scheduling: Forgiveness
Node 1 Node 2
Kubernetes Cluster
Node 3
API
Server
Pod
(t=5m)
Treat pod as
dead & schedule
a new 30m pod
Pod
(t=5m)
Pod
(t=30m)
SCENARIO: Supporting network failure
94. Sophisticated Scheduling: Forgiveness
Node 1 Node 2
Kubernetes Cluster
Node 3
API
Server
Pod
(t=5m)
Treat pod as
dead & schedule
a new 30m pod
Pod
(t=5m)
Pod
(t=30m)
SCENARIO: Supporting network failure
95. Sophisticated Scheduling: Forgiveness
Node 1 Node 2
Kubernetes Cluster
Node 3
API
Server
Pod
(t=5m)
Treat pod as
dead & schedule
a new 30m pod
Pod
(t=5m)
Pod
(t=30m)
SCENARIO: Supporting network failure
96. Sophisticated Scheduling: Forgiveness
Node 1 Node 2
Kubernetes Cluster
Node 3
API
Server
Pod
(t=5m)
Treat pod as
dead & schedule
a new 30m pod
Pod
(t=5m)
Pod
(t=30m)
Pod
(t=30m)
SCENARIO: Supporting network failure
97. Sophisticated Scheduling: Forgiveness
Node 1 Node 2
Kubernetes Cluster
Node 3
API
Server
Pod
(t=5m)
Treat pod as
dead & schedule
a new 30m pod
Pod
(t=5m)
Pod
(t=30m)
Pod
(t=30m)
SCENARIO: Supporting network failure
98. Sophisticated Scheduling: Disruption Budget
Node 1 Node 2
Kubernetes Cluster
Node 3
Two Pod
Set (A)
API
Server
Time to upgrade
to Kubernetes
1.5!
Two Pod
Set (B)
SCENARIO: Cluster upgrades with stateful workloads
99. Sophisticated Scheduling: Disruption Budget
Node 1 Node 2
Kubernetes Cluster
Node 3
Two Pod
Set (A)
API
Server
Two Pod
Set (B)
“Evict
A!”
SCENARIO: Cluster upgrades with stateful workloads
100. Sophisticated Scheduling: Disruption Budget
Node 1 Node 2
Kubernetes Cluster
Node 3
Two Pod
Set (A)
API
Server
Two Pod
Set (B)
“Shut
down”
SCENARIO: Cluster upgrades with stateful workloads
101. Sophisticated Scheduling: Disruption Budget
Node 1 Node 2
Kubernetes Cluster
Node 3
Two Pod
Set (A)
API
Server
Two Pod
Set (B)
SCENARIO: Cluster upgrades with stateful workloads
102. Sophisticated Scheduling: Disruption Budget
Node 1 Node 2
Kubernetes Cluster
Node 3
Two Pod
Set (A)
API
Server
Two Pod
Set (B)
“Evict
B!”
SCENARIO: Cluster upgrades with stateful workloads
103. Sophisticated Scheduling: Disruption Budget
Node 1 Node 2
Kubernetes Cluster
Node 3
Two Pod
Set (A)
API
Server
Two Pod
Set (B)
“Sorry,
can’t!”
SCENARIO: Cluster upgrades with stateful workloads
104. Sophisticated Scheduling: Disruption Budget
Node 1 Node 2
Kubernetes Cluster
Node 3
Two Pod
Set (A)
API
Server
Two Pod
Set (B)
SCENARIO: Cluster upgrades with stateful workloads
105. Sophisticated Scheduling: Disruption Budget
Node 1 Node 2
Kubernetes Cluster
Node 3
Two Pod
Set (A)
API
Server
Two Pod
Set (B)
SCENARIO: Cluster upgrades with stateful workloads
106. Sophisticated Scheduling: Disruption Budget
Node 1 Node 2
Kubernetes Cluster
Node 3
Two Pod
Set (A)
API
Server
Two Pod
Set (B)
“Ok, now
Evict B!”
SCENARIO: Cluster upgrades with stateful workloads
107. Sophisticated Scheduling: Disruption Budget
Node 1 Node 2
Kubernetes Cluster
Node 3
API
Server
Two Pod
Set (B)
“OK!”
Two Pod
Set (A)
SCENARIO: Cluster upgrades with stateful workloads
108. Sophisticated Scheduling: Disruption Budget
Node 1 Node 2
Kubernetes Cluster
Node 3
API
Server
Two Pod
Set (B)
Two Pod
Set (A)
“Shutdown”
SCENARIO: Cluster upgrades with stateful workloads
109. Sophisticated Scheduling: Disruption Budget
Node 1 Node 2
Kubernetes Cluster
Node 3
API
Server
Two Pod
Set (B)
Two Pod
Set (A)
SCENARIO: Cluster upgrades with stateful workloads
110. Network Policy
Problem: Network policy is complicated!
Today:
Use VM tooling to support security (but limit VM utilization)
Managing port level security
Proxy-ing everything
127. Kubernetes Cluster
Network Policy Object
SCENARIO: Two-tier app needs to be locked down
VM 1 VM 2 VM 3
“Green” can talk
to “Red”
✓
✓
128. Problem: I need to deploy complicated apps!
Today:
Manually deploy applications once per cluster
Manually publish global endpoints and load balance
Build a control plane for monitoring application
Helm
143. Accelerating Stateful Applications
Container-optimized servers for
compute and storage
Management of storage and data for
stateful applications on Kubernetes
Management of Kubernetes at
enterprise scale
+
Automated Stateful Apps on K8S
148. What’s Next
Nothing!*
* for large values of “Nothing”
Bringing many features from alpha to beta & GA, including:
Federated deployments and daemon sets
Improved RBAC
StatefulSet upgrades
Improved scaling & etcd 3
Easy cluster setup for high availability configuration
Integrated Metrics API
149. Kubernetes is Open
• open community
• open design
• open source
• open to ideas
Twitter: @aronchick
Email: aronchick@google.com
• kubernetes.io
• github.com/kubernetes/kubernetes
• slack.kubernetes.io
• twitter: @kubernetesio