Building Highly Available Services using Kubernetes

1. Google Cloud Platform
Building Highly Available Services using
Kubernetes
Openstack Days Silicon Valley
Aug 9-10
Allan Naim <anaim@google.com>
Product Manager - Kubernetes and Container Engine
@allannaim

2. Google Cloud Platform
Kubernetes
Greek for “Helmsman”; also the root of the
words “governor” and “cybernetic”
• Manages container clusters
• Inspired and informed by Google’s
experiences and internal systems
• Supports multiple cloud and bare-metal
environments
• Supports multiple container runtimes
• 100% Open source, written in Go
Manage applications, not machines

3. Google Cloud Platform
kubelet
UI
kubeletCLI
API
users master nodes
Single Kubernetes Cluster
etcd
kubelet
scheduler
controllers
apiserver

4. Google Cloud Platform
UI
All you really care about
API
Container
Cluster

5. @briandorsey
Pods
Small group of containers & volumes
Tightly coupled
• same node
The atom of cluster scheduling &
placement
Each pod has its own IP address
• shared namespace: share IP address &
localhost
Ephemeral
• can die and be replaced
Example: data puller & web server
Pod
File Puller Web Server
Volume
Consumers
Content
Manager

6. Google Cloud Platform
Volumes
Pod-scoped storage
Support many types of volume plugins
• Empty dir (and tmpfs)
• Host path
• Git repository
• GCE Persistent Disk
• AWS Elastic Block Store
• Azure File Storage
• iSCSI
• Flocker
• NFS
• vSphere
• GlusterFS
• Ceph File and RBD
• Cinder
• FibreChannel
• Secret, ConfigMap,
DownwardAPI
• Flex (exec a binary)
• ...

7. Google Cloud Platform
Arbitrary metadata
Attached to any API object
Generally represent identity
Queryable by selectors
• think SQL ‘select ... where ...’
The only grouping mechanism
• pods under a ReplicationController
• pods in a Service
• capabilities of a node (constraints)
Labels

8. Google Cloud Platform
ReplicaSets*
A simple control loop
Runs out-of-process wrt API server
One job: ensure N copies of a pod
• grouped by a selector
• too few? start some
• too many? kill some
Layered on top of the public Pod API
Replicated pods are fungible
• No implied order or identity
* The evolution of ReplicationControllers
ReplicaSet
- name = “my-rc”
- selector = {“App”: “MyApp”}
- template = { ... }
- replicas = 4
API Server
How
many?
3
Start 1
more
OK
How
many?
4

9. @briandorsey
Services
A group of pods that act as one == Service
• group == selector
Defines access policy
• only “load balanced” for now
Gets a stable virtual IP and port
• called the service portal
• also a DNS name
VIP is captured by kube-proxy
• watches the service constituency
• updates when backends change
Hide complexity - ideal for non-native apps
Portal (VIP)
Client

10. Customer pain points
● High Availability
○ mitigate zone/provider outages
● Geographic scaling
○ low latency local serving
● Application Migration
○ Vendor lock-in
○ Automating migrations & rollbacks
● Capacity Overflow
○ Utilization / Cost
○ Performance
● Policy Enforcement
○ Data must be stored and processed within
specified political jurisdictions
Cross-cluster
Load Balancer
Your
paying
customer
Cluster 1
Cluster 2
Cluster 3

11. UI
CLI
API
Control Plane Clusters
Kubernetes Federation
API
Users
Kubernetes on
Kubernetes on
Kubernetes on
Premise
Federation

12. Create the Kubernetes Clusters
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API API API API
API CLI
UI

13. Configure the Federated Control Plane
● Federation:
○ Namespace
○ API Server Service with public VIP
○ API Server Deployment Pod with 2 replicas
○ Controller Manager Pod with 1 replica
○ Database key/value store
Deploy the Federated Control Plane on an existing
Kubernetes cluster
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
--context=federation-cluster

14. Add Cluster(s) to Federation
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API API API API
Federation Control Plane
kubectl --context=federation-cluster create -f clusters/gce-asia-east1.yaml
apiVersion: federation/v1beta1
kind: Cluster
metadata:
name: gce-asia-east1
spec:
serverAddressByClientCIDRs:
- clientCIDR: "0.0.0.0/0"
serverAddress: "https://xxx.xxx.194.68"
secretRef:
name: gce-asia-east1CLI

15. Deploying a Federated Service
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API API API API
Federation Control Plane
API CLI
kubectl --context=federation-cluster create -f
services/nginx.yaml
NGINX NGINX NGINX NGINX

16. Each Service Shard is exposed via External Load Balancer
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
Federation Control Plane
API CLI
NGINX NGINX NGINX NGINX

17. Deploy Federated Service Backends
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
DNS
NGINX NGINX NGINX NGINX
DNS DNS DNS

18. Cross Cluster Service Discovery (External Request)
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
DNS
NGINX NGINX NGINX NGINX
DNS DNS DNS
External DNS
nginx.default.federation.svc.federation.com
C3 C4
C3 C4
1
2
3
4

19. Cross Cluster Service Discovery (External Request)
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
DNS
NGINX NGINX NGINX NGINX
DNS DNS DNS
C3 C4
X
External DNS

20. Confidential & ProprietaryGoogle Cloud Platform 20Updated 2016-08-03 (Dataproc HA schedule update), douglasdollars@
Want to learn more?
● Kubernetes Cluster Federation Sneak Peak (link)
● Kubernetes Cluster Federation using GKE (link)
● Cluster Federation Admin Guide (link)
● Cross Cluster Service Discovery Deployment Guide (link)
● Cross Cluster Services - Achieving Higher Availability for your
Kubernetes Applications (link)
Also,
● Participate with us on the Kubernetes #sig-federation
● Post issues or feature requests on GitHub
● Join us in the #federation channel on Slack