3. Google Cloud Platform 3
The Anatomy of a Large-Scale Hypertextual Web Search Engine
1996, Sergey Brin and Lawrence Page
Computer Science Department, Stanford University, Stanford,
CA 94305
13. Google Cloud Platform 13
Hypervisor
Guest environment
app code
libraries
guest kernel
This is vanilla virtualization
14. Google Cloud Platform 14
machine image
locked into a
platform
It has downsides:
Not portable, and opaque
Hypervisor
Guest environment
app code
libraries
guest kernel
15. Google Cloud Platform 15
It has downsides:
No Isolation
Hypervisor
Guest environment
app code
libraries
guest kernel
dependency???
app code
16. Google Cloud Platform 16
It has downsides:
Little Reuse
Hypervisor
Guest environment
app code
libraries
guest kernel
Guest environment
app code
libraries
guest kernel
Guest environment
app code
libraries
guest kernelredundant
17. Google Cloud Platform 17
Containers create
a better abstraction layer
Hypervisor
Guest environment
app code
libraries
guest kernel
cut here
19. Google Cloud Platform 19
• Lightweight Linux environment
• Hermetically sealed,
deployable application
• Introspectable, runnable artifact
• Recently popularized by Docker
Containers
20. Google Cloud Platform 20
Resource
isolation
Predictability
Quality of Service
Batch vs.
interactive
Containers
at Google
Resource
accounting
Efficient overcommit
21. Google Cloud Platform 21
Containers at Google
We launch over
2 Billion
containers per week.
Google Cloud Platform 21
22. Google Cloud Platform 22
Repeatable, runnable
artifact
Loosely coupled
Static application
environment
Images by Connie Zhou
● Portability
● Develop here, run
there
● Pick your cloud
solely on its merits
● Easier to build and
manage
● Compose
applications from
micro-services
● Mix in and extend
third party services
● Reliable
deployments
● No stress
deployment and
update
Why do developers care?
23. Google Cloud Platform 23
Highly automatable = path to active management
● Efficiency: optimized packing, better scaling
● Performance: active environment tuning
● Continuous integration: easy and reliable
● Robustness: active monitoring, self healing
Why do developers care?
24. Google Cloud Platform 24
Containers have taken the world by
storm!
• 2013: LXC is a niche technology
• 2016: “Container” wins buzzword bingo
It’s a huge paradigm shift
• VMs are out, containers are in
It’s happening way faster than
anyone could have predicted
• Adoption is up 5x in 12 months1
The right tech at the right time
1: https://www.datadoghq.com/docker-adoption/
27. Google Cloud Platform 27
Greek for “Helmsman”:
the root of the word “governor” and “cybernetic”
● Container orchestrator
● Builds on Docker containers
○ also supporting other container technologies
● Multiple cloud and bare-metal environments
● Supports existing OSS apps
○ cannot require apps becoming cloud-native
● Inspired and informed by Google’s
● experiences and internal systems
● 100% Open source, written in Go
Let users manage applications, not machines
κυβερνήτης
33. Google Cloud Platform 33
Pods
Small group of containers & volumes
Tightly coupled
same node
The atom of cluster scheduling &
placement
Each pod has its own IP address
shared namespace: share IP address &
localhost
Ephemeral
can die and be replaced
Example: data puller & web server
Pod
File Puller Web Server
Volume
Consumers
Content
Manager
34. Google Cloud Platform 34
Volumes
Pod-scoped storage
Support many types of volume plugins
Empty dir (and tmpfs)
Host path
Git repository
GCE Persistent Disk
AWS Elastic Block Store
Azure File Storage
iSCSI
Flocker
NFS
vSphere
GlusterFS
Ceph File and RBD
Cinder
FibreChannel
Secret, ConfigMap,
DownwardAPI
Flex (exec a binary)
...
35. Google Cloud Platform 35
ReplicaSets*
A simple control loop
Runs out-of-process wrt API server
One job: ensure N copies of a pod
grouped by a selector
too few? start some
too many? kill some
Layered on top of the public Pod API
Replicated pods are fungible
No implied order or identity
* The evolution of ReplicationControllers
ReplicaSet
- name = “my-rc”
- selector = {“App”: “MyApp”}
- template = { ... }
- replicas = 4
API Server
How
many?
3
Start 1
more
OK
How
many?
4
36. Google Cloud Platform 36
Services
A group of pods that act as one == Service
group == selector
Defines access policy
only “load balanced” for now
Gets a stable virtual IP and port
called the service portal
also a DNS name
VIP is captured by kube-proxy
watches the service constituency
updates when backends change
Hide complexity - ideal for non-native apps
Portal (VIP)
Client
37. Google Cloud Platform 37
Customer pain points
High Availability
mitigate zone/provider outages
Geographic scaling
low latency local serving
Application Migration
Vendor lock-in
Automating migrations & rollbacks
Capacity Overflow
Utilization / Cost
Performance
Policy Enforcement
Data must be stored and processed within
specified political jurisdictions
Cross-cluster
Load Balancer
Your
paying
customer
Cluster 1
Cluster 2
Cluster 3
38. Google Cloud Platform 38
UI
CLI
API
Control Plane Clusters
Federation
API
Users
Kubernetes on
Kubernetes on
Kubernetes on
Premise
Federation
39. Google Cloud Platform 39
Create the Kubernetes Clusters
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API API API API
API CLI
UI
40. Google Cloud Platform 40
Deploy the Federated Control Plane on an existing Kubernetes cluster
● Federation:
○ Namespace
○ API Server Service with public VIP
○ API Server Deployment Pod with 2 replicas
○ Controller Manager Pod with 1 replica
○ Database key/value store
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
--context=federation-cluster
41. Google Cloud Platform 41
Add Cluster(s) to Federation
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API API API API
Federation Control Plane
kubectl --context=federation-cluster create -f clusters/gce-asia-east1.yaml
apiVersion: federation/v1beta1
kind: Cluster
metadata:
name: gce-asia-east1
spec:
serverAddressByClientCIDRs:
- clientCIDR: "0.0.0.0/0"
serverAddress: "https://xxx.xxx.194.68"
secretRef:
name: gce-asia-east1CLI
42. Google Cloud Platform 42
Deploying a Federated Service
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API API API API
Federation Control Plane
API CLI
kubectl --context=federation-cluster create -f
services/nginx.yaml
NGINX NGINX NGINX NGINX
43. Google Cloud Platform 43
Each Service Shard is exposed via External Load Balancer
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
Federation Control Plane
API CLI
NGINX NGINX NGINX NGINX
44. Google Cloud Platform 44
Deploy Federated Service Backends
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
DNS
NGINX NGINX NGINX NGINX
DNS DNS DNS
45. Google Cloud Platform 45
Cross Cluster Service Discovery (External Request)
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
DNS
NGINX NGINX NGINX NGINX
DNS DNS DNS
External DNS
nginx.default.federation.svc.federation.com
C3 C4
C3 C4
1
2
3
4
46. Google Cloud Platform 46
Cross Cluster Service Discovery (External Request)
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
DNS
NGINX NGINX NGINX NGINX
DNS DNS DNS
C3 C4
X
External DNS
47. Google Cloud Platform 47
Q2 2016 Q3 2016 Q4 2016 (**) 2017 and beyond (**)
Federation Control plane (beta1)
- Public facing Multi
Region/Could - Cross
Cluster Service Discovery
(internal/external DNS)
- Service object API
support
Federation Control plane (beta2)
- Multi Region Ingress(L7)
Load Balancing across
clusters for GCP only
Federation Control plane (beta3)
- Cross provider Multi
Region Ingress(L7) Load
Balancing
- GKE IAM Integration
Federation Control plane (GA)
- Non Public Facing Cross
Cluster Service Discovery
- Full support for Kubernetes
API objects
- UI support for Federated
Clusters
- Federated IAM
- GKE Hosted Control Plane
Updated 2016-08-03 (Dataproc HA schedule update), douglasdollars@
(**) - this is a proposed roadmap. Items listed here are subject to change.
Cluster Federation roadmap
48. Google Cloud Platform 48
● Kubernetes Cluster Federation Sneak Peak (link)
● Kubernetes Cluster Federation using GKE (link)
● Cluster Federation Admin Guide (link)
● Cross Cluster Service Discovery Deployment Guide (link)
● Cross Cluster Services - Achieving Higher Availability for your Kubernetes Applications (link)
Also,
● Participate with us on the Kubernetes #sig-federation
● Post issues or feature requests on GitHub
● Join us in the #federation channel on Slack
Want to learn more?
50. Google Cloud Platform 50
Stateful services (1.2)
- ReplicaSets ensure N replicas of a pod
template exist
- Only attribute that differs between pods is the
name
- Pods have no stable network identity other
than an assigned pod IP
- e.g. no fixed and predictable hostname
- Best for ‘shared-nothing, zero-coordination
systems’
Photo credit: http://delectabledeliciousness.blogspot.co.uk
51. Google Cloud Platform 51
Persistent Volume Framework
Administrator
Developer
Developer
owned
PV PV PV
Pool of Persistent Volumes
Registers PVs in the pool
claim
claim
Claims a PV from the pool
References claim in pod
Pod
PV
Administrator
owned
52. Google Cloud Platform 52
Init Containers
● Alpha feature
● Run container(s) before the long-running main pod
container(s) are launched
● Executed in sequential order
● Only if init containers succeed do the main container(s)
start
● Use cases:
○ download files / clone repos
○ write out config files
○ etc
init-1 init-2
main-container
53. 53
● New Alpha resource (apps/v1alpha1)
○ Not available on GKE
● Stable and unique identity associated with that
instance of the storage
● Consistent network identity
● Predictable number of instances to ensure that
systems can form a quorum
● Ability to migrate from node to node with stable
network identity (DNS name)
● Scale up/down in a controlled fashion
PetSet
54. Google Cloud Platform 54
PetSet
PetSet
pet-0.
pet.default...
PetSet
Controller
PV-0 PVC-0
API Server
Watches
Creates claim from template
BindsMounts
Creates
and waits
until ready
Service
1:1 mapping
55. Google Cloud Platform 55
PetSet
PetSet
pet-0.
pet.default...
PetSet
Controller
PV-0 PVC-0
API Server
Service
pet-1.
pet.default... PV-1 PVC-1
pet-2.
pet.default... PV-2 PVC-2
57. Google Cloud Platform 57
Top 0.01% of all
Github projects
1200+ external
projects based on
Kubernetes
800+
unique contributors
Community
30,000+ commits
Broad industry support
58. Google Cloud Platform 58
Kubernetes Adoption
Startup to Enterprise Adoption
● 2,000+ external projects based on Kubernetes
● Usage spans all platforms, from on-prem to all public clouds
● Google Container Engine (GKE) usage is doubling every 90 days
59. Google Cloud Platform 59
Kubernetes & Container Engine
Containers are powering a revolution
For Developers:
Easy dev-test environments
Easy to compose micro-services
Update your cluster with one command
For Ops/IT:
Kubernetes is managed for you
Deploying becomes file-copy
Highly portable
Google Cloud Platform 59
60. Google Cloud Platform 60
Thank you
#kubernetes @craigbox
Thanks to Matt Bates from jetstack.io
and Steve Watt from Red Hat
