This document discusses running Cloud Foundry on Kubernetes to provide highly available cloud platforms. It begins with an overview of cloud computing models and introduces Cloud Foundry. It then discusses deploying Cloud Foundry using Kubernetes primitives like pods, services, and stateful sets for high availability. The document demonstrates how to install Cloud Foundry on Kubernetes using Helm charts and configure for high availability. It shows the components have been made highly available to prevent downtime during failures or upgrades. Finally, it provides a demo of deploying a sample application on Cloud Foundry on Kubernetes under chaotic conditions to showcase the high availability.

1. Highly Available Cloud Foundry
on Kubernetes
Open Source Data Center Conference 2018
Cornelius Schumacher <cschum@suse.com>
Distinguished Engineer
SUSE Linux

2. 2
Where are we?

3. 3
Cloud Computing 101
Application
Runtime
Userland
Kernel
Virtualization
Hardware
Infrastructure
(IaaS)
Platform
(PaaS)
Software
(SaaS)
Application
Runtime
Userland
Kernel
Virtualization
Hardware
Application
Runtime
Userland
Kernel
Virtualization
Hardware
Application
Runtime
Userland
Kernel
Virtualization
Hardware
Container
(CaaS)
Managed by user:
Managed by provider:

4. 4
Cloud Foundry

6. Operations
●
Failure & recovery
●
Scaling
●
Security & patching
●
Platform upgrades
●
Zero downtime

7. 7
Cloud Foundry
User Applications
Services
Infrastructure

8. 8
Cloud Foundry
User Applications
Services
Containers
IaaS
Buildpack
Operating System
Virtual Machines
Operating System
Operating System

9. Cloud
Foundry
Application
Runtime

10. Operating System
Buildpack
User application
Container

11. 11
Application lifecycle management

12. 12
User Applications
Containers
Buildpack
Operating SystemOperating System
Stack

13. 13
Staging the runtime
Container
Operating SystemOperating System
Container
Buildpack
Operating SystemOperating System
User Applications
Container
Buildpack
Operating SystemOperating System
OS Base
Container
Buildpack builder Runtime

14. 14
Benefits
●
Separation of concerns
●
Standardized operation
●
Transparency
●
Manage security of OS and middleware independent of application

15. 15
Cloud Foundry Community

18. https://cloudfoundry.org/projects/

19. Extreme Programming

20. 20
Deploying Cloud Foundry with BOSH

22. IaaS
Virtual Machines
Operating System
BOSH

23. 23
BOSH
Deployment:
●
Upload stemcell
●
Upload release
●
Deploy manifest

25. cf-deployment

26. Footprint (HA on OpenStack)
●
22 virtual machines
●
118 GB of RAM
●
56 vCPUs
●
1 TB of storage
●
+ workload specific needs
Source: http://docs.pivotal.io/pivotalcf/2-0/customizing/openstack.html

27. 27
Kubernetes

28. 28
What is Kubernetes?
“Kubernetes is an open-source system for
automating deployment, scaling, and management
of containerized applications.”
– kubernetes.io
●
Nodes, Pods, Containers
●
Network, Storage, Services
●
YAML files to describe desired state
●
API driven service to manage state

29. 29
Platform
(PaaS)
Application
Runtime
Userland
Kernel
Virtualization
Hardware
Application
Runtime
Userland
Kernel
Virtualization
Hardware
Container
(CaaS)
Container Container

30. 30
Running Cloud Foundry on Kubernetes

31. 31
Cloud Foundry
User Applications
Services
Containers
IaaS
Buildpack
Operating System
Virtual Machines
Operating System
Operating System

32. 32
Cloud Foundry
User Applications
Services
Containers
IaaS
Buildpack
Operating SystemOperating System
Containers
Operating System
Kubernetes
CaaS Bare Metal

33. 33
OS Base Image Layer
Stemcell Layer
Operating System (built with Kiwi)
Fissile stemcell
container image
Cloud Foundry
container image
Cloud Foundry release Fissile
Building Cloud Foundry container images
Helm
charts

34. The stemcell pipeline
openSUSE Leap Base OS Image
BOSH stemcell
Fissile container
base image
Kiwi
BOSH Linux
Stemcell Builder
Product Pipeline
(Concourse)

35. 35
Installing with Helm
vi config.yaml
helm repo add suse <suse-helm-repo>
helm install suse/uaa –namespace uaa –values config.yaml
helm install suse/cf –namespace scf –values config.yaml
helm install suse/console –namespace stratos –values config.yaml

36. Footprint (HA on Kubernetes)
●
6 Kubernetes nodes (3 masters, 3 workers)
●
72 GB of RAM
●
300 GB of disk space
●
+ workload specific needs
Source: https://www.suse.com/documentation/cloud-application-platform-1/index.html

37. 37
Kubernetes
User Applications
Containers
Buildpack
Operating SystemOperating System
User Applications
Containers
Buildpack
Operating SystemOperating System
Kubernetes
●
Nested containers
●
Two schedulers: Diego and Kubernetes
●
Reconcile: https://github.com/cloudfoundry-incubator/eirini

38. 38
High Availability

39. Internals
Elastic Runtime
Diego Cell
Diego Cell
Cloud Foundry Services
Diego
Scheduler
APIs and management
Logging
Routing & Access
Services
diego-api
Diego Cell
grootfs garden
runc
buildpack/docker
application
cf-usb
nfs-broker
loggregator
doppler
router
tcp-router
routing-api
api
cc-worker cc-clock
uaa
Internal
Storage
mysql nats
blobstore
cc-uploader
diego-access
diego-brain
diego-locket
mysql-proxy
post-deployment-setup
secret-generation syslog-adapter
syslog-rlp
syslog-scheduler

40. Internals
Elastic Runtime
Diego Cell
Diego Cell
The critical pieces
Diego
Scheduler
APIs and management
Logging
Routing & Access
Services
diego-api
Diego Cell
grootfs garden
runc
buildpack/docker
application
cf-usb
nfs-broker
loggregator
doppler
router
tcp-router
routing-api
api
cc-worker cc-clock
uaa
Internal
Storage
mysql nats
blobstore
cc-uploader
diego-access
diego-brain
diego-locket
mysql-proxy
post-deployment-setup
secret-generation syslog-adapter
syslog-rlp
syslog-scheduler

41. Component Flavors
Simply load balanced
• Cloud Controller
• Routers
• Cells
Things that cluster
• MariaDB
• Nats
Active/Passive
• Routing API
• Diego Database
router-235-xrf 1/1
router-235-rgh 1/1
nats-0 1/1
nats-1 1/1
nats-2 1/1
routing-api-147-ftl 1/1
routing-api-147-zbl 0/1

42. 42
HA: true

43. 43
High Availability
config.yaml:
sizing:
api:
count: 2
nats:
count: 3
diego_cell:
count: 3
...
●
Using Kubernetes primitives
●
Pods
●
Storage classes
●
Services
●
Stateful sets
●
Deployments
●
Liveliness/readiness probes
●
...

44. Kube Services
“A Kubernetes Service is an abstraction
which defines a logical set of Pods and a
policy by which to access them -
sometimes called a micro-service.“
“api.cf.svc.cluster.local:8500”

45. Kube Stateful Sets
“StatefulSets manage Pods that are based
on an identical container spec. However,
although their specs are the same, the Pods
in a StatefulSet are not interchangeable.
Each Pod has a persistent identifier that it
maintains across any rescheduling.“
“nats-1.cf.svc.cluster.local:4222”

46. Kube Probes
“The kubelet uses liveliness probes to
know when to restart a Container.“
“The kubelet uses readiness probes to
know when a Container is ready to start
accepting traffic.“

47. Exposing it using Helm
• Opinionated defaults that allow
you to go from basic to HA by
changing replica count
• Operator is shielded from the
complexity of managing each
HA component individually

48. What we’ve achieved
• We can horizontally scale the critical pieces to make
sure user applications suffer no downtime
• We can make all Cloud Foundry components HA, so
when upgrades or failures occur, all features are still
available, and the service doesn’t need to be degraded
• We can survive the chaos monkey!

49. 49
Demo

50. 50
https://www.youtube.com/watch?v=BZy1L4F66TQ
●
Full deployment of Cloud Foundry on Kubernetes
●
Running user application with 4 instances
●
A chaos monkey script killing random things every 3 minutes
●
Constant application and API monitoring
●
3 hours compressed to 5 minutes

51. 51

52. 52
54 killings
99.5 % application availability
99.8 % API availability

53. 53
What do we get?

54. 54
Operators
●
IT-as-a-service for (internal) customers
●
Automatic provisioning, deployment and scaling of applications
within existing IT constraints
●
Controlled configuration of the middleware (buildpack and service
control)
●
Similar environment that can be run in private or public cloud
●
No more “templating" of VMs required
●
Visibility into the management, monitoring, logging of systems,
users and applications

55. $ cf push
Application Developers

56. “We are able to build a product in the amount of
time it used to take to estimate and decide.”
—Opal Perry, Allstate, Cloud Foundry Summit 2017

57. 57
What are your questions?

58. References
● Cloud Foundry: https://cloudfoundry.org
● Cloud Foundry Documentation:
http://docs.cloudfoundry.org/
● Recordings of talks from last Cloud Foundry Summit:
https://www.youtube.com/playlist?
list=PLhuMOCWn4P9hsn9q-GRTa77gxavTOnHaa
● BOSH: https://bosh.io
● Kubernetes: https://kubernetes.io
● Helm: https://helm.sh
● SUSE Cloud Foundry: https://github.com/SUSE/scf