anynines ran a public PaaS located in a German datacenter based on Cloud Foundry. In more than 12 months of running a Cloud Foundry PaaS man lessons about security, high availability, open stack and many other exciting topics have been learned. See how Bosh can be used and how it shouldn't be used. Learn how to perform Cloud Foundry upgrades and read how to harden Cloud Foundry by adding more fault tolerance with pacemaker.

1. Running Cloud Foundry
An Experience Report

2. About this talk

3. • Receive an opinion about running Cloud
Foundry (CF)
• How to shoot your own leg with CF and
overcommitment settings
• How to perform CF updates
• How to harden CF
• Wise words about CF services

4. Introduction

5. about.me/fischerjulian

6. Running a public
Cloud Foundry
for more than a year.

7. It works.

8. In order to run
Cloud Foundry smoothly …

9. … refer to the package leaflet for
risks and side effects and consult
pivotal, cloudcredo or anynines.“

10. The details

11. The anynines Stack

12. Cloud Foundry
OpenStack
VMware
Hardware

13. We migrated from a
Rented VMware to a
self-hosted OpenStack.

14. For more details on this:
http://rh.gd/a9vmw2sos

15. Proof point made…

16. Cloud Foundry
saves investments into
software development
by being
infrastructure agnostic.

17. Running Cloud Foundry.
What happened.

18. Security Issues

19. • Pivotal informs partners early about
issued
• Usually along with fixes

20. OpenStack Issues

21. • Ext4 vs. Ext3
• DEA MTU
• rsyslogd command not found

22. CF Gotchas

23. DEA evacuate & Bosh
timeout race-condition

24. • Removing a DEA
→ Apps will be evacuated
→ DEA will be stopped
• Bosh deployment will fail when
evacuation takes longer than the Bosh
timeout
• Set your Bosh timeout accordingly!

25. DEA over-commitment

26. Default overcommitment
factor = 4

27. RAM peaks may cause
random errors

28. • Failures during staging
• Random application crashes
• No meaningful log information

29. Reducing over-commitment

30. • Native strategy
• Reduce over-commitment factor
• Bosh deploy

32. • 8 GB VM, OC factor 4
→ Announces 32 GB (V)RAM
• 8 GB VM, OC factor 2
→ Announces 16 GB (V)RAM
• When evacuating a 32 GB (V)RAM host,
another 32 GB (V)RAM host will be
preferred (more free space)

33. Evacuation Wave

34. 1 GB
1 GB
1 GB
1 GB

35. = maximum impact on
running apps!

36. New DEAs (OC 2) will
receive apps when old DEAs
(OC 4) have been stopped.

37. Hints

38. • Create 2nd resource pool for new DEAs
• Deploy the 2nd resource pool before
startup to stop old DEAs
• (-) Needs more resources
• (+) Smoother transition

39. Updating Cloud Foundry

40. Required:
Staging System

41. • Structurally identical
• Less VMs

42. 1.
Determine new features
since last release

43. 2.
Study
deployment manifest
changes

44. 3.
Apply
deployment manifest
changes

45. 4.
First staging attempt

46. 5.
Debug and Fix it!

47. 6.
Simulate the live-upgrade

48. 7.
Schedule maintenance on
status.anynines.com

49. 8.
Perform the upgrade
and cross fingers.

50. CF Hardening

51. Accept that VMs are
ephemeral

52. VM Failover Strategies

53. Resurrect

54. • Monitor VM
• Re-Build VMs automatically
• e.g. using Cloud Foundry Bosh
• + Easy
• - Takes long (minutes not seconds)
• - Open Stack doesn’t release persistent
disks automatically

55. Failover to Standby VM

56. Distribute CF components
across availability zones

57. • Build disjunct networks, racks, etc.
• Each disjunct zone = availability zone
• Tell your IaaS about availability zones
• On provision choose the AZ
• Build Bosh releases accordingly

58. • Provide stand-by VM
• Monitor VM and perform failover
• IP failover using Pacemaker
• + Fast failover (seconds)
• - Pacemaker not easy to use (& boshify)
• - Increased resource usage by stdby
VM(s)

59. • 2 * UAA
• 2 * CC
• 2 * n * DEAs
• 2 * Health Manager
• …

60. UAA & CC DB
=
SPOF

61. HA Postgres

62. • UAA and Cloud Controller database
• Single point of failure for Cloud Foundry

63. • Postgres not inherently clusterable >
failover with standby vm
• Master/slave replication
• Pacemaker/corosync
• IP-Failover using NIC-reattachment

64. That’s half way towards a
PostgreSQL CF Service

65. • Add a V2 Service Broker
• Add a provisioning logic
• Provision 2-node db cluster on
cf create service postgres medium-cluster

66. Services

67. “The best way to find yourself is to lose
yourself in the service of others.”
― Mahatma Gandhi

68. Wardenized Services
(community services)
are cute for pet projects.

69. Not suitable for production.

70. • Implementations are outdated
• One size doesn’t fit all!

71. No production CF without
high quality services.

72. CF Service Design

73. • Use clusterable services if possible
• Implement automatic failover if not
• Autoprovisioning using Bosh
• Organize self-healing
• (Semi-)Automatic recovery from
degraded mode

74. Summary

75. • Bosh & the CF release are powerful, yet
you can cut yourself.
• HA Services are very necessary.
• CF is ready to be used in production.

76. Questions?

77. Thank you!