SpringOne Platform 2018 Securing Pivotal Cloud Foundry by Regularly Rebuilding - Lance Rochelle, Matt Saner

1. Security Through Repaving
Lance Rochelle – Product Owner (Pivotal Cloud Foundry)
August 2018
Public Information
© 2018 Wells Fargo Bank, N.A. All rights reserved.

2. 22
Introductions
Lance Rochelle

3. 33
History of Patching: A Comedy or a Tragedy
~1999
•“Hey, let’s see how long we can keep these servers up and use uptime as a
benchmark for stability.” - Super 1337 SysAdmin
~2002
•“We should probably patch. Once a year seems like a pretty good idea, that way we
know the server can survive a reboot” – Some CIO, probably
~2004
•“You know, this patching thing isn’t so bad. I bet we could do it twice a year.” – The
Business, begrudgingly
~2007
•“About once a quarter there is a new Operating System kernel we should patch to the
new kernel a few months after they come out, let’s do once a quarter” – OS
Engineers, anxious to engineer
~2012
•“Security would like us to patch ONCE A MONTH?! Who does that, whyyyyyyyy…“ –
Everyone and their brother
~2018
•“You know what would be cool, what if we could blow away the entire
environment every day and rebuild it from scratch?” – A super smart person

4. 44
What are the Primary Threats and Concerns?
Advanced Persistent Threats.APT
The state of the environment changing over
time
Configuration
Drift
Exploitable “things” that you don’t want in
your environmentVulnerabilities
Unpatched, out of date, and unmaintained
software
Technical
Debt

5. 55
What is “Repaving”?
Principles…
1) Patch early, patch often
2) Gold Images
3) Deploy via Automation
4) Aim for “Cattle” not “Pets”
5) Redeploy Often - Even when you don’t think you
have to or need to

6. 66
Automate Platform Patching – BOSH with PCF
BOSH is an open source project that unifies release engineering, deployment, and
lifecycle management of small and large-scale cloud software.
BOSH can provision and deploy software over hundreds of virtual appliances and can
also perform monitoring, failure recovery, and software updates with zero-to-minimal
downtime.
While BOSH was developed to deploy Cloud Foundry PaaS, it can also be used to
deploy almost any other software.
BOSH is particularly well-suited for large distributed systems.
In addition, BOSH supports multiple Infrastructure as a Service (IaaS) providers
(VMware vSphere, Google Cloud Platform, Amazon Web Services EC2, Public Azure
and some versions of OpenStack)

7. 77
Application Deployment Process
https://docs.cloudfoundry.org/concepts/images/app_push_flow_diagram_diego.png

8. 88
Security Threats are Increasing at a Rapid Rate
CVE = Common Vulnerabilities and Exposures.
The total number of vulnerabilities identified in the
wild:
• 2015 = 6480
• 2016 = 6447
• 2017 = 14714
• 2018 ≈ 19500+ (estimate from Jan to Aug)
The only way to keep up with threats is to
automate all updates.
https://www.cvedetails.com/browse-by-date.php
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
jan feb mar apr may jun jul aug sep oct nov dec jan feb mar apr may jun jul
By Month and CVSS Score Jan 2017 to Jul 2018
0-2.9 3-6.9 7-10+ Total
0
5000
10000
15000
20000
25000
CVE reported

9. 99
Stemcell Scanning is Still Important!
PivNet
Scanning Agent
IaaS (AWS/GCP/vSphere/Azure/OpenStack)
Continuous Integration Pipeline
Artifact Repository
Representative Cluster
1
2
3
5
7

10. 1010
Platform Repaving with BOSH and PCF
Pivotal Cloud Foundry – Elastic Runtime
Phase 3
Applications Reside on the Diego Cells
Application 2
Instance 3
Application 1
Instance 1
Application 2
Instance 1
Application 1
Instance 2
Application 2
Instance 3
Application 1
Instance 3
Diego Cell
Diego Cell
Diego Cell Diego Cell Diego Cell
Diego Cell
Diego Cell
Diego Cell
Diego Cell
Phase 1
Consul
NATS
ConsulConsul
NATS
Diego BBSDiego BBSDiego BBS
UAA UAA
Cloud Controller
Cloud Ctl WorkerCloud Ctl Worker
Clock Global
Cloud Ctl Worker
Diego Brain Diego BrainDiego Brain
TCP Router TCP Router TCP Router
Doppler ServerDoppler Server Doppler Server
Doppler Server Doppler Server Doppler Server
Logregator Logregator Logregator
Logregator Logregator Logregator
Logregator LogregatorLogregator
Cloud ControllerCloud Controller
Phase 2
Virtual RouterVirtual RouterVirtual Router
Virtual Router Virtual Router Virtual Router
 Key Point: All servers are immutable
Phase 1
All Virtual Appliances are recreated
with a new image based on a
concurrency value.
Phase 2
Traffic is drained automatically from
the virtual appliances then the
virtual appliance is recreated with a
new image and assigned the role of
the virtual router.
Phase 3
Applications Instances are migrated
from a currently running Diego Cell
to another Diego Cell.

11. 1111
Thank you!
Q&A