SpringOne Platform 2018 Developer Secure Containers for the Cyberspace Battlefield Chris Saunders, Pivotal; Jason Scanga, VMware

1. Developer Secure Containers for the
Cyberspace Battlefield
Chris Saunders-Pivotal Federal
Jason Scanga-VMware Federal

2. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Evolution of Military Tactics-Mobility
2

3. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/

4. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Evolution of Military Tactics-Protection and Security

5. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
We still build network security like a castle
Internet
VM VM
VM
VM
VM
VM
VM

6. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Evolution of Military Tactics-Mission Command
“The concept of mission command/command and control allows subordinates to be innovative and operate independently according to clear orders, commander’s
intent, and clearly articulated rules of engagement.” URBAN OPERATIONS Headquarters, Department of the Army Headquarters, United States Marine Corps

7. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Application Fielding With A DoD Customer
7
• Request for Environment Creation
• Systems and Network IT shops rack and stack infrastructure, install middleware, and supporting technologies
• Request ticket for DNS and firewall changes
• Testing for Verification & Acceptance
• Environment available for application hosting in production
• Fix issues/bugs,
• Open Change Control
• Change Control Approval is Given
• Implement Change Control
• Rinse and Repeat
Average Estimated Time: 3 months

8. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
How Can we Help DOD IT Evolve?

9. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Hardware
IaaS
Container Orchestrator
Application Platform
Serverless Functions
Strategic goal (portfolio optimization): Push as many
workloads as technically feasible to the top of the platform
hierarchy
Higher flexibility and less
enforcement of
standards
Lower development complexity
and higher operational
efficiency
Choose the right tool for the job

10. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Stability + Scalability
Deliver enterprise SLAs and breakthrough
operational efficiency
● Four layers of HA
● Zero downtime deployments
● Logging, metrics, and scaling
● Linux + Windows Server
HIGHLIGHTS HIGHLIGHTS HIGHLIGHTS
Typical Customer Outcomes with Pivotal Cloud Foundry
Security
Improve your security posture
with built-in capabilities
● “Secure by default” containers
● Full-stack support
● Rapid fixes to CVEs
● “Repair, repave, rotate”
Speed
Deploy new code
thousands of times a month
● Best platform for Spring Boot
● Container-ready
● Native Windows + .NET
● Full integrated with CI/CD

11. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
What is PKS?

12. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Kubernetes
Kubernetes is an open-source platform designed to automate deploying, scaling, and
operating application containers.
With Kubernetes, you are able to quickly and efficiently respond to customer demand:
https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
• Deploy your applications quickly and predictably.
• Scale your applications on the fly.
• Roll out new features seamlessly.
• Limit hardware usage to required resources only.
• Manage your applications like cattle instead of pets

13. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
> kubectl
Storage NetworkingCompute
Kubernetes Dashboard
Dev / Apps
IT / Ops
App User
Kubernetes is a Runtime for Containerized Workloads

14. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Storage NetworkingCompute
Dev / Apps
App User
IT / Ops
> kubectl
Kubernetes Dashboard
Load Balancing / Routing
Container Image
Registry
App Monitoring
App Logging
OS Updates
OS Images
K8S Updates
K8S Images
Log & Monitor
Recover & Restart
Backup & Restore
External
Data Services
Cluster Provisioning
Provision & Scale
Command Line /
API
Management
GUI
Monitoring GUI
...but Kubernetes alone is not enough for enterprises

15. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Storage NetworkingCompute
Dev / Apps
App User
IT / Ops
> kubectl
Kubernetes Dashboard
Load Balancing / Routing
Container Image
Registry
App Monitoring
App Logging
OS Updates
OS Images
K8S Updates
K8S Images
Log & Monitor
Recover & Restart
Backup & Restore
External
Data Services
Cluster Provisioning
Provision & Scale
Command Line /
API
Management
GUI
Monitoring GUI
...but Kubernetes alone is not enough for enterprises

16. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Storage NetworkingCompute
Pivotal Container Service (PKS) provides what’s missing
Dev / Apps
App User
IT / Ops
> kubectl
Kubernetes Dashboard
Load Balancing / Routing
Container Image
Registry
OS Updates
OS Images
K8S Updates
K8S Images
Log & Monitor
Recover & Restart
Backup & Restore
External
Data Services
Cluster Provisioning
Provision & Scale
App Logging
PKS Control Plane
GCP Service
Broker
> pks
Operations Manager
vRealize Operations*
*integration

17. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Security In PKS

18. Common CI/CD Pipelines for Federal
Dev on Low Side (AWS GovCloud)
End to
End
Tests
Package
Unit
Tests
Static
Analysi
s
Artifact
Repository
Version
Control
PCF Development
(Product Owner
Acceptance)
Dynamic
Analysis
Developers commit code to version control
Deploy
WWW/NIPR
Cross Domain Device (Coming soon) SIPR/JWICS
PCF Production
(Blue/Green Deploy)
Smoke
Tests
Accepted
?
Deploy
Artifact
Repository
End to End Tests (with real
endpoints e.g. UFMS Dev/Test)
PCF Staging (Integration
and Acceptance)
Deploy
Integrity
Check
Prod on High Side (SIPR/JWICS)
1
3
2
6
4
5
7

19. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
A new security patch is released for Kubernetes.
Pivotal releases a new CVE for PKS within a few hours.
The Platform Operator can then apply the CVE with no
platform downtime.

20. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
… or
...
Pivotal
NetworkConcourse
pipelinePlatform Ops
Execute
Verify pre-reqs
Verify current
install
Download
updated binaries
Rolling
Updates
Config
Update PKS

21. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Role-Based Access Control (RBAC)
LDAP/AD Integration
Image Vulnerability Scanning (Clair)
Notary Image Signing
Policy-Based Image Replication
Graphical User Portal & RESTful API
Image Deletion & Garbage Collection
Auditing
An enterprise-class registry server for Docker
images
Build Image
Push
Image
Scan
Image for
CVEs
Sign
Image
kubectl
run
Dev Team
Image
Registry
Clair Notary
R
B
A
C
UAA
Auth
R
E
PL

22. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
How do we bake in Security?

23. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
• Unified VM to Container Networking
• On-demand network virtualization
• Micro-segmentation
• Full Network Visibility
• Enterprise Support
• Pod-Level Container Networking
• Load Balancing
• Network Security policies
• Tenant - level isolation
• Unique logical switch per K8s namespace

24. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Multi-Tenancy
PKSControlPlane
Kubernetes cluster
Kubernetes cluster
Harbor
GCP SB
NSX-T
BOSH
Kubernetes cluster
Master
Worker
Worker
etcd
Worker
Master
etcd
Worker
How to isolate and secure access from different tenants?

25. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Deployment Topologies & Multi-Tenancy
Multi-cluster Single cluster
K8s Cluster A
K8s Cluster
BOSH
Namespace A
Namespace B
Namespace C
BOSH
K8s Cluster B
K8s Cluster C
NSX-T
cluster-based namespace-based
PKSControlPlane
PKSControlPlane

26. Confidential │ ©2018 VMware, Inc.
PKS Technical Overview – NSX-T
Physical Infrastructure
BOSH
NSX-T
Service
Broker(s)
vSANvSphere
etcd worker
Container
Registry
master etcd workermaster
PKS Control Plane
Kubernetes Cluster Kubernetes Cluster
Wavefront
by VMware
vRealize
Automation
vRealize
Log Insight
vRealize
Operations
vRealize
Network
Insight

27. Confidential │ ©2018 VMware, Inc.
Kubernetes NSX Topology
Namespace: foo Namespace: bar
NSX/ K8s topology
• Namespaces: NSX-T builds a separate network
topology per K8s namespace
• Pods: Every Pod has its own logical port on a NSX
logical switch dedicated to its namespace
• Nodes: Every Node can have Pods from different
Namespaces from different IP Subnets / Topologies
• Firewall: Every Pod has DFW rules applied on its
Interface
• Routing: High performant East/West and
North/South routing using NSX’s routing
infrastructure
• Visibility and troubleshooting: The full suite of
NSX-T troubleshooting tools works for pod
networking
• IPAM: NSX-T provides IPAM services enabling
policy-based dynamic IP allocation for all
Kubernetes components
Kubernetes NSX Topology
10.24.0.0/24 10.24.1.0/24 34.14.5.0/24

28. Confidential │ ©2018 VMware, Inc. 28
With most networking technologies, the source IP
of the traffic can't be mapped to the tenancy.
This is the biggest hurdle today to get K8s
integrated in enterprise IT environments

29. Confidential │ ©2018 VMware, Inc.
NSX-T PKS Integration – Namespace/Topology Mapping
admin@k8s-master:~$ kubectl create namespace foo
namespace ”foo" created
admin@k8s-master:~$ kubectl create namespace bar
namespace ”bar" created
admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foo
deployment "nginx-foo" created
admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bar
deployment "nginx-bar" created
Namespace: foo Namespace: bar
NSX / K8s topology
10.24.0.0/24 10.24.1.0/24 10.24.2.0/24
NAT boundary
K8s nodesK8s Masters

30. Confidential │ ©2018 VMware, Inc.
NSX-T & PKS Integration
• Micro-Segmentation in K8s: The data model to describe segmentation policies between Namespaces, and within namespaces is
called ’Network Policies’ and is released on Kubernetes 1.7 (Beta on 1.6)
Firewalling in Kubernetes
• NSX could utilize K8s Network Policies to define Dynamic Security
Groups & Policies.
• Capabilities are limited to K8s Network Policy capabilities.
K8s Network Policy
• Security Groups & Policies could be predefined on NSX. Labels are
used to specify Pods Membership
• Mapping of IP based groups, egress rules, VM based matching could
be available to be used in the policy definition
Pre-Defined Label based rules
• The NSX / K8s integration intends to support both the pre-defined label based rules and K8s network policy.
Firewalling in NSX / K8s

31. Confidential │ ©2018 VMware, Inc. 31
NSX-T & PKS Integration – Pods Micro-Segmentations
Option1: Predefined Label Based Rules
admin@k8s-master:~$ kubectl label pods nginx-foo-3492604561-nltrf secgroup=web -n foo
Pod "nginx-nsx-3492604561-nltrf" labeled
admin@k8s-master:~$ kubectl label pods nginx-bar-2789337611-z09x2 secgroup=db -n bar
pod "nginx-k8s-2789337611-z09x2" labeled
admin@k8s-master:~$ kubectl get pods --all-namespaces -Lsecgroup
NAMESPACE NAME READY STATUS RESTARTS AGE SECGROUP
k8s nginx-foo-2789337611-z09x2 1/1 Running 0 58m web
nsx nginx-bar-3492604561-nltrf 1/1 Running 0 1h db
Namespace: foo Namespace: bar
NSX / K8s topology
10.24.0.0/24 10.24.1.0/24 10.24.2.0/24
NAT boundary
Web
• Security Groups are defined in NSX with ingress and
egress policy
• Each Security Group could be micro-segmented to
protect Pods from each other
DB

32. Confidential │ ©2018 VMware, Inc. 32
NSX-T & PKS Integration – Pods Micro-Segmentations
Option 2: K8s Network Policy
admin@k8s-master:~$ vim nsx-demo-policy.yaml
apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
name: nsx-demo-policy
spec:
podSelector:
matchLabels:
app: web
ingress:
- from:
- namespaceSelector:
matchLabels:
ncp/project: db
ports:
- port: 80
protocol: TCP
admin@k8s-master:~$ kubectl create -f nsx-demo-policy.yaml
• Use Case: Using Network Policy, users can define firewall
rules to allow traffic into and out of a Namespace, and
between Pods. The network policy is a Namespace
property. The default behavior is drop.
Namespace: foo Namespace: bar
NSX / K8s topology
10.24.0.0/24 10.24.1.0/24 10.24.2.0/24
NAT boundary
DB
Label: app=db
Web
Label: app=web

33. Confidential │ ©2018 VMware, Inc.
NSX-T & PKS Integration – Pods Micro-Segmentations
$ kubectl create -f nsx-demo-policy.yaml
Dynamic Creation of
Security Groups
Dynamic Creation of
Security Policy based on
k8s Network Policy
Once the Network Policy is applied, NSX will dynamically create source & destination Security Groups and apply the
right policy

34. Confidential │ ©2018 VMware, Inc. 34
NSX-T & PKS Operational Tools
Why having a centralized SDN Controller enhances visibility
With most other networking technologies in K8s and PCF. So, there's no counters, troubleshooting tools,
'span ports', Firewall Rules Overview, etc.

35. Confidential │ ©2018 VMware, Inc. 35
NSX-T & PKS Operational Tools
Why having a centralized SDN Controller enhances visibility
With NSX-T you are gain deep visibility into the container networks, and you can use the same
troubleshooting tools we created for VM based workloads

36. > Stay Connected.
csaunders@pivotal.io
jscanga@vmware.com
Mission Impossible
Deploying Pivotal Cloud Foundry to Nine Air Operational Sites in
a Year
#springone@s1p