As delivered by Tim Mackey, Senior Technical Evangelist - Black Duck Software, at LinuxCon and ContainerCon in Berlin 2016. Traditionally, when datacenter operators talk about application security, they've tended to focus on issues related to key management, firewalls and data access. By contrast, application developers have a security focus which is more aligned with code analysis and fuzzing techniques. The reality is, secure application deployment principles extend from the infrastructure layer through the application and include how the application is deployed. With the prevalence of continuous deployment of micro-services, it’s imperative to focus efforts on what attackers’ view as vulnerable; particularly in an environment where new exploits are being disclosed almost daily. In this session we’ll present: • How known vulnerabilities can make their way into production deployments • How deployment of vulnerable code can be minimized • How to determine the vulnerability status of a container • How to determine the risk associated with a specific package

1. Secure Application
Development in the Age
of Continuous Delivery
LinuxCon/ContainerCon Europe 2016 – Berlin Germany

2. #whoami – Tim Mackey
Current roles: Senior Technical Evangelist; Occasional coder
• Former XenServer Community Manager in Citrix Open Source Business Office
Cool things I’ve done
• Designed laser communication systems
• Early designer of retail self-checkout machines
• Embedded special relativity algorithms into industrial control system
Find me
• Twitter: @TimInTech ( https://twitter.com/TimInTech )
• SlideShare: slideshare.net/TimMackey
• LinkedIn: www.linkedin.com/in/mackeytim

3. Standard Development Workflow
DEVELOP BUILD PACKAGE DEPLOY PRODUCTION
BUG TRACKING
TEST
AUTOMATION

4. Understanding the
Attacker Model

5. Vulnerability Management Implies Data Breach Management
 89% of data breaches had a
financial or espionage motive
 Legal costs and forensics dominate
remediation expenses
Source: Verizon 2016 Data Breach Report

6. Attackers Decide What’s Valuable …

7. But security investment is often not aligned with actual risks

8. Anatomy of a New Attack
Potential Attack
Iterate
Test against platforms
Document
Don’t forget PR department!
Deploy

9. Exploiting a
Vulnerability

10. CLOSED SOURCE COMMERCIAL CODE
• DEDICATED SECURITY RESEARCHERS
• ALERTING AND NOTIFICATION INFRASTRUCTURE
• REGULAR PATCH UPDATES
• DEDICATED SUPPORT TEAM WITH SLA
OPEN SOURCE CODE
• “COMMUNITY”-BASED CODE ANALYSIS
• MONITOR NEWSFEEDS YOURSELF
• NO STANDARD PATCHING MECHANISM
• ULTIMATELY, YOU ARE RESPONSIBLE
Who is Responsible for Code and Security?

11. Knowledge is Key. Can You Keep Up?
glibc
Bug
Reported
July 2015
Vuln: CVE-2015-7547: glibc getaddrinfo stack-based
buffer overflow

12. Knowledge is Key. Can You Keep Up?
glibc
Vuln
Introduced
May 2008
glibc
Bug
Reported
July 2015
CVE-2015-
7547
CVE
Assigned
Feb 16-2016
Low Security Risk
Vuln: CVE-2015-7547: glibc getaddrinfo stack-based
buffer overflow

13. Knowledge is Key. Can You Keep Up?
glibc
Vuln
Introduced
May 2008
CVE-2015-
7547
CVE
Assigned
Feb 16-2016
glibc
Bug
Reported
July 2015
National
Vulnerability
Database
Vuln
Published
Feb 18-2016
Moderate Security Risk
Low Security Risk
Vuln: CVE-2015-7547: glibc getaddrinfo stack-based
buffer overflow

14. Knowledge is Key. Can You Keep Up?
glibc
Vuln
Introduced
National
Vulnerability
Database
Vuln
Published
You
Find It
May 2008
CVE-2015-
7547
CVE
Assigned
Feb 16-2016 Feb 18-2016
glibc
Bug
Reported
July 2015
Patches
Available
You
Fix It
Highest Security Risk
Moderate Security Risk
Low Security Risk
Vuln: CVE-2015-7547: glibc getaddrinfo stack-based
buffer overflow

15. Understanding Vulnerability Impact

16. 0
500
1000
1500
2000
2500
3000
3500
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
Open Source Vulnerabilities Reported Per Year
BDS-Enhanced Vulnerability Data nvd
Reference: Black Duck Software KnowledgeBase, NVD
Vulnerability Disclosures Trending Upward

17. 0
500
1000
1500
2000
2500
3000
3500
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
Open Source Vulnerabilities Reported Per Year
nvd BDS-Enhanced Vulnerability Data
Reference: Black Duck Software KnowledgeBase, NVD
Vulnerability Disclosures Trending Upward

18. Vulnerabilities Enter Product and Become Deployed
DEVELOP BUILD PACKAGE DEPLOY PRODUCTION
BUG TRACKING
TEST
AUTOMATION

19. Workflow Integrations Matter

20. Understanding the
Deployment
Environment

21. Primary goals
• Protect against BIOS and firmware attacks
• Protect cryptographic host state
• Ensure valid hypervisor kernel
• Validate launch of critical VMs
• Attest to hosts’ trust state
Implemented by
• Intel Haswell and newer
• Cryptographic hashes stored in TPM
Intel TXT – Trusted Execution Protection - Foundational

22. Intel EPT – Extended Page Tables
Page 0
…
Page
13553
Page
13554
…
…
Page 126
Page 127
…
Page
64589
Page
64590
Page
64591
Page 0
…
Page 217
…
Page 31289
…
…
Page 78924
…
Page 97586
…
0→64589
13553→127
13554→64591
App Memory OS Memory
TLB CR3
Virtual Machine
126→31289
127→0
64589→97586
64590→217
64591→78924
Host Memory
EPT

23. Hypervisor
Hypervisor Memory Introspection – Enabled by EPT
Implementation Overview
• Critical memory pages are
assigned permissions in EPT
• Exception handler defined in
hypervisor
• Shadow EPT defined with elevated
privs
Protects Against Attack Techniques
• Rootkit injection
• Buffer overflow
• API hooking
VM Kernel Memory Layout
…
Kernel Code (R/X)
Driver Code (R/X)
…
Driver Data (R/W)
Kernel Code (R/X)
Kernel Data (R/W)
…
126→31289 (R/X)
127→0 (R/X)
64589→97586 (R/W)
64590→217 (R/X)
64591→78924 (R/W)
EPT#1
126→31289 (+W)
127→0 (+W)
64589→97586 (+X)
64590→217 (+W)
64591→78924 (+X)
EPT#2 (Shadow)
Exception
Handler

24. Guest Guest Guest Guest Guest
Critical
Memory
Access
Critical
Memory
Access
Critical
Memory
Access
Critical
Memory
Access
Critical
Memory
Access
Networking StorageCompute
Simplified Hypervisor Introspection Architecture Diagram
Xen Project Hypervisor
Control
Domain
(dom0)
Security
Appliance
(domU)
Memory
Introspection
Engine
Direct Inspect
APIs

25. Virtual Switches as Local Edge Protection – Silent Block
Guest
VM
SSL access
Attack silently blocked
Virtual Switch Rules
Ingress:
HTTPS public
Egress:
Dynamic port to origin
MySQL internal
Private CIDR internal
Port 22 access

26. Virtual Switches as Local Edge Protection – Traffic Monitor
Guest
VM
SSL access
Attack blocked with traffic log
Virtual Switch Rules
Ingress:
HTTPS public
Egress:
Dynamic port to origin
MySQL internal
Private CIDR internal
Port 22 access
ovs Controller
Log SSH Port 22 access
Create port mirror for attacker
Traffic
Monitor
Virtual Switch Rules
Ingress:
HTTPS public
Egress:
Dynamic port to origin
MySQL internal
Private CIDR internal
Mirror:
Port 22 to Traffic Monitor
All attacker traffic to monitor

27. Guest
VM
Virtual Switches as Local Edge Protection – Quarantine
Guest
VM
SSL access
Attack quarantined with full log
Virtual Switch Rules
Ingress:
HTTPS public
Egress:
Dynamic port to origin
MySQL internal
Private CIDR internal
Port 22 access
ovs Controller
Log SSH Port 22 access
Create port mirror for attacker
Quarantine VM for attacker use
Trigger replacement VM for farm
Traffic
Monitor
Virtual Switch Rules
Ingress:
HTTPS attacker
Egress:
Dynamic port to origin
Mirror:
Port 22 to Traffic Monitor
All attacker traffic to monitor

28. Production
Container Usage

29. Are Containers Production Ready?

30. Container Use Cases
Application containers
• Hold a single application
• Can follow micro-services, cloud native design pattern
• Starting point for most container usage
• Short lifespan, many per host
System containers
• Proxy for a VM
• Insulate against core operating system
• Perfect for legacy apps
• Long lifespan, few per host
MySQL
Tomcat
nginx
Kernel
MySQL
Tomcat
nginx
Kernel

31. Securing the
Container
Contents and
Environment

32. Baseline to Limit the Scope of Compromise
• Enable Linux Security Modules
• SELinux
• --selinux-enabled on Docker engine, --security-opt=“label:profile”
• AppArmor
• -- security-opt=“apparmor:profile”
• Apply Linux kernel security profiles
• grsecurity, PaX and seccomp protections for ALSR and RBAC
• Adjust privileged kernel capabilities
• Reduce capabilities with --cap-drop
• Beware –cap-add and –privileged=false, and CAP_SYS_ADMIN
• Use a minimal Linux host OS
• Red Hat Enterprise Linux Atomic Host 7, CoreOS, RancherOS, Intel Clear Linux, Alpine, etc
• Reduce impact of noisy neighbors
• Use cgroups to set CPU shares and memory

33. Red Hat Enterprise Linux Atomic Host 7
• What is Atomic Host?
• Optimized RHEL7 variation designed for use with Docker
• Uses SELinux for safeguards
• Provides atomic upgrade and rollback capabilities via rpm-ostree
• Pre-installed with Docker and Kubernetes
• Atomic App and Atomic Nulecule
• Provides a model for multi-container application definition
• Supports Docker, Kubernetes, OpenShift and Mesos
• OpenShift artifacts run natively or via atomic provider
• Provides security compliance scan capabilities

34. Container Source Trust
Red Hat Atomic Host
AtomicApp
AtomicApp
AtomicApp
Red Hat Registry
MySQL
Redis
Jenkins
Docker Hub
DockerContainer
DockerContainer
DockerContainer
DockerContainer
DockerContainer
Third Party and Custom
Problem: Who to trust, and why?
• Trusted source?
• Unexpected image contents
• Locked application layer
versions (e.g. no yum update)
• Layer dependencies
(monolithic vs micro-services)
• Validated when?

35. OpenSCAP vs. Black Duck Hub
OpenSCAP
• Profile driven compliance policy engine
• Vendor vulnerability data is but one component of policy
• Integrated directly with Red Hat Atomic
• Usage: atomic scan --scanner openscap {container id}
Black Duck Hub integration with Red Hat Atomic
• Broad vulnerability data for most open source components
• Covers vulnerability, license compliance and operational risk
• Integrated with Red Hat Atomic
• Rich tooling integration for development teams
• Installed via: atomic install blackducksoftware/atomic
• Usage: atomic scan --scanner blackduck {container id}

36. A DONATION HAPPILY GIVEN
TO THE DEMO GODS

37. Gating Builds for Vulnerabilities
DEVELOP BUILD PACKAGE
BUG TRACKING
VULNERABILITY
ASSESMENT

38. Control
Domain
NetworkingCompute Storage
Hypervisor
Container
VM
Minimal OS
Understanding Scope of Compromise – Protect From the Inside
Container
Container
Container
Container
VM
Minimal OS
Container
Container
Container
SecurityService
Container

39. Understanding Scope of Compromise – Protect From the Inside
Container VM
Minimal OS
Container
Container
Unpatched
Container
Compromised
Container
Compromised
Container

40. Vulnerability Assessment Post Deployment
DEVELOP BUILD PACKAGE DEPLOY PRODUCTION
BUG TRACKING
TEST
AUTOMATION
VULNERABILITY
ASSESMENT

41. Risk Mitigation Shrinks Scope of Compromise
Open source license compliance
• Ensure project dependencies are understood
Use of vulnerable open source components
• Is component a fork or dependency?
• How is component linked?
Operational risk
• Can you differentiate between “stable” and “dead”?
• Is there a significant change set in your future?
• API versioning
• Security response process for project

42. Vulnerability Analysis Compliments SAST/DAST
Vulnerability Analysis
- Identifies vulnerable dependencies
- 3000+ disclosures in 2015
- Most vulnerabilities found by researchers
All possible security vulnerabilities
Static and Dynamic Analysis
- Discover common security patterns
- Challenged by nuanced bugs
- Focuses on your code; not upstream

43. 7 of the top 10
Software Companies
(44 of the top 100)
6 of the top 8
Mobile Handset Vendors
6 of the top 10
Investment Banks
24
Countries
250+
Employees
1,800Customers
Who is Black Duck Software?
27Founded
2002

44. 8,500
WEBSITES
350
BILLION LINES OF CODE
2,400
LICENSE TYPES
1.5
MILLION PROJECTS
76,000
VULNERABILITIES
• Largest database of open source project
information in the world.
• Vulnerabilities coverage extended through
partnership with Risk Based Security.
• The KnowledgeBase is essential for identifying
and solving open source issues.
Comprehensive KnowledgeBase

45. Black Duck Hub Security Architecture
Hub Scan1 File and Directory Signatures2 Open Source
Component Identified
3
Hub Web Application
Black Duck
KnowledgeBase
On Premise Black Duck Data Center

46. We Need Your Help
Knowledge is power
• Know what’s running and why
• Define proactive vulnerability response process
• Don’t let technology hype cycle dictate security
Invest in defense in depth models
• Don’t rely on perimeter security to do heavy lifting
• Do look at hypervisor & container trends in security
• Make developers and ops teams part of the solution
• Focus attention on vulnerability remediation
Together we can build a more secure data center

47. Free Tools to Help
Free Docker Container Security Scanner
• https://info.blackducksoftware.com/Security-Scan.html
14 Day Free Trial to Black Duck Hub
• https://info.blackducksoftware.com/Demo.html