This document summarizes Tim Mackey's presentation at DevSecCon. It discusses the importance of security driven development practices like using trusted components, continuous integration processes that include security testing, and digitally signing container images. It warns that while infrastructure teams aim to provide security, vulnerabilities can still exist, and advocates continually evaluating the trust of components used. The document predicts disclosure of security issues will increase and outlines penalties for data breaches under new regulations like GDPR. It emphasizes automating awareness of open source dependencies to keep pace with DevOps.
Handwritten Text Recognition for manuscripts and early printed texts
DevSecCon London 2017: when good containers go bad by Tim Mackey
1. Join the conversation #DevSecCon
BY TIM MACKEY – BLACK DUCK SOFTWARE
A Question of Trust – When
Good Containers Go Bad
2. #whoami – Tim Mackey
• Current roles: Senior Technical Evangelist; Occasional coder
• Former XenServer Community Manager in Citrix Open Source Business
Office
• Cool things I’ve done
• Designed laser communication systems
• Early designer of retail self-checkout machines
• Embedded special relativity algorithms into industrial control system
• Find me
• Twitter: @TimInTech ( https://twitter.com/TimInTech )
• SlideShare: slideshare.net/TimMackey
• LinkedIn: www.linkedin.com/in/mackeytim
3. Security Driven Development and Deployment
• Developers are empowered with security information
• Clear security driven release policies exist
• Trusted components are used as dependencies
• CI processes incorporate security testing
• Binary artifacts are only created when release policies are met
• Releasable binaries are digitally signed
• Container images are built from trusted base images
• Produced images are stored in trusted container registries
• Containers are only deployed from trusted registries
4. Data Breaches are Serious Business
• Average cost of data breach: $7.35 Million
• Lost business: $4.03 Million
• Average time to identify and contain a breach: 206 days
Source: 2017 Cost of Data Breach Study – Ponemon Insitute
5. Prediction: Rate of Security Disclosures To Increase
• e.g. GDPR
• Anyone operating with data on EU person
• Corporate location irrelevant
• Violation has fine of 4% global turnover or
20 Million Euro (which ever is greater)
• Applies equally to controllers of data and
processors of data
• Breach notification required within 72 hours
• Requires “Privacy by Design”
8. Question Everything and Continually Evaluate Trust
• Where does your base image actually come from?
• What is the health of that base image?
• You’re updating it at build time, but from what cache?
• You trust your build servers, but who controls them?
• Is there any way a foreign container can start in your environment?
• Who has rights to modify container images?
• What happens if base image registry goes away?
• What happens if base image tag goes away?
• What happens if an update mirror goes down?
• When a security disclosure happens what’s the process to determine impact?
• How are images being updated and deployed in the face of new security disclosures?
10. CLOSED SOURCE COMMERCIAL CODE
• TRADITIONAL PROCUREMENT PROCESS
• ALERTING AND NOTIFICATION INFRASTRUCTURE
• SUPPORT AVAILABLE THROUGH EOL
• STAFFED WITH SECURITY RESEARCHERS
• REGULAR PATCH UPDATES
• DEDICATED SUPPORT TEAM WITH SLA
OPEN SOURCE CODE
• AD-HOC ADOPTION MODEL
• MONITOR NEWSFEEDS YOURSELF
• EOL MAY CREATE DEADEND
• “COMMUNITY”-BASED CODE ANALYSIS
• NO STANDARD PATCHING MECHANISM
• ULTIMATELY, YOU ARE RESPONSIBLE
Proprietary Software Rules Aren’t Open Source
11. Attackers are Clever and You Need to be Cunning
Potential Attack
Iterate
Test against platforms
Document
Don’t forget PR department!
Deploy
14. Patches Available
Media
Coverage
Embargo
Expires
Oct 21 2016
Git://id
Upstream
Patch
Vuln: CVE-2016-5195 – AKA
“Dirty Cow”
Oct 18 2016
National
Vulnerability
Database
Vuln
Published
Nov 10 2016
Highest Security Risk
Timing is Opportunity
15. Security Analysis Isn't Only SAST/IAST/DAST
All possible security vulnerabilities
Static, Injection and Dynamic Analysis
- Discover common security patterns
- Challenged by nuanced bugs
- Focuses on your code; not upstream
Vulnerability Analysis
- Identifies vulnerable dependencies
- 3000+ disclosures in 2015
- 4000+ disclosures in 2016
- Most vulnerabilities found by researchers
16. We’re all Researchers – Report What You Find
• Distributed Weakness Filing
• Open Source specific CAN
• Designed for Open Source projects
without an existing CAN
• Increasing vulnerability
awareness
• Reinforce security collaboration
• Reduce islands of knowledge
https://iwantacve.org
https://github.com/distributedweaknessfiling/
17. Heartbleed: Why in 2017?
Don’t Give Attackers Opportunities
OpenSSH (CVE-2004-1653): AllowTCPForwarding creates open IoT proxyApache Struts (CVE-2017-5638): Vulnerability response time matters
18. 1649 Days 144 Days7 Days
The Tale of CVE-2017-5638 and Equifax
Code Bug
Introduced
August
2012
Struts 2.3
Released
November
2012
Struts 2.5
Released
May
2016
Patches
Available
March 6
2017
March 7
2017
Disclosure
Published
NVD
Details
March 14
2017
Hacks
Successful
May 13
2017
Hacks
Discovered
July 29
2017
20. LEVEL 1 – BLISSFUL IGNORANCE
No policies in place to manage open source
security and licensing risks. Unknown
versions and dependencies. No
documentation of intent.
21. LEVEL 2 – AWAKENING
Inconsistent manual processes to
identify and report on open source
usage. Conceptual awareness of
license requirements. Unaware of
security implications of open source
usage.
22. LEVEL 3 – UNDERSTANDING
Manual review processes, and basic
tooling. Primary focus on license
compliance. Accuracy is difficult to
maintain. Provides limited insight into
security vulnerabilities.
Tools: Spreadsheets, low cost tools,
sporadic security scans
23. LEVEL 4 – ENLIGHTENMENT
Automatic identification of open
source components and versions.
Direct mapping to licenses and
disclosed vulnerabilities.
Integration with developer, issue
management, CI/CD and
deployment tools.
24. Join the conversation #DevSecCon
We Need to Automate This – Don’t
We?
Obtain Enlightenment
and make information flow your friend
25. Focus on Factors Impacting Risk
• Use of vulnerable open source components
• What are my dependencies and where are they coming from?
• Is component a fork or dependency?
• How is component linked?
• Impact of Point in Time Decisions
• Can you differentiate between “stable” and “dead”?
• Is there a significant change set in your future?
• API versioning
• Security response process for project
• Commit velocity and contributors
27. Support Gating of Artifact Builds for Risk Elements
DEVELOP BUILD PACKAGE
RISK ASSESSMENT
BUG TRACKING
28. Build a Risk Profile for All Containers In SDLC
Registry
SCM Trigger
Deployment
TriggerGit
Build Pipelines
Production
Trigger
Registry Registry
Security
Scan
Staging
Tests
SCM TriggerGit
29. Support Ongoing Monitoring for Changes in Risk
DEVELOP BUILD PACKAGE DEPLOY PRODUCTION
BUG TRACKING
TEST
AUTOMATION
RISK ASSESSMENT
30. Evaluate Security Information Throughout SDLC
Developer Experience
IDEs
Release Engineering
SAST
DAST
Artifact
Storage
Production
Deployment
Package
management
CI
33. Layer Container Security For Maximum Impact
Secure Platform with Red Hat OpenShift Container Platform and Atomic Host
Administer DISA STIG: CVE, CCE, CPE, CVSS, OVAL, and XCCDF
OVAL formatted patch definitions for Red Hat products
Scan all container images in an OpenShift deployment as the are created, modified and used
Provide visibility into open source components regardless of source
Annotate images and image streams with vulnerability information
Annotations automatically updated as new disclosures occur – without the need for rescan
34. 10,000
DATA SOURCES
530
TERABYTES OF CONTENT
2,500
LICENSE TYPES
12
YEARS OF OSS ACTIVITY
100,000
OSS VULNERABILITIES
KnowledgeBase is Critical
• Designed with Open Source behavior traits
including forks and merges
• Vulnerability information enhanced through
dedicated security research team
• Real time updates as security issues occur
• Map vulnerabilities to public exploits
35. Managing Open Source Security Requires End-End Visibility
INVENTORY
Open Source
Components
MAP
To Known
Vulnerabilities
IDENTIFY
Open Source
Risks
MANAGE
Open Source
Governance Policies
ALERT
For New
Vulnerabilities
Automation and workflow
Integration with DevOps tools and processes
37. • 2 ½ days of keynotes, breakout sessions, and networking
• Four conference tracks: Dev & DevOps, Security,
Legal & Compliance, Research & Innovation
Register at: https://www.blackducksoftware.com/flight
FLIGHT 2017
Join us at the Boston Seaport Hotel & World Trade Center
November 7-9, 2017
Register using the code TIM99 to
save $1196 on the conference price
https://www.cesg.gov.uk/guidance/open-source-software-%E2%80%93-exploring-risk-good-practice-guide-38
If you’re using commercial software, the vendor is responsible for best practice deployment guidance, the notification of any security vulnerabilities and ultimately patches and workarounds for disclosed vulnerabilities. This is part of the deliverable they provide in return for their license fee.
If you’re using open source software, that process becomes partly your responsibility. To illustrate the level of information you have to work with, let’s look at a media-wiki maintenance release from December 2015.
“various special pages resulted in fata errors” – this clearly is something which needs resolution, but which pages? How do you test?
“1.24.6 marks the end of support for 1.24.x” – this is good to know, but I hope it was published elsewhere.
“However, 1.24.5 had issues (along with other versions) so it was thought fair to fix them” – This is a good thing, but can we expect this treatment in the future? From the title, we also have a fix for 1.23.x, but what other versions?
Let’s take a little bit of time and look at how an attack is created. Potential attackers have a number of tools at their disposal, and use a number of different tactics. In this case, the attacker wishes to create an attack on a given component. In order to be effective, they have two primary models. First they can actively contribute code in a highly active area of the component with an objective of planting a back door of some form. The hope being that their code will fail to be recognized as suspect given how quickly the area of code is evolving.
Second they can look for areas of code which are stable, and the longer they’ve bene stable, the better. The reason for this is simple, old code is likely written by someone who isn’t with the project any longer, or perhaps doesn’t recall all assumptions present at the time the code was written. After all, its been long understood that even with the best developers, assumptions change and old code doesn’t keep up.
The goal in both cases being to create an attack against the component, so they test, and fail, and iterate against the component until they’re successful or move on. Assuming they’re successful, they create a deployment tool and document the tool for others. Of course, given the publicity received by some recent vulnerabilities, a little PR goes a long way.
Now there are responsible researchers who follow a similar workflow, and they legitimately attempt to work with component creators to disclose vulnerabilities. They too will publish results, but are less interested in creating the an attack beyond a proof of concept.
http://www.istockphoto.com/photo/person-in-hooded-sweater-using-a-laptop-on-wooden-table-gm464503138-58544934?st=cf78f31
http://www.istockphoto.com/photo/cloud-computing-gm518556682-90104967
https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html
https://sourceware.org/bugzilla/show_bug.cgi?id=18665
https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT)
http://cve.mitre.org/cve/cna.html
https://openclipart.org/detail/200681/primary-patch
https://www.youtube.com/watch?v=hkryI6eapOA
https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html
https://sourceware.org/bugzilla/show_bug.cgi?id=18665
https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT)
http://cve.mitre.org/cve/cna.html
https://openclipart.org/detail/200681/primary-patch
https://www.youtube.com/watch?v=hkryI6eapOA