Securing Application Deployments in
Multi-tenant CI/CD Environments
O’Reilly Security Conference NY 2016

HELLO!
I am Binu Ramakrishnan
Principal Security Engineer
at Yahoo
You can find me at:
@securitysauce
github.com/prbinu

Overview
▹ A brief introduction to CI/CD platform
▹ Challenges with multi-tenant CI/CD environments
▹ Threat modeling
▹ Security patterns and best practices

CI/CD
A CI/CD pipeline is a set of jobs chained to achieve continuous delivery, and deployment of
applications to production.

CI/CD Deployments
Single-tenant | Multi-tenant

Single-tenant
▹ Dedicated build environments
▹ Distributed
▹ Tenant and provider are the same
▹ Limited security impact

Multi-tenant
▹ Shared build environments
▹ Driven by economies of scale
▹ Focused on operational efficiency
▹ Allow enterprise to centrally
enforce dev stds & compliance
controls
▹ Gateway to enterprise networks

Multi-tenant CI/CD - Security Risks
▹ This platform can be an attractive target for attackers because:
▸ Consolidation of large number of long-lived keys in one place
▸ It acts as a corridor that directly connects to large number of machines, bypassing
firewalls
▹ A successful attack can have a significant impact to an enterprise
▸ Leads to exposure of user’s personal info, application credentials, TLS private keys etc.

Denial of Service (DoS) Attacks
* http://bit.ly/2exRpWm
“Enterprises relying on SaaS apps had no choice
but to sit and wait and hope their providers got
back online as soon as possible.” *

CI/CD in Nutshell

Enterprise CI/CD

Threat Modeling

CI/CD Admin
Exclusive access to the CI/CD
platform
Actors
User/Developer
Person who uses the platform
and run their build pipelines.
CI/CD Platform
Developer
Limited admin access to the
platform.

Trust Boundaries
▹ Application - web interface, REST APIs
▹ Host - isolation between host OS and the build job
▹ Network - segment build machines from other
machines and from Internet

Entry Points
▹ Build web interface
▹ Commit handlers
▹ Internal endpoints
▸ Build slave endpoint (to launch
builds; SSH)
▹ Deployable artifacts
▹ Build notifications - eg. emails, IRC
and Hipchat messages
▹ Build console logs
.
Attack Surface
Exit Points

Threats
16

Common Threats
▹ User account compromise & insider threats
▸ User and platform admins
▹ Network Intrusion
▸ CI/CD internal and external endpoints

Local Storage of Non-ephemeral Keys
▹ Long-lived keys are stored locally to allow build jobs to access
protected services.
▹ OAuth tokens, Shared secrets, SSH private keys etc.

Weak Security Controls
Build jobs require access to services as part of their execution (eg. artifacts
store). Protection mechanism used by those services are less effective in
following cases:
▸ IP whitelist based protection
▸ Shared secrets: shared (and accessible) by all build jobs
The above controls are susceptible to spoofing in multi-tenant environments.

Build Slave Compromise
Large attack surface, spread across multiple networks (iphone, Android,
Server app etc.)
How?
▹ A network level compromise, exploiting a vulnerability on build slaves
▹ Jobs break out of build container
Impact
▹ Listen to the network, spoof identity and access locally stored
credentials
▹ Gain access to unauthorized machines and steal sensitive data

Building External Code
An attacker can take this path to get into internal network, either by
adding backdoors or exploiting known vulnerabilities with open
source software.

Security Patterns for
Risk Reduction

Ephemeral Keys
https://www.flickr.com/photos/articnomad/241620406
JWT, OAuth, SSH-CA, AWS access key etc.
▹ Automated deployments require delegation
▹ Token Delegation: a controlled replay of tokens
▹ Ephemeral keys go well with token delegation
model

Capability Delegation &
Stateless Auth
https://www.flickr.com/photos/18946008@N06/14551311971/
By passing capabilities (in the form of tokens)
with build events, we eliminate the need to
store long-lived keys in CI/CD platform

▹ Separate out auth components from the core build platform
▹ The Auth service intercepts and authenticates build events, and issues job tokens to downstream
build platform
▹ Stateless: Build platform no longer stores job specific long-lived keys
▹ Distributed: Per tenant Auth service for cloud based environments, hosted inside tenant’s network
▹ No consolidation of keys in one place

Audit Trails
▹ Immutable and append-only
▹ Traceability: logs should capture build pipeline
events end to end from commit to final deploy.
▹ Trail traits: who, how, when, sequence of events,
start and end state.
▹ Audit trails produced by the Auth service can
provide a verifiable chain of trust from commit to
deploy.

Container &
ToolChain
Hardening

Network Segmentation
Colocating CI/CD platform with other
corporate machines allow malicious build
jobs to raid your internal network.
The recommendation is to zone CI/CD
platform from other corporate systems.

Minimal Builds
▹ Do not expose PII or other sensitive info to build machines.
▹ Do complex tests outside of build, in a separate environment you
control.

Few more...
SSH: Use
Restricted
Shells
Headless SSH access for
automated deployment
should use a restricted shell.
Roll Keys
Periodically
Establish a process to
periodically roll trust anchor
keys (and do it periodically).
Restrict Job
Console Logs
Restrict build job console logs
only to authorized users
Enable 2FA
Admins must follow good
security hygiene and use 2FA
to access platform
application and hosts.
Prune Admin
Access List
Keep admin list small for
build systems and Git repo
access.
Vulnerability
Patch Mgmt
Maintain an inventory of all
packages in use and have a
mechanism to patch the system
in response to a disclosure.

Acknowledgements
▹ Christopher Harrell
▹ St John Johnson
▹ Mike Shema
▹ Jeremiah Wuenschel

Let’s recap major concepts
Ephemeral Keys
Replace long-lived keys with
time-bound scoped tokens.
Service providers should start
supporting tokens for auth &
authz.
Capability
Delegation
Delegate job capabilities with
build events, and eliminate the
need to store long-lived keys in
CI/CD platform.
Audit Trails
A verifiable chain of trust
based on traceable audit logs
is a foundational requirement
for CI/CD.
Minimal Builds
Avoid pulling PII or other
production sensitive data to
build environments. Keep the
builds to minimum.
Network
Segmentation
Network level Isolation of CI/CD
machines.
Build ToolChain
Hardening
Build tools and Docker
containers must be adequately
hardened.

THANKS!
Any questions?
You can find me at:
@securitysauce
github.com/prbinu

CREDITS
Special thanks to all the people who made and
released these awesome resources for free:
▹ Presentation template by SlidesCarnival
▹ Photographs by Death to the Stock Photo
(license)