1.
Hacking DevOps
December 1, 2022
APPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED. CASE NUMBER 22-3754
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

2.
# whoami
 Security Engineer:
GSE #263, SANS Master’s Degree, 11+ security certifications
 Hacker:
Speaker at DEF CON Cloud & AppSec Villages
 DevOps Engineer:
Automating build, test, deployment, and monitoring
 Systems Engineer:
Focus on the overall system and process to deliver the system
 Developer:
 10+ years
 Cloud Engineer:
Designed and built both AWS and Azure environments for
large teams
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

3.
3
Why should I care about DevOps?
 DevOps is popular: ~83% adoption in one
Puppet survey
 Subverting this process is powerful because
you can get inside an adversary’s
development cycle to achieve superior
visibility and persistence in their
environment
 Some of the attacks and defenses happen
to open-source projects, providing an
opportunity and a risk for “crowdsourced
conflict”
 Security is not consistently included in
DevOps, if it is included at all
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

4.
4
Major Impacts of DevOps Compromises
 SolarWinds Orion was modified during build
time to include the malicious payload
reaching many government organizations
and costing millions to resolve
 1500+ exposed Slack credentials on GitLab
led to hackers being able to read proprietary
info from major companies
 A compromised JS package allowed remote
code execution and exfiltration of package
management credentials for up to 3.5 million
installs
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.
https://www.gao.gov/products/gao-22-104746

5.
5
Overview
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.
https://dodcio.defense.gov/Portals/0/Documents/DoD%20Enterprise%20DevSecOps%20Reference%20Design%20v1.0_Public%20Release.pdf?ver=2019-09-26-115824-583

6.
Protecting the Pipeline
 Practice good DevOps for the
pipeline itself (and related
services) to ensure its integrity
 The bootstrap problem can be
somewhat mitigated and controlled
through use of Infrastructure as
Code (IaC) and continuous
monitoring
 Don’t forget the value of basic
server hardening for internal
resources
 Consider using traditional tactics to
compromise the pipeline, which
can be a very valuable target for
recon and increased access
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 6

7.
Source Control
 Identity, Credential, & Access
Management (ICAM)
 Users – integrated with SSO?
 Groups – where are they managed?
 Use “Protected Branches” to
enforce least privilege
 Limit and/or protect potentially
sensitive environment variables
 Use one user’s access to pivot to or
within SCM
 Break developer expectations and
insert malicious code into unprotected
branches
 Snoop on poorly protected
environment variables, which may
contain credentials
 Exfiltrate development history to look
for non-public vulnerabilities or
incorrectly removed sensitive data
or credentials
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 7

8.
Artifact Control
 Permissions need to have a
managed lifecycle
 Who needs Read/Write
 At what granularity are those
permissions needed?
 How are Non-Person Entities (NPEs)
managed?
 Have an artifact retention plan
 Examples: left-pad, mimemagic
 Use one actor’s access to pivot to
additional data by inserting
malicious artifacts
 Disrupt operations or restoration
capabilities by removing artifacts
 If no artifact retention plan is in place,
this can be done without internal access
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 8

9.
Pipeline
 Limit access to modify sensitive or
critical test steps
 Alert and review when the pipeline
changes
 Include integrity checks when passing
products or test results across
security boundaries
 Ensure sensitive details and
credentials are appropriately secured
 Ensure build steps are protected
against disclosure of sensitive data
 Change what the pipeline runs
 Insert malicious steps
 Remove security tests
 Change test results
 Change the built product to be
different from the tested one
 Find sensitive environment details or
credentials
 Pivot to ChatOps, test servers, prod
servers
 Recon for the rest of the DevOps
process
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 9

10.
Build Servers
 Manage which projects share build
servers to limit the impact of a
compromise
 Use protected security boundaries
between projects using a shared
server
 If relying on pre-installed software,
verify its integrity
 Access code and/or artifacts of
new projects through poorly
protected shared build servers
 If a project uses pre-installed
software or libraries, these can be
compromised to produce false
results
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 10

11.
Secrets
 Manage permissions lifecycle
 Put sensitive data into a Vault
 For example: Passwords, Private Keys,
API Tokens
 Ensure secrets are not put into source
control with a tool like Gitleaks
 Avoid writing secrets to disk
 Clear secrets when no longer needed,
but be careful not to use these
variables after they’re cleared
 Find credentials and other
sensitive data in:
 Source code, using a tool like truffleHog
 Configuration files
 Environment variables in CI tools
 Environment variables in test and
production deployments
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 11

12.
Test & Evaluation
 Limit/manage who can alter test cases
and policy conditions
 Consider more permissive
development tests that become more
strictly managed prior to making
deployment decisions
 Manage who has access to
vulnerability information
 N.B., developers need access to
vulnerability results in order to remediate
problems quickly
 Monitor sensitive test steps for
unusual activity
 Change test results
 Change what tests are run
 Change pass/fail conditions
 Exfiltrate results
 For example, of security tests disclosing
new vulnerabilities which may be used
to further compromise the development
and test environment
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 12

13.
ChatOps
 Use a Privileged Access
Management (PAM) solution to
share credentials if it is necessary
to do so
 Practice good OPSEC if you don’t
have control of the chat platform
 Limit scope of access credentials
provided to NPEs – for example to
a #chatops-channel
 Access to development
discussions including bugs,
vulnerabilities, architecture,
schedules, and possibly
credentials
 May be able to trigger rebuild or
redeploy, which could be useful to
deny service or cover your tracks
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 13

14.
Infrastructure as Code
 Test and validate infrastructure as code
(IaC) using similar techniques to other
code in the pipeline (e.g., static analysis)
 Use IaC to consistently apply updates,
including security patches
 Consider adopting immutable
infrastructure where infrastructure is
destroyed and recreated instead of
updated
 Use IaC verification methods to monitor
for unexpected changes to infrastructure
and applications
 Leverage IaC to reduce Mean Time
To Recovery (MTTR) as part of a
recovery plan
 Compromising IaC provides a fantastic
persistence mechanism
 Include malicious infrastructure
configurations to subvert application
security
 May be able to force deployment of older,
vulnerable versions of the application
 May be able to eliminate or bypass any
mitigations and security services deployed
as IaC
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 14

15.
15
Key Takeaways
 DevOps tools are:
 Common
 Contain valuable info for an attacker
 Often poorly understood and protected
 DevOps practices can be used to protect
both the application being developed and the
tooling used to evaluate it
 Protecting development tooling is important
for both open source and internal
development teams
 Failure to protect DevOps tools and
processes can result in major impacts to an
organization
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

16.
Phillip Marlow
pmarlow@mitre.org
@wolramp
linkedin.com/in/phillipmarlow
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.
Thank You!