Securing your
delivery pipelines
with a slight shift
to the left

I’m OK at Computers.

Can you imagine…

We should do better.
We can do better.

Supply Chain Levels for Software Artefacts
(SLSA)
A framework designed to help
organisations improve the integrity of
their software supply chains.

Developer Burnout
Recommendations
Performance

The Secure Software Development Framework
(SSDF) is a set of fundamental, sound, and
secure software development practices based
on established secure software development
practice documents from organizations such as
BSA, OWASP, and SAFECode. Few software
development life cycle (SDLC) models explicitly
address software security in detail, so practices
like those in the SSDF need to be added to and
integrated with each SDLC implementation.

The SSDF outlines solid practices for
embedding secure software
development practices in the delivery
lifecycle, that don’t just identify
threats but actually address them.
Source: https://csrc.nist.gov/Projects/ssdf

33% of respondents described their
security strategy as having a mix of
prevention and detection.
Source: Left and Right of Boom in Cybersecurity ,Elastic, 2022
82% said they plan to implement, are
implementing or have implemented.

33% of respondents described their
security strategy as having a mix of
prevention and detection.
Source: Left and Right of Boom in Cybersecurity ,Elastic, 2022
82% said they plan to implement, are
implementing or have implemented.

The road to hell is paved
with good intentions.

“would pursue laws to establish
liability for software companies
that sell technology that lacks
cybersecurity protections”
The Biden-Harris National Cybersecurity Strategy

Security is our Responsibility

• Linting
• Unit tests
• Integration tests
Continuous
Integration
• Staging deploy
• Prod deploy
• Smoke tests
Continuous
Deployment
Commit &
Push
Git
Hello
World

CI CD
Git

Top 10 CI/CD
SECURITY RISKS
SECURITY RISKS
The Open Worldwide Application Security Project (OWASP)

SECURITY RISKS
SECURITY RISKS
1 — Insufficient Flow Control Mechanisms
2— Inadequate Identity and Access Management
3— Dependency Chain Abuse
4— Poisoned Pipeline Execution (PPE)
5 — Insufficient PBAC (Pipeline-Based Access Controls)
6 — Insufficient Credential Hygiene
7 — Insecure System Configuration
8— Ungoverned Usage of 3rd Party Services
9 — Improper Artifact Integrity Validation
10 — Insufficient Logging and Visibility

SECURITY RISKS
SECURITY RISKS
1 — Insufficienct Flow Control Mechanisms
2— Inadequate Identity and Access Management
3— Dependency Chain Abuse
4— Poisoned Pipeline Execution (PPE)
5 — Insufficient PBAC (Pipeline-Based Access Controls)
6 — Insufficient Credential Hygiene
7 — Insecure System Configuration
8— Ungoverned Usage of 3rd Party Services
9 — Improper Artifact Integrity Validation
10 — Insufficient Logging and Visibility
4 — Poisoned Pipeline Execution (PPE)
5 — Insufficient PBAC (Pipeline-Based Access Controls)
2 — Inadequate Identity and Access Management
6 — Insufficient Credential Hygiene
1 — Insufficient Flow Control Mechanisms
3 — Dependency Chain Abuse
2— Inadequate Identity and Access Management

Our goal is to limit the blast radius.

Is executing build scripts within
all build contexts okay?

Executing scripts within
all build contexts is not ok.

How about running
`terraform plan`
in all build contexts?

Executing arbitrary code
in all build contexts is not ok.

SECURITY RISKS
SECURITY RISKS
1 — Insufficient Flow Control Mechanisms
2— Inadequate Identity and Access Management
3— Dependency Chain Abuse
5 — Insufficient PBAC (Pipeline-Based Access Controls)
6 — Insufficient Credential Hygiene
Poisoned Pipeline Execution (PPE)
7 — Insecure System Configuration
8— Ungoverned Usage of 3rd Party Services
9 — Improper Artifact Integrity Validation

Poisoned Pipeline Execution (PPE)
• Have isolated pipeline environments and contexts
• Sensitive and Non-Sensitive contexts
• Use branch protection rules in GitHub/GitLab/BitBucket
etc.

Upload Pipeline Build Docker Image
Linting Security Scans RSpec
Jest Code Coverage Bundle Analysis
Branch Build
Non-sensitive context
- no access to secrets
- no pipeline to prod
Sensitive context
- access to secrets
- additional permissions
Upload Pipeline Build Docker Image
Linting Security Scans RSpec
Jest Code Coverage Bundle Analysis
Main Build
Prepare for Deploy Deploy to Prod

SECURITY RISKS
SECURITY RISKS
1 — Insufficient Flow Control Mechanisms
2— Inadequate Identity and Access Management
3— Dependency Chain Abuse
4— Poisoned Pipeline Execution (PPE)
6 — Insufficient Credential Hygiene
7 — Insecure System Configuration
8— Ungoverned Usage of 3rd Party Services
9 — Improper Artifact Integrity Validation
Insufficient PBAC (Pipeline-Based Access Controls)

• Restrict the scope of a pipeline's access & permissions
• Use granular access controls
Insufficient PBAC (Pipeline-Based Access Controls)

ECS Service
Agent
Job ECS deploy role
Agent API (Pipelines)

ECS Service
Agent
Job
Agent API (Pipelines) OIDC provider
OIDC token

eyJhbGciOiJSUzI1NiIsImtpZCI6IjFlOWdkazcifQ.ew
ogImlzcyI6ICJodHRwOi8vc2VydmVyLmV4YW1w
bGUuY29tIiwKICJzdWIiOiAiMjQ4Mjg5NzYxMDAx
IiwKICJhdWQiOiAiczZCaGRSa3F0MyIsCiAibm9u
Y2UiOiAibi0wUzZfV3pBMk1qIiwKICJleHAiOiAxM
zExMjgxOTcwLAogImlhdCI6IDEzMTEyODA5Nz
AKfQ.ggW8hZ1EuVLuxNuuIJKX_V8a_OMXzR0E
HR9R6jgdqrOOF4daGU96Sr_P6qJp6IcmD3HP99
Obi1PRscwh3LOp146waJ8IhehcwL7F09JdijmBqk
vPeB2T9CJNqeGpegccMg4vfKjkM8FcGvnzZUN4
_KSP0aAp1tOJ1zZwgjxqGByKHiOtX7TpdQyHE5lc
MiKPXfEIQILVq0pc_E2DzL7emopWoaoZTF_m0
_N0YzFC6g6EJbOEoRoSK5hoDalrcvRYLSrQAZZ
KflyuVCyixEoV9GfNQC3_os.jzw2PAithfubEEBLu
VVk4XUVrWOLrLl0nx7RkKU8NXNHq-rvKMzqg
Header

eyJhbGciOiJSUzI1NiIsImtpZCI6IjFlOWdkazcifQ.ew
ogImlzcyI6ICJodHRwOi8vc2VydmVyLmV4YW1w
bGUuY29tIiwKICJzdWIiOiAiMjQ4Mjg5NzYxMDAx
IiwKICJhdWQiOiAiczZCaGRSa3F0MyIsCiAibm9u
Y2UiOiAibi0wUzZfV3pBMk1qIiwKICJleHAiOiAxM
zExMjgxOTcwLAogImlhdCI6IDEzMTEyODA5Nz
AKfQ.ggW8hZ1EuVLuxNuuIJKX_V8a_OMXzR0E
HR9R6jgdqrOOF4daGU96Sr_P6qJp6IcmD3HP99
Obi1PRscwh3LOp146waJ8IhehcwL7F09JdijmBqk
vPeB2T9CJNqeGpegccMg4vfKjkM8FcGvnzZUN4
_KSP0aAp1tOJ1zZwgjxqGByKHiOtX7TpdQyHE5lc
MiKPXfEIQILVq0pc_E2DzL7emopWoaoZTF_m0
_N0YzFC6g6EJbOEoRoSK5hoDalrcvRYLSrQAZZ
KflyuVCyixEoV9GfNQC3_os.jzw2PAithfubEEBLu
VVk4XUVrWOLrLl0nx7RkKU8NXNHq-rvKMzqg
Payload

eyJhbGciOiJSUzI1NiIsImtpZCI6IjFlOWdkazcifQ.ew
ogImlzcyI6ICJodHRwOi8vc2VydmVyLmV4YW1w
bGUuY29tIiwKICJzdWIiOiAiMjQ4Mjg5NzYxMDAx
IiwKICJhdWQiOiAiczZCaGRSa3F0MyIsCiAibm9u
Y2UiOiAibi0wUzZfV3pBMk1qIiwKICJleHAiOiAxM
zExMjgxOTcwLAogImlhdCI6IDEzMTEyODA5Nz
AKfQ.ggW8hZ1EuVLuxNuuIJKX_V8a_OMXzR0E
HR9R6jgdqrOOF4daGU96Sr_P6qJp6IcmD3HP99
Obi1PRscwh3LOp146waJ8IhehcwL7F09JdijmBqk
vPeB2T9CJNqeGpegccMg4vfKjkM8FcGvnzZUN4
_KSP0aAp1tOJ1zZwgjxqGByKHiOtX7TpdQyHE5lc
MiKPXfEIQILVq0pc_E2DzL7emopWoaoZTF_m0
_N0YzFC6g6EJbOEoRoSK5hoDalrcvRYLSrQAZZ
KflyuVCyixEoV9GfNQC3_os.jzw2PAithfubEEBLu
VVk4XUVrWOLrLl0nx7RkKU8NXNHq-rvKMzqg
Signature

• Restrict the scope of a pipeline's access & permissions
• Apply granular access controls:
• job-tokens
• OIDC
• Use these things with a dedicated Secrets Manager:
• Hashicorp Vault (Buildkite plugin)
• AWS Secure Secrets Manager (Buildkite plugin)
• Have ingress/egress filters to the internet:
• Tailscale
• Cloudflare etc.
• Always terminate agents and wipe VMs/Machines!
Insufficient PBAC (Pipeline-Based Access Controls)

SECURITY RISKS
SECURITY RISKS
1 — Insufficient Flow Control Mechanisms
2— Inadequate Identity and Access Management
3— Dependency Chain Abuse
4— Poisoned Pipeline Execution (PPE)
6 — Insufficient Credential Hygiene
7 — Insecure System Configuration
8— Ungoverned Usage of 3rd Party Services
9 — Improper Artifact Integrity Validation
Insufficient PBAC (Pipeline-Based Access Controls)

SECURITY RISKS
SECURITY RISKS
1 — Insufficient Flow Control Mechanisms
2— Inadequate Identity and Access Management
3— Dependency Chain Abuse
4— Poisoned Pipeline Execution (PPE)
5 — Insufficient PBAC (Pipeline-Based Access Controls)
7 — Insecure System Configuration
8— Ungoverned Usage of 3rd Party Services
9 — Improper Artifact Integrity Validation
Insufficient Credential Hygiene

• Limit the blast radius of potential breaches.
• Reduce risk of Poisoned Pipeline Execution (PPE):
• Limit what code is executed in certain contexts
• Have sensitive/non-sensitive build contexts
• Have strong Pipeline-Based Access Controls (PBAC):
• Limit scope of what builds/pipelines have access to
• Use ephemeral/tightly scoped access tokens
Insufficient Credential Hygiene

Let machines do the work!

• Use a dedicated secret manager:
• HashiCorp Vault, AWS Secure Secrets Manager etc.
• Automatically scan for leaked keys and credentials:
• GitGuardian, GitHub’s configurable Secret Scanning etc.
Insufficient Credential Hygiene

Alerts are only useful if
they’re seen and acted on.

SECURITY RISKS
SECURITY RISKS
2— Inadequate Identity and Access Management
3— Dependency Chain Abuse
4— Poisoned Pipeline Execution (PPE)
5 — Insufficient PBAC (Pipeline-Based Access Controls)
6 — Insufficient Credential Hygiene
7 — Insecure System Configuration
8— Ungoverned Usage of 3rd Party Services
9 — Improper Artifact Integrity Validation
Insufficient Flow Control Mechanisms

we accept mistakes are part of
software delivery.
CI/CD exists because

Insufficient Flow Control Mechanisms
LGTM
• Unreviewed code can’t trigger deployment pipelines
• Code reviews & approvals should be part of the merge
process.
• Configure this process in your Source Control Manager:
• 2 human approvals prior to a PR being merged
• For teams with additional compliance regulations
consider using a `block step` in your pipeline.

SECURITY RISKS
SECURITY RISKS
1 — Insufficient Flow Control Mechanisms
2— Inadequate Identity and Access Management
4— Poisoned Pipeline Execution (PPE)
5 — Insufficient PBAC (Pipeline-Based Access Controls)
6 — Insufficient Credential Hygiene
7 — Insecure System Configuration
8— Ungoverned Usage of 3rd Party Services
9 — Improper Artifact Integrity Validation
Dependency Chain Abuse

Open Source
NPM, Yarn, PyPi, RubyGems, all the things…

Dependency Chain Abuse
• Get visibility into CVEs and act on them, use tools like:
• GitHub Dependabot (Enterprise orgs)
• Identifies & notifies users about vulnerable dependencies
• Opens PRs to keep dependencies updated
• Use services like Snyk or Cloudsmith who:
• Integrate with most CI/CD providers
• Do application/container scanning
• Asset Discovery and tagging (so you can pin versions)
• Avoid latest versions
• Verify the checksum

Software Bill of Materials
An immutable list of what’s in an application:
• Open source libraries (languages, imports/dependencies)
• Plugins, extensions, add-ons used
• Application code (versioned)
• Information about versions, licensing status and patch status of
these components
An SBOM for a SaaS application can include info like:
• APIs
• 3rd party services required to run the SaaS application.

SBOM > F-BOMB

CC/CD
CI/

Create actionable SBOMs

Dependency Chain Abuse
• Get visibility into packages + CVEs with tools and act on them
• GitHub Dependabot
• Snyk
• Avoid latest versions
• Verify the checksum
• Practice Continous Compliance (Put a CC in CI/CD)
• Generate SBOMs for your applications, using:
• GitHub, Cloudsmith, SonaType, JFrog, ReversingLabs
• Create action oriented workflows around your SBOMs!

Aim to
limit the blast radius

Establish
Strict Boundaries

Lean on tooling & automation

Work together to create and
adapt the human processes.

GAME OVER
GAME OVER

@MelissaKaulfuss

OWASP Top 10 CI/CD Security risks
2022 State of DevOps Report
Supply Chain Levels for Software Artifacts (SLSA)
Secure Software Development Framework (SSDF)
US National Cybersecurity Strategy (March 2023)
Auth0's Open ID Connect Handbook
Software Bill of Materials (SBOM)
Automating Governance Risk and Compliance
Creating Actionable SBOMs with Cloudsmith & Buildkite
GitHub self-service SBOM
Resources