DevSecOps journey
ABN AMRO
Stefan Simenon / Dominik de Smit
Sonatype DevSecOps Leadership Forum
25th May 2018, Amsterdam

About
2
Stefan Simenon
 Head of COE Software Development
 Focussing on: CICD, Tooling, Pipelines,
Software Quality & Security, Database
Automation, Cloud Native
 stefan.simenon@nl.abnamro.com
 +31 6 51478665
Dominik de Smit
 Teamlead Secure Coding
 Software engineer with specialization in Software
Security
 Focussing on: DevSecOps, AppSec, Cloud security,
Security awareness
 dominik.de.smit@nl.abnamro.com
 +31 6 43888985
 @dominikdesmit

CASE FOR CICD
3
Produce automated
builds and detect
errors as soon as
possible, by
integrating and
testing all changes
on a regular (daily)
basis.
High frequency
delivery of a
tested functional
piece of software
that can be
deployed to
production rapidly.
Fully automated
process including
deployment to
production without
human interaction.
Continuous
Integration
Continuous
Deployment
Continuous
Delivery
Many manual handovers
and approvals
Long lead time for
software delivery
Software quality
issues found at a late
stage
Code merging happening
at a late stage
Inefficient
cooperation between
DEV and OPS
Big non-frequent
releases to Production
 It is not only about tooling but mainly mindset & behavior, a
changed Way of Working and process improvements.
• Increase maturity of teams
• Set up the conditions (tooling, pipelines, generic building
blocks) for the teams to get working.
• Train the blocks on applying the right mindset, knowledge
and appropriate tooling
 We know other large companies which need 3 - 8 years, and
changed their approach along the way.
 Therefore we keep the overall stages in mind, but plan for the
coming three months. Focus on learning and improving instead
of long term planning.

Midrange Build & Delivery pipeline: orchestration
4

DWH
ETL
Java
Front End
BPM/
TIBCO
MicroSoft
Docker
Image
Java IIB
Mainframe
CoTS
Mobile
Pipelines within ABN AMRO

Test environment uptime improved
Improved code quality & secure coding
Improved cooperation across stakeholders
Improved time to market
Improved development processes
Source code
mgt
Build
&Unittest
Code quality
review
PackageDevelop
Compo-nent
mgt
Deploy Release tests
(ET)
Deploy
Continuousintegration
Continuousdelivery
Continuousdeployment
ProdchecksDeploy Test(ST)
Zerotouchplatforms
Code pushflow DeploymentflowBuild,QAandpackage flow
x3 deployments to UT x2,5 deployments to ET+20% successful Builds -100% Package creation time -75% Testing time
We never thought it would
be possible to develop, test
and deploy something
completely in one sprint
I-Markets doubled velocity
after 1 sprint containing
CICD improvements only
From 4 Internet
Banking releases to 18
releases per year
Core review times have
been shortened and
violations when
merging are being
prevented
Changes are being
rolled out as soon as
they are available
Increased velocity
Private Banking
Interlnational team reduced
build from 5 hours to 5
minutes
First continuous deployment
realised by identity access
mgmt team
Release times halved for
teams using XL Release
Realised benefits within ABN AMRO

DevSecOps: How to include Security in our DevOps?
7
Open Source Software
Hybrid cloud and Container
Security
Credentials Management

Open Source Software: Risks
8
 “Open source and third-party
components can help organizations
speed development and reduce cost,
but they can also increase risk
exposure when organizations do not
manage them effectively. Evaluating
and approving standard components
helps organizations streamline their
software supply chains, improve
quality, and reduce risk and cost.”
 By 2020, 50% to 80% of code in new
applications will come from external
third-party channels.”
 “By 2020, 50% of organizations will
have suffered damage caused by
failing to manage trust in their or
their partners', software development
life cycles — causing revenue loss of
more than 15%.”
 “31% suspect or have verified a
breach related to open source
components in the last 12 months

Standard CI pipelines and build breakers
9
Dependency
scan
Check out
project from
SCM
Developer
triggers build
Build project and
execute unit
tests
Code quality
scan
Secure coding
scan
Publish
Deployable
artifact
N
Y
ABN AMRO has introduced a set of quality gates and build breakers. The principle is
that the Jenkins build is broken once the required quality or security is not met and
the developer needs to fix the defect in order to proceed. The developer has access to
software quality in his IDE so defects can be detected and fixed in an early stage

DevSecOps Maturity Model
10
CULTURE
Surprising with a lots
of push back
Full awareness but
feeling helpless
Integrated and talked
about by Execs
Measured by Execs
Context driven
decisions
SKILLS
Skills developed
outside job function
Skills lining up with
job functions
Skill development
paired with the job
Proactive skill
development to meet
roadmap needs
Knowledge evolves
inline / Lessons
savored
PROGRAMS /
OUTCOMES
Just getting by
Orderly processes &
faster reactions
Reduced number of
incidents
Measurable
difference in attacks
Predictive &
Proactive
SECURITY
PRIORITIES
P0/critical waiting for
attackers
P0 and P1s some
hygiene, getting fixed
P0 and P1
compliance
Attack surface driven
& measured
Staying ahead of bad
guys
1
2
3
4
5
Burp Crawl Walk Run Fly
Insanity
Reactive
Proactive
Measurable
ContinuousCurrent state ABN AMRO

Nexus Life Cycle status and way forward
11
Status
 Updated open source policy
 Nexus LifeCycle choosen and implemented
 Automated onboarding pipeline, security scans included in standard Java,Front End,
Mobile and MicroSoft pipelines.
 Application security training conducted
 Awareness sessions about OSS and Nexus Lifecycle conducted
Way forward
 CICD metrics dashboard to visualize security issues per grid/domain, both for security
issues in Development and Production environment.
 Track progress via senior management meetings
 Additional application security training
 Guiding dev teams to fix security issues coming
 Increase security awareness via senior management
 Reward teams who have the right focus on security

An hybrid cloud strategy has been defined to support our
digital transformation
12
Private Cloud (IBM CMS)
 Consists of hardware deployed
within ABN AMRO data centres
 ABN AMRO controls the underlying
infrastructure, IBM manages CMS
 IaaS solution, provisioning of
Virtual Machines (VM)
 Limited PaaS and lacks SaaS
capability
 Currently majority of
application run on CMS
Public Cloud (AWS, Azure)
 Offers advanced SaaS, PaaS, IaaS
 Increased developer productivity due to large
suite of out of the box technologies and
services (e.g.,150+ services in AWS)
 Delivers an automated and native platform for
Agile and DevOps
 Pay as you go: pay only for the used services
 50-100 applications will land on Public Cloud
in 2018, further increase foreseen in 2019

7 work streams have been defined to prepare further scaling
13

Cloud Native
14
Define a microservice architecture structures an
application as a collection of loosely coupled
services, which implement business capabilities.
The microservice architecture enables the
continuous delivery/deployment of large, complex
applications.Define a Serverless architecture allowing
developers to build and run applications and
services without thinking about servers.
Select, implement and maintain a PaaS that
provides a platform allowing developers to
develop, run, and manage applications without the
complexity of building and maintaining the
infrastructure typically associated with
developing and launching an app. .
Define container security on multiple levels.
Define and implement standard dockerized
pipelines to land on public cloud.
Cloud Native: Design
and develop cloud
native applications
that can take
advantage of the
technologies and
services offered by
our preferred cloud
platforms.

Container security – Lifecycle
15
BUILDCODE SHIP RUN
Security is needed in each step of
the lifecycle

Container security – Tools …
16
Anchore Navigator
AppArmor
AquaSec
BlackDuck Docker security
Cavirin
Cilium
CoreOS Clair
Docker-bench security
Dockscan
Falco
HashiCorp Vault
NeuVector
Notary
OpenSCAP
REMnux
SELinux
Seccomp
StackRox
Sysdig Secure
Sysdig
Tenable Flawcheck
Twistlock

Container security – Preventing the bad
17

Container security – Pipeline
18
Smoke test
Jenkinsfile +
Dockerfile from
SCM
Developer
triggers Docker
image build
Build docker
image
Docker lint
syntax check
Docker container
dependencies
check
Sign + Publish
Docker image in
trusted registry
N
Y
Docker container
OS check
Apply security
profiles
(AppArmor)

Container security – Focus area’s
19
• Docker host and kernel security (prevent
container breakout)
• Container image authenticity (signing)
• Container resource abuse
• Docker security vulnerabilities present in the
static image
• Docker credentials and secrets management
• Docker runtime security monitoring
• Secure container orchestrator
• and many more…

Credentials management – What?
20
Credentials Management are all
processes and techniques to securely
access secrets. A secret is anything
that you want to tightly control access
to, such as API keys, passwords, or
certificates.

Credentials management – Risks
21
• A hacker accessed a docker registry that contained the
entire Vine source code, API keys and secrets
• An insider stored a secret key in a public repository. This
key was used to compromise a database containing
personal information about drivers
• Developers stored privileged credentials and keys in
source code

Credentials management – Secrets
22

Credentials management – Secrets
23
• 75% organizations do not have
a privileged account security
strategy for DevOps
• Fewer than half report that
DevOps and security teams
consistently work together
• Nearly all (99%) of security
pros and DevOps respondents
failed to identify all places
where privileged accounts or
secrets exist
Source: CYBERARK GLOBAL ADVANCED THREAT LANDSCAPE REPORT 2018: FOCUS ON DEVOPS

Credentials management – Focus area’s
24
• Key rolling
• Granular access permissions
• Secure storage
• Detailed audit logs
• Monitoring
• Must fit seamlessly in the DevOps environment

Questions?
25