What an AppSec Pipeline is, why it's going to change AppSec, how to take good ideas from DevOps and Agile into AppSec Programs and various stages of maturity for AppSec Pipelines. All done with the hope that others will start on their AppSec Pipeline journey.

1. Building a Secure DevOps Pipeline
(for your AppSec Program)
Aaron Weaver
Matt Tesauro
AppSec USA 2017

2. Building a Secure
DevOps Pipeline
(for your appSec program)
Matt Tesauro
Aaron Weaver
AppSec USA 2017

3. Hello!
I am Aaron Weaver
All around good guy and craftsman of wood and coffee
aaron@sec.training / @weavera

4. Hello!
I am Matt Tesauro
I think AppSec needs to change and
I’m going to tell you how I see it changing
matt.tesauro@owasp.org / @matt_tesauro

5. AppSec
Evolution

7. ✣ Radically changed travel in the US
✣ Travel time across the US
Pre-train: 6 months + $1,000
Post-train: 1 week + $150
✣ Towns that had a stopping prospered
Those that didn’t, faded away
The Iron Horse Straddles
America

8. Trains == Change
✣ Changed the landscape for better or
worse
The US ‘got smaller’ - travel was in reach
Expanded markets, more customers
‘Cost’ of going west went way down

9. Trains <==> DevOps
✣ Changed the landscape for better or
worse
DevOps changed IT for better or worse
The US ‘got smaller’ - travel was in reach
Batch / change size got smaller (CI/CD)
Expanded markets, more customers
Increased agility, more customers
’Cost’ of going west went way down
Cost of experiments goes way down

10. When will we see this?
DevOps AppSec

11. Genis: Pipeline
Species: AppSec
A new evolution of AppSec

12. AppSec Pipelines
Using CI/CD as inspiration,
figure out your AppSec workflow

13. Key Features of
AppSec Pipelines
✣ Designed for iterative improvement
✣ Provides a reusable path for AppSec activities
✣ Provides a consistent process for both the team
and our constituency
✣ One way flow with well-defined states
✣ Relies heavily on automation
✣ Grow in functionality organically over time
✣ Gracefully interconnects with the development
process

15. What we want is
ala carte
- Just with a limited
number of choices

16. Pipeline

17. Gen 1 Pipelines
Look at your team's purpose
and those processes
which aid it

18. Spending time optimizing
anything other than
the critical resource is an
illusion.
W. Edwards Deming

19. AppSec Personnel
✣ They are the critical resource
- optimize their work
Automate the things that don’t require a human brain
Drive up consistency
Increase tracking of work status
Increase flow through the system
Increase visibility and metrics
Reduce any dev team friction with application security

20. Gen 1: Inside out focused AppSec

21. Then,
once your house
is in order...

22. Gen 2 Pipelines
Look outside your team's
purpose and those
processes which aid it

23. DevOps Pipeline AppSec Pipeline
Integrate with Dev & Ops teams

24. Weaponizing Jenkins
✣ Zero false positives
Anaphylactic shock
✣ Health Checks vs Scanning
Run these all the time
✣ Home of specific issue tests
Find a vuln, write a test
✣ Cadence for longer running tests
These NEVER break the build
Every X builds or every Y days

25. OWASP & AppSec Pipelines

26. OWASP Defect Dojo
✣ Single source of truth for findings
✣ AppSec Programs, QA, Pen Testers
Custom report generation
Metrics and Dashboards
App & Infrastructure findings supported
✣ New-ish OWASP Project
Code base is 3+ years - started at Rackspace
✣ Community and contributor friendly
Bugs triaged, verified and fixed quickly
11 contributors from multiple companies
✣ Github: 262 stars, 106 forks, 199 watchers

27. Species: AppSec
Family: Automation
Evolving AppSec faster

28. What went hand in
hand with the
transcontinental
railroad expansion?

29. The Telegraph

30. Telegraph | Automation
✣ Sped up signaling and communication
Enhanced benefits of easier travel
Followed the existing tracks
✣ Linked cities using a standard protocol
Morse code

31. Telegraph | Automation
✣ Sped up routine tasks
Enhances benefits of existing AppSec Pipeline
Follows the same path aka consistent
✣ Links software using a standard protocol
HTTP / REST

32. A call to action...

33. AppSec
Chat Ops
Making chat the way
you do security

34. FYI: You’re being attacked

35. FYI: You’re being blocked

36. Advice for Devs - 24x7

37. Static Analysis Integration
Recurring static analysis in about 10 minutes!

38. Scaling with
Docker
Containers

39. docker run -it --name kali-pipeline
kali-pipeline /bin/bash
/usr/local/bin/run.sh
'nikto localhost -h localhost -T 58'
results.txt

40. Docker Security
Tool Launch
(python, Go)
ZAP
Nikto
Return ZAP IP
Run Scan, Push
Results to S3

41. Benefits
✣ Effectively Scales
✣ Build security tools once,
run anywhere
✣ Ease of deployment

42. Pull in or scale out,
your choice
Pull in Docker containers
to your build server
ZAP
Nikto
Scale out to Docker Swarm
ZAP
Nikto

43. Jenkins Pipeline

44. Pipeline as Code

45. Conduct your own
AppSec Pipeline Experiment
Come down out of the
traditional AppSec trees

46. Pick a language
✣ To do AppSec well, you need to know
something about coding
✣ Don’t care what language, pick one
and stark hacking away
✣ Most Pipeline code is glue code
- l33t algorithms need not apply

47. Case Studies

48. AppSec Pipeline - Company #1
✣ Security Findings
Turn each into a self-contained test
✣ Add those tests to Jenkins
Run hourly or at least daily
Turn green when they are fixed
✣ Tied alerts / Chat ops to those tests
Let them tell you when they are fixed
✣ Developer knows release X fixed finding Y
Bonus points for connecting Jenkins test passing to
closing Jira bug
✣ 2 FTEs assessed 35 Apps in year 1

49. 2014
✣ 44 assessments
~5x increase
2015
✣ ~200 assessments
Changes from 2014 to 2015:
- Created the AppSec Pipeline - initial launch in March 2015
- AppSec team numbers dropped
- lost a couple of key people approx 3.5 FTEs
- Two of the AppSec team members went meta for most of 2015
Company #2

50. 2015
✣ ~200 assessments
~2x increase
2016
✣ 414 assessments
Changes from 2015 to 2015:
- Lost 2 key FTE engineers
- AppSec team numbers dropped
- not every vacant FTE position was filled
Company #2

51. 2014
✣ 44 assessments
9.4x
increase
2016
✣ 414 assessments
Things to remember
- Year 1 may go slow - you need to build a solid foundation
- Get your house in order, THEN reach out to other teams
- Divide tests into
- Quick, low false-positive - these go into CI/CD
- Longer, less accurate tests
Company #2

52. @weavera@weavera

53. @weavera@weavera
“I am a nice shark, not a mindless
eating machine. If I am to change this
image, I must first change myself. Fish
are friends, not food.”
-Bruce, Chum and Anchor

54. @weavera@weavera
“I am a nice security professional,
not a mindless vulnerability spewing
machine. If I am to change this image,
I must first change myself.
Developers are friends, not fools.”
-Bruce, Aaron and Matt

55. @weavera@weavera
I’m with Bruce
@BruceSecDevOps
#BruceSecDevOpsTM

56. Thanks!
Any questions?
Aaron Weaver
@weavera
aaron@sec.training
/in/aweaver
github.com/aaronweaver
Matt Tesauro
@matt_tesauro
matt.tesauro@owasp.org
/in/matttesauro
github.com/mtesauro

57. References
Presentation template:
http://www.slidescarnival.com/dolabella-free-presentation-template/840
Black and White train image (original)
https://www.youtube.com/watch?v=-80sFvilSXs
Train/Transport facts from:
https://gtgtechnologygroup.com/transcontinental-railroad/
https://www.thoughtco.com/effect-of-railroads-on-the-united-states-104724
http://www.american-rails.com/transcontinental.html
Meeting of the railroads image (original) public domain image at
https://www.thoughtco.com/effect-of-railroads-on-the-united-states-104724
Model Train cars (original)
https://www.pinterest.com/njaredmartin99/trains/
What color is your parachute (original)
http://earthconservant.com/100-ways-earthfit-day-37-color-parachute/

58. Telegraph poles by a railroad picture (original)
https://www.pinterest.com/pin/412360909600598272/
Telegraph key
https://openclipart.org/detail/188507/telegraph-key

59. Instructions for use
EDIT IN GOOGLE SLIDES
Click on the button under the presentation
preview that says "Use as Google Slides
Theme".
You will get a copy of this document on
your Google Drive and will be able to edit,
add or delete slides.
You have to be signed in to your Google
account.
EDIT IN POWERPOINT®
Click on the button under the presentation
preview that says "Download as PowerPoint
template". You will get a .pptx file that you can
edit in PowerPoint.
Remember to download and install the fonts
used in this presentation (you’ll find the links to
the font files needed in the Presentation design
slide)
More info on how to use this template at
www.slidescarnival.com/help-use-presentation-template
This template is free to use under Creative Commons Attribution license. You can keep the Credits slide or
mention SlidesCarnival and other resources used in a slide footer.

60. Credits
Special thanks to all the people who made and
released these awesome resources for free:
✣ Presentation template by SlidesCarnival
✣ Photographs by Unsplash
✣ Paper texture by GraphicBurguer