The document is a transcript from a presentation given by Joshua Corman and Gene Kim at a security conference in San Francisco in September 2012. The presentation discusses the problems with current security practices, introduces the concepts of DevOps and Rugged DevOps, and provides three ways ("systems thinking", "amplifying feedback loops", and "culture of continual experimentation") to implement Rugged DevOps practices to improve security. The overall message is that cultural and process changes are needed, not just technical fixes, to build more secure software.
TeamStation AI System Report LATAM IT Salaries 2024
Rugged DevOps: Security at Ludicrous Speed
1. SECURITY IS DEAD.
LONG LIVE RUGGED DEVOPS:
SEPTEMBER 12 – 14, 2012
IT AT LUDICROUS SPEED…
GRAND HYATT, SAN FRANCISCO
Joshua Corman
TRUTH, LIES Gene Kim
AND DECISIONS
Moving Forward in an Insecure World September 2012
Organized by
2. Gene Kim: Two Truths and a Lie
Please fill out the table below with two statements that are true and one lie
about yourself. I will put the information into the polling system to go live
before your presentation.
Statement Truth or lie?
I didn't know that Purdue University was in Indiana, Truth
otherwise I wouldn't have gone there
I still carry around a J. R. R. Tolkien book in my Lie
briefcase everywhere I go
I have an outrageous man-crush on my co-presenter, Truth
Josh Corman
3. About Joshua Corman
• Director of Security Intelligence for Akamai Technologies
- Former Research Director, Enterprise Security [The 451 Group]
- Former Principal Security Strategist [IBM ISS]
• Industry:
- Expert Faculty: The Institute for Applied Network Security (IANS)
- 2009 NetworkWorld Top 10 Tech People to Know
- Co-Founder of “Rugged Software” www.ruggedsoftware.org
- BLOG: www.cognitivedissidents.com
• Things I’ve been researching:
- Compliance vs Security
- Disruptive Security for Disruptive Innovations
- Chaotic Actors
- Espionage
- Security Metrics
3
4. Josh Corman: Two Truths and a Lie
Please fill out the table below with two statements that are true and one lie
about yourself. I will put the information into the polling system to go live
before your presentation.
Statement Truth or lie?
My philosophy thesis was entitled "Schizophrenic Truth
Alienated Tennis Pros in Love"
I'm the president of my local zombie survivalist Lie
chapter
I have a life sized statue of Spider-Man in my foyer Truth
5. About Gene Kim
• Researcher, Author
• Industry:
- Invented and founded Tripwire, CTO (1997-2010)
- Co-author: “Visible Ops Handbook”(2006), “Visible Ops Security” (2008)
- Co-author: “When IT Fails: The Novel,” “The DevOps Cookbook” (Coming May 2012)
• Things I’ve been researching:
- Benchmarked 1300+ IT organizations to test effectiveness of IT controls vs. IT
performance
- DevOps, Rugged DevOps
- Scoping PCI Cardholder Data Environment
5
6. PART 1: THE PROBLEM
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
Joshua Corman
TRUTH, LIES Gene Kim
AND DECISIONS
Moving Forward in an Insecure World September 2012
Organized by
7.
8. Consequences: Value & Replaceability
http://blog.cognitivedissidents.com/2011/10/24/a-replaceability-continuum/
8
16. SOFTWARE
AS
VULNERABILITY
SEPTEMBER 12 – 14, 2012
Grand Hyatt, San Francisco Organized by
17. CONNECTED
AS
EXPOSED
SEPTEMBER 12 – 14, 2012
Grand Hyatt, San Francisco Organized by
18. OUR CHALLENGES ARE NOT
TECHNICAL,
BUT CULTURAL
SEPTEMBER 12 – 14, 2012
Grand Hyatt, San Francisco Organized by
19. WE CAN DO
BETTER
SEPTEMBER 12 – 14, 2012
Grand Hyatt, San Francisco Organized by
20. PART 2: DEVOPS
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
Joshua Corman
TRUTH, LIES Gene Kim
AND DECISIONS
Moving Forward in an Insecure World September 2012
Organized by
35. PART 3: RUGGED
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
Joshua Corman
TRUTH, LIES Gene Kim
AND DECISIONS
Moving Forward in an Insecure World September 2012
Organized by
36. WHAT IS RUGGED?
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
36 Organized by
37. WHAT IS RUGGED?
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
37 Organized by
38. SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
TRUTH, LIES
AND DECISIONS
Moving Forward in an Insecure World
RUGGED SOFTWARE DEVELOPMENT
Joshua Corman, David Rice, Jeff Williams
2010
Organized by
64. PART 4: ROCKING INFOSEC WITH
SEPTEMBER 12 – 14, 2012
RUGGED DEVOPS
GRAND HYATT, SAN FRANCISCO
Joshua Corman
TRUTH, LIES Gene Kim
AND DECISIONS
Moving Forward in an Insecure World September 2012
Organized by
67. The First Way:
Systems Thinking (Left To Right)
Understand the flow of work
Always seek to increase flow
Never unconsciously pass defects downstream
Never allow local optimization to cause global degradation
Achieve profound understanding of the system
68. Create One Step Environment Creation Process
Make environments available early in the Development
process
Make sure Dev builds the code and environment at the same
time
Create a common Dev, QA and Production environment
creation process
69. Embed Into Automated Infrastructure Team
Get educated on open source tools like puppet and chef
Provide them your hardening guidance
Add your monitoring tools
70. Break Things Early And Often
“Do painful things more frequently, so you can make it less
painful… We don’t get pushback from Dev, because they
know it makes rollouts smoother.”
-- Adrian Cockcroft, Architect, Netflix
71. Break Things Early And Often
Enforce consistency in code, environments and configurations
across the environments
Add your ASSERTs to find misconfigurations, enforce https,
etc.
Add static code analysis to automated continuous integration
and testing process
72. The First Way:
Systems Thinking: Infosec Insurgency
Have someone attend the daily Agile standups
• Gain awareness of what the team is working on
Define what changes/deploys cannot be made without
triggering full retest
74. The First Way:
Outcomes
Determinism in the release process
Creating single repository for code and environments
Consistent Dev, QA, Int, and Staging environments, all
properly built before deployment begins
Decreased cycle time
• Reduce deployment times from 6 hours to 45 minutes
• Refactor deployment process that had 1300+ steps spanning 4 weeks
Faster release cadence
76. The Second Way:
Amplify Feedback Loops (Right to Left)
Understand and respond to the needs of all customers,
internal and external
Shorten and amplify all feedback loops: stop the line when
necessary
Create quality at the source
Create and embed knowledge where we need it
77. “We found that when we woke up developers at 2am, defects
got fixed faster than ever”
-Patrick Lightbody,
CEO, BrowserMob
78. Phase 2: Extend Release Process And Create Right ->
Left Feedback Loops
Invite Dev to post-mortems/root cause analysis meeting
Have Dev and Infosec cross-train IT Operations
Ensure application monitoring/metrics to aid in Ops and
Infosec work (e.g., incident/problem management)
79. The Second Way:
Amplify Feedback Loops: Infosec Insurgency
Give production feedback to developers: being attacked is a gift
• Capture all instances of “UNION ALL” in user input and graph it, show it to
developers
• Show all instances of segfaults
Create reusable Infosec use and abuse stories that can be added to every
project
• “Handle peak traffic of 4MM users and constant 4-6 Gb/sec Anonymous DDoS
attacks”
Pre-enable, shield streamline successful audits
• Document separation of duty and compensating controls
• Don’t let them disrupt the work
80. The Second Way:
Outcomes
Defects and security issues getting fixed faster than ever
Reusable Ops and Infosec user stories now part of the Agile
process
All groups communicating and coordinating better
Everybody is getting more work done
82. The Third Way:
Culture Of Continual Experimentation And Learning
Foster a culture that rewards:
• Experimentation (taking risks) and learning from failure
• Repetition is the prerequisite to mastery
Why?
• You need a culture that keeps pushing into the danger zone
• And have the habits that enable you to survive in the danger zone
83. “The best way to avoid failure is to fail constantly”
84. An Innovation Culture
“By installing a rampant innovation culture, they now do 165
experiments in the three months of tax season.
Our business result? Conversion rate of the website is up 50 percent.
Employee result? Everyone loves it, because now their ideas can make
it to market.”
--Scott Cook, Intuit Founder
85
87. Phase 3: Organize Dev and Ops To Achieve
Organizational Goals
Allocate 20% of Dev cycles to non-functional requirements
Integrate fault injection and resilience into design,
development and production (e.g., Chaos Monkey)
88. The Third Way:
Culture Of Continual Experimentation And Learning:
Infosec
Infosec remediation projects in the Agile backlog
• Make technical debt visible
• Help prioritize work against features and other non-functional
requirements
Release your Chaos Monkey
• Evil/Fuzzy/Chaotic Monkey
• Eridicate SQLi and XSS defects in our lifetime
Find processes that waste everyone’s time
Eliminate needless complexity
89. The Third Way:
Outcomes
Technical debt is being paid off
Exploitable attack surface area decreases
Continual reduction of unplanned work
More cycles for planned work
More resilient code and environments
Balancing nimbleness and practiced repetition
Enabling wider range of risk/reward balance
90. PART 5: WHY?
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
Joshua Corman
TRUTH, LIES Gene Kim
AND DECISIONS
Moving Forward in an Insecure World September 2012
Organized by
91.
92.
93.
94. When IT Fails: The Novel and The DevOps Cookbook
Coming in July 2012
“In the tradition of the best MBA case studies, this
book should be mandatory reading for business and
IT graduates alike.” -Paul Muller, VP Software
Marketing, Hewlett-Packard
“The greatest IT management book of our
generation.” –Branden Williams, CTO Marketing, RSA
95. When IT Fails: The Novel and The DevOps Cookbook
If you would like these slides, the “Top 10
Things You Need To Know About DevOps,”
Rugged DevOps resources, and updates on
the book:
Sign up at http://itrevolution.com
Email genek@realgenekim.me
Give me your business card
96. END
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
Joshua Corman
TRUTH, LIES Gene Kim
AND DECISIONS
Moving Forward in an Insecure World September 2012
Organized by
Notes de l'éditeur
There are many ways to react to this: like, fear, horror, trying to become invisible… All understandable, given the circumstances…Because infosec can no longer take 4 weeks to turn around a security review for application code, or take 6 weeks to turnaround a firewall change. But, on the other hand, I think it’s will be the best thing to ever happen to infosec in the past 20 years. We’re calling this Rugged DevOps, because it’s a way for infosec to integrate into the DevOps process, and be welcomed. And not be viewed as the shrill hysterical folks who slow the business down.
Tell story of Amazon, Netflix: they care about, availability, securityIt’s not a push, it’s a pull – they’re looking for our help (#1 concern: fear of disintermediation and being marginalized)
At RSA 2009, Josh Corman, Jeff Williams, and David Rice were chatting at the Greylock cocktail party.
So software not only need
…fast, and…
…agile, but it also needs to be…
…rugged. Capable of withstanding…
…the harshest conditions…
…and most unfriendly environments…
From Rugged Handbook: https://www.ruggedsoftware.org/documents/
From Rugged Handbook: https://www.ruggedsoftware.org/documents/
This story is about how Bill, the thoughtful and methodical VP IT Operations, who saves some of the largest problems of the company. It’s a story about a Visible Ops and DevOps style transformation. It’s how Bill saves the company, helping it achieves their project goals, operational goals, security and compliance goals.And Steve the CEO realizes that Bill, the lowly VP of IT Operations, is the person who saved the company.
This story is about how Bill, the thoughtful and methodical VP IT Operations, who saves some of the largest problems of the company. It’s a story about a Visible Ops and DevOps style transformation. It’s how Bill saves the company, helping it achieves their project goals, operational goals, security and compliance goals.And Steve the CEO realizes that Bill, the lowly VP of IT Operations, is the person who saved the company.
This story is about how Bill, the thoughtful and methodical VP IT Operations, who saves some of the largest problems of the company. It’s a story about a Visible Ops and DevOps style transformation. It’s how Bill saves the company, helping it achieves their project goals, operational goals, security and compliance goals.And Steve the CEO realizes that Bill, the lowly VP of IT Operations, is the person who saved the company.