3. Nick Galbreath (@ngalbreath)
! Spoken at Black Hat, DEFCON, OWASP
! Book on cryptography
! but really...
! Engineering Management and Software
Development for high growth startups.
! Personal site http://www.client9.com/
4. libinjection
! Author of libinjection
! A very different way of doing SQLi detection
! Right now in another room, Vladimir
Vorontsov is showing how to bypass it (to
be fixed shortly)
! Check it out and file bugs on github
http://libinjection.client9.com/
! Found a bypass?
What database
and query is needed
to exploit it?
RU ♥
SQLi
5. IPONWEB
! customized online advertising infrastructure
and exchanges
! engineering offices in Moscow, with
business offices in London, New York and
Tokyo.
! YEAH IN MOSCOW
! YEAH WE ARE HIRING
! Send email to nickg@iponweb.net
6. Well that's a bold statement...
Fixing Security by
Fixing Development
Using Continuous Deployment
7. and here's another
For web applications, our release-based
software development lifecycle is still
based on a pre-Internet model and is
harmful to organizations and
particularly harmful for security.
8. What needs fixing?
! SQLi dropped from #8 to #14 in the latest
White Hat "The State of Web Security"
report. Good news, right?
! This means SQLi is only 7% of websites.
That's 1 in 15.
And this is the #14 vulnerability!
! And time to fix was on average 196 days.
That's embarrassing.
Veracode claims 32% of incoming web applications have SQLi
https://info.veracode.com/state-of-software-security-report-volume5.html
https://reg.whitehatsec.com/
WPstats0513
9. Even worse...
! Number 1 driver to fix security
problems...
compliance.
! Number 1 reason to not fix security is...
compliance.
! Not..
! keeping our employees and customers safe
! protecting corporate interests.
! improving quality
! being good at what we do.
10. Security Products #1 .. in security bugs
VeraCode: State of
Software Security, V4
December 2011
Security Product
74% Fail Rate
11. Let's Just Give Up
! “You could spend all your resources chasing
such things as this,” William Ribich, the former
president of Technology Solutions Group
[ QinteliQ ], said in an interview in January.
Ribich, who retired in November 2009, shortly
after the discovery of a major data theft, said he
needed to balance the uncertain risk that the
hackers could use what they stole against a
growing shopping list of security products and
consulting fees.
! "You finally have to reach a point where you
say ’let’s move on,’” he said.
http://www.bloomberg.com/news/2013-05-01/china-cyberspies-outwit-u-s-stealing-military-secrets.html
^^^ The Russian and Chinese hackers did not move on ^^^^.
12. I would call that broken
! But preventing SQLi isn't a technically
hard problem.
! And most security patches are very small.
! How did we get here?
13. Software Product Model
! Code flows between functional groups
! Product Managers spec code
! Engineers write code
! QA engineers tests code
! Release engineers package code
! Operations runs code
! ... and Security does something too.
14. High Distribution Cost
! The Software Product Model is designed for
software where the cost of distribution is
high. "High" might be financial, risk, time,
resources, customer annoyance.
! Retail, physical product, CD/DVD
! Embedded of Exotic Hardware
! Safety, Medical or Defense Systems
! Operating Systems (desk or phone)
! Homework (1-time deploy)
17. Web Applications Year 2000
! Mostly followed Software Product
Model since that's all we knew.
! High barrier to entry
! Specialized Hardware, Software and
People needed to get started.
! Lots of engineering needed to keep
things running.
! (side note: CERT/CVE started in
1999)
18. True Story #1
! "Can't push out the spelling error fix
– it's too risky"
! "That code as already been through
QA, it's locked down."
! "Product has to prioritize that
change, else we aren't touching it."
19. True Story #2
! We'll do an iteration, where we try to fix
as many things as possible.
! This won't be a scheduled iteration,
it will be done because so many things
are piled up.
! So the spelling error will get fixed...
uhh, who knows when.
20. Web Applications 2013
! Almost no barrier to entry
! Commodity hardware
! Programming not that hard
! Scaling problems can
be mostly outsourced
(mostly)
21. Cost of Distribution 2013
! Frequently no compile step
or it's very fast.
! Moving to production a few
kilobytes or megabytes of code
over 1Gbps, 10Gbps link.
! In other words... free
22. Failure is very different however
! Most web applications are data-driven.
! Frequently have social features, APIs,
user-generated content.
! Failures might be due to algorithmic
problems... but...
! Most likely to due to user input, bad data
in database or operational load.
! this means data in past can cause
problems in the future.
23. Releases and Problems
! When a web-release goes out, and
has problems....
! Next week is spent tracking down
who changed what, where.
! Re-QA
! Re-Push
! meanwhile new code is piling up.
24. When SPM meets Web Apps
! A long time between code being written
and code being released.
! Might be weeks or months
! Feedback loop between code-in-dev
and code-in-production is broken
! When security or bug reports come in,
the author is likely on a different project.
25. Hypothesis
! It is impossible to simulate the production
environment in development, either due to
operational differences or data
differences.
! No amount of QA or Security Testing can
prove you don't have bugs, vulnerabilities,
or cause severe operational problems.
! You have bugs and vulnerabilities,
right now, in your application.
26. Impedance Mismatch
! Easy to write code, +
! Long release cycles +
! Security as an end-of-line or out
of band process ==
! no one cares
! Something is going to break,
and most people don't care.
28. So the Answer is...
! Going slower? I'm sure your boss
will love that suggestion
! More steps and process? In other
words, slower.
! Asking for more people? Sure but
good luck hiring them. Doesn't
scale.
! Asking for more products? Since
the others have worked so well.
29. Continuous Deployment
! Also known as Continuous Delivery.
! A System of Software Production
Characterized by Numerous Small
Changes the Production Environment,
initiated by the author of the change.
You change it, you push it to prod.
30. Deployment != Feature Release
time
change
New code goes out all the time.
New features get turned on in
a separate process.
31. "Writing Software"
! Software Developers think their job is
writing software.
! And so, they love to make things perfect
before anyone else sees it.
! Impolite: "data hiding"
! code is hiding on developer's computer
! or on some branch
! in other words invisible until it's ready.
32. Actually
! The software engineer's job is actually
writing running software, that works well.
! This idea is so alien, that companies have
to remind the engineers of this.
33. Rackspace Haiku
writing code is hard
if you cannot deploy
it does not matter
@paulvx from DevOpsDays Austin 2013
34. Facebook's Analog Labs Poster
"Move Fast and
Break Things....
Except "Push" (deployment system)
via http://mitadmissions.org/blogs/entry/
move_fast_and_break_things
36. Today's goal
! but for today the goal is getting the
developer to care about their code
in production.
! If you don't have that, I don't think you can
really solve security problems.
37. How does this work?
! Really?
Developers push their own code
out?
! How is this not a disaster.
! How is this not a security disaster?
38. The Deploy Button
! What is you had a button that said
"DEPLOY"
! That pushed to production, whatever
is current in your source control
system.
! And took about a minute
! The change and who pressed the
button is logged, but that's it.
39. Part 1: Fear
! No one is going to push
it ;-)
! Meanwhile code is piling
up
Real example: A new hire I had at Etsy
was afraid of deploying an HTML change
that they made.
"But I don't want to break the site!"
40. Part 2: First Push
! Someone brave will press the button
! And very likely the site will explode,
and a rollback will need to be done.
! They'll know since someone else will
have told them.
41. Part 3: With Graphs
! Let's get all those operational graphs
out in the open. And put them right
next to the button.
http://codeascraft.com/2011/02/15/measure-
anything-measure-everything/
42. Part 4: Push #2
! Repush
! Site might still explode
! But the developer is aware and
can rollback.
43. Take 5: Isolation
! Hmmm, the developer notices that in the
change set, a million things are going out.
! Maybe just pushing out a smaller change
will help isolate the issue.
44. Take 6: Success!
! Yes, the developer just pushed out
some code and made the site better.
! The secret about continuous
deployment is small changes that
can be easily understood.
45. Take 7: Dark Pushes
! Now we got some bugs fixed, let's push
a feature.
! First let's push out all the supporting
files. Since they aren't being called,
they do nothing and are safe to push
out.
! Now everyone can see them
46. Take 8: Getting the feature live
! Instead of "all at once", we slowly ramp up a
feature.
! if (user_id % 20 == 0):
do new feature;
! we change change the percentage easily with
another code push.
! or turn it down. Much nicer change log.
! While the site didn't explode, it's hard to see if
the feature is being used or not.
47. Take 9: Application Level Graphs
! Allow developers to instrument their
code so they can see what is
happening in production.
! Enter StatsD and other
UDP-based tools
! Enter centralized logging and in-
application method to make it easy
to log problems.
48. Take 10: Communication
! So far good for one developer.
! To scale up, you'll need a system to allow
developers.
! IRC-like tools work well (e.g. "the push
channel") – skype, jabber, hangouts, etc
49. Along the way
! Expose production logs to developers
! Add in a staging-step where the code
goes to faction of the cluster, so
developers can test with real traffic
! Try to make development closer to prod.
! Make "smoke tests" to catch basic errors
! Add syntax checkers to eliminate obvious
issues.
! Use static analysis to find bugs
50. Mistakes will happen
! Do postmortem analysis
! Everyone thought they were doing
the right thing at the time.
! "How can the environment be
changed to prevent this" and build
tools to enforce it.
! (Rarely can you truly change people)
52. That guy who pushes at 3am
! Courtesy and convention will
converge very quickly when the site
goes down at 3am and the
developer starts getting calls ;-)
! Of hours pushes of course can
happen, when they notify operations.
53. What About Code Reviews?
! Yes, please do them.
! Nothing here prevents code reviews.
! In fact code reviews are easier since
! they are small
! they are in mainline not some branch
54. What about Security Reviews
! Please do them.
! Nothing here eliminates
architectural planning or review.
! This actually doesn't change the
SDLC very much.
55. What about Agile Methods
! (everyone seems to have a different idea
of what Agile is but..)
! Agile methodologies typically work to
improve the business spec / development
cycle. (are you building what the
customer wants)
! But doesn't address code deployment.
! They are complimentary practices.
56. What about Customer Service?
! "Don't they freak out with all the
changes?"
! Remember: deployment != feature release
! Most deployments do very little from the
customer point of view
! Feature releases (frequently controlled by
ramp-ups or flags) always needs to be
coordinated with product and customer
service.
57. What about Compliance? PCI?
! Let me tell you about compliance...
! mechanism not policy
! compliance is a lot easier when it's done
every day instead of a once-a-year audit.
59. Obvious Benefit to Security
! Security patches can go out quickly
! You know this since they are now
just part of a normal development
cycle and code goes out regularly.
! Why not clear out those low-priority
security problems?
60. More Importantly
! That Engineer who previously did not
push code is now sensitized that their
code has consequences and are
responsive and empowered to fix it.
! It’s amazing how interested engineers
become in security when you find
problems with their code when they are
able to fix it quickly themselves.
61. New Security Math
! Instead of focusing only on increasing MTTF,
which will never be infinite
! more firewalls, more process, more magic
! You can focus on how fast can you detect faults,
and how fast can you fix them.
! How low can you go?
! MTTD - Mean time to Detect
! MTTR - Mean time to Repair
62. Hack The Stack
! A side effect of this you now have
tools to repurpose for security and
monitoring of production
! Note that most changes are not
security problems.
63. Logging
! Due to allow developers to see
application logging, it's now very
easy to instrument the application to
log security events.
! Or add logs to times when you are
under attack.
64. Graphing
! Make dashboards of
! SQLi and XSS attacks
! Every type of log-in failure
! Core Dumps
! Database Syntax Errors
65. Static Analysis
! You now have a place to insert them.
! Work with QA group to add more code
quality tests.
66. Post-Commit Checks
! Alert on when sensitive areas of the code
are changed (auth, login)
! Alert on crypto usage (why is developer
using MD5.. hmmm)
! Alert on your programming language's
"dangerous functions"
This allows you to engage the developer
at the start of the cycle.
67. Faster is Better
! You could do most of this in a normal
release-cycle software lifecycle.
! The difference is you are finding
problems at the start instead of 10m
before the launch and telling
everyone to stop.
! The feedback loop works.
68. New Roles, Less Silos
! Developers: works with operations
! QA: works on building systems for
testing, to empower others to write
better tests
! Release Engineering: tools to enable
code to flow faster
! Security: in-house consultancy,
secure-by-default architecture,
monitoring
70. Goal: 50% reduction in deploy time
! Whatever your state of deployment is,
no matter how many people are
involved, no matter how long it
currently takes, make a goal of cutting
it in half.
! This is an easy sell to management
just on cost basis.
! Everything else flows from this.
71. Mechanism not Policy
! Strive for the fastest deployment
mechanism for possible
! But you define the "continuous" in
Continuous Deployment
! Yes, Etsy was 60+ deploys per day, with
each having multiple authors.
! Current gig? we have rules of no more
than 3 per week since our customer have
asked for that, and only deployed at
"low-tide"
73. In other contexts: Operations
! How fast can you deploy OS changes
to you production environment?
! How fast can you deploy router
changes?
! How fast can you deploy patches to
the desktop
You probably don't do it that often since
it's really painful and time consuming!
That's exactly the problem.
74. In other contexts: software product
! here "production" might be getting code
into the main branch and running
automated build / test.
! It's the flow of code: little changes vs big.
75. In other contexts: silicon
! Continuous deployment already done for
silicon! wut?
! Only small changes, with tests are
allowed to be committed!
! Big changes are rejected.
Learned the hard way that big changes
are completely unmanageable