SlideShare utilise les cookies pour améliorer les fonctionnalités et les performances, et également pour vous montrer des publicités pertinentes. Si vous continuez à naviguer sur ce site, vous acceptez l’utilisation de cookies. Consultez nos Conditions d’utilisation et notre Politique de confidentialité.
SlideShare utilise les cookies pour améliorer les fonctionnalités et les performances, et également pour vous montrer des publicités pertinentes. Si vous continuez à naviguer sur ce site, vous acceptez l’utilisation de cookies. Consultez notre Politique de confidentialité et nos Conditions d’utilisation pour en savoir plus.
Fixing Security by Fixing Software Development Using Continuous Deployment
Do you have an effective release cycle? Is your process long and archaic? Long release cycle are typically based on assumptions we haven't seen since the 1980s and require very mature organizations to implement successfully. They can also disenfranchise developers from caring or even knowing about security or operational issues. Attend this session to learn more about an alternative approach to managing deployments through Continuous Deployment, otherwise known as Continuous Delivery. Find out how small, but frequent changes to the production environment can transform an organization’s development process to truly integrate security. Learn how to get started with continuous deployment and what tools and process are needed to make implementation within your organization a (security) success.
Fixing Security by Fixing Software DevelopmentUsing Continuous DeploymentNick Galbreath, VP Eng IPONWEB, http://www.iponweb.com/@ngalbreath http://www.client9.com/May 14-15, San Francisco, California
Follow AlongLatest version posted online: http://slidesha.re/144OIv3 http://www.client9.com/
Nick Galbreath (@ngalbreath)! Spoken at Black Hat, DEFCON, OWASP! Author of libinjection: advanced SQLidetection used in > 2 WAFs, 1 Honeypot! Book on cryptography! but really...! Engineering Management and SoftwareDevelopment for high growth startups.! Personal site http://www.client9.com/
IPONWEB! customized online advertisinginfrastructure and exchanges! engineering oﬃces in Moscow, withbusiness oﬃces in London, New York andTokyo.
Original AbstractFixing Security by Fixing Software Development Using ContinuousDeploymentDo you have an eﬀective release cycle? Is your process long and archaic? Longrelease cycle are typically based on assumptions we havent seen since the1980s and require very mature organizations to implement successfully. Theycan also disenfranchise developers from caring or even knowing about securityor operational issues. Attend this session to learn more about an alternativeapproach to managing deployments through Continuous Deployment, otherwiseknown as Continuous Delivery. Find out how small, but frequent changes to theproduction environment can transform an organization’s development processto truly integrate security. Learn how to get started with continuous deploymentand what tools and process are needed to make implementation within yourorganization a (security) success.
Well thats a bold statement..."Fixing Security by Fixing DevelopmentUsing Continuous Deployment"
and heres anotherFor web applications, our release-basedsoftware development lifecycle is stillbased on a pre-Internet model and isharmful to organizations and particularly harmful for security.
What needs ﬁxing?! SQLi dropped from #8 to #14 in the latestWhite Hat "The State of Web Security"report. Good news, right?! This means SQLi is only 7% of websites.Thats 1 in 15. And this is #14vulnerability!! And time to ﬁx was on average 196 days. Thats embarrassing.Veracode claims 32% of incoming web applications have SQLi https://info.veracode.com/state-of-software-security-report-volume5.htmlhttps://reg.whitehatsec.com/WPstats0513
Even worse...! Number 1 driver to ﬁx security problems... compliance.! Number 1 reason to not ﬁx security is... compliance.! Not.. ! keeping our employees and customers safe! protecting corporate interests.! improving quality! being good at what we do.
Security Products #1 .. in security bugsVeraCode: State ofSoftware Security, V4December 2011
Lets Just Give Up! “You could spend all your resources chasing suchthings as this,” William Ribich, the former president ofTechnology Solutions Group [ QintelIQ ], said in aninterview in January. Ribich, who retired in November 2009,shortly after the discovery of a major data theft, said heneeded to balance the uncertain risk that the hackers coulduse what they stole against a growing shopping list ofsecurity products and consulting fees.! "You ﬁnally have to reach a point where you say ’let’s move on,’” he said. http://www.bloomberg.com/news/2013-05-01/china-cyberspies-outwit-u-s-stealing-military-secrets.html
I would call that broken! But preventing SQLi isnt a technicallyhard problem.! And most security patches are very small.! How did we get here?
High Distribution Cost! The Software Product Model is designedfor software where the cost of distributionis high. "High" might be ﬁnancial, risk,time, resources, customer annoyance.! Retail, physical product, CD/DVD! Embedded of Exotic Hardware! Safety, Medical or Defense Systems! Operating Systems (desk or phone)! Homework (1-time deploy)
Web Applications Year 2000! Mostly followed Software Product Modelsince thats all we knew.! High barrier to entry! Specialized Hardware, Software andPeople needed to get started.! Lots of engineering needed to keep thingsrunning.! (side note: CERT/CVE started in 1999)
True Story #1 ! "Cant push out the spelling error ﬁx – itstoo risky"! "That code as already been through QA,its locked down."! "Product has to prioritize that change, elsewe arent touching it."
True Story #2! Well do an iteration, where we try to ﬁx asmany things as possible.! This wont be a scheduled iteration, it willbe done because things are so bad! So the spelling error will get ﬁxed... uhh, who knows when.
Web Applications 2013 ! Almost no barrier to entry! Commodity hardware! Programming not that hard! Scaling problems can be mostly outsourced (mostly)
Cost of Distribution 2013! Frequently no compile step or its very fast.! Moving to production a few kilobytes ormegabytes of code over 1Gbps, 10Gbpslink.! In other words... free
Failure is very diﬀerent however! Most web applications are data-driven.! Frequently have social features, APIs,user-generated content.! Failures might be due to algorithmicproblems... but...! Most likely to due to user input, bad datain database or operational load.! this means data in past can causeproblems in the future.
Releases and Problems! When a web-release goes out, and hasproblems....! Next week is spent tracking down whochanged what, where.! Re-QA! Re-Push! meanwhile new code is piling up.
When SPM meets Web Apps! A long time between code being writtenand code being released.! Might be weeks or months! Feedback loop between code-in-dev andcode-in-production is broken! When security or bug reports come in, theauthor is likely on a diﬀerent project.
Hypothesis! It is impossible to simulate the productionenvironment in development, either due tooperational diﬀerences or data diﬀerences.! No amount of QA or Security Testing canprove you dont have bugs, vulnerabilities,or cause severe operational problems.! You have bugs and vulnerabilities, right now, in your application.
Impedance Mismatch! Easy to write code, +! Long release cycles +! Security as an end-of-line or out of bandprocess ==! no one cares
So the Answer is...! Going slower? Im sure your boss will lovethat suggestion! More steps and process? In other words,slower.! Asking for more people? Sure but goodluck hiring them. Doesnt scale.! Asking for more products? Since theothers have worked so well.
Continuous Deployment ! Also known as Continuous Delivery. ! A System of Software ProductionCharacterized by Numerous SmallChanges the Production Environment,initiated by the author of the change.You change it, you push it to prod.
"Writing Software"! Software Developers think their job iswriting software.! And so, they love to make things perfectbefore anyone else sees it.! Impolite: "data hiding"! code is hiding on developers computer! or on some branch! in other words invisible until its ready.
Actually! The software engineers job is actuallywriting running software, that works well.! This idea is so alien, that companies haveto remind the engineers of this.
Rackspace Haikuwriting code is hard if you cannot deployit does not matter@paulvx from DevOpsDays Austin 2013
Facebook "Move Fast and Break Things....Except "Push" (deployment system)via http://mitadmissions.org/blogs/entry/move_fast_and_break_things
Todays goal! but for today the goal is getting thedeveloper to care about their code in production.! If you dont have that, I dont think you canreally solve security problems.
How does this work?! Really? Developers push their own codeout?! How is this not a disaster.! How is this not a security disaster?
The Deploy Button! What is you had a button that said "DEPLOY"! That pushed to production, whatever iscurrent in your source control system.! And took about a minute! The change and who pressed the button islogged, but thats it.
Part 1: Fear! No one is going to push it ;-)! Meanwhile code is piling upReal example: A new hire I had at Etsy was afraid of deploying an HTML change that they made. "But I dont want to break the site!"
Part 2: First Push! Someone brave will press the button! And very likely the site will explode, and arollback will need to be done.! Theyll know since someone else will havetold them.
Part 3: With Graphs! Lets get all those operational graphs outin the open. And put them right next tothe button.
Part 4: Push #2! Repush! Site might still explode! But the developer is aware and canrollback.
Take 5: Isolation! Hmmm, the developer notices that in thechange set, a million things are going out.! Maybe just pushing out a smaller changewill help isolate the issue.
Take 6: Success!! Yes, the developer just pushed out somecode and made the site better.! The secret about continuous deploymentis small changes that can be easilyunderstood.
Take 7: Dark Pushes! Now we got some bugs ﬁxed, lets push afeature.! First lets push out all the supporting ﬁles.Since they arent being called, they donothing and are safe to push out.! Now everyone can see them
Take 8: Getting the feature live! Instead of "all at once", we slowly ramp upa feature.! if (user_id % 20 == 0): do new feature; ! we change change the percentage easilywith another code push.! or turn it down. Much nicer change log.! While the site didnt explode, its hard tosee if the feature is being used or not.
Take 9: Application Level Graphs! Allow developers to instrument their codeso they can see what is happening inproduction.! Enter StatsD and other UDP-based tools! Enter centralized logging and in-application method to make it easy to logproblems.
Take 10: Communication! So far good for one developer.! To scale up, youll need a system to allowdevelopers.! IRC-like tools work well (e.g. "the pushchannel") – skype, jabber, hangouts, etc
Along the way! Expose production logs to developers! Add in a staging-step where the codegoes to faction of the cluster, sodevelopers can test with real traﬃc! Try to make development closer to prod.! Make "smoke tests" to catch basic errors! Add syntax checkers to eliminate obviousissues. ! Use static analysis to ﬁnd bugs
Mistakes will happen! Do postmortem analysis! Everyone thought they were doing theright thing at the time.! "How can the environment be changed toprevent this" and build tools to enforce it.! (Rarely can you truly change people)
That guy who pushes at 3am! Courtesy and convention will convergevery quickly when the site goes down at3am and the developer starts gettingcalls ;-)! Of hours pushes of course can happen,when they notify operations.
What About Code Reviews?! Yes, please do them.! Nothing here prevents code reviews.! In fact code reviews are easier since! they are small! they are in mainline not some branch
What about Security Reviews! Please do them.! Nothing here eliminates architecturalplanning or review.! This actually doesnt change the SDLCvery much.
What about Agile Methods! (everyone seems to have a diﬀerent idea ofwhat Agile is but..)! Agile methodologies typically work toimprove the business spec / developmentcycle. (are you building what the customerwants)! But doesnt address code deployment.! They are complimentary practices.
What about Customer Service?! "Dont they freak out with all thechanges?"! Remember: deployment != feature release! Most deployments do very little from thecustomer point of view! Feature releases (frequently controlled byramp-ups or ﬂags) always needs to becoordinated with product and customerservice.
What about Compliance? PCI?! Let me tell you about compliance...! mechanism not policy
Obvious Beneﬁt to Security! Security patches can go out quickly! You know this since they are now just partof a normal development cycle.
More Importantly! That Engineer who previously didn’t pushcode is now sensitized that their code hasconsequences and are responsive andempowered to ﬁx it.! It’s amazing how interested engineersbecome in security when you ﬁndproblems with their code when they areable to ﬁx quickly themselves.
Hack The Stack! A side eﬀect of this you now have tools torepurpose for security and monitoring of production! Note that most changes are not securityproblems.
Logging! Due to allow developers to see applicationlogging, its now very easy to instrumentthe application to log security events.! Or add logs to times when you are underattack.
Graphing! Make dashboards of! SQLi and XSS attacks! Every type of log-in failure! Core Dumps! Database Syntax Errors
Static Analysis ! You now have a place to insert them.! Work with QA group to add more codequality tests.
Post-Commit Checks! Alert on when sensitive areas of the codeare changed (auth)! Alert on crypto usage (why is developerusing MD5.. hmmm)! Alert your programming languages"dangerous functions"This allows you to engage the developer at the start of the cycle.
Faster is Better! You could do most of this in a normalrelease-cycle software lifecycle.! The difference is you are finding problemsat the start instead of 10m before thelaunch and telling everyone to stop.! The feedback loop works.
New Roles, Less Silos! Developers: works with operations! QA: works on building systems for testing,to empower others to write better tests! Release Engineering: tools to enable codeto ﬂow faster! Security: in-house consultancy, secure-by-default architecture, monitoring
Goal: 50% reduction in deploy timestimes ! Whatever your state of deployment is, nomatter how many people are involved, nomatter how long it currently takes, make agoal of cutting it in half.! This is an easy sell to management just oncost basis.! Everything else ﬂows from this.
Mechanism not Policy! Strive for the fastest deployment mechanism forpossible! But you deﬁne the "continuous" in ContinuousDeployment! Yes, Etsy was 60+ deploys per day, with each havingmultiple authors.! Current gig? we have rules of no more than 3 per weeksince our customer have asked for that, and onlydeployed at "low-tide"
In other contexts: Operations! How fast can you deploy OS changes toyou production environment?
In other contexts: IT! How fast can you deploy desktopchanges?
In other contexts: software product! here "production" might be getting codeinto the main branch and runningautomated build / test.! Its the ﬂow of code: little changes vs big.
In other contexts: silicon! Continuous deployment already done forsilicon! wut?! Only small changes, with tests are allowedto be committed!! Big changes are rejected. Learned the hard way that big changes arecompletely unmanageable