Much attention has been given to the need for increased automation in security, given the sheer volume of attackers and attacks, the overload of information security pros must wrangle, and the continued high demand for security expertise. But can automation solve all of security’s most serious problems? If not, why not? Will there always be a need for human involvement? These slides were used in a live webcast featuring, 451 Research Information Security Research Director Scott Crawford and Cigital Managing Principal Nabil Hannan. You can watch this and other webcasts by visiting https://www.cigital.com/resources/.

1. Can you really automate yourself secure?
Facts vs. Fantasies
Scott Crawford, Research Director, 451 Research
Nabil Hannan, Managing Principal, Security, Cigital

2. Security teams are struggling
• “Lack of staff expertise” the most
common obstacle to multiple
aspects of security ops
• “Organizational politics/Lack of
attention” the second most reported
infosec pain point1
• More data than teams can handle
• SIEM: Events per second into the 5-
digit range
• One-fourth of security orgs still can’t
understand and baseline normal
behavior2
DAST/SAST inhibitors: What inhibitors has your
organization encountered in adopting or fully utilizing
your vendor’s technology?
1 451 Research, Voice of the Enterprise: Information Security, Q3
2015
2 SANS Institute 2015 Analytics and Intelligence Survey
Source: 451 Research, Voice of
the Enterprise: Information
Security, Q3 2015

3. Automation can help with these burdens
• Continued growth in use of security
analytics
• Assessment and testing as well as
operational monitoring and
control
• Applications of machine learning
• Ability to handle data at speed
and scale
• Advantages of the cloud
• Rise of security task automation
• Today:
• “Playbook” approach to orchestration
• Auto-generation of code (.py)
• Incident response workflow & data
aggregation
• Testing and assessment
• Tomorrow:
• Containers and microservices
• “Infrastructure-as-code”
What CAN’T automation do?

4. What CAN’T automation do?

5. First of all, what’s your strategy?
• Case in point: Application security covers a lot
of ground
• Static, Dynamic, Interactive testing?
• IDE-integrated real-time code analysis/guidance?
• Source supply chain? Runtime protection?
• What do you hope to achieve with
automation?
• Who will be responsible? Developers? Operations
teams? Security experts?
• Limited requirements for test data and automated test suites? Developers
implementing more secure code?
• Limited requirements for modifying the production environment?
• Automation without a plan and a strategy runs the risk of
automating the wrong things

6. Goal Setting

7. The reality of automation tools
• They are designed to address “likely” use
cases out of the box
• BUT…Every application is different
• Specific implementations require one (or
both) of two things:
• Adaptation of the technology to the
environment (tool tuning)
• Adaptation of the environment to the tool (assuring that tools have access to
all relevant functionality in the target application)
• Who or what makes this happen?
• Do you know what the cost in time and expertise will be to make sure
your automation tools deliver expected results?

8. Adapting to application differences

9. Automation has its limits
• Do you know how far automation can go in your
case?
• Application security coverage:
• How much direction do your tools require?
• Do they follow all the logic and branches you
expect? What are the tradeoffs if they do?
• Caveat: Be aware when benchmarks may mask these factors!
• Those that favor techniques such as IAST by scripting the assessment of all inputs, vs.
SAST or DAST tools which must drive their own coverage
• Scope/scale of test cases: How many does each type of benchmark evaluate? Are you
measuring a sprint against a marathon?
• Do the limits of your tools fit your expectations? Do you know how to
manage the gaps if they don’t?

10. Automation has its own information overload risks
• False positives aren’t the only issue. Say:
• A given run of your tool finds 500 possible issues
• With a true positive rate of 20% (so 100 out of those
500 reported issues are actually legit)
• …but your team only has time to evaluate 200 of the
500 findings
• If that 20% of true positives is evenly distributed,
your team will ID only 40 TP’s out of the 200
possibilities they examine (200 x 20% = 40)
• ...Meaning that your team will miss the
remaining 60 true positives found
• Does your team really have the
resources to fully assess the results of
automation without missing important
findings?
All
findings
The
findings
our
team
can get
to
All true
positives
The
TPs
our
team
found
The TPs
our team
missed

11. Where will automation fit into your processes?
• Traditional application security testing:
• Periodic
• DAST may be “monolithic” – run against an
entire application or large/comprehensive
components
• In a DevOps environment:
• Frequent updates, releases – moving toward
CI/CD
• Testing may need to be closer to continuous
• Broken into smaller chunks – more “unit”
oriented than monolithic
• Are your teams ready for the impact of
integrating secure development and
testing into DevOps?
How well do your security pros
understand the DevOps toolchain?
How well do your developers & ops
teams understand security needs?

12. Congratulations, you have results! …Now what?
• Don’t forget: Discovery is only half
the battle!
• Once your automation tools reveal
exposures, will you be able to close
them successfully?
• Security task automation can close
operational exposures…
• ...but can they remediate
vulnerabilities or implementations in
the applications themselves?
• If your automated testing tools
produce results, what’s your
strategy for closing the loop on
remediation?

13. Automation solves a lot of problems…but
• Only people can:
• Understand your business and security
goals and objectives
• Know the limits of automation tools…and
how and where to close their gaps
• Invest the effort required to assess the
results of automation and apply it correctly
• Integrate security automation into
development and operational practices
• Assure that remediation answers the security needs revealed by the
automation of monitoring and assessment
• Help you develop the right automation strategy based on
experience

14. Solutions for the full SDLC