We will discuss the what, why and the how of running modern security operations. We will take a look at the pain points in a DevOps life cycle and see the benefits of pragmatic security solutions. Attendees will get an idea about where and how to start devsecops for secure devops pipeline.
This talk is focused on the what, why and the how of running security operations in the modern world. The way attacks are changing and developers are moving ahead with the next generation technologies is blazingly fast. However, traditional operations still exist. It then becomes imperative to make changes in the way security operations should run to defend against attackers and work with developers and modern businesses. In this talk, we will see what are the real world problems faced by organisations, how we can rapidly adapt to changes by modifying the culture and methodologies while relying on processes, tools and techniques.
Comparing Sidecar-less Service Mesh from Cilium and Istio
Modern Security Operations aka Secure DevOps @ All Day DevOps 2017
1. October 24, 2017
Modern Security Operations
aka
Secure DevOps
Madhu Akula
Automation Ninja @ Appsecco
2. October 24, 2017
About Me
• Automation Ninja at Appsecco
• Interested in Security, DevOps and Cloud
• Speaker & Trainer : Defcon, All Day DevOps, DevSecCon,
c0c0n, null, etc.
• Discovered security vulnerabilities in Google, Microsoft,
Yahoo, Adobe, etc.
• Never ending learner
• Follow me (or) Tweet to me @madhuakula
3. October 24, 2017
Modern Security Operations
• To improve collaboration between Developers, Operations
and Security
• Applying security into each phase of DevOps lifecycle
• Practice of developing and deploying safer software sooner
• Building secure defaults and following best practices
• Proactive monitoring & defence
• Performing redteam activities, before real attacks happen
• Learning & sharing with community
4. October 24, 2017
What is DevOps?
There are many definitions for this term.
I personally follow CAMS by Damon Edwards and John Willis
‘Implementing a culture of sharing between
Development and Operations’
● Culture
● Automation
● Measurement
● Sharing
7. October 24, 2017
Security Misconfiguration
Source: https://www.upguard.com/breaches/cloud-leak-accenture
8. October 24, 2017
Components with known security vulnerabilities
Source: https://github.com/blog/2447-a-more-connected-universe
9. October 24, 2017
Insecure Defaults
Source: https://hackernoon.com/capturing-all-the-flags-in-bsidessf-ctf-by-pwning-our-infrastructure-3570b99b4dd0
10. October 24, 2017
Secret keys in public github
Source: https://www.theregister.co.uk/2015/01/06/dev_blunder_shows_github_crawling_with_keyslurping_bots
15. October 24, 2017
DevSecOps moto
“The purpose and intent of DevSecOps is to build on the
mindset that “everyone is responsible for security” with the goal
of safely distributing security decisions at speed and scale to
those who hold the highest level of context without sacrificing
the safety required.”
Source: http://www.devsecops.org/blog/2015/2/15/what-is-devsecops
17. October 24, 2017
Thinking about security from the outset
“Companies that consider security from the start assess their
options and make reasonable choices based on the nature of
their business and the sensitivity of the information involved.
Threats to data may transform over time, but the fundamentals
of sound security remain constant.”
- Federal Trade Commision
Source: https://www.ftc.gov/system/files/documents/plain-language/pdf0205-startwithsecurity.pdf
18. October 24, 2017
Planning
● The most important phase of development is planning
● Involve all parties (Dev, Sec, Ops) from the beginning, it will
enable everyone to understand and speed up things without
compromising quality & security
● Build things with the mindset of secure defaults with built-in
security
19. October 24, 2017
Planning
● This applies to all teams
○ Developers need to think about secure coding best practices,
using secure libraries and keeping up to date with latest
vulnerabilities.
○ Operations teams need to be aware of technology specific
security configurations, best practices and hardening
guidelines.
○ Security teams have to understand the workflow, create
suitable standards and apply them throughout the lifecycle
20. October 24, 2017
● Version control gives the power of moving traditional
operations to modern DevOps shops
● Managing things will be super-easy and efficient
● This way everything can be audited, tracked and can be
rolled back if required
Version control
21. October 24, 2017
● Some of the things that can be version controlled include
○ Documentation, knowledge bases, etc
○ Developers’ code
○ Op’s configurations and playbooks
○ Custom scripts and snippets
○ Many more...
Version control
22. October 24, 2017
● Infrastructure as code aims to make operations more
efficient and remove human errors
● By doing this, we can achieve
○ Version controlled and codified versions of secure
infrastructure
○ We can perform continuous integration with the deployment
process
○ We can improve the inventory by building Configuration
Management DataBases (CMDB)
Infrastructure as code
23. October 24, 2017
● This requires process and tools
○ Identifying the all manual repetitive tasks and structuring
them for automation
○ With tools like Ansible, Chef, Terraform, etc.
● We can validate our infrastructure as code against security &
compliance
● We can create security playbooks for hardening & patching
Infrastructure as code
24. October 24, 2017
Ansible playbook snippet for MySQL hardening
- name: Secures the MySQL root user
mysql_user:
user: root
password: "{{ mysql_root_password }}"
host: "{{ item }}"
login_password: "{{ mysql_root_password }}"
login_user: root
with_items:
- 127.0.0.1
- localhost
- ::1
- "{{ ansible_fqdn }}"
- name: Removes the MySQL test database
mysql_db:
db: test
state: absent
login_password: "{{ mysql_root_password }}"
login_user: root
25. October 24, 2017
● The practice of integrating work frequently, which requires
quick verification to process next steps using automated
build processes
● In this phase you include your test cases and security checks,
performing them before going to production
● It allows us to integrate existing tool sets using web hooks
and plugins into the build process
Continuous everything
26. October 24, 2017
● Ensures that the quality of the code and configurations
remains the same by using automated test cases and
validation checks
● This requires defining the steps each team needs to perform
to speed up the delivery process without compromising
security
Continuous everything
27. October 24, 2017
● Deployment is the phase where things are made live; aka
production
● Using a standard baseline-OS and containers, which reduces
the level of security risk
● Hardening configuration and environments with best
practice and against known vulnerabilities (Using your
security playbooks)
Secure deployments
28. October 24, 2017
● Managing secrets and data is a key part while deploying to
production, use secure communication channels and storage
like Vault
● Verify deployments by running security scans against them
for misconfigurations
● Also using modern tool-set like Moby project, LinuxKit, etc.
for docker containers
Secure deployments
29. October 24, 2017
● To make an important decisions (or) to troubleshoot things,
monitoring is the place to start
● Monitoring needs to apply to every phase of the DevOps
lifecycle
● Health checks of applications & infrastructure to know how
things are going
● Security monitoring of applications, servers, network devices
Proactive monitoring & alerting
30. October 24, 2017
● Alerting based on thresholds and attack anomalies
● Fine-tuning and improving the alerting system gives more
control
● Automating actions against known repetitive alerts can be
efficient, but take care
Proactive monitoring & alerting
31. October 24, 2017
● Define baseline security
○ Test against it
○ And run tests continuously
● Drive testing from the DevOps pipeline
● Never deploy sub-standard code
● Requires tests to be passed in order to deploy into
production
● Empower DevOps teams to fix issues
● Apply feedback loops
Test driven security
Source: https://docs.google.com/presentation/d/1H0Ym0bJ4TgVz7BRkoeun7ndGuTIteVyFGDDxguXmff0
32. October 24, 2017
● This requires you to have proactive monitoring in place,
which includes building centralised logging and monitoring
systems
● Build your defences from an offensive mindset and start by
focusing on your critical infrastructure
● Enable DevOps teams to better understand and identify
what security attacks look like by red teaming (we can also
use this log data to train defence systems)
Attack driven defence
Source: https://www.slideshare.net/zanelackey/attackdriven-defense
33. October 24, 2017
● Identify patterns and anomalies for alerting and take action
against them using automated defence
● Apply data science and machine learning techniques for data
sets
● Build defence systems with real attack data and defend like
an attacker
Attack driven defence
Source: https://www.slideshare.net/zanelackey/attackdriven-defense
34. October 24, 2017
● Clear communication enables us to be more productive
● Collaboration between teams makes things faster. It should
start from outset!
● Break requirements into actionable items and assign them
to respective teams
● Eliminate the barriers between Devs, Ops and Security
teams and work towards a DevSecOps approach (everyone
is responsible for security)
Communication & collaboration
35. October 24, 2017
● Use task and project management tools for collaboration,
this will help showcase dependencies between teams
● Spread awareness of different roles and skills by conducting
social events; learning by lunch, etc.
Communication & collaboration
36. October 24, 2017
Training people
Training developers and operations about how attackers work,
by using vulnerable labs and applications, will give them a better
understanding
OWASP Vulnerable Web Applications Directory Project
37. October 24, 2017
● We must learn from each other, the best way to do this is
sharing with others
○ For example, security teams can write a playbook to harden
infrastructure to meet the policies and standards rather
pointing out that it’s an ops issue
● Rather than working as a big teams, we can mix the different
teams into smaller groups and work together to achieve
great results
Culture & innovation
38. October 24, 2017
● Simplicity, documentation and clear communication is a
win-win
● When things go wrong transparency and open contributions
is vital
● Attending conferences and meetups and being part of the
community helps us to know how the world is doing things
differently to us
Culture & innovation
39. October 24, 2017
● Fail fast and early, so there is less cost and damage for the
business
● Maintain secure backups and validate the restore process
● Test for resiliency and recoverability using tools like chaos
monkey and security monkey
● Conduct internal hackathons and bug bounty programs
● Perform redteam activities, simulate how real attacks
happen
Think about failures, before they occur
40. October 24, 2017
Wardly maps for DevSecOps
Source: https://github.com/devsecops/wardley-maps
44. October 24, 2017
Takeaways
● Everyone is responsible for security (Dev + Sec + Ops)
● Clear communication, active collaboration is key to success
● Build with secure defaults mindset
● Test driven development & Attack driven defence
● Hack your applications, infra, etc. like real attackers
● Keep learning and sharing