This talk was given at DevSecOps Days Boston and DevOps & Security Meetup Vienna in 2021 Automatic Release Validation, aka Quality Gates, is not a new concept but often only covers functional or performance metrics. Keptn’s open SLO-based evaluation allows DevSecOps to have their favorite security tool report SLOs such as number of detected vulnerabilities as part of delivery automation

1. Adding Security to your
SLO-based Release Validation
with
Andreas Grabner
DevOps Activist at Dynatrace
DevRel for Keptn
@grabnerandi
https://www.linkedin.com/in/grabnerandi
Star us @ https://github.com/keptn/keptn
Follow us @keptnProject
Slack Us @ https://slack.keptn.sh
Visit us @ https://keptn.sh
DevSecOps Boston, July 26 2021
+ Vienna DevOps & Security, Nov 9 2021

2. What is Keptn
Whichproblemdoesitsolve?
Service Level Objectives
Forautomatedreleasevalidation
Extending Keptn
Howtointegrateyoursectools
Talk outline: so you know whats gonna happen 

3. What is Keptn
Which problemdoes it solve

4. DevOps & SREs must automate many tasks through pipelines
Deployment
Performance Engineering
Quality Gates Security & Vulnerability Scans
Chat Notifications
Configure Monitoring
Chaos Testing
Integrate Approval Process
Canary
Blue/Green Feature Flagging
SLO Monitoring
Setup Alerting
Auto Remediation
API Test
SLA Validation
Functional Test
Rollbacks

5. ChristianHeckelmann
SeniorDevOps Engineer
2800
projects
966
CI/CDs
„I am constantly reacting to
‚Pipeline Broken – please fix!‘“
DIY (Do It Yourself) can become very complex and hard to maintain

6. pipeline {
stages {
stage('Deploy to dev namespace') {
steps {
container(‘helm’) {
}
}
}
stage('Run tests') {
steps {
container(‘jmeter’) {
}
}
}
stage(‘Evaluate performance’) {
steps {
container(‘curl’) {
}
}
}
if (evaluation.passed) {
stage('Deploy to staging') {
steps {
container(‘helm’) {
}
}
}
}
}
}
pipeline {
stages {
stage('Deploy to dev namespace') {
steps {
container(‘kustomize’) {
}
}
}
stage('Run tests') {
steps {
container(‘jmeter’) {
}
}
}
stage(‘Evaluate performance’) {
steps {
container(‘curl’) {
}
}
}
if (evaluation.passed) {
stage('Deploy to staging') {
steps {
container(‘helm’) {
}
}
}
}
}
}
1 Service =
1 Pipeline
1 Project = x Pipelines n Teams = n*x Pipelines
pipeline {
stages {
stage('Deploy to dev namespace') {
steps {
container(‘helm’) {
}
}
}
stage('Run tests') {
steps {
container(‘jmeter’) {
}
}
}
stage(‘Evaluate performance’) {
steps {
container(‘curl’) {
}
}
}
if (evaluation.passed) {
stage('Deploy to staging') {
steps {
container(‘helm’) {
}
}
}
}
}
}
25
services
96
workloads
DieterLadenhauf
SeniorACE Engineer
„Onboarding or updating
pipelines is manual & error
prone!‘“
Scaling DIY to many projects often highlights technical debt in your automation

7. Oh Keptn, my Keptn!

8. classical-automation-script {
} // 1000+ lines later
Keptn removes hard-coded dependencies of classical automation approaches
Prepare
Deploy
Test
SLO Evaluation
Promote
Deployment Tool D1
Testing Tool T1
Monitoring Tool M1
Notification Tool N1
config

9. stage: automation-seqeuence
SLO Evaluation
Keptn separates process, tooling and configuration and connects via events
Deployment Tool D1
Testing Tool T1
Monitoring Tool M1
Notification Tool N1
Eventing
Prepare
Deploy
Test
Promote
config
Git

10. Example #1: automate performance sequence in staging
Deployment Tool D1
Testing Tool T1
Monitoring Tool M1
Notification Tool N1
Eventing
Deploy
Test
SLO Evaluation
Git
$ keptn trigger performance --stage=staging --image=myapp:2.0
staging: performance
Task: get-sli
Service:myapp
Stage:staging
Time: last 10min
Task: Test
Stage:staging
Strategy:Performance
Prepare
Promote
Shipyard
Sequence of automation tasks grouped in stages
CloudEvents
Task specific with metadata
Uniform
Subscribed Keptn Services
Config
Tool specific
Task: Deploy
Image:myapp:2.0
Stage:staging
Strategy:Blue/Green

11. Example #2: remediate production security issue
Infra Autom. Tool I1
Monitoring Tool M1
Notification Tool N2
Eventing
Action: block-traffic
SLO Evaluation
SLO Evaluation
Escalate
Git
$ keptn trigger remediation --stage=production --problem=high-vulnerability-attack --rootcause=bad-library-version
production: remediation
Task: Remedation
Action:block-traffic
Stage:production
Data:172.31.0.0/24
Task: Remediation
Action: rollback
Stage:production
Image:myapp:1.0
Action: rollback
Shipyard
Sequence of automation tasks grouped in stages
CloudEvents
Task specific with metadata
Uniform
Subscribed Keptn Services
Config
Tool specific
Deployment Tool D2
Task: Escalate
Reason: Failure Rate
Stage:production Ticketing Tool T1
remediation.yaml (bad-library-version)

12. Reduce your pipeline‘s complexity by letting orchestrate declarative, data-driven delivery and ops automation
90% less automation code
SLOs built-in
Connects with all your tools
Separation of process & tool
GitOps: all config in git
Keptn brings opinionated cloud native automation to all your projects

13. Service Level Objectives
For automated release validation

14. Applying SRE Best Practices across the lifecycle
Authentication Service
0.89s
0.5%
Oct 2021 Nov 2021
0.61s
2.5%
1000/s 1600/s
Service X
xxs
xx% yys
yy%
xx/s yy/s
Production
Shift-Left
Continuous Delivery
Authentication Service
Commit
#1
Commit
#2
Commit
#3
Commit
#4
SLO Evaluation
Service X

15. An example: SLOs defined by DevSecOps validated as part of automated delivery
Overall Failure Rate
Query: builtin:service.errors.total
Test Step LOGIN Response Time
Query: calc:service.teststeprt:filter(Test, LOGIN)
Test Step LOGIN # Service Calls
Query: calc:service.testsvc:filter(tx, LOGIN)
<= 5%
<= 2%
<=150ms & <=+10%
<= 400ms
<= +0%
Build 1
0%
80ms
100ms
SLO: Overall Score Goal 90% 75%
Response Time 95th Perc
Query: builtin:service.responsetime(p95)
<=100ms
<= 250ms
SLO
SLIs (Service Level Indicators) warn
pass
1
100%
Build 2
4%
120ms
90ms
2
50%
Build 3
1%
90ms
120ms
1
70.0%
Build 4
0%
95ms
95ms
1
100%
Build 1 Build 2 Build 3 Build 4
$ keptn send event start-evaluation myproject myservice starttime=build1_deploy endtime=build1_testsdone
$ keptn send event start-evaluation myproject myservice starttime=build2_deploy endtime=build2_testsdone
$ keptn send event start-evaluation myproject myservice starttime=build3_deploy endtime=build3_testsdone
$ keptn send event start-evaluation myproject myservice starttime=build4_teststart endtime=build4_testsend
Open Security Vulnerabilities
Query: calc:secproblems:filter(risk,CRITICAL)
<=0
0 0 1 0

16. How SLOs work in Keptn behind the scene!
sli.yaml (Dynatrace)
indicators:
error_rate: "builtin:service.errors.total.count"
count_dbcalls: "calc:service.toptestdbcalls"
jvm_memory: "builtin:tech.jvm.memory.pool.committed"
sec_critical: "calc:secproblems:filter(risk,CRITICAL)"
slo.yaml (SLI Provider independent)
objectives:
- sli: error_rate
pass:
- criteria:
- "<=1“ # We expect a max error rate of 1%
- sli: jvm_memory
- sli: count_dbcalls
pass:
- criteria:
- "=+2%" # We allow a 2% increase in DB Calls between builds
warning:
- criteria:
- "<=10" # We expect no more than 10 DB Calls per TX
- sli: sec_critical
pass:
- criteria:
- "<=0" # We do not allow any critical security issues
total_score:
pass: "90%"
warning: "75%"
sli.yaml (Prometheus)
indicators:
error_rate: "http_requests_total{status=“error"}"
jvm_memory: "jvm_memory_used_bytes{area="heap"}[1m]"
sec_critical: "rate(falco_events[5m])"
SLI Providers: Query SLIs based on sli.yaml and return individual values
Lighthouse Service: Retrieves SLIs and compares them against SLOs
...
*get-sli*
*evaluation*
count_dbcalls: 5 jvm_memory: 360MB
error_rate: 4.3% sec_critial: 1

17. triggers an automation sequence orchestrates monitoring config, deployment, test execution, SLO evaluation & remediation
Integrating SLO evaluation (and other Keptn use cases) with your existing tooling

18. Release Readiness for Austrian Online Banking
#1 List of release
relevant SLOs
#3 Link back to Jenkins
https://medium.com/keptn/keptn-automates-release-readiness-validation-for-austrian-online-banking-software-eaaab7ad7856

19. Extending Keptn
How to integrate your sec tools

20. • Build your own Keptn Service in minutes!
• https://github.com/keptn-sandbox/keptn-service-template-go
• https://github.com/keptn-contrib/job-executor-service
Integrate your security testing tools with Keptn
type: sh.keptn.event.test.triggered
data:
- test.strategy: functional
- deployment.url: https://yoururl
type: sh.keptn.event.test.started
data:
- status: succeeded | failed
type: sh.keptn.event.test.finished
data:
- status: succeeded | failed
- result: pass | fail
Run Test
Distributor:
Subscribing
to
*.test.*
Your Tool ...
$ zap.sh cmd -quickurl
https://yoururl -quickprogress

21. Integrate your security results/reports as an SLI Provider
type: sh.keptn.event.get-sli.triggered
data:
- timeframe: 30m
- sli: sec_critical
type: sh.keptn.event.get-sli.started
data:
- status: succeeded | failed
type: sh.keptn.event.get-sli.finished
data:
- status: succeeded | failed
- values: sec_critical=4
Query Data
Distributor:
Subscribing
to
*.get-sli.*
Your Tool ...
• Build your own Keptn Service in minutes!
• https://github.com/keptn-sandbox/keptn-service-template-go
• https://github.com/keptn-contrib/job-executor-service

22. Wrap Up
AutomatingSecuritySLOswith Keptnandyourfavoritesecuritytools

23. Confidential 2
5
Get startedwithour tutorials:tutorials.keptn.sh
live public demo

24. Adding Security to your
SLO-based Release Validation
with
Andreas Grabner
DevOps Activist at Dynatrace
DevRel for Keptn
@grabnerandi
https://www.linkedin.com/in/grabnerandi
Star us @ https://github.com/keptn/keptn
Follow us @keptnProject
Slack Us @ https://slack.keptn.sh
Visit us @ https://keptn.sh
DevSecOps Boston, July 26 2021