Keynote presentation at DevOps Con Munich, December 3, 2019, presented by Damon Edwards, co-founder of Rundeck. Responding to incidents has always been the core job of Operations. With the rise of DevOps and SRE, how Operations work gets done — and who is doing the work — is changing. This talk will look at how high-performing organizations are applying DevOps and SRE practices to shorten incidents and reduce escalations. See a Demo of Rundeck Enterprise : https://www.rundeck.com/see-demo --or-- Download Rundeck Open Source here: https://rundeck.com/open-source Connect: Stack Overflow community: https://stackoverflow.com/questions/tagged/rundeck Github: https://github.com/rundeck/rundeck/issues Twitter: https://twitter.com/Rundeck Facebook: https://www.facebook.com/RundeckInc/ LinkedIn: www.linkedin.com › company › rundeck-inc

1. Damon Edwards
Incident Management
in the Age of DevOps and SRE

2. Incident Management
in the Age of DevOps and SRE
@damonedwards rundeck.com/dc-munich-19
Damon Edwards

3. Assertion:
The ability to respond to and resolve incidents is the true
indicator of an organization’s operational capabilities

4. Assertion 2:
Everybody now works in “Operations"

5. What Is an Incident?
An unplanned disruption impacting
customers or business operations

6. What Is an Incident?
An unplanned disruption impacting
customers or business operations
Outages
Service Degradation

7. What Is an Incident?
An unplanned disruption impacting
customers or business operations
Outages
Service Degradation
Work interruption
Delay/Waiting
“Short-Notice” Requests

11. Board

12. Integrated
Board

13. Integrated
Responsive
Board

14. Integrated
Responsive
Everywhere
Board

15. Integrated
Responsive
Everywhere
Always
Board

16. Integrated
Responsive
Everywhere
Always
Board
Tech Org Execution

17. Integrated
Responsive
Everywhere
Always
Board
Tech Org Execution

19. Kubernetes
AWS GCP Azure
Docker
Consul
Terraform Istio
Zipkin
Envoy
Serverless
OpenShift
KafkaLamba
Prometheus
Containerd
Helm
Cloud Foundry
Linkerd
Etcd
CoreDNS
MongoDB
Redis
InfluxDB
Jaeger
gRPC
CRI-O
Cognito
Fargate
Cloud Functions
Cosmos
BigQuery Spark
Rook
Ceph
NGINXHAProxy
Open vSwitch
NSX Sensu
Vault
Aurora
Nomad

20. Kubernetes
AWS GCP Azure
Docker
Consul
Terraform Istio
Zipkin
Envoy
Serverless
OpenShift
KafkaLamba
Prometheus
Containerd
Helm
Cloud Foundry
Linkerd
Etcd
CoreDNS
MongoDB
Redis
InfluxDB
Jaeger
gRPC
CRI-O
Cognito
Fargate
Cloud Functions
Cosmos
BigQuery Spark
Rook
Ceph
NGINXHAProxy
Open vSwitch
NSX Sensu
Vault
Aurora
Nomad

21. Kubernetes
AWS GCP Azure
Docker
Consul
Terraform Istio
Zipkin
Envoy
Serverless
OpenShift
KafkaLamba
Prometheus
Containerd
Helm
Cloud Foundry
Linkerd
Etcd
CoreDNS
MongoDB
Redis
InfluxDB
Jaeger
gRPC
CRI-O
Cognito
Fargate
Cloud Functions
Cosmos
BigQuery Spark
Rook
Ceph
NGINXHAProxy
Open vSwitch
NSX Sensu
Vault
Aurora
Nomad

22. Kubernetes
AWS GCP Azure
Docker
Consul
Terraform Istio
Zipkin
Envoy
Serverless
OpenShift
KafkaLamba
Prometheus
Containerd
Helm
Cloud Foundry
Linkerd
Etcd
CoreDNS
MongoDB
Redis
InfluxDB
Jaeger
gRPC
CRI-O
Cognito
Fargate
Cloud Functions
Cosmos
BigQuery Spark
Rook
Ceph
NGINXHAProxy
Open vSwitch
NSX Sensu
Vault
Aurora
Nomad
SAIL/cornell.edu

23. Adrian Cockcroft
Developer
Developer
Developer
Developer
Developer
Old Release Still
Running
Release Plan
Release Plan
Release Plan
Release Plan
Deploy
Feature to
Production
Deploy
Feature to
Production
Deploy
Feature to
Production
Deploy
Feature to
Production
Bugs
Deploy
Feature to
Production
Immutable microservice deployment
scales, is faster with large teams and
diverse platform components
DockerCon EU 2014 Architecture enables speed.
Speed is the advantage.

26. The Three Ways (2013)

27. The Three Ways (2013) The Five Ideals (2019)

29. DEV

30. Go! Go! Go!DEV

31. Go! Go! Go!DEV …OPS?

32. 0000
Go! Go! Go!DEV …OPS?

33. 0000
Go! Go! Go!DEV …OPS?
Operations:
The Last Mile

36. 1. SRE needs Service Level Objectives, with consequences
2. SREs have time to make tomorrow better than today
3. SRE teams have the ability to regulate their workload
Principles of SRE

37. 1. SRE needs Service Level Objectives, with consequences
2. SREs have time to make tomorrow better than today
3. SRE teams have the ability to regulate their workload
Principles of SRE

38. 1. SRE needs Service Level Objectives, with consequences
2. SREs have time to make tomorrow better than today
3. SRE teams have the ability to regulate their workload
Principles of SRE

40. DevOps + SRE
Product,
Not Project
Continuous
Delivery
Shift
Left
Error
Budgets
0
100
!!
Toil
Limits
Cloud
Native+ + + + +

41. DevOps + SRE
Product,
Not Project
Continuous
Delivery
Shift
Left
Error
Budgets
0
100
!!
Toil
Limits
Cloud
Native+ + + + +

42. Dev Ops
Cross-Functional Team
Cross-Functional Team
DevOps + SRE
Product,
Not Project
Continuous
Delivery
Shift
Left
Error
Budgets
0
100
!!
Toil
Limits
Cloud
Native+ + + + +

43. Dev Ops
Cross-Functional Team
Cross-Functional Team
DevOps + SRE
Product,
Not Project
Continuous
Delivery
Shift
Left
Error
Budgets
0
100
!!
Toil
Limits
Cloud
Native+ + + + +
“Value-Aligned” and Self-Regulating
Shared
Responsibility
Model

45. Traditional ITSM

46. Traditional ITSM
ITIL
1989 - ?

47. Traditional ITSM
ITIL
1989 - ?

48. Traditional ITSM
Unintentionally Encourages Silos
ITIL
1989 - ?

49. Traditional ITSM
X X X XX X
Unintentionally Encourages Silos
ITIL
1989 - ?

50. Traditional ITSM
X X X XX X
Unintentionally Encourages Silos
Encourages command
& control management
ITIL
1989 - ?

51. Traditional ITSM
X X X XX X
Unintentionally Encourages Silos
Encourages command
& control management
ITIL
1989 - ?

52. Old Way
New Way

53. Old Way
New Way

55. +

69. REDeploy.io

70. There is no root cause.
(That’s just a political distinction)
REDeploy.io

71. Why?
Why?
Why?
Why?
Why?
There is no root cause.
(That’s just a political distinction)
REDeploy.io

72. Why?
Why?
Why?
Why?
Why?
There is no root cause.
(That’s just a political distinction)
Right,
Wrong,
Safety II,
and You.
REDeploy.io

73. Why?
Why?
Why?
Why?
Why?
There is no root cause.
(That’s just a political distinction)
Right,
Wrong,
Safety II,
and You.
Incidents = unplanned investments
REDeploy.io

75. You
Not

78. Col. John Boyd
OODA Loop

81. Monitoring
Spotting the knowns

82. Monitoring
Spotting the knowns
Observability
Interrogating the unknowns

83. Observability
Interrogating the unknowns

84. Observability
Interrogating the unknowns
Logging: The event

85. Observability
Interrogating the unknowns
Logging: The event
Metrics: Data points over time

86. Observability
Interrogating the unknowns
Logging: The event
Metrics: Data points over time
Tracing: Events in context of a single request

87. Observability
Interrogating the unknowns
Logging: The event
Metrics: Data points over time
Tracing: Events in context of a single request

88. Automated Governance
Objective automated attestation of
GRC controls

89. Automated Governance
Objective automated attestation of
GRC controls

90. Automated Governance
Objective automated attestation of
GRC controls

91. Automated Governance
Objective automated attestation of
GRC controls

92. Monitoring
Observability
Governance
Everyone
Everyone
Everyone
Everyone

93. Incident Command
Mobilization, Coordination, Communication

94. Incident Command
Mobilization, Coordination, Communication
Incident Command System
(FEMA)

95. Incident Command
Mobilization, Coordination, Communication
Incident Command System
(FEMA)

96. Incident Command
Mobilization, Coordination, Communication
Incident Command System
(FEMA)

97. Incident Command
Mobilization, Coordination, Communication
Incident Command System
(FEMA)

98. Incident Command
Mobilization, Coordination, Communication
Incident Command System
(FEMA)
GitHub: PagerDuty/incident-response-docs

99. Ops = Platform Eng + SRE
Divide and conquer

100. Ops = Platform Eng + SRE
Divide and conquer

101. Ops Platform Eng + SRE
Divide and conquer
SRE: Expert Operators (distributed)
Platform Eng: Build and Operate Platform Services (centralized)

102. Ops Platform Eng + SRE
Divide and conquer
SRE: Expert Operators (distributed)
Platform Eng: Build and Operate Platform Services (centralized)

103. Ops Platform Eng + SRE
Divide and conquer
SRE: Expert Operators (distributed)
Platform Eng: Build and Operate Platform Services (centralized)

104. New Views on Escalations
Avoid… but swarm if you do
Support at
the edge
Swarm

105. Diagnose: Health checks, exploratory actions
Take Action!
Restore: Restart, repair actions, rollback

106. The Return of Runbooks
Awhile ago Not that long ago Now

107. The Return of Runbooks
Awhile ago Not that long ago Now
Runbooks
(Mostly Manual)

108. The Return of Runbooks
Awhile ago Not that long ago Now
Runbooks
(Mostly Manual) …

109. The Return of Runbooks
Awhile ago Not that long ago Now
Runbooks
(Mostly Manual)
Runbooks
(Automate!…How?)…
Thanks SRE!

110. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

111. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

112. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

113. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.
Moving the bits is the easy part!

114. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

115. Empower those closest to the action!
Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

116. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

117. De-risk!
Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

118. Before Runbook Automation…

119. Before Runbook Automation…
3 options:

120. 1. Decipher the wiki
Before Runbook Automation…
3 options:

121. 1. Decipher the wiki
2.Ad-hoc tool/script usage
Before Runbook Automation…
3 options:

122. 1. Decipher the wiki
2.Ad-hoc tool/script usage
3.ESCALATE!
Before Runbook Automation…
3 options:

123. …with Runbook Automation

124. Shorter Incidents. Fewer Escalations.
Before RBA

125. Shorter Incidents. Fewer Escalations.
Before RBA

126. With RBA
Shorter Incidents. Fewer Escalations.

127. With RBA
Shorter Incidents. Fewer Escalations.

128. Before RBA
Shorter Incidents. Fewer Escalations.

129. With RBA
Shorter Incidents. Fewer Escalations.

130. Solve Difficult Security & Compliance Problems
Before RBA

131. Solve Difficult Security & Compliance Problems
With RBA

132. Everything Through a SDLC
Promote

133. Runbooks as a Service

138. Incidents = unplanned investments …the ROI is up to you.

139. Recap!
Elevate the Human.

140. @damonedwards
damon@rundeck.com
Let’s talk…
rundeck.com/dc-munich-19