Presented by Damon Edwards, co-founder of Rundeck, at QCon San Francisco 2019. See a Demo of Rundeck Enterprise : https://www.rundeck.com/see-demo --or-- Download Rundeck Open Source here: https://rundeck.com/open-source Connect: Stack Overflow community: https://stackoverflow.com/questions/tagged/rundeck Github: https://github.com/rundeck/rundeck/issues Twitter: https://twitter.com/Rundeck Facebook: https://www.facebook.com/RundeckInc/ LinkedIn: www.linkedin.com › company › rundeck-inc

1. Damon Edwards
Incident Management
in the Age of DevOps and SRE
November 12, 2019

2. Damon Edwards
Incident Management
in the Age of DevOps and SRE
@damonedwards
November 12, 2019

4. Assertion:
The ability to respond to and resolve incidents is the true
indicator of an organization’s operational capabilities

6. Assertion 2:
Everybody now works in “Operations"

7. What Is an Incident?
An unplanned disruption impacting
customers or business operations

8. What Is an Incident?
An unplanned disruption impacting
customers or business operations
Outages
Service Degradation

9. What Is an Incident?
An unplanned disruption impacting
customers or business operations
Outages
Service Degradation
Work interruption
Delay/Waiting
“Short-Notice” Requests

13. Board

14. Integrated
Board

15. Integrated
Responsive
Board

16. Integrated
Responsive
Everywhere
Board

17. Integrated
Responsive
Everywhere
Always
Board

18. Integrated
Responsive
Everywhere
Always
Board
Tech Org Execution

19. Integrated
Responsive
Everywhere
Always
Board
Tech Org Execution

21. Kubernetes
AWS GCP Azure
Docker
Consul
Terraform Istio
Zipkin
Envoy
Serverless
OpenShift
KafkaLamba
Prometheus
Containerd
Helm
Cloud Foundry
Linkerd
Etcd
CoreDNS
MongoDB
Redis
InfluxDB
Jaeger
gRPC
CRI-O
Cognito
Fargate
Cloud Functions
Cosmos
BigQuery Spark
Rook
Ceph
NGINXHAProxy
Open vSwitch
NSX Sensu
Vault
Aurora
Nomad

22. Kubernetes
AWS GCP Azure
Docker
Consul
Terraform Istio
Zipkin
Envoy
Serverless
OpenShift
KafkaLamba
Prometheus
Containerd
Helm
Cloud Foundry
Linkerd
Etcd
CoreDNS
MongoDB
Redis
InfluxDB
Jaeger
gRPC
CRI-O
Cognito
Fargate
Cloud Functions
Cosmos
BigQuery Spark
Rook
Ceph
NGINXHAProxy
Open vSwitch
NSX Sensu
Vault
Aurora
Nomad

23. Kubernetes
AWS GCP Azure
Docker
Consul
Terraform Istio
Zipkin
Envoy
Serverless
OpenShift
KafkaLamba
Prometheus
Containerd
Helm
Cloud Foundry
Linkerd
Etcd
CoreDNS
MongoDB
Redis
InfluxDB
Jaeger
gRPC
CRI-O
Cognito
Fargate
Cloud Functions
Cosmos
BigQuery Spark
Rook
Ceph
NGINXHAProxy
Open vSwitch
NSX Sensu
Vault
Aurora
Nomad

24. Kubernetes
AWS GCP Azure
Docker
Consul
Terraform Istio
Zipkin
Envoy
Serverless
OpenShift
KafkaLamba
Prometheus
Containerd
Helm
Cloud Foundry
Linkerd
Etcd
CoreDNS
MongoDB
Redis
InfluxDB
Jaeger
gRPC
CRI-O
Cognito
Fargate
Cloud Functions
Cosmos
BigQuery Spark
Rook
Ceph
NGINXHAProxy
Open vSwitch
NSX Sensu
Vault
Aurora
Nomad
SAIL/cornell.edu

25. Adrian Cockcroft
Developer
Developer
Developer
Developer
Developer
Old Release Still
Running
Release Plan
Release Plan
Release Plan
Release Plan
Deploy
Feature to
Production
Deploy
Feature to
Production
Deploy
Feature to
Production
Deploy
Feature to
Production
Bugs
Deploy
Feature to
Production
Immutable microservice deployment
scales, is faster with large teams and
diverse platform components
DockerCon EU 2014 Architecture enables speed.
Speed is the advantage.

28. The Three Ways (2013)

29. The Three Ways (2013) The Five Ideals (2019)

31. DEV

32. Go! Go! Go!DEV

33. Go! Go! Go!DEV …OPS?

34. 0000
Go! Go! Go!DEV …OPS?

35. 0000
Go! Go! Go!DEV …OPS?
Operations:
The Last Mile

38. 1. SRE needs Service Level Objectives, with consequences
2. SREs have time to make tomorrow better than today
3. SRE teams have the ability to regulate their workload
Principles of SRE

39. 1. SRE needs Service Level Objectives, with consequences
2. SREs have time to make tomorrow better than today
3. SRE teams have the ability to regulate their workload
Principles of SRE

40. 1. SRE needs Service Level Objectives, with consequences
2. SREs have time to make tomorrow better than today
3. SRE teams have the ability to regulate their workload
Principles of SRE

42. DevOps + SRE
Product,
Not Project
Continuous
Delivery
Shift
Left
Error
Budgets
0
100
!!
Toil
Limits
Cloud
Native+ + + + +

43. DevOps + SRE
Product,
Not Project
Continuous
Delivery
Shift
Left
Error
Budgets
0
100
!!
Toil
Limits
Cloud
Native+ + + + +

44. Dev Ops
Cross-Functional Team
Cross-Functional Team
DevOps + SRE
Product,
Not Project
Continuous
Delivery
Shift
Left
Error
Budgets
0
100
!!
Toil
Limits
Cloud
Native+ + + + +

45. Dev Ops
Cross-Functional Team
Cross-Functional Team
DevOps + SRE
Product,
Not Project
Continuous
Delivery
Shift
Left
Error
Budgets
0
100
!!
Toil
Limits
Cloud
Native+ + + + +
“Value-Aligned” and Self-Regulating
Shared
Responsibility
Model

47. Traditional ITSM

48. Traditional ITSM
ITIL
1989 - ?

49. Traditional ITSM
ITIL
1989 - ?

50. Traditional ITSM
Unintentionally Encourages Silos
ITIL
1989 - ?

51. Traditional ITSM
X X X XX X
Unintentionally Encourages Silos
ITIL
1989 - ?

52. Traditional ITSM
X X X XX X
Unintentionally Encourages Silos
Encourages command
& control management
ITIL
1989 - ?

53. Traditional ITSM
X X X XX X
Unintentionally Encourages Silos
Encourages command
& control management
ITIL
1989 - ?

54. Old Way
New Way

55. Old Way
New Way

57. +

71. REDeploy.io

72. There is no root cause.
(That’s just a political distinction)
REDeploy.io

73. Why?
Why?
Why?
Why?
Why?
There is no root cause.
(That’s just a political distinction)
REDeploy.io

74. Why?
Why?
Why?
Why?
Why?
There is no root cause.
(That’s just a political distinction)
Right,
Wrong,
Safety II,
and You.
REDeploy.io

75. Why?
Why?
Why?
Why?
Why?
There is no root cause.
(That’s just a political distinction)
Right,
Wrong,
Safety II,
and You.
Incidents = unplanned investments
REDeploy.io

77. You
Not

83. 18Million
IT Ops
22.3Million
Developers

86. Col. John Boyd
OODA Loop

89. Monitoring
Spotting the knowns

90. Monitoring
Spotting the knowns
Observability
Interrogating the unknowns

91. Observability
Interrogating the unknowns

92. Observability
Interrogating the unknowns
Logging: The event

93. Observability
Interrogating the unknowns
Logging: The event
Metrics: Data points over time

94. Observability
Interrogating the unknowns
Logging: The event
Metrics: Data points over time
Tracing: Events in context of a single request

95. Observability
Interrogating the unknowns
Logging: The event
Metrics: Data points over time
Tracing: Events in context of a single request

96. Automated Governance
Objective automated attestation of
GRC controls

97. Automated Governance
Objective automated attestation of
GRC controls

98. Automated Governance
Objective automated attestation of
GRC controls

99. Monitoring
Observability
Governance
Everyone
Everyone
Everyone
Everyone

100. Incident Command
Mobilization, Coordination, Communication

101. Incident Command
Mobilization, Coordination, Communication
Incident Command System
(FEMA)

102. Incident Command
Mobilization, Coordination, Communication
Incident Command System
(FEMA)

103. Incident Command
Mobilization, Coordination, Communication
Incident Command System
(FEMA)

104. Incident Command
Mobilization, Coordination, Communication
Incident Command System
(FEMA)

105. Incident Command
Mobilization, Coordination, Communication
Incident Command System
(FEMA)
GitHub: PagerDuty/incident-response-docs

106. Ops = Platform Eng + SRE
Divide and conquer

107. Ops = Platform Eng + SRE
Divide and conquer

108. Ops Platform Eng + SRE
Divide and conquer
SRE: Expert Operators (distributed)
Platform Eng: Build and Operate Platform Services (centralized)

109. Ops Platform Eng + SRE
Divide and conquer
SRE: Expert Operators (distributed)
Platform Eng: Build and Operate Platform Services (centralized)

110. Ops Platform Eng + SRE
Divide and conquer
SRE: Expert Operators (distributed)
Platform Eng: Build and Operate Platform Services (centralized)

111. New Views on Escalations
Avoid… but swarm if you do
Support at
the edge
Swarm

112. Diagnose: Health checks, exploratory actions
Take Action!
Restore: Restart, repair actions, rollback

113. The Return of Runbooks
Awhile ago Not that long ago Now

114. The Return of Runbooks
Awhile ago Not that long ago Now
Runbooks
(Mostly Manual)

115. The Return of Runbooks
Awhile ago Not that long ago Now
Runbooks
(Mostly Manual) …

116. The Return of Runbooks
Awhile ago Not that long ago Now
Runbooks
(Mostly Manual)
Runbooks
(Automate!…How?)…
Thanks SRE!

117. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

118. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

119. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

120. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.
Moving the bits is the easy part!

121. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

122. Empower those closest to the action!
Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

123. Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

124. De-risk!
Runbook Automation
Safe self-service access to the expert knowledge
you need to take action.

125. Before Runbook Automation…

126. Before Runbook Automation…
3 options:

127. 1. Decipher the wiki
Before Runbook Automation…
3 options:

128. 1. Decipher the wiki
2.Ad-hoc tool/script usage
Before Runbook Automation…
3 options:

129. 1. Decipher the wiki
2.Ad-hoc tool/script usage
3.ESCALATE!
Before Runbook Automation…
3 options:

130. …with Runbook Automation

131. Shorter Incidents. Fewer Escalations.
Before RBA

132. Shorter Incidents. Fewer Escalations.
Before RBA

133. With RBA
Shorter Incidents. Fewer Escalations.

134. With RBA
Shorter Incidents. Fewer Escalations.

135. Before RBA
Shorter Incidents. Fewer Escalations.

136. With RBA
Shorter Incidents. Fewer Escalations.

137. Solve Difficult Security & Compliance Problems
Before RBA

138. Solve Difficult Security & Compliance Problems
With RBA

139. Everything Through a SDLC
Promote

140. Runbooks as a Service

145. Incidents = unplanned investments …the ROI is up to you.

146. Recap!
Elevate the Human.

147. @damonedwards
damon@rundeck.com
Let’s talk…
Special thanks to