The Upside of Downtime (Velocity 2010)

http://gapingvoid.com/
Sunday, June 20, 2010

The Upside of Downtime
Turning disaster into opportunity


Who’s had a site go down?


Who’s hasn’t had a site go
down?


There’s always
that one guy!


Downtime
sucks

Source: http://www.motivatedphotos.com/?id=8080

Why downtime sucks
Business $3,000

$2,250

$1,500
Sales
$750

$0
0 2 4 6 8 10 12 14 16 18 20 22


Why downtime sucks
Business
Brand


Why downtime sucks
Business
Brand
You


Why downtime sucks
Business
Brand
You
Users


Downtime = Bad! (Duh)


Approach #1
Don’t fail


Source: http://kansansforlife.ﬁles.wordpress.com/2009/12/titanic.jpg

“Everything fails all the time”
-- Werner Vogels (Amazon, CTO)


Your site
will fail

Werner Vogels
(Amazon, CTO)

Why?!?


Why Failure Happens
Risk Homeostasis

Source: http://joshuahind.ﬁles.wordpress.com/2009/09/bicycle-crash.jpg


Why Failure Happens
Risk Homeostasis
Black Swan

Source: Amazon.com

Why Failure Happens
Risk Homeostasis
Black Swan
Unknown unknowns

Source: http://www.apoliticus.com/wp-content/uploads/2009/01/6_21_080306_rumsfeld.jpg

Why Failure Happens
Risk Homeostasis
Black Swan
Unknown unknowns
Change

Source: http://bozark.net/wordpress/wp-content/uploads/2008/09/barack_obama_change_fairey.jpg

Why Failure Happens
Risk Homeostasis
Black Swan
Unknown unknowns
Change
Many small failures

Source: http://www.biojobblog.com/uploads/image/dominos.jpg


Why Failure Happens
Risk Homeostasis
Black Swan
Unknown unknowns
Change
Many small failures
Humans
Source: http://www.librarian.net/talks/clc/CLC.key/SJ_Shoulder_Shrug.jpg

Polisher
blocked

Not unusual

Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm

Polisher Moisture leaks into
blocked air system

Not unusual Not expected


Polisher Moisture leaks into Flow of cold water
blocked air system stopped

Not unusual Not expected Not good



Backup disabled



Backup disabled

Doh! Indicator blocked



Backup disabled


Dammit Relief valve broken



Backup disabled



WTF Gauge broken



Backup disabled



Meltdown Gauge broken


Source: http://support.rightscale.com/09-Clouds/AWS/02-Amazon_EC2/Designing_Failover_Architectures_on_EC2/03-Advanced_Failover_Architecture

“accidental power failure”

Source: http://www.datacenterknowledge.com/archives/2010/06/16/power-failure-kos-intuit-sites-for-24-hours/

“trafﬁc accident damaged a nearby
utility transformer”
Source: http://www.datacenterknowledge.com/archives/2007/11/13/truck-crash-knocks-rackspace-ofﬂine/

“unfortunate code change”
Source: http://www.datacenterknowledge.com/archives/2010/06/11/errant-code-change-crashes-10-million-blogs/

“Unhappy customers may get some
attention, but unhappy networked
customers can quickly impact your
business”
-- Clay Shirky

Source: http://happenupon.ﬁles.wordpress.com/2009/02/technology-guru-clay-shir-001.jpg, http://scholarlykitchen.sspnet.org/2010/03/02/shirky-at-nfais-how-abundance-breaks-everything/

http://labs.webmetrics.com/crowdsourceduptime

Recap


Your site will fail


Your site will fail
+
Downtime is bad


Your site will fail
+
Downtime is bad
+
Everyone will ﬁnd out


Your site will fail
+
Downtime is bad
+
Everyone will ﬁnd out
=
Screw it, I’ll become a
lumberjack
Source: http://sbadrinath.ﬁles.wordpress.com/2009/03/different26rqcu3.jpg

“Embrace fear of outages and
degradation. Use it to guide your
architecture, your code, your
infrastructure. So lean into it.”
-- John Allspaw, VP Tech. Ops at Etsy


Approach #2
Prepare for downtime


Disclaimer:
Try hard to avoid downtime


Learning by example...


Case Study #1
Facebook


“The larger issue here isn't just that a portion of
Facebook's platform has gone down - numerous web
services have issues from time to time, including
everything from Gmail to Twitter. An outage of this
length, however, with no ofﬁcial communication
from the company itself is disturbing.”
-- N.Y. Times


Facebook

Downtime Disturbing


Case Study #2
Google App Engine


Google App Engine

Downtime Kudos


Case Study #3
Atlassian


Atlassian

Downtime Bravo


http://atlassian.com/


Downtime:
Opportunity to Build Trust


Downtime:
Opportunity to Destroy Trust


How To:
Prepare for Downtime


Something > Nothing


Upside of Downtime Framework 1.0

Life is good Oh crap That sucked
Time



Prepare Communicate Explain
Time



1. Communication channel




Something is Can’t tell if it’s I’ll assume it’s
wrong me or you you

You suck




Something is Can’t tell if it’s I’ll assume it’s
wrong me or you you

Tell me when You suck a lot
I know it’s you
you’re back less



Easy to ﬁnd



Easy to ﬁnd
Hosted off-site



Easy to ﬁnd
Hosted off-site
Real-time / automated


7 keys for public health dashboards

1. Must show current status for each “service”
2. Data must be accurate and timely
3. Must be easy to ﬁnd
4. Must provide details for events in real time
5. Provide historical uptime and performance data
6. Provide a way to be notiﬁed of status changes
7. Provide details on the data is gathered

Source: http://www.transparentuptime.com/2008/11/rules-for-successful-public-health.html



Easy to ﬁnd
Hosted off-site

2. Process



Easy to ﬁnd
Hosted off-site

2. Process
Authority



Easy to ﬁnd
Hosted off-site

2. Process
Authority
Mean-Time-To-Communicate (MTTC)



Easy to ﬁnd
Hosted off-site

2. Process
Authority
Mean-Time-To-Communicate (MTTC)
On-call/drills/escalations/etc.

Your servers



1. Communicate



1. Communicate
Use communication channel



1. Communicate
MTTC



1. Communicate
MTTC
Who/what is affected



1. Communicate
MTTC
When the incident started



1. Communicate
MTTC
ETA



1. Communicate
MTTC
ETA
Update regularly



1. Communicate
MTTC
ETA
Update regularly

2. Fix it!

Phew, close
one!



1. Postmortem



1. Postmortem
Admit failure

Source: http://en.blog.wordpress.com/2010/02/19/wp-com-downtime-summary/


1. Postmortem
Admit failure
Sound like a human

Source: http://www.bureauofcommunication.com/compose/apology


“We apologize for any
inconvenience this may
have caused”



1. Postmortem
Admit failure
Sound like a human
Start time and end time

Source: https://groups.google.com/group/google-appengine/browse_thread/thread/a7640a2743922dcf


1. Postmortem
Admit failure
Sound like a human
Who/what was impacted

Source: http://techcrunch.com/2009/11/02/large-scale-downtime-at-rackspace-cloud/


1. Postmortem
Admit failure
Sound like a human
What went wrong

Source: http://www.zendesk.com/2010/03/tuesday-double-whammy.html


1. Postmortem
Admit failure
Sound like a human
What went wrong
Lessons learned

Source: http://graysky.org/2010/02/downtime-postmortem/


1. Postmortem
Admit failure
Sound like a human
What went wrong
Lessons learned



“I was completely overwhelmed by
the amount of positive feedback and
support I received.”


1. Postmortem
Admit failure
Sound like a human
What went wrong
Lessons learned

2. Improve for the future


“Google is not just saying sorry, they are
actually implementing serious changes which
probably represents millions of dollars of
development to help make sure this doesn't
happen again.”

Source: http://news.ycombinator.com/item?id=1168493



Source: https://groups.google.com/group/google-appengine/browse_thread/thread/a7640a2743922dcf


Be human



Be authentic



Be transparent



Accept responsibility



Learn and improve



Trust



1. Communication channel 1. Communicate 1. Post-mortem
- Easy to ﬁnd - Use channel - Admit failure
- Off-site - M.T.T.C. - Sound like a human
- Real-time - Who/what affected - Start time and end time
- When started - Who/what was impacted
2. Process - ETA to resolution - What went wrong
- Give authority - Update regularly - Lessons learned
- M.T.T.C.
- On-call/escalations 2. Fix it! 2. Learn and improve



- M.T.T.C.

Be Prepared + Be Transparent + Be Human



- M.T.T.C.

Be Prepared + Be Transparent + Be Human =

Trust

Disclaimer:
Don’t screw up too often


Downtime Prisoner’s Dilemma

Transparent Not Transparent

Caught

Not
Caught




Caught

Not
Caught Win




Caught Big Loss

Not
Caught Win




Caught Big Win Big Loss

Not
Caught Win




Caught Big Win Big Loss

Not
Caught Win Win


Beneﬁts
Gain trust
Reduce churn, increase loyalty
Reduce support costs
Ability to control the message
Competitive advantage
More time to focus on the actual problem
Reduce stress


Change != Easy


Change != Impossible


Keys to Adoption
Getting past a culture of “hide the problem”


Keys to Adoption
Overriding commitment to want to improve


Keys to Adoption
Available resources to improve


Keys to Adoption
Pain


Keys to Adoption
Pain
Buy-in


Product
Management

Support

Engineering/
Operations

Sales/
Marketing


Product Default: Lets wait for complaints
Management

Support

Engineering/
Operations

Sales/
Marketing


Management Reality: Proactiveness => Forgiveness

Support

Engineering/
Operations

Sales/
Marketing



Default: Too much work
Support

Engineering/
Operations

Sales/
Marketing



Support
Reality: More upfront, less when it matters

Engineering/
Operations

Sales/
Marketing



Support

Engineering/ Default: Don’t want to look bad
Operations

Sales/
Marketing



Support

Operations Reality: Opportunity to learn/improve

Sales/
Marketing



Support


Sales/ Default: I don’t want my customers to know
Marketing



Support


Sales/ Default: I don’t want my customers to know
Marketing Reality: They’ll ﬁnd out, better from us


Source: http://delicious.com/lennysan/healthdashboard


Simple as that!


Your site
will still fail!


“The measure of a society is how
well it transforms pain and suffering
into something worthwhile.”
-- Fredrick Nietzsche


“The measure of a company is how
well it transforms pain of downtime
into something worthwhile.”
-- Lenny Rachitsky

Source: Original quote inspired by Fredrick Nietzsche

Bare minimum:
Register a Twitter account


Thank You

Slides: http://bit.ly/upside-of-downtime

Lenny Rachitsky
@lennysan
http://www.transparentuptime.com/

Webmetrics/Neustar
@webmetrics
http://www.webmetrics.com/

Bonus


- M.T.T.C.


- M.T.T.C.

"Unlikely that an accidental surface or subsurface
oil spill would occur from the proposed activities"
-- Exploration and environmental impact plan

Source: http://en.wikipedia.org/wiki/Deepwater_Horizon_drilling_rig_explosion


“Be not afraid of transparency;
some are born transparent,
some achieve transparency,
and others have transparency

thrust upon them.”
-- Burrowed from William Shakespeare


Making change
1. Find the bright spots - (this presentation has a bunch)


Making change
2. Script the critical moves - (framework)


Making change
3. Point to the destination - (W.W.G.D.)


Making change

4. Find the feeling - (how would you feel?)


Making change

5. Shrink the change - (start small)


Making change

6. Grow your people - (everyone is learning as they go)


Making change

7. Tweak the environment - (create a simple process)


Making change

8. Build habits - (build process organically)


Making change

8. Build habits - (build process organically)
9. Rally the herd - (get buy in, rest will follow)

The Upside of Downtime (Velocity 2010)

Recommended

Recommended

More Related Content

More from Lenny Rachitsky

More from Lenny Rachitsky (11)

Recently uploaded

Recently uploaded (20)

The Upside of Downtime (Velocity 2010)