Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
AdvancedPostMortem Fu& Human Error 101                    John Allspaw                    VP, Tech Ops                    ...
We Want    YOU    etsy.com/careers
ScienceTimeTravelMythbustingReadingHomework
Here To Challenge You
Resilience EngineeringDr. Erik Hollnagel   Dr. David Woods Dr. Sidney Dekker Dr. Richard Cook
Complex, Dynamic
Fundamental Surprises
E.T.T.O.    Efficiency   Thoroughness
Organizations, Policies, Procedures,           Regulations                                       BLUNT       Resources & C...
Organizations, Policies, Procedures,                  Regulations                                                    BLUNT...
FUTURE  +PAST
Why Do Them?
Why?Understand the                 failure
Why?Understand the                 system
Where “System” =                     Networks                        Servers                   Applications               ...
Where “System” =                     Networks                        Servers                   Applications               ...
People
(Anticipation)    Knowing      What    To Expect
(Anticipation)    Knowing         Knowing      What            What    To Expect      To Look For                 (Monitor...
(Anticipation)                   (Response)    Knowing         Knowing        Knowing      What            What          W...
(Anticipation)                   (Response)    Knowing         Knowing        Knowing      Knowing      What            Wh...
(Anticipation)                   (Response)    Knowing         Knowing        Knowing      Knowing      What            Wh...
Microphones are ON?
Event AwarenessCode Deploys
TIMELINE      IRC logging = Rich Data
TIMELINE       Traces of Data      IRC logging = Rich Data
IRC Logs Fed Into Solr
Status BlogTwitter Feed
Flight Data Recorder
Annotation Traces
Investigation Basics
Start?
TTDHow?
TTR
Stable(“all clear”)
Impact Time = TTR - Start
SEVERITY
Severity 1A. Total loss of serviceB. Severe degradation, effectively unusableC. Loss of a critical feature
Severity 2A. Major degradation/feature loss for SUBSET of membersB. Minor degradation/feature loss for ALL members
Severity 3Noticeable non-critical feature loss or degradation
Severity 4No visible impact, loss of redundancy or capacityheadroom
Severity 5No-impact but unexpected failure
5/11/2011 - Payments/Checkout system issue   Start                                     4:10pm   TTD                       ...
•   Basic metrics•   Timeline with details•   Remediations/Observations
“normal”            Incident   PostMortemoperation              Time
How?Why?Prevention?
How
Crisis PatternsProblem Starts                              PostMortem                 Time
Crisis PatternsProblem Starts      Detection                               PostMortem                  Time
Crisis PatternsProblem Starts      Detection             Evaluation                                       PostMortem      ...
Crisis PatternsProblem Starts      Detection             Evaluation                     Response                          ...
Crisis PatternsProblem Starts      Detection             Evaluation                     Response                          ...
Crisis PatternsProblem Starts      Detection             Evaluation                     Response                          ...
Crisis PatternsProblem Starts      Detection             Evaluation                     Response                          ...
Crisis PatternsProblem Starts                          Stress      Detection             Evaluation                     Re...
Crisis PatternsProblem Starts                              PostMortem                 Time
Crisis PatternsProblem Starts      Detection                               PostMortem                  Time
Crisis PatternsProblem Starts      Detection   Evaluation                               PostMortem                  Time
Crisis PatternsProblem Starts      Detection   Evaluation                    Response                               PostMo...
Crisis PatternsProblem Starts      Detection   Evaluation                    Response                          Stable     ...
Crisis PatternsProblem Starts      Detection   Evaluation                    Response                          Stable     ...
Crisis PatternsProblem Starts      Detection   Evaluation                    Response                          Stable     ...
Crisis PatternsProblem Starts                 Stress      Detection   Evaluation                    Response              ...
Crisis PatternsProblem Starts                              PostMortem                 Time
Crisis PatternsProblem Starts      Detection                               PostMortem                  Time
Crisis PatternsProblem Starts      Detection          Evaluation                                    PostMortem            ...
Crisis PatternsProblem Starts      Detection          Evaluation               Response                                   ...
Crisis PatternsProblem Starts      Detection          Evaluation               Response                                 St...
Crisis PatternsProblem Starts      Detection          Evaluation               Response                                 St...
Crisis PatternsProblem Starts      Detection          Evaluation               Response                                 St...
Crisis PatternsProblem Starts                      Stress      Detection          Evaluation               Response       ...
Problem Starts      Detection          Evaluation               Response                                 Stable           ...
Problem Starts      Detection          Evaluation               Response                                 Stable           ...
Problem Starts      Detection          Evaluation               Response                                 Stable           ...
Crisis PatternsProblem Starts      Detection          Evaluation               Response                                 St...
Crisis Patterns
Crisis PatternsForced beyond learned roles
Crisis PatternsForced beyond learned rolesActions whose consequences are both important anddifficult to see
Crisis PatternsForced beyond learned rolesActions whose consequences are both important anddifficult to seeCognitively and p...
Crisis PatternsForced beyond learned rolesActions whose consequences are both important anddifficult to seeCognitively and p...
Thematic Vagabonding“butterfly minds”       NOT STUCK ENOUGH
Goal Fixation                 (encystment)TOO STUCK
HeroismNon-communicating lone wolf-isms
DistractionIrrelevant noise in comm channels
IMPROVISATIONREQUIREMENT for troubleshootingcomplex systems
IMPROVISATION
IMPROVISATION
Why?
Root Cause Analysis“With the unknown, one is confronted with danger,discomfort, and care; the first instinct is to abolisht...
Hindsight BiasKnowledge of the outcome influences theanalysis of the processMakes steps towards failure appearforeseeable a...
After The Fact
After The Fact
Reality: Before and During
Hindsight Bias“Should have known better”“All the signs were there, you just needed topay attention”
Hindsight BiasBEFORE accident:        The future seems implausibleAFTER accident:   Obviously clear: “how could they not s...
Hindsight Bias“...people’s need to be rightis stronger than their abilityto be objective.”                          N. Cra...
Outcome BiasJudging a past decisionbased on its outcome.
ID          Fishbone/Ishikawa     FMEA                         Five Whys                Fault Tree      CED      CRT
why?       OUTAGE
why?because  this             OUTAGE
why? why?  because    this            OUTAGE
why? why?because   because  this      this                    OUTAGE
why?     why? why?   because   because     this      this                       OUTAGE
why?       why? why?because   because   because  this      this      this                              OUTAGE
why?        why? why?because    because   because  this       this      this                               OUTAGE         ...
which                       causedSome       Caused       otherAction   Some Things   things                              ...
Sequence-Of-Events
• Satisfyingly simple, easy to explain and document
•   Satisfyingly simple, easy to explain and document•   Solves for a specific case•   Ignorant of surrounding circumstance...
NOT HELPFUL•   Satisfyingly simple, easy to explain and document•   Solves for a specific case•   Ignorant of surrounding c...
Epidemiological        (adapted from Reason, 1990)
(adapted from Reason, 1990)
Holes = Active/Latent Failures,Bad Things™ Waiting to Happen                  (adapted from Reason, 1990)
Holes = Active/Latent Failures,                            Bad Things™ Waiting to HappenCheese = Safety Barriers,   Layers...
(adapted from Reason, 1990)
Code   Servers           (adapted from Reason, 1990)
Code        ServersSchedule     Training   (adapted from Reason, 1990)
Unmonitored Disk Space   (latent condition)
Capacity                     Unmonitored Disk Space  Miscalculation                        (latent condition)(latent condi...
Capacity                     Unmonitored Disk Space  Miscalculation                        (latent condition)(latent condi...
Violation of known coding     Capacity                standards         Unmonitored Disk Space  Miscalculation         (la...
Violation of known coding     Capacity                          standards         Unmonitored Disk Space  Miscalculation  ...
Violation of known coding     Capacity                          standards         Unmonitored Disk Space  Miscalculation  ...
Violation of known coding     Capacity                          standards         Unmonitored Disk Space  Miscalculation  ...
Violation of known coding     Capacity                          standards         Unmonitored Disk Space  Miscalculation  ...
Violation of known coding     Capacity                          standards         Unmonitored Disk Space  Miscalculation  ...
Violation of known coding     Capacity                          standards         Unmonitored Disk Space  Miscalculation  ...
Violation of known coding     Capacity                          standards         Unmonitored Disk Space  Miscalculation  ...
Violation of known coding     Capacity                          standards         Unmonitored Disk Space  Miscalculation  ...
Violation of known coding     Capacity                          standards         Unmonitored Disk Space  Miscalculation  ...
Capacity  Miscalculation(latent condition)                                                            Unit Test In        ...
Capacity  Miscalculation(latent condition)                                                            Unit Test In        ...
Capacity  Miscalculation(latent condition)                                                            Unit Test In        ...
Capacity  Miscalculation(latent condition)                                                            Unit Test In        ...
• Better than dominoes, but still linear layers of “defense”• Helps uncover multiple contributors and latent failures (at ...
• Better than dominoes, but still linear layers of “defense”• Helps uncover multiple contributors and latent failures (at ...
Better, but still                                               NOT HELPFUL• Better than dominoes, but still linear layers...
Multiple Contributors “each necessary but only jointly sufficient”
Resultant Versus Emergent
Systemic                DatabaseRouter                     Memcache    Webserver
Systemic                DatabaseRouter                        Memcache    Webserver                            Feature    ...
Systemic                DatabaseRouter                        Memcache                                  Last Round    Webs...
Systemic                        Dashboard                                 Design                DatabaseRouter            ...
Systemic                          Dashboard                                    Design                   Database  Router  ...
Systemic                          Dashboard                                    Design                   Database  Router  ...
Systemic                           Dashboard                                     Design                    Database  Route...
Systemic                           Dashboard                                     Design                    Database  Route...
Systemic                               Dashboard                                         Design                    Databas...
Systemic Hiring                                        Dashboard                                         Design        Dif...
Systemic Hiring                          S3 is slow    Dashboard                                         Design        Dif...
Systemic Hiring                          S3 is slow    Dashboard                                         Design        Dif...
Systemic
Systemic
Systemic
Systemic
Systemic
Systemic
Systemic
Functional ResonanceIn isolation, components act within bounds.Interconnected, they produce emergingbehaviors.
Causes Are Constructed,Not Found WYLFIWYF Pre-conceived notions on “causes” and behaviors
Contributors, not causes
There is no root cause.
LEARNING
Quantifying ResponseTime to detect?Time for escalation, internal notification?Time to notify the public?Time to graceful de...
Qualifying ResponseHigh signal:noise in comm channels?Troubleshooting fatigue?Troubleshooting handoff?All tools on-hand?Met...
All Together Now•   Start/TTD/TTR/Stable/etc.•   Severity•   DATA (graphs, IRC, etc.)•   Description (timeline, etc.)•   O...
(Anticipation)                   (Response)    Knowing         Knowing        Knowing      Knowing      What            Wh...
(Anticipation)                   (Response)    Knowing         Knowing        Knowing      Knowing      What            Wh...
Human Error“...knowledge and error flow from thesame mental sources, only successcan tell one from the other.”             ...
Human ErrorNobody comes to work to do a bad job.
Human ErrorUseless as a label and ending point.
Human ErrorHuman error isn’t a cause, it’san effect.
Why did it make senseto the personat the time?
Error Categories• Slips• Lapses• Mismatches• Violations
Error Categories
First Stories“Human error” seen as root cause.Counterfactuals: saying what they “should”have done.Prevention: be “more car...
Second Stories“Human error” seen as systemic vulnerabilities,deeper inside the organization.Digging into why it made sense...
Why did it make senseto the personat the time?
Why did it make senseto the personat the time?
Why did it make senseto the personat the time?
Substitution TestCould peers have made thesame error under the samecircumstances?
WHERE to learn from ?
Two Propositions
100 deploys6 deploy-related issues
100 > 6
Proposition #1“Ways in which things go right are special casesof the ways in which things go wrong.”
Proposition #1Successes = failures gone wrongStudy the failures, generalize from that.       data sources: 6 out of 100
Proposition #2“Ways in which things go wrong are specialcases of the ways in which things go right.”
Proposition #2Failures = successes gone wrongStudy the successes, generalize from thatdata sources:   94 out of 100
94/100 ?   OR6/100 ?
What and WHY Do ThingsGo RIGHT?
Not just:                why did we fail?But also:            why did we succeed?
Near MissesHey everybody -Don’t be like me. I tried to X, butbecause it was no good.It almost exploded everyone.So, don’t ...
Taking the New ViewRecognize that human error isan attribution.
Taking the New ViewPursue Second Stories.
Taking the New ViewEscape Hindsight Bias.
Taking the New ViewUnderstand work as performedat the sharp end.
Taking the New ViewExamine how changes (at alllayers) will produce newvulnerabilities.
Taking the New ViewUse technology to support andenhance human expertise.
Taking the New ViewTame complexity through newforms of feedback.
Taking the New ViewRealize that your systems arenot inherently safe.
Taking the New ViewHuman error is an inevitableby-product of strainedcomplex systems.
Taking the New ViewHuman error isn’t at the root ofyour safety problems.
Taking the New ViewHuman error isn’t random.
(Anticipation)                   (Response)    Knowing         Knowing        Knowing      Knowing      What            Wh...
Just Culture
Just CultureBalancing accountability withlearning.
Intentional Malice
Negligence  Found         Severity of the Outage
NameBlameShame
NameBlameShame
Name  WHY? Blame#!@%$&#Shame
“Must set an example!”
“Has to be some fear that notdoing one’s job correctly couldlead to punishment.”
“Must set an example!”       Punishing       Deterrents is a not“Has to be some fear that            Losing coulddoing one...
Holding People Accountable          !=     Blaming People
Accountability    and  Learning                 Punishment For Errors
No Bad ApplesOnly Bad Theories of Error
Name  WHY? Blame#!@%$&#Shame
Signs Of Old View“Gross Misconduct”“Carelessness”“Negligence”“Egregious Behavior”“Willful Violations”
Discretionary Spaces
AcceptableUnacceptable
AcceptableUnacceptable
AcceptableUnacceptable
AcceptableUnacceptable
Acceptable   (who draws this subjective line?)Unacceptable
Increase Accountability BySupporting Learning
Empower PeopleLet them own their own stories.Don’t make people pay penalties.Allow them to educate the organization.
Reduce UncertaintyMake it clear who definesacceptable behavior.
Organizational RootsAccountability =Responsibility + Requisite Authority
(Anticipation)                   (Response)    Knowing         Knowing        Knowing      Knowing      What            Wh...
(Thanks, Fellas)Dr. Erik Hollnagel   Dr. David Woods Dr. Sidney Dekker Dr. Richard Cook
Homework!
We Want    YOU    etsy.com/careers
•                                             Photo Credits    http://www.flickr.com/photos/51035644987@N01/2678090600/•   ...
THE END
• Photo Credits http://www.flickr.com/photos/51035644987@N01/2678090600/• http://www.flickr.com/photos/67196253@N00/29416559...
• Photo Credits http://www.flickr.com/photos/51035644987@N01/2678090600/• http://www.flickr.com/photos/67196253@N00/29416559...
• Photo Credits http://www.flickr.com/photos/51035644987@N01/2678090600/• http://www.flickr.com/photos/67196253@N00/29416559...
• Photo Credits http://www.flickr.com/photos/51035644987@N01/2678090600/• http://www.flickr.com/photos/67196253@N00/29416559...
• Photo Credits http://www.flickr.com/photos/51035644987@N01/2678090600/• http://www.flickr.com/photos/67196253@N00/29416559...
• Photo Credits http://www.flickr.com/photos/51035644987@N01/2678090600/• http://www.flickr.com/photos/67196253@N00/29416559...
• Photo Credits http://www.flickr.com/photos/51035644987@N01/2678090600/• http://www.flickr.com/photos/67196253@N00/29416559...
• Photo Credits http://www.flickr.com/photos/51035644987@N01/2678090600/• http://www.flickr.com/photos/67196253@N00/29416559...
• Photo Credits http://www.flickr.com/photos/51035644987@N01/2678090600/• http://www.flickr.com/photos/67196253@N00/29416559...
• Photo Credits http://www.flickr.com/photos/51035644987@N01/2678090600/• http://www.flickr.com/photos/67196253@N00/29416559...
• Photo Credits http://www.flickr.com/photos/51035644987@N01/2678090600/• http://www.flickr.com/photos/67196253@N00/29416559...
Prochain SlideShare
Chargement dans…5
×

• Photo Credits http://www.flickr.com/photos/51035644987@N01/2678090600/• http://www.flickr.com/photos/67196253@N00/2941655917/• Advanced PostMortem Fu and Human Error 101 (Velocity 2011)

68 837 vues

Publié le

• Photo Credits http://www.flickr.com/photos/51035644987@N01/2678090600/• http://www.flickr.com/photos/67196253@N00/2941655917/• http://www.flickr.com/photos/stirwise/417629641• http://www.flickr.com/photos/38383999@N06/3888057995/• http://www.flickr.com/photos/94443490@N00/361543080/• http://www.flickr.com/photos/7729940@N06/4333396494/• http://www.flickr.com/photos/cpstorm/167418602• http://www.flickr.com/photos/63474264@N00/4366221069/• http://www.flickr.com/photos/proimos/4199675334• http://www.flickr.com/photos/30475691@N07/2862060992/• http://www.flickr.com/photos/14663487@N00/797755046/• http://www.flickr.com/photos/25080113@N06/5361445631/

Publié dans : Technologie
  • DOWNLOAD FULL eBOOK INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookeBOOK Crime, eeBOOK Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD FULL eBOOK INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookeBOOK Crime, eeBOOK Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD FULL. BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • Nice !! Download 100 % Free Ebooks, PPts, Study Notes, Novels, etc @ https://www.ThesisScientist.com
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • hello
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

×