The Journey of Chaos Engineering Begins with a Single Step

Bruce Wong
Bruce WongR&D Leadership at Twilio Inc. à Twilio Inc.
#PDSummit16#PDSummit16
The Journey of Chaos Engineering
Begins with a Single Step
#PDSummit16#PDSummit16
Bruce WongSenior Engineering Manager
Twilio
@bruce_m_wong
https://www.linkedin.com/in/brucemwong
#PDSummit16#PDSummit16
#PDSummit16#PDSummit16
2009
2012
2014
http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html
https://github.com/Netflix/SimianArmy
http://techblog.netflix.com/2015/09/chaos-engineering-upgraded.html
#PDSummit16
http://readwrite.com/2014/09/17/netflix-chaos-engineering-for-everyone/
http://techblog.netflix.com/2014/09/introducing-chaos-engineering.html
#PDSummit16
https://www.twilio.com/
#PDSummit16#PDSummit16
https://customers.twilio.com/
#PDSummit16#PDSummit16
The journey of a thousand
miles begins with a single
step.
-Lao Tzu
#PDSummit16#PDSummit16
James BurnsTech Lead
Twilio
@1mentat
#PDSummit16
https://www.linkedin.com/in/james-burns-7816a82
#PDSummit16#PDSummit16
Preparation
Pre-Launch Log Aggregation System
-Stage env
-Synthetic Traffic
#PDSummit16
The Master of Disaster
•Network Issues
•Partitions
•Thundering Herds
•Cascading Failures
•Resource Starvation
•CPU
•Memory
•Disk IO
•Network IO
•Application Load
> sudo halt
#PDSummit16
Incident Start
#PDSummit16
Impact?
#PDSummit16
Post-Mortem
#PDSummit16
#PDSummit16#PDSummit16
#PDSummit16
Round 2
•Network Issues
•Partitions
•Thundering Herds
•Cascading Failures
•Resource Starvation
•CPU
•Memory
•Disk IO
•Network IO
•Application Load
> sudo halt
#PDSummit16
Third-Party API Failure
#PDSummit16
#PDSummit16
Well, that’s not what
I expected to see
#PDSummit16
Outcomes
Instrument
Instrument
Instrument
API SLAs
Architectural
Change!
#PDSummit16
Recap
• Start Simple
• Instrumentation
Gaps
• Understand your
dashboards
• Prevent outages
#PDSummit16
http://www.crisistextline.org/
http://polarisproject.org/befree-textline
http://trekmedics.org/
https://www.twilio.org/
#PDSummit16
When you wish upon a
blue moon…
#PDSummit16#PDSummit16
Please provide
feedback for this
session by filling out
the feedback survey
1 sur 25

Recommandé

DESIGN IT! Talk #001 (UX) Yukio Andoh par
DESIGN IT! Talk #001 (UX) Yukio AndohDESIGN IT! Talk #001 (UX) Yukio Andoh
DESIGN IT! Talk #001 (UX) Yukio AndohYukio Andoh
1.6K vues103 diapositives
Web 2.0 Tools for PR Pros: Online Tools that won't break the Bank par
Web 2.0 Tools for PR Pros: Online Tools that won't break the BankWeb 2.0 Tools for PR Pros: Online Tools that won't break the Bank
Web 2.0 Tools for PR Pros: Online Tools that won't break the BankJil Wright
777 vues19 diapositives
Responsive Process Joomla World Conference 2012 par
Responsive Process Joomla World Conference 2012Responsive Process Joomla World Conference 2012
Responsive Process Joomla World Conference 2012Steve Fisher
1.4K vues151 diapositives
WebGL demos showcase par
WebGL demos showcaseWebGL demos showcase
WebGL demos showcaseYukio Andoh
2.3K vues34 diapositives
Smidig kontrakt par
Smidig kontraktSmidig kontrakt
Smidig kontraktchristingorman
307 vues22 diapositives
LivingFoods par
LivingFoodsLivingFoods
LivingFoodsJordan Smith
96 vues5 diapositives

Contenu connexe

En vedette

The Case for Chaos par
The Case for ChaosThe Case for Chaos
The Case for ChaosBruce Wong
13.4K vues33 diapositives
Chaos Patterns par
Chaos PatternsChaos Patterns
Chaos PatternsBruce Wong
1.3K vues47 diapositives
Chaos Patterns Twilio SIGNALCONF 2016 par
Chaos Patterns Twilio SIGNALCONF 2016Chaos Patterns Twilio SIGNALCONF 2016
Chaos Patterns Twilio SIGNALCONF 2016Bruce Wong
1.2K vues32 diapositives
Chaos Driven Development par
Chaos Driven DevelopmentChaos Driven Development
Chaos Driven DevelopmentBruce Wong
1.1K vues44 diapositives
Chaos Engineeringのススメ par
Chaos EngineeringのススメChaos Engineeringのススメ
Chaos EngineeringのススメAtsushi Kojima
1.2K vues7 diapositives
Principles of Chaos Engineering par
Principles of Chaos EngineeringPrinciples of Chaos Engineering
Principles of Chaos Engineeringh_marvin
979 vues30 diapositives

En vedette(20)

The Case for Chaos par Bruce Wong
The Case for ChaosThe Case for Chaos
The Case for Chaos
Bruce Wong13.4K vues
Chaos Patterns par Bruce Wong
Chaos PatternsChaos Patterns
Chaos Patterns
Bruce Wong1.3K vues
Chaos Patterns Twilio SIGNALCONF 2016 par Bruce Wong
Chaos Patterns Twilio SIGNALCONF 2016Chaos Patterns Twilio SIGNALCONF 2016
Chaos Patterns Twilio SIGNALCONF 2016
Bruce Wong1.2K vues
Chaos Driven Development par Bruce Wong
Chaos Driven DevelopmentChaos Driven Development
Chaos Driven Development
Bruce Wong1.1K vues
Principles of Chaos Engineering par h_marvin
Principles of Chaos EngineeringPrinciples of Chaos Engineering
Principles of Chaos Engineering
h_marvin979 vues
Open Business Conference: Continuous Delivery At Netflix -- Powered by Open S... par Dianne Marsh
Open Business Conference: Continuous Delivery At Netflix -- Powered by Open S...Open Business Conference: Continuous Delivery At Netflix -- Powered by Open S...
Open Business Conference: Continuous Delivery At Netflix -- Powered by Open S...
Dianne Marsh1.7K vues
Web Scale Applications using NeflixOSS Cloud Platform par Sudhir Tonse
Web Scale Applications using NeflixOSS Cloud PlatformWeb Scale Applications using NeflixOSS Cloud Platform
Web Scale Applications using NeflixOSS Cloud Platform
Sudhir Tonse1.9K vues
Principles Of Chaos Engineering - Chaos Engineering Hamburg par Nils Meder
Principles Of Chaos Engineering - Chaos Engineering HamburgPrinciples Of Chaos Engineering - Chaos Engineering Hamburg
Principles Of Chaos Engineering - Chaos Engineering Hamburg
Nils Meder960 vues
Embracing Failure - Fault Injection and Service Resilience at Netflix par Josh Evans
Embracing Failure - Fault Injection and Service Resilience at NetflixEmbracing Failure - Fault Injection and Service Resilience at Netflix
Embracing Failure - Fault Injection and Service Resilience at Netflix
Josh Evans3.1K vues
Release the Monkeys ! Testing in the Wild at Netflix par Gareth Bowles
Release the Monkeys !  Testing in the Wild at NetflixRelease the Monkeys !  Testing in the Wild at Netflix
Release the Monkeys ! Testing in the Wild at Netflix
Gareth Bowles3.9K vues
What's New in Java 8 par javafxpert
What's New in Java 8What's New in Java 8
What's New in Java 8
javafxpert42.3K vues
Scalable Microservices at Netflix. Challenges and Tools of the Trade par C4Media
Scalable Microservices at Netflix. Challenges and Tools of the TradeScalable Microservices at Netflix. Challenges and Tools of the Trade
Scalable Microservices at Netflix. Challenges and Tools of the Trade
C4Media10.9K vues
Scaling the Cloud - Cloud Security par Bill Burns
Scaling the Cloud - Cloud SecurityScaling the Cloud - Cloud Security
Scaling the Cloud - Cloud Security
Bill Burns8.1K vues
AWS Re:Invent 2012 - Chaos Monkey & The Netflix Simian Army par Ariel Tseitlin
AWS Re:Invent 2012 - Chaos Monkey & The Netflix Simian ArmyAWS Re:Invent 2012 - Chaos Monkey & The Netflix Simian Army
AWS Re:Invent 2012 - Chaos Monkey & The Netflix Simian Army
Ariel Tseitlin2.8K vues
From resilient to antifragile - Chaos Engineering Primer DevSecCon par Sergiu Bodiu
From resilient to antifragile - Chaos Engineering Primer DevSecConFrom resilient to antifragile - Chaos Engineering Primer DevSecCon
From resilient to antifragile - Chaos Engineering Primer DevSecCon
Sergiu Bodiu3K vues

Similaire à The Journey of Chaos Engineering Begins with a Single Step

The Soul in The Machine - Developing for Humans par
The Soul in The Machine - Developing for HumansThe Soul in The Machine - Developing for Humans
The Soul in The Machine - Developing for HumansChristian Heilmann
1.4K vues83 diapositives
The Promise of a Cloud Native Platform (Code Freeze conf) par
The Promise of a Cloud Native Platform (Code Freeze conf)The Promise of a Cloud Native Platform (Code Freeze conf)
The Promise of a Cloud Native Platform (Code Freeze conf)bridgetkromhout
621 vues39 diapositives
39 Nofollow Insant Approval Backlinks With HIgh DA & DR 2020 par
39 Nofollow Insant Approval Backlinks With HIgh DA & DR 202039 Nofollow Insant Approval Backlinks With HIgh DA & DR 2020
39 Nofollow Insant Approval Backlinks With HIgh DA & DR 2020EmmaRose49
15 vues2 diapositives
AMKSS Career Conference 2018: Software Engineering par
AMKSS Career Conference 2018: Software EngineeringAMKSS Career Conference 2018: Software Engineering
AMKSS Career Conference 2018: Software EngineeringMelvin Zhang
105 vues35 diapositives
2010 And Beyond par
2010 And Beyond2010 And Beyond
2010 And BeyondJeremy Johnson
23.2K vues201 diapositives
To always be shipping (SPS) par
To always be shipping (SPS)To always be shipping (SPS)
To always be shipping (SPS)bridgetkromhout
492 vues17 diapositives

Similaire à The Journey of Chaos Engineering Begins with a Single Step(20)

The Soul in The Machine - Developing for Humans par Christian Heilmann
The Soul in The Machine - Developing for HumansThe Soul in The Machine - Developing for Humans
The Soul in The Machine - Developing for Humans
Christian Heilmann1.4K vues
The Promise of a Cloud Native Platform (Code Freeze conf) par bridgetkromhout
The Promise of a Cloud Native Platform (Code Freeze conf)The Promise of a Cloud Native Platform (Code Freeze conf)
The Promise of a Cloud Native Platform (Code Freeze conf)
bridgetkromhout621 vues
39 Nofollow Insant Approval Backlinks With HIgh DA & DR 2020 par EmmaRose49
39 Nofollow Insant Approval Backlinks With HIgh DA & DR 202039 Nofollow Insant Approval Backlinks With HIgh DA & DR 2020
39 Nofollow Insant Approval Backlinks With HIgh DA & DR 2020
EmmaRose4915 vues
AMKSS Career Conference 2018: Software Engineering par Melvin Zhang
AMKSS Career Conference 2018: Software EngineeringAMKSS Career Conference 2018: Software Engineering
AMKSS Career Conference 2018: Software Engineering
Melvin Zhang105 vues
Beyond Ad-hoc Automation - To Structured Platforms par bridgetkromhout
Beyond Ad-hoc Automation - To Structured PlatformsBeyond Ad-hoc Automation - To Structured Platforms
Beyond Ad-hoc Automation - To Structured Platforms
bridgetkromhout623 vues
Beyond Ad-hoc Automation: Leveraging Structured Platforms par bridgetkromhout
Beyond Ad-hoc Automation: Leveraging Structured PlatformsBeyond Ad-hoc Automation: Leveraging Structured Platforms
Beyond Ad-hoc Automation: Leveraging Structured Platforms
bridgetkromhout813 vues
Rethinking the Public Cloud user experience par OVHcloud
Rethinking the Public Cloud user experienceRethinking the Public Cloud user experience
Rethinking the Public Cloud user experience
OVHcloud102 vues
Overboard.js - where are we going with with jsconfasia / devfestasia par Christian Heilmann
Overboard.js - where are we going with with jsconfasia / devfestasiaOverboard.js - where are we going with with jsconfasia / devfestasia
Overboard.js - where are we going with with jsconfasia / devfestasia
Christian Heilmann5.4K vues
Doing Less with More - Invisible Design and User Research par Bernadette Irizarry
Doing Less with More - Invisible Design and User ResearchDoing Less with More - Invisible Design and User Research
Doing Less with More - Invisible Design and User Research
#NoAgile - Dan Suciu par ITCamp
 #NoAgile - Dan Suciu #NoAgile - Dan Suciu
#NoAgile - Dan Suciu
ITCamp822 vues
Como criar histórias em quadrinho online par Márcio Martins
Como criar histórias em quadrinho onlineComo criar histórias em quadrinho online
Como criar histórias em quadrinho online
Márcio Martins2.2K vues
Demand Attention Now! Create Your Visual Moment of Truth par Michael Pranikoff
Demand Attention Now!  Create Your Visual Moment of TruthDemand Attention Now!  Create Your Visual Moment of Truth
Demand Attention Now! Create Your Visual Moment of Truth
Michael Pranikoff14.7K vues
QCT fantastici e dove trovarli - Crafted Software par Thomas Rossetto
QCT fantastici e dove trovarli - Crafted SoftwareQCT fantastici e dove trovarli - Crafted Software
QCT fantastici e dove trovarli - Crafted Software
Thomas Rossetto257 vues
Adapting to Input — Smashing Conference NYC par Jason Grigsby
Adapting to Input — Smashing Conference NYCAdapting to Input — Smashing Conference NYC
Adapting to Input — Smashing Conference NYC
Jason Grigsby938 vues
Redefining Contract Manufacturing Through the Use of Robotics par Lora Cecere
Redefining Contract Manufacturing Through the Use of RoboticsRedefining Contract Manufacturing Through the Use of Robotics
Redefining Contract Manufacturing Through the Use of Robotics
Lora Cecere2.7K vues

Dernier

Data Integrity for Banking and Financial Services par
Data Integrity for Banking and Financial ServicesData Integrity for Banking and Financial Services
Data Integrity for Banking and Financial ServicesPrecisely
56 vues26 diapositives
MVP and prioritization.pdf par
MVP and prioritization.pdfMVP and prioritization.pdf
MVP and prioritization.pdfrahuldharwal141
38 vues8 diapositives
Scaling Knowledge Graph Architectures with AI par
Scaling Knowledge Graph Architectures with AIScaling Knowledge Graph Architectures with AI
Scaling Knowledge Graph Architectures with AIEnterprise Knowledge
53 vues15 diapositives
PharoJS - Zürich Smalltalk Group Meetup November 2023 par
PharoJS - Zürich Smalltalk Group Meetup November 2023PharoJS - Zürich Smalltalk Group Meetup November 2023
PharoJS - Zürich Smalltalk Group Meetup November 2023Noury Bouraqadi
141 vues17 diapositives
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti... par
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...ShapeBlue
46 vues29 diapositives
Migrating VMware Infra to KVM Using CloudStack - Nicolas Vazquez - ShapeBlue par
Migrating VMware Infra to KVM Using CloudStack - Nicolas Vazquez - ShapeBlueMigrating VMware Infra to KVM Using CloudStack - Nicolas Vazquez - ShapeBlue
Migrating VMware Infra to KVM Using CloudStack - Nicolas Vazquez - ShapeBlueShapeBlue
96 vues20 diapositives

Dernier(20)

Data Integrity for Banking and Financial Services par Precisely
Data Integrity for Banking and Financial ServicesData Integrity for Banking and Financial Services
Data Integrity for Banking and Financial Services
Precisely56 vues
PharoJS - Zürich Smalltalk Group Meetup November 2023 par Noury Bouraqadi
PharoJS - Zürich Smalltalk Group Meetup November 2023PharoJS - Zürich Smalltalk Group Meetup November 2023
PharoJS - Zürich Smalltalk Group Meetup November 2023
Noury Bouraqadi141 vues
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti... par ShapeBlue
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
ShapeBlue46 vues
Migrating VMware Infra to KVM Using CloudStack - Nicolas Vazquez - ShapeBlue par ShapeBlue
Migrating VMware Infra to KVM Using CloudStack - Nicolas Vazquez - ShapeBlueMigrating VMware Infra to KVM Using CloudStack - Nicolas Vazquez - ShapeBlue
Migrating VMware Infra to KVM Using CloudStack - Nicolas Vazquez - ShapeBlue
ShapeBlue96 vues
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or... par ShapeBlue
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...
ShapeBlue88 vues
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda... par ShapeBlue
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...
ShapeBlue63 vues
DRBD Deep Dive - Philipp Reisner - LINBIT par ShapeBlue
DRBD Deep Dive - Philipp Reisner - LINBITDRBD Deep Dive - Philipp Reisner - LINBIT
DRBD Deep Dive - Philipp Reisner - LINBIT
ShapeBlue62 vues
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De... par Moses Kemibaro
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...
Moses Kemibaro29 vues
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit... par ShapeBlue
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
ShapeBlue57 vues
Why and How CloudStack at weSystems - Stephan Bienek - weSystems par ShapeBlue
Why and How CloudStack at weSystems - Stephan Bienek - weSystemsWhy and How CloudStack at weSystems - Stephan Bienek - weSystems
Why and How CloudStack at weSystems - Stephan Bienek - weSystems
ShapeBlue111 vues
Business Analyst Series 2023 - Week 4 Session 7 par DianaGray10
Business Analyst Series 2023 -  Week 4 Session 7Business Analyst Series 2023 -  Week 4 Session 7
Business Analyst Series 2023 - Week 4 Session 7
DianaGray1080 vues
Keynote Talk: Open Source is Not Dead - Charles Schulz - Vates par ShapeBlue
Keynote Talk: Open Source is Not Dead - Charles Schulz - VatesKeynote Talk: Open Source is Not Dead - Charles Schulz - Vates
Keynote Talk: Open Source is Not Dead - Charles Schulz - Vates
ShapeBlue119 vues
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R... par ShapeBlue
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...
ShapeBlue54 vues
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue par ShapeBlue
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlueCloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue
ShapeBlue46 vues
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online par ShapeBlue
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineKVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
ShapeBlue102 vues
Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P... par ShapeBlue
Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P...Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P...
Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P...
ShapeBlue82 vues

The Journey of Chaos Engineering Begins with a Single Step