The Journey of Chaos Engineering Begins with a Single Step

•

1 j'aime•3,038 vues

PagerDuty Summit 2016 Presenters: Bruce Wong, James Burns https://www.pagerduty.com/pagerduty-summit-2016/ Heard of Netflix' Chaos Engineering & the Simian Army? Google's legendary DiRT exercises? Hear about how Twilio is getting started on its journey with Chaos Engineering. This talk is the story of how Twilio got started with Chaos Engineering, lessons learned, and the impact to our engineering culture.

Technologie

#PDSummit16#PDSummit16
The Journey of Chaos Engineering
Begins with a Single Step

#PDSummit16#PDSummit16
Bruce WongSenior Engineering Manager
Twilio
@bruce_m_wong
https://www.linkedin.com/in/brucemwong

#PDSummit16#PDSummit16
2009
2012
2014
http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html
https://github.com/Netflix/SimianArmy
http://techblog.netflix.com/2015/09/chaos-engineering-upgraded.html

#PDSummit16
http://readwrite.com/2014/09/17/netflix-chaos-engineering-for-everyone/
http://techblog.netflix.com/2014/09/introducing-chaos-engineering.html

#PDSummit16#PDSummit16
https://customers.twilio.com/

#PDSummit16#PDSummit16
The journey of a thousand
miles begins with a single
step.
-Lao Tzu

#PDSummit16#PDSummit16
James BurnsTech Lead
Twilio
@1mentat
#PDSummit16
https://www.linkedin.com/in/james-burns-7816a82

#PDSummit16#PDSummit16
Preparation
Pre-Launch Log Aggregation System
-Stage env
-Synthetic Traffic

#PDSummit16
The Master of Disaster
•Network Issues
•Partitions
•Thundering Herds
•Cascading Failures
•Resource Starvation
•CPU
•Memory
•Disk IO
•Network IO
•Application Load

#PDSummit16
Round 2
•Network Issues
•Partitions
•Thundering Herds
•Cascading Failures
•Resource Starvation
•CPU
•Memory
•Disk IO
•Network IO
•Application Load

#PDSummit16
Well, that’s not what
I expected to see

#PDSummit16
Outcomes
Instrument
Instrument
Instrument
API SLAs
Architectural
Change!

#PDSummit16
Recap
• Start Simple
• Instrumentation
Gaps
• Understand your
dashboards
• Prevent outages

#PDSummit16
http://www.crisistextline.org/
http://polarisproject.org/befree-textline
http://trekmedics.org/
https://www.twilio.org/

#PDSummit16
When you wish upon a
blue moon…

#PDSummit16#PDSummit16
Please provide
feedback for this
session by filling out
the feedback survey

Contenu connexe

En vedette

The Case for ChaosBruce Wong

Chaos PatternsBruce Wong

Chaos Patterns Twilio SIGNALCONF 2016Bruce Wong

Chaos Driven DevelopmentBruce Wong

Chaos EngineeringのススメAtsushi Kojima

Principles of Chaos Engineeringh_marvin

Open Business Conference: Continuous Delivery At Netflix -- Powered by Open S...Dianne Marsh

Web Scale Applications using NeflixOSS Cloud PlatformSudhir Tonse

I Don't Test Often ...Gareth Bowles

Principles Of Chaos Engineering - Chaos Engineering HamburgNils Meder

Embracing Failure - Fault Injection and Service Resilience at NetflixJosh Evans

Release the Monkeys ! Testing in the Wild at NetflixGareth Bowles

Automated Fault Tolerance TestingAjay Kumar Vaddadi

What's New in Java 8javafxpert

Intro to Netflix's Chaos MonkeyMichael Whitehead

Scalable Microservices at Netflix. Challenges and Tools of the TradeC4Media

Scaling the Cloud - Cloud SecurityBill Burns

Mini-Training: Netflix Simian ArmyBetclic Everest Group Tech Team

AWS Re:Invent 2012 - Chaos Monkey & The Netflix Simian ArmyAriel Tseitlin

From resilient to antifragile - Chaos Engineering Primer DevSecConSergiu Bodiu

En vedette (20)

The Case for Chaos

Chaos Patterns

Chaos Patterns Twilio SIGNALCONF 2016

Chaos Driven Development

Chaos Engineeringのススメ

Principles of Chaos Engineering

Open Business Conference: Continuous Delivery At Netflix -- Powered by Open S...

Web Scale Applications using NeflixOSS Cloud Platform

I Don't Test Often ...

Principles Of Chaos Engineering - Chaos Engineering Hamburg

Embracing Failure - Fault Injection and Service Resilience at Netflix

Release the Monkeys ! Testing in the Wild at Netflix

Automated Fault Tolerance Testing

What's New in Java 8

Intro to Netflix's Chaos Monkey

Scalable Microservices at Netflix. Challenges and Tools of the Trade

Scaling the Cloud - Cloud Security

Mini-Training: Netflix Simian Army

AWS Re:Invent 2012 - Chaos Monkey & The Netflix Simian Army

From resilient to antifragile - Chaos Engineering Primer DevSecCon

Similaire à The Journey of Chaos Engineering Begins with a Single Step

The Soul in The Machine - Developing for HumansChristian Heilmann

The Promise of a Cloud Native Platform (Code Freeze conf)bridgetkromhout

39 Nofollow Insant Approval Backlinks With HIgh DA & DR 2020EmmaRose49

AMKSS Career Conference 2018: Software EngineeringMelvin Zhang

2010 And BeyondJeremy Johnson

To always be shipping (SPS)bridgetkromhout

Beyond Ad-hoc Automation - To Structured Platformsbridgetkromhout

Beyond Ad-hoc Automation: Leveraging Structured Platformsbridgetkromhout

Rethinking the Public Cloud user experienceOVHcloud

2020 06-03 cukenfest-bdd-and-sl_osAbigail Bangser

MmIT Webinar - Essential tools and technologies for the library and informati...MmIT - Multimedia Information Technology Group for CILIP

Doing Less with More - Invisible Design and User ResearchBernadette Irizarry

#NoAgile - Dan SuciuITCamp

Como criar histórias em quadrinho onlineMárcio Martins

Demand Attention Now! Create Your Visual Moment of TruthMichael Pranikoff

QCT fantastici e dove trovarli - Crafted SoftwareThomas Rossetto

Adapting to Input — Smashing Conference NYCJason Grigsby

Redefining Contract Manufacturing Through the Use of RoboticsLora Cecere

Beyond Squishy: The Principles of Adaptive DesignBrad Frost

SMAC projects - The best summer internship experience I ever had!Raul Chong

Similaire à The Journey of Chaos Engineering Begins with a Single Step (20)

The Soul in The Machine - Developing for Humans

The Promise of a Cloud Native Platform (Code Freeze conf)

39 Nofollow Insant Approval Backlinks With HIgh DA & DR 2020

AMKSS Career Conference 2018: Software Engineering

2010 And Beyond

To always be shipping (SPS)

Beyond Ad-hoc Automation - To Structured Platforms

Beyond Ad-hoc Automation: Leveraging Structured Platforms

Rethinking the Public Cloud user experience

2020 06-03 cukenfest-bdd-and-sl_os

MmIT Webinar - Essential tools and technologies for the library and informati...

Doing Less with More - Invisible Design and User Research

#NoAgile - Dan Suciu

Como criar histórias em quadrinho online

Demand Attention Now! Create Your Visual Moment of Truth

QCT fantastici e dove trovarli - Crafted Software

Adapting to Input — Smashing Conference NYC

Redefining Contract Manufacturing Through the Use of Robotics

Beyond Squishy: The Principles of Adaptive Design

SMAC projects - The best summer internship experience I ever had!

Dernier

Boost PC performance: How more available memory can improve productivityPrincipled Technologies

How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j

Histor y of HAM Radio presentation slidevu2urc

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science

2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong

How to convert PDF to text with Nanonetsnaman860154

08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls

Scaling API-first – The story of a global engineering organizationRadu Cotescu

Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC

🐬 The future of MySQL is Postgres 🐘RTylerCroy

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

Automating Google Workspace (GWS) & more with Apps Scriptwesley chun

TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo

CNv6 Instructor Chapter 6 Quality of Servicegiselly40

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia

08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls

Slack Application Development 101 Slidespraypatel2

Dernier (20)

Boost PC performance: How more available memory can improve productivity

How to Troubleshoot Apps for the Modern Connected Worker

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Histor y of HAM Radio presentation slide

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

2024: Domino Containers - The Next Step. News from the Domino Container commu...

How to convert PDF to text with Nanonets

08448380779 Call Girls In Friends Colony Women Seeking Men

Scaling API-first – The story of a global engineering organization

Breaking the Kubernetes Kill Chain: Host Path Mount

🐬 The future of MySQL is Postgres 🐘

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Automating Google Workspace (GWS) & more with Apps Script

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

CNv6 Instructor Chapter 6 Quality of Service

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

Slack Application Development 101 Slides

The Journey of Chaos Engineering Begins with a Single Step

1. #PDSummit16#PDSummit16 The Journey of Chaos Engineering Begins with a Single Step

2. #PDSummit16#PDSummit16 Bruce WongSenior Engineering Manager Twilio @bruce_m_wong https://www.linkedin.com/in/brucemwong

3. #PDSummit16#PDSummit16

4. #PDSummit16#PDSummit16 2009 2012 2014 http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html https://github.com/Netflix/SimianArmy http://techblog.netflix.com/2015/09/chaos-engineering-upgraded.html

5. #PDSummit16 http://readwrite.com/2014/09/17/netflix-chaos-engineering-for-everyone/ http://techblog.netflix.com/2014/09/introducing-chaos-engineering.html

6. #PDSummit16 https://www.twilio.com/

7. #PDSummit16#PDSummit16 https://customers.twilio.com/

8. #PDSummit16#PDSummit16 The journey of a thousand miles begins with a single step. -Lao Tzu

9. #PDSummit16#PDSummit16 James BurnsTech Lead Twilio @1mentat #PDSummit16 https://www.linkedin.com/in/james-burns-7816a82

10. #PDSummit16#PDSummit16 Preparation Pre-Launch Log Aggregation System -Stage env -Synthetic Traffic

11. #PDSummit16 The Master of Disaster •Network Issues •Partitions •Thundering Herds •Cascading Failures •Resource Starvation •CPU •Memory •Disk IO •Network IO •Application Load

12. > sudo halt #PDSummit16

13. Incident Start #PDSummit16

14. Impact? #PDSummit16

15. Post-Mortem #PDSummit16

16. #PDSummit16#PDSummit16

17. #PDSummit16 Round 2 •Network Issues •Partitions •Thundering Herds •Cascading Failures •Resource Starvation •CPU •Memory •Disk IO •Network IO •Application Load

18. > sudo halt #PDSummit16

19. Third-Party API Failure #PDSummit16

20. #PDSummit16 Well, that’s not what I expected to see

21. #PDSummit16 Outcomes Instrument Instrument Instrument API SLAs Architectural Change!

22. #PDSummit16 Recap • Start Simple • Instrumentation Gaps • Understand your dashboards • Prevent outages

23. #PDSummit16 http://www.crisistextline.org/ http://polarisproject.org/befree-textline http://trekmedics.org/ https://www.twilio.org/

24. #PDSummit16 When you wish upon a blue moon…

25. #PDSummit16#PDSummit16 Please provide feedback for this session by filling out the feedback survey