Chaos Engineering session from the FWDays Ukraine 2023

2. GEERT VAN DER CRUIJSEN
@GEERTVDC
CLOUD NATIVE ARCHITECT
#DOEPICSHIT
FULL CYCLE DEVELOPER
DEVOPS COACH

3. CHAOS ENGINEERING ?
WHY DO WE NEED

7. MAYBE YOUR APPLICATION
LANDSCAPE ISN’T *THAT*
COMPLEX

8. “IN A COMPLEX LANDSCAPE
YOUR APPLICATION IS
NEVER FULLY UP”

9. TRADITIONAL MONITORING
TOOLS ARE DEAD!

10. MEASURE
USER IMPACT

11. MEASURE
USER IMPACT RELIABILITY
AVAILABILITY LATENCY
THROUGHPUT
CORRECTNESS
FRESHNESS
COVERAGE
QUALITY
DURABILITY

12. RESILIENT APPLICATIONS
INFRASTRUCTURE
NETWORK
APPLICATION
PEOPLE

13. GRACEFUL DEGRADATION
FAIL OPEN

14. CHAOS ENGINEERING ?
WHAT IS

15. CHAOS ENGINEERING
IS NOT
RANDOMLY BREAKING
STUFF IN PRODUCTION

16. CHAOS ENGINEERING
“Chaos Engineering is the discipline of
experimenting on a distributed system
in order to build confidence in the
system’s capability to withstand
turbulent conditions in production.”
https//principlesofchaos.org

17. CHAOS ENGINEERING
“Chaos Engineering is the discipline of
experimenting on a distributed system
in order to build confidence in the
system’s capability to withstand
turbulent conditions in production.”
https//principlesofchaos.org

18. SERVICE
INPUT OUTPUT
SERVICE

19. CHAOS ENGINEERING EXPERIMENTS
HOST FAILURE
RESOURCE CAPACITY ATTACKS
APPLICATION FAILURE
NETWORK ATTACKS
BRENT ATTACK

20. CHAOS ENGINEERING
ONLY IN PRODUCTION?

21. YOUR FIRST EXPERIMENT
HOW TO START

22. GAME DAY

23. INCIDENT RESPONSE LEARNING
OUTAGE
NORMAL
DETECT &
ANALYSIS
FIX
LEARN
IMPROVE

24. CHAOS GAME DAY
CHAOS
EXPERIMENT
NORMAL
DETECT &
ANALYSIS
FIX
LEARN
IMPROVE

25. CHAOS EXPERIMENT PHASES
STEADYSTATE
DEFINE
HYPOTHESIS
DESIGN&
EXECUTE
LEARN FIX

26. STEADY STATE
STEADYSTATE
DEFINE
HYPOTHESIS
DESIGN&
EXECUTE
LEARN FIX

27. STEADY STATE
MEASURE BUSINESS METRICS
100ms extra load time drop Amazon’s sale by 1%

28. STEADY STATE
SERVICE
UNDER TEST
ROUTING SERVICE B

29. STEADY STATE
SERVICE
UNDER TEST
ROUTING SERVICE B
CONTROL
SERVICE
EXPERIMENT
SERVICE

30. STEADY STATE
SERVICE
UNDER TEST
ROUTING SERVICE B
CONTROL
SERVICE
EXPERIMENT
SERVICE
98%
1%
1%

31. ALWAYS BE ABLE TO ABORT

32. DEFINE HYPOTHESIS
STEADYSTATE
DEFINE
HYPOTHESIS
DESIGN&
EXECUTE
LEARN FIX

33. DEFINE HYPOTHESIS
BRAINSTORM WHAT CAN GO WRONG
BRING EVERYONE
DEVELOPERS
SRE /OPERATIONS
NETWORKS
BUSINESS
INFRASTRUCTURE
TESTERS
WHAT CAN GO WRONG?
WHAT IFDATABASE ISDOWN?
WHAT IFSERVICE RESPONDSSLOWER?
WHAT IFMY CACHE RESPONDSSLOW?
WHAT IFA PODDIES?
WHAT IFLOADBALANCER STOPS?
WHAT IF….?

34. STOP IF YOU KNOW THE
EXPERIMENT WILL BREAK

35. DESIGN & EXECUTE EXPERIMENT
STEADYSTATE
DEFINE
HYPOTHESIS
DESIGN&
EXECUTE
LEARN FIX

36. DESIGN & EXECUTE EXPERIMENT
START SMALL
NOTIFY PEOPLE INVOLVED
SLOWLY INCREASE BLAST RADIUS
TOOLS:
AZURE.MICROSOFT.COM/SERVICES/CHAOS-STUDIO
GREMLIN.COM
CHAOSTOOLKIT.ORG
GITHUB.COM/NETFLIX/SIMIANARMY
GITHUB.COM/ASOBTI/KUBE-MONKEY
Azure Chaos Studio

37. AZURE CHAOS STUDIO DEMO

38. LEARN
STEADYSTATE
DEFINE
HYPOTHESIS
DESIGN&
EXECUTE
LEARN FIX

39. LEARN
HOW FAST DID WE RECOVER?
HOW FAST DID WE DETECT?
DO NOT BLAME!

40. FIX
STEADYSTATE
DEFINE
HYPOTHESIS
DESIGN&
EXECUTE
LEARN FIX

41. FIX
IMPLEMENT FIX
RERUN EXPERIMENT

42. PATTERNS
RESILIENT ARCHITECTURE
MULTI PARALELLISM
ASYNC COMMUNICATION
OUTBOX PATTERN
QUEUE BASED LOAD LEVELING
IDEMPOTENCY
BULKHEAD
CIRCUIT BREAKERS

43. WRAP UP
BIG CULTURE CHANGE
FULL CYCLE DEVELOPERS
PRODUCTION ACCESS
START EXPERIMENTING
STARTSMALL CHECKOUT TOOLS
OBSERVABILITY

44. “CHAOS ENGINEERING DOESN’T CAUSE
PROBLEMS, IT JUST REVEALS THEM”
NORA JONES– CHAOSENGINEERINGLEADSLACK

45. GEERT VAN DER CRUIJSEN
@GEERTVDC
THANK YOU!
ALL PICTURES USED ARE FROM UNSPLASHED.COM
RESOURCES
BOOKS:
Chaosengineering-O’Reilly
Chaosengineeringobservability-O’Reilly
RESOURCES:
principlesofchaos.org
github.com/dastergon/awesome-chaos-engineering
docs.microsoft.com/en-us/azure/architecture/patterns/category/resiliency