Recommandé
Recommandé
Contenu connexe
Similaire à Observability - From fire fighting to smoke detection
Similaire à Observability - From fire fighting to smoke detection (20)
Dernier
Dernier (20)
Observability - From fire fighting to smoke detection
- 1. THE IMPORTANCE OF OBSERVABILITY From Fire Fighting to Smoke Detection @ShaneCarroll84
- 7. OBSERVABILITY IS LIKE A FITNESS TRACKER, BUT FOR YOUR SYSTEM @ShaneCarroll84
- 10. Track content opens, clicks, views, interactions… Get this data and send it on for other teams. MACGYVER’S JOB? @ShaneCarroll84
- 11. CHALLENGES Considerable amount of old enterprise environments… Ancient systems that are not easily testable. @ShaneCarroll84
- 12. IMPACT OUR TEAM? HOW DID THE LACK OF OBSERVABILITY @ShaneCarroll84
- 13. IT ALL STARTED WITH A BUG… Missing a large number of tracking events for one customer… 🤔 A customer is telling us we’ve a problem with our system. @ShaneCarroll84
- 14. WE WERE IN FIRE FIGHTING MODE Scrambling to try find the cause of the issue… Don’t know the full impact, causing stress for the team. @ShaneCarroll84
- 15. @ShaneCarroll84
- 16. LET’S LOOK AT THE LOGS… Logs were noisy and not easily searchable… Not all useful information to help isolate the issue was logged. @ShaneCarroll84
- 17. MACGYVER, WE HAVE A PROBLEM Issue with titles containing special characters… 🤦 Lots of customers impacted and over a million tracking events lost. @ShaneCarroll84
- 18. WHY DID WE NOT GET AN ALERT? IT WAS LOST IN A SEA OF ALERTS. @ShaneCarroll84
- 19. OUR PROCESS WAS BROKEN! Took weeks to gather missing data and reprocess events… Problem continued to impact customers and other teams. @ShaneCarroll84
- 20. WE DIDN’T KNOW OUR SYSTEM’S HEALTH What can we learn from this? Use a bad experience such as a bug to learn from and spark change. @ShaneCarroll84
- 21. WHEN YOU LOOK AT YOUR CURRENT SYSTEM, HOW DO YOU KNOW IT’S HEALTHY? @ShaneCarroll84
- 22. TESTING AT POPPULO Rob Meaney, Head of Testing CODS model 10 P’s of Testability @ShaneCarroll84
- 23. LEARNING REVIEW Detection Impact Isolation Fix & Retest Repair Minimise Impact Prevention @ShaneCarroll84
- 26. THREE AMIGOS George Dinwiddie first came up with this strategy in his blog. The Three Amigos – Product, Developer, and Tester – discuss the new feature @ShaneCarroll84
- 27. THREE AMIGOS Allow for discussion on risk before beginning work on a feature… We now ‘Three Amigo’ each new story before beginning any new development. @ShaneCarroll84
- 28. REFINING ALERTS Reduce noise and only alert on what is important to the team. Entire team takes responsibility for investigating and fixing alerts. @ShaneCarroll84
- 29. DAILY STAND-UP We changed our standup, to include a new question… Small change but now it’s part of our team process! Any new alerts today? @ShaneCarroll84
- 30. IMPROVED LOGGING Removed noise. Logged everything that helped isolate potential issues. Used structured logging. @ShaneCarroll84
- 34. DASHBOARDS Identified critical metrics Highlight failures Show trends Added important tests to the dashboard @ShaneCarroll84
- 39. WATCHING TV AT WORK Why invest time and effort into monitoring tools if no one looks at them? Dashboards are now in constant view! @ShaneCarroll84
- 42. BUT BE CAREFUL Don’t introduce a dependency in your system when adding monitoring tools! @ShaneCarroll84
- 52. SMOKE DETECTION MODE ACTIVATED Risk is discussed early. Alerts are raised and actioned. Dashboards show our critical metrics. @ShaneCarroll84
- 53. SMOKE DETECTION MODE ACTIVATED Investigating logs and isolating issues is easier. Quickly see who's impacted. Replay scripts are available. @ShaneCarroll84
- 54. HOW DO YOU GO FROM FIRE FIGHTING TO SMOKE DETECTION? LEARN FROM THE FIRE! @ShaneCarroll84