Monitoring involves collecting logs, metrics and alerts to detect issues, while observability provides insight into internal system states. The presenter faced problems determining causes of performance drops. They will discuss starting with monitoring basics like logging, tracing and metrics. They will then explain how to transition to domain-oriented observability through techniques like aspect-oriented programming to better understand the system. Observability aims to answer any questions about internal states using monitoring tools.

1. From Monitoring to
Domain-Oriented
Observability
What’s the difference between monitoring and
observability and why does it matter?

2. 2. About me
● Took part in developing of microservice architecture based on
Event Sourcing and CQRS
● I have obtained a position of Tech Lead. I implemented Canary
release and Feature Toggles (aka Feature Flags), migrated
microservice from REST to Event-Driven
● Made a webinar about microservices testing.
● I am trying to apply best engineering practices to Safeguard Cyber project
● I want to build such a process in a company (team) in which it will be pleasant to work and develop
professionally. Where ideas will be heard, where a person will not need to sacrifice his family or
health for professional growth.

3. 3. Parts of presentation
● Why you should start thinking about monitoring and our case
● Theoretical minimum about monitoring
● How to switch from monitoring to Domain-oriented Observability

4. 4. Part 1
● Why you should start thinking about monitoring and our case

5. 5. About our product

6. 6. Threat Detection

7. 7. Cyber Defense

8. 8. Machine Learning and AI

9. 9. Fire on Production

10. 10. Fire on production

11. 11. Fire on production

12. 12. Fire on production

13. 12. Fire on production

14. 14. Problems we faced with
● What is the cause of performance drop?
● What can lead to poor system performance?
● How can certain changes influence the system?
● Our product is difficult to fit in SLA

15. 15. Part 2
● Theoretical minimum about monitoring

16. 16 Monitoring
1) Logging
2) Tracing
3) Metrics
4) Alerts

17. “A log is an immutable,
timestamped record of event
describing what happened over
time “
17 Logging

18. 18. Kibana filters

19. 19. Kibana filters result

20. 20. Errors frequency analysis

21. “A trace is a representation of series
of causally related distributed events
that encode the end-to-end request
flow through a distributed system”
21. Tracing

22. 22. Tracing

23. “Metrics are a numeric
representation of data
measure over intervals of
time”
23. Metrics

24. 24. Metrics

25. 25. Metrics

26. 26. Metrics

27. ● throughput
● success
● error
● performance
27. Metrics subtypes

28. ● Rate - the number of requests, per second, you services are serving.
● Errors - the number of failed requests per second.
● Duration - distributions of the amount of time each request takes.
28. Three key metrics by RED methodology

29. 31. Trends are very important.

30. 30. Trends are very important

31. Automated alerts are essential to
monitoring. They allow you to spot
problems anywhere in your
infrastructure, so that you can
rapidly identify their causes and
minimize service degradation and
disruption. Alerts draw human
attention to the particular systems
that require observation,
inspection, and intervention.
31. Alerts

32. ● There should be people’s reaction
● Alert should have priority
● There should be possibility to disable notifications
● Alert should provide further instructions
32. Alert rules

33. 33. Alerts in Kibana

34. 34. Part 3
● How to switch from monitoring to Domain-oriented Observability

35. Definition:
“In control theory, observability is a measure of how well internal states of a
system can be inferred from knowledge of its external outputs. The observability
and controllability of a system are mathematical duals.”
- Wikipedia
In English:
Can you understand what’s happening inside your code and system, simply by
asking questions using your tools? Can you answer any new question you think
of, or only the ones you prepared for?
35. Observability

36. 36. Black and White box Component View

37. 37. Observability code example

38. 38. Observability code example

39. 39. Cleanup the mess

40. 40. Cleanup the mess

41. 41. Moving code to class

42. 42. Domain Probe: DiscountInstrumentation

43. ● AOP
● DECORATOR
● λάμβδα
43. Other opportunities

44. 44. Testing + Monitoring

45. 45. Testing + Monitoring

46. Start being proactive
Don’t be firefighters

47. ● The RED Method
● Monitoring Distributed Systems
● Domain-Oriented Observability
● Distributed Systems Observability by Cindy Sridharan
● Testing in Production, the safe way
● Deploy != Release part1 and part2
● SRE: Observability: Metric Namespaces and Structures
● Observability: Metric, Logging, and Tracing
● Decorator
● Monitoring in the time of Cloud Native
● https://www.elastic.co/learn
46. Resources:

48. Questions