Are you looking at your organization's efforts to enter or expand into the cloud native landscape and feeling a bit daunted by the vast expanse of information surrounding cloud native observability? When you're moving so fast with agile practices across your DevOps, SRE's, and platform engineering teams, it's no wonder this can seem a bit confusing. Unfortunately, the choices being made have a great impact on both your business, your budgets, and the ultimate success of your cloud native initiatives. That hasty decision up front leads to big headaches very quickly down the road. In this talk, I'll introduce the problem facing everyone with cloud native observability followed by 3 common mistakes that I'm seeing organizations make and how you can avoid them!

1. chronosphere.io
chronosphere.io
3 Pitfalls
Eric D. Schabell
Director Evangelism
@ericschabell{@fosstodon.org}
10 Jan 2024
Everyone Should Avoid with Cloud Native Observability
Cloud Native London

2. chronosphere.io
Cloud Native Observability

3. chronosphere.io
Cloud Native

4. chronosphere.io
Data volume
Experiment:
- Hello World app on 4 node
Kubernetes cluster with
Tracing, End User Metrics
(EUM), Logs, Metrics
(containers / nodes)
- 30 days == +450 GB

5. chronosphere.io
Retention
Retention
Retention
Retention
Retention
Retention
Retention

6. chronosphere.io
Cloud Native at Scale

7. chronosphere.io
Observability…

8. chronosphere.io
Cloud Native Observability
at Scale

9. chronosphere.io
O11y at Scale (need)

10. chronosphere.io
1. Ignoring costs in application landscape
2. Focusing on The Pillars
3. Sneaky sprawling tooling mess
4. Controlling costs
5. Losing your way in the protocol jungles
6. Underestimating cardinality
(Click on a pitfall to jump to that section)
Picking Your Pitfalls

11. chronosphere.io
1. Ignoring costs in
application landscape

12. chronosphere.io

13. chronosphere.io
Prometheus for metrics, alerting, queries

14. chronosphere.io
Prometheus auto discovery

15. chronosphere.io
Manual instrumentation (java client lib)

16. chronosphere.io
Short link:
bit.ly/prom-workshop

17. chronosphere.io
Applications (Java)
OTel Auto Instrumentation (libraries)
OTel API
OTel SDK
OTel Collector
OTLP
OTLP
OTLP
OpenTelemetry (Auto) instrumentation

18. chronosphere.io
Host
Observability Backend
(Prometheus, Jaeger, Fluent Bit, etc.),
Applications
OTel Auto Instrumentation
OTel API
OTel SDK
OTel Collector Agent
OTLP
OTLP
OTLP
OTLP
OTLP
OpenTelemetry Collector (agent)

19. chronosphere.io
Host
Host
Host
Observability Backend
(Prometheus, Jaeger, Fluent Bit, etc.),
Applications
OTel Auto Instrumentation
OTel API
OTel SDK
OTel Collector Agent
OTLP
OTLP
OTLP
OTLP
Collector (gateway)
OTel Collector Gateway

20. chronosphere.io
Short link:
bit.ly/opentelemetry-workshop

21. chronosphere.io
1. Ignoring costs in application landscape
2. Focusing on The Pillars
3. Sneaky sprawling tooling mess
4. Controlling costs
5. Losing your way in the protocol jungles
6. Underestimating cardinality
(Click on a pitfall to jump to that section, or jump to end)
Picking Your Next Pitfall

22. chronosphere.io
2. Focusing on The Pillars

23. chronosphere.io
Pillars Phases

24. chronosphere.io
Developer
Technology
Bottom up

25. chronosphere.io
Pillar problems…

26. chronosphere.io
Car is on fire…

27. chronosphere.io
Better outcomes…
Faster remediation…
Easier detection…
Happier customers…

28. chronosphere.io
Phase 1
Know something is
happening as fast
as possible…

29. chronosphere.io
Phase 2
Triage with specific
information…

30. chronosphere.io
Phase 3
Understand to
ensure never
happens again…

31. chronosphere.io

32. chronosphere.io

33. chronosphere.io
1. Ignoring costs in application landscape
2. Focusing on The Pillars
3. Sneaky sprawling tooling mess
4. Controlling costs
5. Losing your way in the protocol jungles
6. Underestimating cardinality
(Click on a pitfall to jump to that section, or jump to end)
Picking Your Next Pitfall

34. chronosphere.io
3. Sneaky sprawling tooling mess

35. chronosphere.io
Over 66% of organizations
use more than 10 different
observability tools
– ESG report over exploding data volumes

36. chronosphere.io

37. chronosphere.io
Know
Triage
Understand

38. chronosphere.io

39. chronosphere.io
1. Ignoring costs in application landscape
2. Focusing on The Pillars
3. Sneaky sprawling tooling mess
4. Controlling costs
5. Losing your way in the protocol jungles
6. Underestimating cardinality
(Click on a pitfall to jump to that section, or jump to end)
Picking Your Next Pitfall

40. chronosphere.io
4. Controlling costs

41. chronosphere.io
“It’s remarkable how common
this situation is, where an
organization is paying more for
their observability data, than
they do for their production
infrastructure.”
?

42. chronosphere.io
O11y data storage costs
are broken.
Keeping everything
model?

43. chronosphere.io
Know the cost of
observability
metrics data?

44. chronosphere.io
Traces
Metrics
Events
CHRONOSPHERE LENS
CONTROL PLANE
Developer-centric observability
Real-time insight and transformation
of observability data
PURPOSE-BUILT DATA
STORES PER TELEMETRY TYPE
Single tenant architecture, proven
industrial reliability and scalability

45. chronosphere.io

46. chronosphere.io
1. Ignoring costs in application landscape
2. Focusing on The Pillars
3. Sneaky sprawling tooling mess
4. Controlling costs
5. Losing your way in the protocol jungles
6. Underestimating cardinality
(Click on a pitfall to jump to that section, or jump to end)
Picking Your Next Pitfall

47. chronosphere.io
5. Losing your way in the
protocol jungles

48. chronosphere.io
Without open standards,
you’ll not find a way back…

49. chronosphere.io

50. chronosphere.io
Host
Observability Backend
(Prometheus, Jaeger, Fluent Bit, etc.),
Applications
OTel Auto Instrumentation
OTel API
OTel SDK
OTel Collector Agent
OTLP
OTLP
OTLP
OTLP
OTLP
OpenTelemetry Collector (agent)

51. chronosphere.io
Prometheus for metrics, alerting, queries

52. chronosphere.io
1. Ignoring costs in application landscape
2. Focusing on The Pillars
3. Sneaky sprawling tooling mess
4. Controlling costs
5. Losing your way in the protocol jungles
6. Underestimating cardinality
(Click on a pitfall to jump to that section, or jump to end)
Picking Your Next Pitfall

53. chronosphere.io
6. Underestimating cardinality

54. chronosphere.io
The struggle is real
“I don't yet collect spans/traces because I can hardly get our devs to care about basic metrics, let alone
traces.”
“This is a large enterprise with approx. 1000 developers. Cultivating a culture of engineering that cares
about availability is a challenge that we need to solve alongside any technical implementations.”

55. chronosphere.io
10 hours
on average, per week,
trying to triage and
understand incidents -
a quarter of a
40 hour work week

56. chronosphere.io
33%
said those issues
disrupted their
personal life
39%
admitting they are
frequently
stressed out

57. chronosphere.io
Cloud Native
Observability
at Scale

58. chronosphere.io
Traces
Metrics
Events
CHRONOSPHERE LENS
CONTROL PLANE
Developer-centric observability
Real-time insight and transformation
of observability data
PURPOSE-BUILT DATA
STORES PER TELEMETRY TYPE
Single tenant architecture, proven
industrial reliability and scalability

59. chronosphere.io
1. Ignoring costs in application landscape
2. Focusing on The Pillars
3. Sneaky sprawling tooling mess
4. Controlling costs
5. Losing your way in the protocol jungles
6. Underestimating cardinality
(Click on a pitfall to jump to that section, or jump to end)
Picking Your Next Pitfall

60. chronosphere.io
What should be
the #1 item on
cloud wishlist?
What should be #1 item on your
cloud native observability
wishlist?

61. chronosphere.io
chronosphere.io
Questions?
Eric D. Schabell
Director Evangelism
@ericschabell{@fosstodon.org}