The document discusses the importance of observability at a cloud provider like DigitalOcean. It provides examples of how DigitalOcean uses logging, metrics, and tracing to monitor services pre and post release. Specifically, it details two case studies where tracing helped troubleshoot latency issues during load testing of a virtual private cloud service and how log-based alerts helped find network bugs in a DHCP service. The key takeaways are that observability is necessary before and after releasing services, and the pillars of observability - metrics, logging, and tracing - should be leveraged, including using the "four golden signals" and "USE" metrics framework.

2. digitalocean.com
whoami?
A software engineer
Background:
casino company, 3D printing, ed-tech
@DigitalOcean:
Delivery → Observability → Networking
Nomad

3. digitalocean.com

4. digitalocean.com
@DigitalOcean
12 data centers
1.15 million droplets
300+ microservices

5. digitalocean.com
We need observability.

6. digitalocean.com
300+ microservices

7. digitalocean.com
Teams are on-call for live, production-ready services.

8. digitalocean.com
We need to test new services.

9. digitalocean.com
Observability at a Cloud Provider
Monitoring for Pre and Post Release

10. digitalocean.com
What is observability?

11. digitalocean.com
a measure of how well internal states of a
system can be inferred from knowledge
of its external output

12. digitalocean.com
pillars of observability
metrics logging tracing

13. digitalocean.com
Logging
Record of discrete events over time

14. digitalocean.com
Tracing
Propagating metadata through
different requests, threads, and
processes

15. digitalocean.com
Metrics
Time series of sampled data

16. digitalocean.com
4 golden signals

17. digitalocean.com
latency: time to service a request
traffic: requests/second
error: error rate of requests
saturation: fullness of a service

18. digitalocean.com
USE metrics

19. digitalocean.com
Utilization
Saturation
Error rate

20. digitalocean.com
What about
alerting?

21. digitalocean.com
May the queries flow,
and the pager stay silent.

22. digitalocean.com
Metrics-based alerts
[state]
Is my service up and/or scrapeable?
absent(up{kubernetes_name="doccserver
"}) or
sum(up{kubernetes_name="doccserver"})
== 0
Do I have the # of loadbalancers I
expect?
sum(up{kubernetes_name="loadbalancer"
}) < 3

23. digitalocean.com
Metrics-based alerts
[threshold]
Is our loadbalancer at 50% capacity in
terms of sessions?
max(haproxy_frontend_current_sessions
/ haproxy_frontend_limit_sessions) BY
(kubernetes_node_name, frontend) *
100 > 50
Are 50 percent of tests taking longer
than 10 minutes?
max(test_duration_seconds{quantile="0
.5",result="pass"}) BY (test_name) >
600

24. digitalocean.com
Log-based alerts
[threshold]

25. digitalocean.com
Teams send their page-worthy alerts to their
on-call rotation and their important but
subcritical alerts to their ticket queues. All
other alerts should be retained as
informational data for status dashboards.
- Google SRE Book

26. digitalocean.com
What do we use at
DigitalOcean?

27. digitalocean.com

28. digitalocean.com
tracecollectors

29. digitalocean.com

30. digitalocean.com
Case Studies

31. VPC: Virtual Private Cloud

32. digitalocean.com
Service A
Service B
Service C
Queue
Datastore
RPC calls
Service B
Service B
Service C
Service C

33. digitalocean.com
VPC: load testing
load tester repeatedly
makes various RPC calls

34. digitalocean.com
VPC: load testing
as load testing continued,
started to notice latency in
different rpc calls

35. digitalocean.com
VPC: load testing
use tracing to take a look
at the /SyncInitialChassis
call

36. digitalocean.com
VPC: load testing
“The fix was to be smarter
and do the queries more
efficiently. The repetitive
loop of queries to rnsdb
really stood out in the
lightstep data.”
- Bob Salmi

37. digitalocean.com
VPC: load testing
can queue be replaced
with simple
request-response
system?

38. digitalocean.com
VPC: chaos testing
Check the recoverability
and failover of multiple
components in the stack
Service A
Service B
Service C
QueueDatastore
Service B
Service B
Service C
Service C

39. digitalocean.com
VPC: configure alerts
Based on the load and chaos
test, configure alerts for when
the service is shipped to
production

40. DHCP:
Dynamic Host Configuration Protocol

41. digitalocean.com
Service B
DHCP server
Virtual
Switch
Service A RPC calls
Iface Y
DHCPv4
gRPC
DHCPv6
Iface X dropletX
DHCP traffic
RPC calls
Hypervisor

42. digitalocean.com
DHCP: production alerts
The DHCP rate-limiter blacklists
abusive clients and logs to
syslog. Elastalert sends alerts to
slack or pagerduty.
Rate Limiter
Centralized
LoggingCentralized
LoggingCentralized
LoggingCentralized
Logging
Elastalert
emits log line

43. digitalocean.com
DHCP: production alerts
DHCP log-based alerting
allowed us to find unexpected
network flow bugs.

44. digitalocean.com
Conclusion:
● Observability is necessary pre and
post release!
● Use logging, tracing, and metrics
● Leverage the 4 golden signals and
USE metrics

45. Thank you!