This document provides an overview of debugging distributed systems. It begins with definitions of distributed systems and why they are difficult to debug due to factors like concurrency, lack of a global clock, and independent failures. It then outlines a structured 8 step approach to debugging distributed systems: 1) observe and document issues, 2) create a minimal reproducer, 3) debug client side, 4) check DNS and routing, 5) check connections, 6) inspect traffic and messages, 7) debug server side, and 8) wrap up with a post mortem. The document concludes with examples of distributed systems war stories and questions.

1. bertjan@openvalue.eu
Debugging distributed systems
Bert Jan Schrijver
@bjschrijver

2. bertjan@openvalue.eu
Debugging distributed systems
Bert Jan Schrijver
@bjschrijver
Networking 101
How the internet works

3. Why?

4. Bert Jan Schrijver
L e t ’ s m e e t
@bjschrijver

5. Why are distributed
systems difficult?
Networking 101
What?
Why?
✅
Demo
War stories
Conclusion
W h a t ‘ s n e x t ?
Outline
A structured approach
@bjschrijver

6. • Level 1: Non-concurrency
• Level 2: Concurrency
• Level 3: Distribution
The hierarchy of complexity for debugging
Source: https://maximilianmichels.com/2020/debugging-distributed-systems

7. What is a distributed system?

8. A distributed system is a system whose
components are located on different
networked computers, which
communicate and coordinate their actions
by passing messages to one another.

9. • Concurrency of components
• Lack of a global clock
• Independent failure of components
• Distributed systems are harder to reason
about
Characteristics of distributed systems
Source: http://www.nasa.gov/images/content/218652main_STOCC_FS_img_lg.jpg

10. Working with distributed systems is
fundamentally different from writing
software on a single computer - and the
main difference is that there are lots of new
and exciting ways for things to go wrong.
- Martin Kleppmann
“
”
Photo: Dave Lehl

11. Why do things go wrong?
“ ”
Photo: Dave Lehl

12. The fallacies of distributed computing
are a set of assertions made by L Peter
Deutsch and others at Sun Microsystems
describing false assumptions that
programmers new to distributed
applications invariably make.

13. 1. The network is reliable;
2. Latency is zero;
3. Bandwidth is infinite;
4. The network is secure;
5. Topology doesn't change;
6. There is one administrator;
7. Transport cost is zero;
8. The network is homogeneous.
Fallacies of distributed computing

14. What could possibly go wrong?
“ ”
Photo: Dave Lehl

15. OSI & TCP/IP
Source: https://www.guru99.com/difference-tcp-ip-vs-osi-model.html

16. .. in your browser’s address bar and press Enter
What happens when you type google.com…
Source: https://github.com/alex/what-happens-when

17. 17

18. In a distributed system, there may well be
some parts of the system that are broken in
some unpredictably way, even though other
parts of the system are working fine…
“
”
Source: Martin Kleppmann

19. “
”
Source: Martin Kleppmann
… but in a system with thousands of
nodes, it is reasonable to assume that
something is always broken.
- Martin Kleppmann

20. Are we done yet?

21. Source: https://7216-presscdn-0-76-pagely.netdna-ssl.com/wp-content/uploads/2011/12/confused-man-single-good-men.jpg
Where do I start?

22. A structured approach
to debugging distributed systems
@bjschrijver
Check DNS & routing
Check connection
Debug client side
Create minimal reproducer
Debug server side
Observe & document
Wrap up & post mortem
Inspect traffic / messages

23. Step 1: Observe & document
• What do you know about the problem?
• Inspect logging, errors, metrics, tracing
• Draw the path from source to target - what’s
in between? Focus on details!
• Document what you know
• Can we reproduce in a test?
• By injecting errors, for example

24. Step 1: Observe & document

25. Step 2: Create minimal reproducer
• Goal: maximise the amount of debugging
cycles
• Focus on short development iterations /
feedback loops
• Get close to the action!

26. Step 3: Debug client side
• Focus on eliminating anything that could be
wrong on the client side
• Are we connecting to the right host?
• Do we send the right message?
• Do we receive a response?
• Not much different from local debugging

27. Step 4: Check DNS & routing
• DNS:
• Make sure you know what IP address the
hostname should resolve to
• Verify that this actually happens at the client
• Routing:
• Verify you can reach the target machine

28. Step 5: Check connection
• Can we connect to the port?
• If not, do we get a REJECT or a DROP?
• Does the connection open and stay open?
• Are we talking TLS?
• What is the connection speed between us?

29. Step 6: Inspect traffic / messages
• Do we send the right request?
• Do we receive the right response?
• How do we know?
• How do we handle TLS?
• Are there any load balancers or proxies in
between?

30. Step 7: Debug server side
• Inspect the remote host
• Can we attach a remote debugger?
• See https://youtube.com/OpenValue
• Profiling
• Strace

31. Step 8: Wrap up & post mortem
• Document the issue:
• Timeline
• What did we see?
• Why did it happen?
• What was the impact?
• How did we find out?
• What did we do to mitigate and fix?
• What should we do to prevent repetition?

32. If you really want a reliable system, you
have to understand what its failure modes
are. You have to actually have witnessed
it misbehaving.
- Jason Cahoon
“
”

33. Distributed systems war stories

34. The time where it worked half of the time…

35. The one where two services didn’t speak
the same language…

36. The one with expensive logging…

37. The one at a school…

39. The one where only one country was
affected…

40. The one where breaking news broke
something else…

41. Summary: a structured approach
to debugging distributed systems
@bjschrijver
Check DNS & routing
Check connection
Debug client side
Create minimal reproducer
Debug server side
Observe & document
Wrap up & post mortem
Inspect traffic / messages

42. Source: https://cdn2.vox-cdn.com/thumbor/J9OqPYS7FgI9fjGhnF7AFh8foVY=/148x0:1768x1080/1280x854/cdn0.vox-cdn.com/uploads/chorus_image/image/46147742/cute-success-kid-1920x1080.0.0.jpg
THAT’S IT.
NOW GO KICK SOME ASS!

43. Questions?
@bjschrijver

44. Thanks for your time.
Got feedback? Tweet it!
All pictures belong
to their respective
authors
@bjschrijver