Using Sysdig Tracers, a lightweight open source APM like syscall tracing

1. Jorge Salamero Sanz
@bencerillo
Trace Everything
When APM meets SysAdmins

2. How Linux SysAdmins (used to)
do (performance) troubleshooting?
• uptime
• free -m
• ps aux
• vmstat 1
• netstat -putan
• top
• tcpdump

3. But something changed

4. But something changed

5. How developers are doing (distributed)
tracing and profiling?
• OpenTracing and alternatives:
• Zipkin, Dapper, HTrace, X-Trace
• Commercial:
• New Relic, AppDynamics, Dyn, SPM
• Self-brewed and hacks:
• statsd, JMX
• print
• logs can bite you in the ass

6. What if we had something...
• Open source
• Simple and easy to use (trade-off vs eBPF/bcc)
• Lightweight
• Could work everywhere (including containers)

7. Sysdig
Open Source system troubleshooting
with native container support
(htop, vmstat, netstat, lsof, tcpdump…)
Monitoring, alerting,
troubleshooting tool for
Docker, Kubernetes, Mesos,
RancherOS, GCE, ECS

8. Sysdig

9. Sysdig

10. Sysdig

11. Sysdig

12. Sysdig
• Capture system events, filter and run scripts
• Trace dumps for analysis
• Container support
• CLI or curses interface

13. Event stream
Open
Read
Close
Connect
Read
Write
Read
Read
Write
Close
Dump to Disk
Filter
Analyze

14. Sysdig Tracers
Sysdig Tracers:
System call tracing

15. Sysdig Tracers: System call tracing
• Inject markers inside Sysdig event stream
• Mark being and end of spans
• Function calls
• Network request
• Arbitrary piece of code
• From any language (write to /dev/null)
• Low overhead (<1us)

16. Event stream
Open
Read
Close
Connect
Read
Write
Read
Read
Write
Close

17. Event stream
Open
Read
Close
Connect
Read
Write
Read
Read
Write
Close
> Request
< Request

18. Event stream
Open
Read
Close
Connect
Read
Write
Read
Read
Write
Close
> Request
< Request
Enter Tracer
Exit Tracer

19. Event stream
Open
Read
Close
Connect
Read
Write
Read
Read
Write
Close
> Request
< Request
Span

20. Event stream
Open
Read
Close
Connect
Read
Write
Read
Read
Write
Close
> Request
< Request
> Request.Download
< Request.Download
Parent
Span
Child
Span

21. Format
Simple format:
<dir>:<ID>:<tag1>.<tag2>...:<arg1>=<val1>,<arg2>=<val2>...
JSON Format:
["<direction>", <ID>, ["tag1", "tag2"...],
[{"arg1":"val1"}, {"arg2":"val2"}...]]

22. Example
while :
do
# Start a trace named 'website-latency'
echo ">::website-latency::" > /dev/null
# Download the sysdig home page
curl -s http://sysdig.org > /dev/null
# End the the trace
echo "<::website-latency::" > /dev/null
done

23. Fun things you can do
● Measure latencies in your code
● Save and filter traces with sysdig
● Analyze traces with csysdig
● Inspect system activity inside execution spans
● Trace-aware Log Monitoring
● Export trace latencies using statsd

24. Demo time!

25. Further reading
● Documentation:
○ https://github.com/draios/sysdig/wiki/Tracers
○ https://sysdig.com/blog/tracking-down-application-bottlene
cks-with-tracers/
● Integrations
○ Python https://github.com/draios/tracer-py
○ Node: https://https://github.com/tj/node-trace
○ Go: https://github.com/tj/go-trace

26. Thank You!