Low Overhead System Tracing with eBPF

1. Low-Overhead System Tracing With eBPF May 2018 Akshay Kapoor DevOps Engineer @ SAP Labs

2. Low-Overhead System Tracing With eBPF

5. You don't need to know how to operate an X-ray machine, but you do need to know that if you swallow a penny, an X-ray is an option! ~ www.bredangregg.com

6. ----------------------------------------------------------------------- CLASSIC PKT. FILTERING BPF VIRTUAL MACHINE EXTENDED BPF BPF SYSTEM CALL ADDTL. PROBES 1993 Before 1992 2013 2014 Today EVOLUTION OF BPF STACK BASED KERNEL -> USPACE COPIES REGISTER BASED (2) LESSER COPIES IMPROVED ISA & eBPF MAPS MORE REGISTERS (10) EXPOSED TO USER SPACE KERNEL FILTERS UPROBES, KPROBES USDT, TRACEPOINTS

7. tcpdump -n "dst host 192.168.1.1 and dst port 23"

8. # Credits : https://suchakra.wordpress.com/ HOW BPF WORKS ?

9. BCC (BPF COMPILER COLLECTION) • https://github.com/iovisor/bcc • Lead Developer – Brenden Blanco #Credits : Sasha Goldshtein

10. BCC Tools [examples…]

14. Flamegraphs [ BCC/BPF Visualizations ]

15. Flamegraphs [ BCC/BPF Visualizations] (Source : https://blog.cloudflare.com/tracing-system-cpu-on-debian-stretch/) CallStacks No. of Samples

16. BPF and Containers

17. • Namespaces BPF and containers… PID Y PID X CONTAINER HOST Restricts Visibility • mnt • pid • net • . . .

18. • Namespaces • CGroups BPF and containers… CONTAINER 1 HOST Restricts Quota/Usage • cpu • mem • blkio • . . . CONTAINER 2 CPU SHARES

19. • Namespaces • CGroups • Analysis from the host • PID Mappings (/sys/fs/cgroup/docker/*) • Symbol file locations BPF and containers… 0x7f82b510ddda 0x7f82b510999d 0x7f82b510f665 0x7f82b510t546

20. • Namespaces • CGroups • Analysis from the host • PID Mappings (/sys/fs/cgroup*) • Symbol file locations • Deployment Methodologies BPF and containers… APP CONTAINER 1 APP CONTAINER 2 BCC TOOLS ON HOST KERNEL EVENTS

21. • Namespaces • CGroups • Analysis from the host • PID Mappings (/sys/fs/cgroup*) • Symbol file locations • Deployment Methodologies BPF and containers… APP CONTAINER 1 APP CONTAINER 2 KERNEL EVENTS BCC CONTAINER 3

22. DEMO OBSERVABILITYNETWORKING SECURITY

23. BPF Implementations… - Seccomp • Control system calls made by a process - Cilium • Controls Networking, Security and Load Balancing for containers - Weavescope • Observability into containerized application stacks like Docker and Kubernetes - Iptables • Bpfilter implementations to optimize ingress/outgress security rules - Systemtap • BPF backend for optimizations

24. References Sasha Goldshtein (goldshtn) Brendan Gregg (brendangregg) Suchakra (tuxology) Julia Evans (b0rk) @Follow https://github.com/iovisor/bcc http://man7.org/linux/man-pages/man2/bpf.2.html http://brendangregg.com/ebpf.html https://github.com/goldshtn/linux-tracing-workshop https://suchakra.wordpress.com/ - eBPF https://blog.yadutaf.fr/ - Networking & eBPF https://jvns.ca/blog/2017/07/05/linux-tracing-systems/ https://www.youtube.com/watch?v=aaTQM7wcmfk – Kernel Meetup | eBPF https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/ https://blog.yadutaf.fr/2016/03/30/turn-any-syscall-into-event-introducing-ebpf-kernel-probes/ https://lwn.net/Articles/740157/ - Thorough eBPF intro https://developers.redhat.com/blog/2017/12/13/introducing-stapbpf-systemtaps-new-bpf-backend/ https://lwn.net/Articles/747551/ - BPF comes to firewalls

25. Thank You ! akshay.kapoor@sap.com akskap akskap akskap