Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2Mb3oWl. Thomas Graf talks about a new efficient in-kernel programming language called eBPF. It allows everyone to extend existing kernel components or glue them together in new forms without requiring to change the kernel itself. Filmed at qconsf.com. Thomas Graf is co-founder & CTO at Covalent and creator of the Cilium project. Before this, he has been a Linux kernel developer at Red Hat for many years. Over the more than 15 years working on the Linux kernel, he was involved in a variety of networking and security subsystems. For the past couple of years, he has been involved in the development of BPF and XDP.

1. BPF
Turning Linux into a
Microservices-aware
Operating System

2. InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
linux-cilium-ebpf

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. About the Speaker
Thomas Graf
● Linux kernel developer for ~15 years working on
networking and security
● Helped write one of the biggest monoliths ever
● Worked on many Linux components over the years (IP,
TCP, routing, netfilter/iptables, tc, Open vSwitch, …)
● Creator of Cilium to leverage BPF in a cloud native and
microservices context
● Co-Founder & CTO of the company building Cilium
2

5. Agenda
● Evolution of running applications
○ From single task processes to microservices
● Problems of the Linux kernel
○ The kernel
● What is BPF?
○ Turning Linux into a modern, microservices-aware operating system
● Cilium - BPF-based networking security for microservices
○ What is Cilium?
○ Use Cases & Deep Dive
● Q&A
3

6. Evolution: Running applications
Split the CPU and
memory. Shared
libraries, package
management, Linux
distributions.
4
Virtualization
Microservices
Containers
Multi tasking
Ship the OS together
with application and run
it in a VM for better
resource isolation.
Virtualized hardware
and software defined
infrastructure.
Dark Age:
Single tasking
The simple age. Back to a shared
operating system.
Applications directly
interact with the host
operating system again.

7. Problems of the
Linux Kernel in the
age of microservices
5

8. Problem #1: Abstractions
6
ProcessProcess
HW
System Call Interface
IPv4
Netdevice / Drivers
Sockets
Ethernet
TCP
IPv6
Netfilter
UDP Raw
Traffic Shaping
Bridge OVS ..
The Linux kernel is split into layers to
provide strong abstractions.
Pros:
● Strong userspace API compatibility
guarantee. A 20 years old binary still
works.
● Majority of Linux source code is not
hardware specific.
Cons:
● Every layer pays the cost of the
layers above and below.
● Very hard to bypass layers.

9. Problem #2: Per subsystem APIs
7
ProcessProcess
HW
System Call Interface
IPv4
Netdevice / Drivers
Sockets
Ethernet
TCP
IPv6
Netfilter
UDP Raw
Traffic Shaping
Bridge OVS
iptablesseccomp tcethtool
..
ip
brctl /
ovsctl
tcpdump

10. 8
Problem #3: Development Process
The Good:
● Open and transparent process
● Excellent code quality
● Stability
● Available everywhere
● Almost entirely vendor neutral
The Bad:
● Hard to change
● Shouting is involved (getting better)
● Large and complicated codebase
● Upstreaming code is hard, consensus has to
be found.
● Upstreaming is time consuming
● Depending on the Linux distribution,
merged code can take years to become
generally available
● Everybody maintains forks with 100-1000s
backports

11. 9
Problem #4: What is a container?
What the kernel knows about:
● Processes & thread groups
● Cgroups
○ Limits and accounting of CPU,
memory, network, … Configured by
container runtime.
● Namespaces
○ Isolation of process, CPU, mount,
user, network, IPC, cgroup, UTS
(hostname). Configured by container
○ runtime
● IP addresses & port numbers
○ Configured by container networking
● System calls made & SELinux context
○ Optionally configured by container
runtime
What the kernel does not know:
● Containers or Kubernetes pods
○ There is no container ID in the kernel
● Exposure requirements
○ The kernel no longer knows whether
an application should be exposed
outside of the host or not.
● API calls made between containers/pods
○ Awareness stops at layer 4 (ports).
While SELinux can control IPC, it can’t
control service to service API calls.
● Servicemesh, huh?

12. What now? Alternatives?
Linus was wrong. The
app should provide its
own OS.
10
Move OS to
Userspace
Rewrite
Everything?Unikernel
We don’t need kernel
mode for most of the
logic. Build on top of a
minimal Linux.
Examples: ClickOS,
MirageOS, Rumprun, ...
Give user
space access
to hardware
Examples: User mode
Linux, gVisor, ...
Expose the hardware
directly to user space.
It will be fine.
Examples: DPDK,
UDMA, ..
Total Estimated Cost
to Develop Linux
(average salary =
$75,662.08/year,
overhead = 2.40).
$1,372,340,206

13. What is BPF?
Highly efficient sandboxed
virtual machine in the Linux
kernel making the Linux kernel
programmable at native
execution speed.
Jointly maintained by Cilium
and Facebook with
collaborations from Google,
Red Hat, Netflix, Netronome,
and many others.
11
$ clang -target bpf -emit-llvm -S
32-bit-example.c
$ llc -march=bpf 32-bit-example.ll
$ cat 32-bit-example.s
cal:
r1 = *(u32 *)(r1 + 0)
r2 = *(u32 *)(r2 + 0)
r2 += r1
*(u32 *)(r3 + 0) = r2
exit

14. The Linux kernel is event driven
12
Process Process
CPU RAM MMU NIC Disk Disk
System Call Interface
USB
Drivers
12M lines of source code
Process Process
System calls
Interrupts

15. Run BPF program on event
13
Process
NICDisk
Process
BPF
BPF
BPF
IORead
Sendnetwork
packet
connect()
Sockets
TCP/IP
Network Device
BPF
TCP
retrans
BPF
read()
File Descriptor
VFS
Block Device
Attachment points
● Kernel functions (kprobes)
● Userspace functions (uprobe)
● System calls
● Tracepoints
● Network devices (packet level)
● Sockets (data level)
● Network device (DMA level) [XDP]
● ...

16. Process
BPF Maps
14
BPF
BPF Maps
BPF map use cases:
● Hold program state
● Share state between programs
● Share state with user space
● Export metrics & statistics
● Configure programs
Map types:
● Hash tables
● Arrays
● LRU (Least recently used)
● Ring buffer
● Stack trace
● LPM (Longest prefix match)

17. BPF Helpers
15
bpf_get_prandom_u32()
BPF
BPF helpers:
● Stable kernel API exposed to BPF
programs to interact with the kernel
● Includes ability to:
○ Get process/cgroup context
○ Manipulate network packets
and forwarding
○ Access BPF maps
○ Access socket data
○ Send metrics to user space
○ ...
bpf_skb_store_bytes()
bpf_redirect()
bpf_get_current_pid_tgid()
bpf_perf_event_output()

18. BPF Tail Calls
16
BPF
BPF
BPF tail calls:
● Chain logical programs together
● Implement function calls
● Must be within same program type
BPF
BPF
BPF

19. BPF JIT Compiler
17
JIT Compiler
● Ensures native execution
performance without requiring to
understand CPU
● Compiles BPF bytecode to CPU
architecture specific instruction set
Supported architectures:
● X86_64, arm64, ppc64, s390x, mips64,
sparc64, arm
Byte
code
Byte
code
x86_64
generic
Byte
code
generic
JIT

20. BPF Contributors
380 Daniel Borkmann (Cilium, Maintainer)
161 Alexei Starovoitov (Facebook, Maintainer)
160 Jakub Kicinski Netronome
110 John Fastabend (Cilium)
96 Yonghong Song (Facebook)
95 Martin KaFai Lau (Facebook)
94 Jesper Dangaard Brouer (Red Hat)
74 Quentin Monnet (Netronome)
45 Roman Gushchin (Facebook)
45 Andrey Ignatov (Facebook)
Top contributors of
the total 186
contributors to BPF
from January 2016 to
November 2018.
18

21. BPF Use Cases
● L3-L4 Load balancing
● Network security
● Traffic optimization
● Profiling
https://code.fb.com/open-s
ource/linux/
● QoS & Traffic optimization
● Network Security
● Profiling
● Replacing iptables with BPF
(bpfilter)
● NFV & Load balancing (XDP)
● Profiling & Tracing
● Performance
Troubleshooting
● Tracing & Systems Monitoring
● Networking
19

22. Simple Kprobe Example
20
Example: BPF program using gobpf/bcc:

23. What is Cilium?
At the foundation of Cilium is the new Linux kernel
technology BPF, which enables the dynamic insertion
of powerful security, visibility, and networking control
logic within Linux itself. Besides providing traditional
network level security, the flexibility of BPF enables
security on API and process level to secure
communication within a container or pod.
Read More
Cilium is open source software for transparently
providing and securing the network and API
connectivity between application services deployed
using Linux container management platforms like
Kubernetes, Docker, and Mesos.
21

24. Project Goals
22
Approachable BPF
● Make the efficiency and flexibility of BPF
available in an approachable way
● Automate program creation and
management
● Provide an extendable platform
Microservices-aware Linux
● Use the flexibility of BPF to make the Linux
kernel aware of cloud native concepts
such as containers and APIs.
Security
● Use the additional visibility of BPF to
provide security for microservices
including:
○ API awareness
○ Identity based enforcement
○ Process level context enforcement
Performance
● Leverage the execution performance and
JIT compiler to provide a highly efficient
implementation.

25. Cilium Use Cases
23
Container Networking
● Highly efficient and flexible
networking
● CNI and CMM plugins
● IPv4, IPv6, NAT46, direct routing,
encapsulation
● Multi cluster routing
Service Load balancing:
● Highly scalable L3-L4 load balancing
implementation
● Kubernetes service implementation or
API driven.
Microservices Security
● Identity-based L3-L4 network security
● Accelerated API-aware security via
Envoy (HTTP, gRPC, Kafka, Cassandra,
memcached, ..)
● DNS aware policies
● SSL data visibility via kTLS
Servicemesh acceleration:
● Minimize overhead when injecting
servicemesh sidecar proxies

26. BPF-based servicemesh
Acceleration
24
Service
Container
Sidecar proxy
Service
Container
Sidecar proxy
How it really looks:

27. BPF-based servicemesh
Acceleration
25
Accelerate the service to
sidecar communication
~3.5x performance improvement

28. Other BPF projects
26
Tracing / Profiling:
● BPFTrace - DTrace for Linux (Brendan
Gregg, et al.)
● bpfd - Load BPF programs into entire
clusters (Joel Fernandes, Google)
Frameworks:
● gobpf - Go based framework to write BPF
programs
● BCC - Python framework to write BPF
programs
Load balancing:
● Katran - Source code of Facebook’s
primary L3-L4 LB (Facebook team)
Security:
● Seccomp - Advanced BPF version of
Seccomp (Kernel team)
DDoS mitigation:
● bpftools - DDOS mitigation tool with
iptables like syntax (Cloudflare)
… and many more

29. Thank you!
Source Code:
https://github.com/cilium/cilium
BPF reference guide:
http://docs.cilium.io/en/stable/bpf/
Twitter:
@ciliumproject
Website:
https://cilium.io/

30. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
linux-cilium-ebpf