+
A Cloud Native SDN for Kubernetes
Juergen Brendel, Stas Kraev
Kubecon, London, March 2016

romana.io A cloud native SDN for Kubernetes @romanaproject
Agenda
● “Cloud native”, why does it matter?
● A better network for cloud native architectures
● New things in Kubernetes
● Demos

romana.io A cloud native SDN for Kubernetes @romanaproject
About us
● Team background:
– Data center networks
– Low-level traffic management
● Created L2 overlay network startup
– Bought by Cisco
● OpenStack networking
● There's got to be a better way
– Time is right

What is 'cloud native'?

romana.io A cloud native SDN for Kubernetes @romanaproject
The past: Enterprise networking
● Full control
● Applications need L2 and L3
– May need hard-wired IP addresses
– Broadcasts
● Servers are pets, not cattle: “Careful!”
– VM migration
● Complex!
– Complexity in the applications
– Because apps may do anything, network needs to support
everything!

romana.io A cloud native SDN for Kubernetes @romanaproject
Cloud native applications
● Automate all the things!
– Infrastructure as code
– Cattle, not pets: “Meh... just kill it.”
– Workloads come and go quickly
– Build for resiliance
● IP is all you need
– No hardcoded IP addresses, discovery
– No special network requirements
– Basic IP connectivity
● Restrictions
– Accept them and get clarity and simplicity in return

The problem

romana.io A cloud native SDN for Kubernetes @romanaproject
We have a mismatch
● Building cloud native applications…
● … on top of enterprise networking
– SDN controllers use overlay L2 domains
– VLAN, VXLAN, OVS, etc.
● Complexity and brittleness
– Lose benefits of simplicity
– Lose performance (encap, blinded hardware)
– Difficult to maintain and trouble shoot

romana.io A cloud native SDN for Kubernetes @romanaproject
The price you pay: Complexity
VXLAN Decap
VXLAN Decap
VXLAN Encap
VXLAN Encap
2 Top of Rack Round
Trips
East/West Traffic
Per Instance Security

romana.io A cloud native SDN for Kubernetes @romanaproject
The price you pay: Performance
Router
Endpoint A Endpoint B
Router
L2 overlay A
L2 overlay B
VRouter

romana.io A cloud native SDN for Kubernetes @romanaproject
Why do we do this to ourselves?
● We don't need any L2 features
● Except traffic segmentation
– Multi tenancy
– Tiers and policies

The solution

romana.io A cloud native SDN for Kubernetes @romanaproject
Cloud native SDNs
● Use native L3 capabilities
● No overlays
● De-emphasize IP address ranges
● Still provides segmentation, multi tenancy
● Simple, clear and scalable network setup

romana.io A cloud native SDN for Kubernetes @romanaproject
A truly cloud native SDN: Romana
● Project Romana
● Open source
● Apache 2.0 license
● Mostly written in Go
● Kubernetes and OpenStack

romana.io A cloud native SDN for Kubernetes @romanaproject
A truly cloud native SDN: Romana
● Use only IP routing
– No overlays
– All workload addresses are 'real'
– Simplicity!
● Use smart addressing
– Encode tenant or segment in IP address
– Assign “virtual” addresses with host prefixes
– Massive (!) collapse of route table
● Routes are static
– No route updates, no broadcasts for new endpoint

romana.io A cloud native SDN for Kubernetes @romanaproject
Routing and route aggregation
Host A
eth0:
192.168.8.11
romana-gw:
10.0.0.1/16
10.0.0.5
10.0.1.7
10.0.1.19
10.0.5.3
Host B
eth0:
192.168.8.22
romana-gw:
10.1.0.1/16
10.1.3.52
10.1.9.2
Host C
eth0:
192.168.8.33
romana-gw:
10.2.0.1/16
10.2.0.16
10.2.3.81
10.2.4.6
Routes:
10.1/16 → 192.168.8.22
10.2/16 → 192.168.8.33
Routes:
10.0/16 → 192.168.8.11
10.2/16 → 192.168.8.33
Routes:
10.0/16 → 192.168.8.11
10.1/16 → 192.168.8.22

romana.io A cloud native SDN for Kubernetes @romanaproject
Architecture
Host A Host B Host C
Agent Agent Agent
Tenant
Topology
IPAM
Root
Kubernetes

romana.io A cloud native SDN for Kubernetes @romanaproject
Architecture
Host A Host B Host C
Agent Agent Agent
Tenant
Topology
IPAM
Root
OpenStack

Romana / Kubernetes integration

romana.io A cloud native SDN for Kubernetes @romanaproject
Integration points
● CNI (Container Network Interface)
– Developed last year by CoreOS
– Supported by Kubernetes since version 1.1
● Third party resources
– Develop Kubernetes extensions via external
processes
● Network Policies
– Still under development by networking SIG
– Different proposals under discussion

romana.io A cloud native SDN for Kubernetes @romanaproject
CNI_COMMAND (ADD | DEL)
CNI_CONTAINERID
CNI_NETNS
CNI_IFNAME
CNI_ARGS
...
CNI: Interface creation workflow
Host A
eth0:
192.168.8.11
Romana
CNI plugin
Kubelet Create interface

romana.io A cloud native SDN for Kubernetes @romanaproject
CNI: Interface creation workflow
Host A
eth0:
192.168.8.11
Romana
CNI plugin
Kubelet
Romana
IPAM
Romana
Tenant
Romana
Topology
Host
Tenant
Segment

romana.io A cloud native SDN for Kubernetes @romanaproject
CNI: Interface creation workflow
Host A
eth0:
192.168.8.11
Romana
CNI plugin
Kubelet
Romana
Agent
10.0.0.5
connectivity
policies
Romana
IPAM
Romana
Tenant
Romana
Topology
IP address

romana.io A cloud native SDN for Kubernetes @romanaproject
Third party resources
● Tell Kubernetes about your new resource
$ kubectl create ­f third­party­resource­definition.yml
● Start listening for events on new URLs
/apis/romana.io/demo/v1/namespaces/default/networkpolicys/
metadata:
name: network­policy.romana.io
apiVersion: extensions/v1beta1
kind: ThirdPartyResource
description: "Network policy"
versions:
­ name: demo/v1

romana.io A cloud native SDN for Kubernetes @romanaproject
Kubernetes network polices
● Recognized need for policies
– Grant / deny access, isolate tiers and tenants
– Basically: ACLs
– Different proposals exist
– Implementations use Kubernetes 3rd party resources
● Namespaces
– Use namespace as 'tenant'
– Add 'isolation' flag to namespace

romana.io A cloud native SDN for Kubernetes @romanaproject
Example network policy
POST /apis/romana.io/demo/v1/namespaces/tenant­a/networkpolicys/
{
"kind": "NetworkPolicy",
"metadata": {
"name": "pol1"
},
"spec": {
"allowIncoming": {
"from": [
{ "pods": { "segment": "frontend" } }
],
"toPorts": [
{ "port": 80, "protocol": "TCP" }
]
},
"podSelector": { "segment": "backend" }
}
}
Gets
applied to
namespace
“segments”:
Natural fit
for Romana

romana.io A cloud native SDN for Kubernetes @romanaproject
Network policy workflow
Kubernetes master
Kubernetes API
3rd
party resource
type definition
kubectl

romana.io A cloud native SDN for Kubernetes @romanaproject
Network policy workflow
Kubernetes master
Kubernetes API
URLs
New URLs for this
resource type, per
namespace

romana.io A cloud native SDN for Kubernetes @romanaproject
Host
Romana
Agent
iptables
Host
Romana
Agent
iptables
Network policy workflow
Kubernetes master
Romana
K8S listener
Kubernetes API
Host
Romana
Agent
New Romana
policy definition
URLs
Events
streamed
through GET
request
Some client
POST /…..
{ new policy }
iptables

Demo

romana.io A cloud native SDN for Kubernetes @romanaproject
Conclusion
● Cloud native architectures simplify things
● Need a cloud native SDN to enjoy benefits
● Romana:
– Cloud native without compromises
– Native network performance
– Mostly static config: Solid network
– Very easy to work with and understand
● Easy to try:
– Simple installers for Kubernetes and OpenStack

romana.io A cloud native SDN for Kubernetes @romanaproject
Thank you!
● Romana Links
– http://romana.io - Project home
– http://romana.io/blog - Blog
– https://github.com/romana/romana - Sources
● Contact
– @romanaproject - Twitter
– info@romana.io - Email
– https://romana.slack.com/ - Slack channel
● Kubernetes links
– http://bit.ly/1RMVkrr - CNI spec

Appendix: Romana technical notes

romana.io A cloud native SDN for Kubernetes @romanaproject
Semantic and topological addressing
3
1
3
0
2
9
2
8
2
7
2
6
2
5
2
4
2
3
2
2
2
1
2
0
1
9
1
8
1
7
1
6
1
5
1
4
1
3
1
2
1
1
1
0
9 8 7 6 5 4 3 2 1 0
0 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1
10
Network prefix bits
The network prefix.
In this example, we
are using the 10/8
address space.
6
Host ID Segment ID
We currently
store tenant ID in
upper bits of
segment ID.
4 67
Endpoint ID
Widths are configurable, don't have to use byte boundaries.

romana.io A cloud native SDN for Kubernetes @romanaproject
Segment and tenant bits
3
1
3
0
2
9
2
8
2
7
2
6
2
5
2
4
2
3
2
2
2
1
2
0
1
9
1
8
1
7
1
6
1
5
1
4
1
3
1
2
1
1
1
0
9 8 7 6 5 4 3 2 1 0
0 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1
10
Network prefix bits
6
Host ID Segment ID
4 67
Endpoint ID
Encode the
tenant ID

romana.io A cloud native SDN for Kubernetes @romanaproject
Romana: Traffic segmentation
● Tenant traffic separated:
– Tenants don't get whole CIDR prefix or L2 domain
– But fully isolated from other tenants' traffic
● Tenants can define segments:
– Like tiers, provide isolation and policies
● Use segment and tenant bits in IP addresses:
– Apply policies (iptables) based on that
– Segments can stretch across hosts

romana.io A cloud native SDN for Kubernetes @romanaproject
Host BHost A
Allowing traffic within tenant
10.0.0.5 10.1.0.12
iptables:
check src/dst addrs
“tenant/segment bits
must match”
Src: 10.0.0.5
Dst: 10.1.0.12
Same
tenant/segment bits

romana.io A cloud native SDN for Kubernetes @romanaproject
Host BHost A
Isolating tenant traffic: Default
10.0.0.5 10.1.128.9
iptables:
check src/dst addrs
“tenant/segment bits
must match”
Src: 10.0.0.5
Dst: 10.1.128.9
Different
tenant/segment bits
Different
tenant

romana.io A cloud native SDN for Kubernetes @romanaproject
Host BHost A
Apply network policy between
segments (full isolation as default)
10.0.0.5 10.1.1.9
iptables:
Does policy chain
exist?
Otherwise: DROP
Src: 10.0.0.5
Dst: 10.1.1.9
Same tenant,
different segment
policy-chain:
From segment 0?
Protocol TCP?
To port 80?