true

2. Prod
Prod
QA DEV
PCI
Secure
DEV
QA QA DEV
Copyright eBay Inc. 2012 2

3.  Any Application Anywhere
 Dedicated physical environments cause fragmentation
 Soft Cabling
 Datacenter reconfiguration is costly and cannot be automated
 Shared Standardized Infrastructure
 Simplifies automation and improves supply chain efficiency
 Virtualize everything
 White space between applications and infrastructure helps agility
 Automate everything
 Automation helps agility and efficiency
Copyright eBay Inc. 2012 3

4. • Translation of physical environment properties into configurations
• Assigned to projects (logical environments), drives scheduling and policies
• For example, network selection
Production DEV
Obligations Restrictions Capabilities Obligations Restrictions Capabilities
QA Approved Builds No Login Access Core DB access Certified OS versions Limited Prod Full root
Access
Prod OS version No Corp Access 24/7 Incident Mgt
Limited QA Access
Monitoring No QA Access Site traffic Access
No site Traffic Filtered Internet
External
Obligations Restrictions Capabilities
No Prod Access Private DB
Certified OS Versions No Corp Access 24/7 Incident Mgt
Monitoring No QA Access Site traffic Access
Copyright eBay Inc. 2012 4

5. Core
4 spines
(Nx10Gb)
Spine
N leaves
(48x1Gb)
Leaves
48 -> N “½ racks”
M servers
2x1Gb
Flat L3 (all switches are routers too)
Line rate from any server to any server (oversubscription = 48/40)
OSPF/ECMP to advertise routes
Copyright eBay Inc. 2012 5

6. Dedicated Network VLAN Based
VLAN trunk
vlan 1
Prod
QA vlan n
Production QA
- physical network build out + Physical isolation - Limited scale (n = 4096) + L2 isolation
- Fragmentation + fool proof - Large fault domain (STP) + somewhat soft Cabling
- coarse grained isolation
Copyright eBay Inc. 2012 6

7. Security Groups or Virtual Firewall
+ no/minimal infrastructure requirement - Difficult to combine provider policies and user policies
+ good for user policies (ip tables) - Management of rules
- Impact of group membership modification
- Aggregation/summarization difficult/impossible
Copyright eBay Inc. 2012 7

8. Virtual Networks using Software Defined Networks
Overlay 1
Prod
Other
Networks
QA
Overlay n
Cloud Fabric
+ L2 isolation + Can complement L3 isolation
+ compatible with large scale networks + large number of networks (n>4096)
+ can be fully automated - Tunnel overhead
+ firewall can be interposed between - L2 size limited by # of tunnels and their mgt
virtual networks
Copyright eBay Inc. 2012 8

9. Traditional SDN
The The
Network Network
Network protocols
Network protocols
Routing/switching engine Routing/switching engine
controls The Switch/Router
controls
Logic
Logic API
The Switch/Router Controller
Copyright eBay Inc. 2012 9

10. Wizard Physical Switches OSPF/ECMP,…
Traffic Engineering
Virtual + Physical switches
Ninja Overlay Networks
Virtual Switches ARP + L2 protocols
Nerdy
Overlay Networks
Copyright eBay Inc. 2012 10

11.  A logical environment defined as a class of service on top of shared infrastructure
 Self Service VM for developers.
 Access must be similar to their desktops (access to QA, Corp, …)
 Should allow collaboration
 Implemented as a set of L2 networks (/24) with in a given L3 (/20)
 No private networks : all developers on same shared networks
 No private IP space: traffic is routed within core, no need for floating Ips
 Isolated from infrastructure
 Overlay network using OpenVswitch / STT tunneling
 Nicira NVP controllers integrated with Quantum (Essex)
 Routed out through perimeter firewall
Copyright eBay Inc. 2012 11

12. From 10.9.1.0/24 default->10.9.0.1 10.9.0.0/20 ->10.9.0.10
From 10.9.2.0/24 default->10.9.0.1
Standby Gateway
Eth1/vlan 1
Dev Cloud : 10.9.0.0/20
Eth0/vlan 2 Corp
10.9.1.0/24 10.9.1.1 N
gtw-xxxx
trunk
gtw-xxxx 10.9.0.10 10.9.0.1
Internet
10.9.2.0/24 N
M
10.9.2.1
gtw-xxxx
QA
vswitch M Eth1/vlan 1
Eth0/vlan 2
vswitch
Nicira
default->10.9.2.1 Nicira Nicira
Active Gateway Service Nicira
Service controllers
Nodes controllers
Nodes
vif
K C Hypervisor S A Q
N:Nova-network+dnsmasq K:Ubuntu + KVM
vswitch C:Nova-compute A:Nova-api
S:Nova-scheduler Q:Quantum
M:Metadata
Infrastructure/Internal Virtual network
Infrastructure/External
Copyright eBay Inc. 2012 12

13. Developer Admin
Create network
(project = admin, Create routes
eBay Cloud Portal Cidr=10.9.x.0/24)
Create instance
1 (COS,OS, size)
Nova-manage Gateway
2 Get Free Networks
eBay IaaS
Create DNS Boot Instance Nova Network
(A,PTR) (Image ID,Flavor, NIC)
Create
4 3 gtw-xxxx
DNS Nova API Quantum
nova
Management db
Create Create
Nova Scheduler port lswitch
13
Get IP
Create port Nicira Controller
Nova Compute
Copyright eBay Inc. 2012

14. 250 100
Instance
200 80
Requests
150 60 Success
Failed
100 40 rate
50 20
0 0
Copyright eBay Inc. 2012 14

15.  Perimeter firewalls configured once, not  No capacity/policy based assignment of
dependent on the instance networks – had to be implemented outside.
creation/deletion/movement Moving it to nova scheduler.
 Network are pre-created using nova-  One network flavor supported in Essex.
manage, good for provider networks
Cannot have, e.g., one gateway per
network, with different behavior (dhcp)
 Can be extended with other COS using same
pattern
 Scale out requires bigger links out of the
gateway, or more gateways
 Stability of both Nicira NVP and Openstack +
Ubuntu + KVM
 Upset the separation of concern
 Looking forward to new features in Folsom – requirement: Netsec + Networking + Sys
Quantum v2 Admins in same box = ‘interesting’
15

16.  New classes of service
 External : private networks + VIP and Floating IP on the Internet
 Production : Bridged network
 Scale out
 80 today, going to a lot more
 More gateways/10Gb
 Folsom upgrade
 L3 Routers
 Load Balancers
 Cleaner Openstack integration
 Network Allocation
 DNS configuration
 AuthN/AuthZ
16

17. We are Hiring !
http://www.ebaycareers.com/
Copyright eBay Inc. 2012 17