Marian Marinov is the chief system architect and head of the DevOps department at SiteGround.com. He discussed the challenges of high-density networks including large broadcast domains, limited MAC/ARP tables, and bandwidth constraints. Some solutions proposed were using VLANs, layered network designs, and overlay technologies like VXLAN and NVGRE to divide the network into smaller segments and increase scalability.
2. ❖❖ Who am I?Who am I?
- Chief System Architect of SiteGround.com- Chief System Architect of SiteGround.com
- Sysadmin since 1996- Sysadmin since 1996
- Organizer of OpenFest, BG Perl- Organizer of OpenFest, BG Perl
Workshops, LUG-BG and othersWorkshops, LUG-BG and others
- Teaching Network Security and- Teaching Network Security and
Linux System AdministrationLinux System Administration
courses in Sofia Universitycourses in Sofia University
and SoftUniand SoftUni
3. - Physical Machines
- with a few hundreds IPs per
machine
A bit of historyA bit of history
4. - Physical Machines
- with a few hundreds IPs per
machine
- Virtual Machines
- with a tens of IPs per VM
- with hundreds of VMs per machine
A bit of historyA bit of history
5. - Physical Machines
- with a few hundreds IPs per
machine
- Virtual Machines
- with a tens of IPs per VM
- with hundreds of VMs per machine
- Containers
A bit of historyA bit of history
6. - Containers
- with several IPs per container
- with thousand containers per
machine
A bit of historyA bit of history
7. - ONE 42U Rack
- with physical machines
42 * 100 = 4200 IPs
- with VMs
42 * (100 * 10) = 42 000 IPs
- with containers
42 * (1000 * 2) = 84 000 IPs
A bit of historyA bit of history
9. - ARP/ICMPv6
- DHCP/mdns
- HA and Gossip like protocols
- 42 machines...
“Not Great, Not Terrible”
- 420 or 4200... that is a problem
Broadcast domainsBroadcast domains
18. BandwidthBandwidth
Assumptions:
- a single physical machine avg.
200Mbps
- a single VM avg. 100Mbps
(req. 10Gbps physical uplink)
- a single container avg. 100Mbps
(req. 100Gbps physical uplink)
19. BandwidthBandwidth
42 * 200Mbps = 8.4 Gbps
42 * 100VM * 100Mbps = 420 Gbps
42 * 1000LXC * 100Mbps = 4.2 Tbps
- a single physical machine avg.
200Mbps
- a single VM avg. 100Mbps
- a single container avg. 100Mbps
20. BandwidthBandwidth
- linking these switches
- typical DataCenter switches link
with multiple 10 or 40Gbps uplinks
- its rear to see 100Gbps uplinks
23. VLANs/QinQ/MPLSVLANs/QinQ/MPLS
- reduces the broadcast domains
- if switches support mac address
tables per-vlan, it also reduces the
mac address table issues
- it does not solve the capacity/BW
issues
- it introduces complexity in the
setup
31. VXLANVXLAN
- Point-to-Point or Multicast
- 50 bytes overhead
- Jumbo frames are preferred
- statically with iproute2
- Linux Documentation vxlan.txt
- dynamically with OpenVswitch
- Supported in switches from Arista
and Brocade
32. NVGRENVGRE
- Developed by Microsoft
supported only by Microsoft
- working over GRE tunnels
- Point-to-Point or Multicast
- 42 bytes
33. GeneveGeneve
- Point-to-Point or Multicast
- Unify NVGRE, VXLAN and
STT(Stateless Transport Tunneling)
- supported in OpenVswitch
35. ConclusionConclusion
- Layer 2 should end in ToR
switches
- Layer 3 should be used for
anything more complex
- Overlays can be used to
accommodate specific client
requirements