Data center networks generally follow regular topologies, but these topologies can have various unique configurations, from a simple two-tier leaf and spine to a massive multi-tier scale-out model. The large amount of physical interconnections and the various patterns with which they connect introduce complexity into the management of the wiring plant. And this complexity can lead to errors during the physical build-out of the network.
Cumulus Networks created the Prescriptive Topology Manager (PTM) to give data center operators a new tool with which to perform a strict wiring validation and more. PTM introduces a software abstraction layer that ensures certain wiring rules are followed by doing a simple runtime verification of connectivity as determined by an operator’s specified wiring plan. This “prescriptive” layer dynamically ensures the desired logical topology and can take some defined actions based on the results of the topology verification, including running scripts and communicating with the Quagga routing protocol suite.
View webinar here: go.cumulusnetworks.com/ptm
1. v
Running the House That CLOS Built
Configuration of CLOS Fabrics
Dinesh Dutt, Chief Scientist
17th September, 2014
2. Cumulus Networks: Making Networks Accessible
Vision
Bringing the Linux Revolution to Networking
Mission
To enable high capacity networks that are easy to deploy and affordable helping
customers realize the full promise of the software-defined data center.
September 17, 2014 cumulusnetworks.com 2
3. A Key Transformation
SPINE
LEAF
AGGREGATION L3
cumulusnetworks.com 3
CORE
L2
ACCESS
September 17, 2014
4. Simplifications Wrought By L3
§ Fewer protocols to configure
§ Single routing protocol vs many L2 protocols
§ No FHRP required
§ Standard, inter-operable protocols
§ Fewer protocols to troubleshoot
§ You can traceroute across the network
September 17, 2014 cumulusnetworks.com 4
5. Agenda
§ Intro
§ Configuration & Automation
§ PTM
§ Outro
September 17, 2014 cumulusnetworks.com 5
7. Routing vs Bridging
§ Forwarding based on IP address, not MAC
§ Loop-free protection in data plane and control plane
§ Not just determined by a control protocol (STP)
§ Routing protocol assumes peer is down and not
reachable if it doesn’t hear from it
§ STP takes ports into forwarding if it doesn’t hear from the
other end
§ Routing protocols can work with simple and
sophisticated use cases
September 17, 2014 cumulusnetworks.com 7
8. Basic Parts To A Routing Routing Protocol
§ Three basic parts
§ Who do I communicate with (neighbor, peer etc.)
§ What do I tell them (IP prefixes usually)
§ Tuning the conversation (timers, various protocol
specific knobs)
§ But first, who am I ?
§ Router ID
September 17, 2014 cumulusnetworks.com 8
9. Comparing STP vs Routing Configuration
cumulusnetworks.com 9
§ auto br0
§ iface br0 inet static
§ bridge-ports swp1 swp2 swp3
§ bridge-stp on
§ mstpctl-treeprio 32768
§ mstpctl-portautoedge swp1=yes
§
§ router ospf
§ router-id 0.0.0.1
§ interface swp1
§ ip ospf area 0.0.0.0
§ ip ospf network point-to-point
§ interface swp2
§ ip ospf area 0.0.0.0
§ ip ospf network point-to-point
§ router bgp 65535
§ bgp router-id 0.0.0.7
§ neighbor 1.2.3.4 remote-as 65534
§ neighbor 1.2.3.4 activate
§ redistribute connected
Who Am I
Who Do I Talk To
What Do I Tell Them
Fine Points Of Conversation
September 17, 2014
10. The Nice Thing About Routing Protocols…
§ Link state vs Distance Vector
§ In distance vector, you tell your neighbors what you know
about the world
• More precisely, a node provides a summarized view of the world to its
neighbors and each neighbor then examines the summaries from all
its available neighbors to pick the best paths to use for
communication
§ In link state, you tell the world about your neighbors
• More precisely, every node has an exact copy of what every other
node said and uses this to compute reachability to each of the other
nodes
• Comparatively chatty protocol
• Network design to configure protocol to reduce chattiness
September 17, 2014 cumulusnetworks.com 10
12. Sample Topology
CLUSTER or POD CLUSTER or POD
S1 S2
M1 M2 M3 M4
cumulusnetworks.com 12
INTER-CLUSTER SPINE
T1 T2 T3 T4
September 17, 2014
13. Using OSPF
§ Popular link-state protocol used in
enterprises
§ Separate versions for IPv4 and IPv6
§ Chattiness is reduced by defining
neighborhoods (called areas)
§ All nodes in a neighborhood have a
precise copy of the info from the other
neighbors in the neighborhood
§ Information outside the neighborhood
is summarized by the border nodes
§ M1& M2 are the border nodes for T1
and T2. Similarly M3 and M4 for T3
and T4.
§ Two level neighborhood hierarchy
support only
cumulusnetworks.com 13
S1 S2
M1 M2 M3 M4
T1 T2 T3 T4
September 17, 2014
14. Using OSPF (example configuration)
§ router ospf
§ ! On M1
§ log-adjacency-changes detail
§ router-id 0.0.1.1
§ passive-interface lo
§ interface swp1
§ ip ospf area 0.0.0.1
§ ip ospf network point-to-point
§ interface swp2
§ ip ospf area 0.0.0.1
§ ip ospf network point-to-point
§ interface swp3
§ ip ospf area 0.0.0.0
§ ip ospf network point-to-point
§ interface swp4
§ ip ospf area 0.0.0.0
§ ip ospf network point-to-point
§ interface lo
§ ip ospf area 0.0.0.0
§ ip ospf network loopback
cumulusnetworks.com 14
S1 S2
area 0.0.0.0
M1 M2 M3 M4
Area 0.0.0.1. Area 0.0.0.1.
T1 T2 T3 T4
September 17, 2014
15. Using BGP
§ Swiss-army knife of routing
§ Powers the Internet
§ Quite a simple path-vector (variant of
distance vector) protocol
§ Configuration can be made hairy, but
can be kept simple
§ Multi-protocol support
§ Used without any other routing
protocol in play in the DC
cumulusnetworks.com 15
S1 S2
M1 M2 M3 M4
T1 T2 T3 T4
September 17, 2014
17. • Simplifying The Configuration
cumulusnetworks.com 17
September 17, 2014
18. Key Characteristics of Any Automatable Configuration
§ Cookie cutter configuration
§ As little node-specific variation as possible
• Nothing more than a single IP address, node name, for example
§ As little duplication of information as possible
• Specifying IP addresses of interfaces AND in OSPF/BGP network
statements
§ As much configuration as necessary, not more
September 17, 2014 cumulusnetworks.com 18
19. OSPF Unnumbered Interfaces
§ IP links are configured to have an IP address at either end in
traditional network design
§ Adds to node-specific config
§ Adds to IP addresses to be allocated
§ IPv6 is very cool with link local address
§ If these addresses are advertised, they can become attack vectors
§ OSPFv2 has a concept called unnumbered interfaces
§ Single IP address per node, usually assigned to loopback interface,
inherited by all the others
§ Simplifies interface configuration, if not routing configuration
September 17, 2014 cumulusnetworks.com 19
20. Simplifying BGP Configuration
§ BGP uses TCP as its transport which implies the
need for an IP address in the specification
§ Violates multiple cookie cutter configuration
metrics:
§ IP address specified in multiple places (neighbor
statement, IP interface definition etc.)
§ Configuration all become node specific
September 17, 2014 cumulusnetworks.com 20
22. Unnumbered BGP: More Details
§ Uses IPv6 RA to determine link local address of
remote peer
§ Sets up session over link local address
§ Works well for IPv6 today, IPv4 in the works
§ Interoperates with other implementations that
can’t support this mechanism, but can peer over
link local address
September 17, 2014 cumulusnetworks.com 22
23. Using iBGP For Further Simplification
§ ASN forces routing configuration to be node-specific
§ iBGP can be used to force configuration to be a
simple three template model:
§ One set of configuration each for leaf, pod-level spine
and inter-pod level spine
§ All leaves get the same configuration
§ All pod-level spines get the same configuration etc.
September 17, 2014 cumulusnetworks.com 23
24. Using iBGP to Get Rid of ASN
cumulusnetworks.com 24
S1 S2
M1 M2 M3 M4
T1 T2 T3 T4
§ router bgp 65400
§ ! M1 config
§ bgp router-id 0.0.0.1
§ neighbor TOR peer-group
§ neighbor TOR remote-as 64500
§ neighbor TOR route-reflector-client
§ neighbor TOR next-hop-self-all
§ neighbor TOR activate
§ neighbor swp1 TOR
§ neighbor swp2 TOR
§ neighbor IPS peer-group
§ neighbor IPS remote-as 64500
§ neighbor IPS next-hop-self
§ neighbor IPS activate
§ neighbor swp3 IPS
§ neighbor swp4 IPS
§ maximum-paths ibgp 64
September 17, 2014
26. Sorting through the Cables
§ Networking cabling complexity
§ As networks scale, the number of cables increases
§ How do we ensure cabling correctness in an easy way?
cumulusnetworks.com 26
Expected
Topology
Design
Network
Blueprint
Cable
Install
Observed
Steady
State Failures Re-
Cabling
September 17, 2014
27. Network Cabling Errors
§ Improper cabling causes
§ Reachability issues
§ Unpredictable and low
performance
“To err is human”
--Alexander Pope
cumulusnetworks.com 27
September 17, 2014
28. Prescriptive Topology Manager (PTM)
§ Define expected topology using DOT language
§ Verify connectivity per topology plan using LLDP
§ Take dynamically defined actions based on match of expected & actual
Graph
G
{
S1:p1
–
M1:p3;
S1:p2
–
M2:p3;
S1:p3
–
M3:p3;
S1:p4
–
M4:p3;
S2:p3
–
M3:p4;
S2:p4
–
M4:p4;
M1:p1
–
T1:p1;
...
M4:p2
–
T4:p2;
}
Topology Graph
cumulusnetworks.com 28
S1 S2
M1 M2 M3 M4
T1 T2 T3 T4
September 17, 2014
29. Prescriptive Topology Manager (PTM) – Work Flow
cumulusnetworks.com 29
Graphviz
Topology.dot Configuration
Management
Tool
Matching
design workflow
NOT matching
design workflow
September 17, 2014
30. Prescriptive Topology Manager (PTM) – Work Flow (contd)
Matching design workflow
• Notify engineers
• Work around problem
LLDP Topology.dot
PTMd
cumulusnetworks.com 30
neighbors
• Configure interfaces
• Add self to monitoring…
NOT matching design workflow
September 17, 2014
31. PTM Pre-defined Hookups
§ With Quagga
§ Quagga acts as PTM client
§ Listens to link-state and BFD notifications
cumulus@S1:~$
sudo
vtysh
-‐c
'conf
t'
-‐c
'ptm-‐enable'
cumulus@S1:~$
sudo
vtysh
-‐c
'show
interface
swp1'
Interface
swp1
is
up,
line
protocol
is
up
PTM
status:
pass
index
3
metrix
1
mtu
1500
flags:
<UP,BROADCAST,RUNNING,MULTICAST>
HWaddr:
00:02:00:00:00:11
inet
21.0.0.2/24
broadcast
21.0.0.255
inet6
fe80::202:ff:fe00:11/64
cumulus@S1:~$
cumulusnetworks.com 31
32. PTM: Running State
§ ptmctl (contd)
§ Shows expected and observed topology
§ Comparison test results – pass or fail
cumulusnetworks.com 32
root@cumulus:~# ptmctl
------------------------------------------
port cbl BFD BFD
status status peer
------------------------------------------
swp13 pass pass 51.0.0.41
swp14 pass pass 52.0.0.41
swp15 pass pass 53.0.0.41
swp20 fail N/A N/A
33. ptmd: Running At A Git Near You
§ Open
source,
published
under
Eclipse
Public
License
(EPL)
§
h6ps://github.com/CumulusNetworks/ptm
September 17, 2014 cumulusnetworks.com 33
35. Mapping Gods Eye View To Box Specific Config
§ A fundamental problem with networking
configuration is the inability to map network-wide
view to box-specific configuration
§ PTM provides a network-wide configuration
mechanism
September 17, 2014 cumulusnetworks.com 35
37. A Summing Up
§ CLOS topology allows for a simplified configuration
§ Routing configuration can be fairly simple
§ Coupled with PTM, BGP unnumbered interface and
other like innovations can make configuring L3
networks a snap
September 17, 2014 cumulusnetworks.com 37
39. Bringing the Linux Revolution to Networking
§ Thank You!
CUMULUS, the Cumulus Logo, CUMULUS NETWORKS, and the Rocket Turtle Logo (the “Marks”) are trademarks and service marks of Cumulus Networks, Inc. in the U.S. and other
countries. You are not permitted to use the Marks without the prior written consent of Cumulus Networks. The registered trademark Linux® is used pursuant to a sublicense from
LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis. All other marks are used under fair use or license from their respective owners.
cumulusnetworks.com 39
September 17, 2014