Setting the stage for TRILL, rethinking data center switching. By Brad Hedlund

1. Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing (TRILL) v 1.0 Brad Hedlund Cisco Systems, Inc. INTERNETWORK EXPERT .ORG 5/7/2010

2. - About the Author - Brad Hedlund Technical Solutions Architect, Data Center Cisco Systems, Inc. CCIE #5530 http://www.internetworkexpert.org/about/ Blog: http://www.internetworkexpert.org Twitter: http://twitter.com/bradhedlund E-mail: bhedlund@cisco.com Comments welcome. INTERNETWORK EXPERT .ORG 5/7/2010

3. Why Evolve Data Center Switching? Transformational Paradigm Shifts From Connectivity to Virtualization New Requirements Large, Flat, Scalable L2 fabrics Simpler, Smarter Networks Ultra High Availability Storage/Ethernet consolidation More bandwidth The Server is a fluid object The virtual machine is the new Server The physical machine is the new Network Miniaturization & Scale Any server, any VLAN, anywhere, anytime What’s Next? The “Data Center” becomes a fluid object INTERNETWORK EXPERT .ORG 5/7/2010

4. Revisiting Classic Ethernet What works? What needs to be improved? SW1 SW2 SW3 INTERNETWORK EXPERT .ORG 5/7/2010 Narrative of this section located at: http://www.internetworkexpert.org/2010/05/07/setting-the-stage-for-trill/

6. Assists in MAC Learning

7. Foundational Behavior

8. Assists in Discovery

9. Assists in MAC Learning

10. Plug & PlayA L L A L L A L L C SW1 SW1 D D D C A A INTERNETWORK EXPERT .ORG 5/7/2010

12. InefficientImprovement Area (TRILL*) SW1 MAC Table A -- Eth 1 B -- Eth 1 C -- Eth 2 D -- Eth 2 B SW1 Unnecessary 2 1 A L L A A L L A A L L A A L L A A L L A A L L A A L L D A L L C A L L A L L B A L L A L L A L L A L L A L L A L L SW3 MAC Table A -- Eth 1 B -- Eth 2 C -- Eth 3 D -- Eth 3 SW4 MAC Table A -- Eth 3 B -- Eth 3 C -- Eth 1 D -- Eth 2 SW4 SW3 3 3 1 2 2 1 C D Useful A INTERNETWORK EXPERT .ORG 5/7/2010

13. Flooding with Multiple Paths Plug & Play loop prevention is needed… SW1 SW2 SW3 SW4 No Ethernet TTL Infinite Loop Broadcast Unknown Unicast Flooding Required A B Improvement Area (TRILL) INTERNETWORK EXPERT .ORG 5/7/2010

15. Loop Free for flooded traffic

16. Single Topology for all traffic

17. Single Path for Unicast

18. Single Path for Multicast

19. 50% Bandwidth Unused

20. Timer based recoveryImprovement Areas (TRILL*) A B INTERNETWORK EXPERT .ORG 5/7/2010

21. Scaling the data center network Challenges, Approaches, SW1 SW2 SW3 SW4 SW5 SW6 INTERNETWORK EXPERT .ORG 5/7/2010 Narrative of this section located at: http://www.internetworkexpert.org/2010/05/07/setting-the-stage-for-trill/

23. Larger Tier 2 Scalability

24. Smaller Tier 1 switches

25. Spread Risk (RAID)

26. Pay as you grow

27. Lower latencyTier 2 SW3 SW4 INTERNETWORK EXPERT .ORG 5/7/2010

29. Larger Tier 2 Scalability

30. Smaller Tier 1 switches

31. Spread Risk

32. Pay as you grow

33. Low latency

34. Spanning Tree still required

35. Blocking on all but one path

36. Did not gain scalability

37. Did not gain bandwidthA B INTERNETWORK EXPERT .ORG 5/7/2010

39. Larger Tier 2 Scalability

40. Smaller Tier 1 switches

41. Pay as you grow

42. Low latency

43. Spanning Tree still required

44. Blocking on all but one path

45. Did not gain scalability

46. Did not gain bandwidthA B INTERNETWORK EXPERT .ORG 5/7/2010

48. Larger Tier 2 Scalability

49. Smaller Tier 1 switches

50. Pay as you grow

51. Low latency

52. Unbalanced at Tier 2

53. Only 1 switch with all paths

54. Did not gain scalability

55. Did not gain bandwidthA B INTERNETWORK EXPERT .ORG 5/7/2010

56. Scaling UP Tier 1 with Classic Ethernet Rigid (2) Switch Tier 1 design constraint forces Scaling UP – not OUT SW2 SW1 Improvement Area (TRILL) SW1 SW2 SW3 SW4 INTERNETWORK EXPERT .ORG 5/7/2010

58. MAC learning

59. LACP

60. Interface states/config

61. Split Brain detectionNot a trivial accomplishment! Several different states must be synced Improvement area (TRILL) Multi path with minimal state/sync complexity INTERNETWORK EXPERT .ORG 5/7/2010

62. Scaling Tier 2 with Classical Ethernet Scale Bandwidth or Size? – Pick one, you can’t have both! SCALING BANDWITH SCALING SIZE OR SW1 SW2 SW2 SW1 Tier 1 SW4 SW3 Tier 2 SW5 SW6 SW4 SW3 Trade-off between Bandwidth or Size Tier 1 switch density key scaling factor Improvement Area (TRILL) INTERNETWORK EXPERT .ORG 5/7/2010

64. MAC learning

65. LACP

66. Interface states/config

67. Split Brain detectionSW3 SW4 SW3 SW4 STP still present Increased Complexity = Difficult & Fragile scaling INTERNETWORK EXPERT .ORG 5/7/2010

68. Scaling out Tier 1 with MPLS Replacing L2 switching with L3 + MPLS and L2 pseudo wire full mesh Overlay via VPLS Sound complicated? That’s because it is. SW1 SW2 SW6 SW5 IP + MPLS +VPLS L2 pseudo wires (full mesh required) SW3 SW4 SW7 SW8 SW9 SW10 N * (N-1) / 2 Complex overlay of L2 services over L3 MPLS skill sets? Configuration Intense NOT Plug & Play! Increased Complexity = Difficult & Fragile scaling INTERNETWORK EXPERT .ORG 5/7/2010

69. TRILL - Layer 2 multi Pathing An Introduction SW1 SW2 SW3 INTERNETWORK EXPERT .ORG 5/7/2010 Narrative of this section to be posted at: http://www.internetworkexpert.org/topic/trill/

70. Design Goals for TRILL TRILL Switching Routing Minimal Configuration Plug & Play Auto Discovery Auto Learning Flat Addressing Spanning Tree Protocol (STP) Slow Convergence Single Path Edge-to-Root Rigid Design Single Multicast Tree Constrained Scaleability Configuration Intense Configured Learning Configured Discovery Plan & Play Fast Convergence Multiple Paths Load Balancing Multiple Multicast Trees Hierarchical Forwarding Any-to-any Flexible Design Highly Scalable INTERNETWORK EXPERT .ORG 5/7/2010

73. Hierarchical, Efficient

74. ScalableImprovement Area (TRILL*) SW1 MAC Table A -- Eth 1 B -- Eth 1 C -- Eth 2 D -- Eth 2 B SW3 -- Eth 1 SW4 -- Eth 2 Conversation Learning SW1 Unnecessary 2 1 SW3 MAC Table A -- Eth 1 B -- Eth 2 C -- Eth 3 D -- Eth 3 SW4 MAC Table A -- Eth 3 B -- Eth 3 C -- Eth 1 D -- Eth 2 B – SW3 AFTER SW4 SW3 3 3 BEFORE C -- SW4 1 2 2 1 C D Useful A INTERNETWORK EXPERT .ORG 5/7/2010

76. Flat & InefficientImprovement Area (TRILL*) SW1 MAC Table A -- Eth 1 B -- Eth 1 C -- Eth 2 D -- Eth 2 B SW1 Unnecessary 2 1 SW3 MAC Table A -- Eth 1 B -- Eth 2 C -- Eth 3 D -- Eth 3 SW4 MAC Table A -- Eth 3 B -- Eth 3 C -- Eth 1 D -- Eth 2 SW4 SW3 3 3 BEFORE C -- SW4 1 2 2 1 C D Useful A INTERNETWORK EXPERT .ORG 5/7/2010

78. Hierarchical, Efficient

79. ScalableSW1 MAC Table B SW3 -- Eth 1 SW4 -- Eth 2 Conversation Learning SW1 2 1 SW3 MAC Table A -- Eth 1 B -- Eth 2 SW4 MAC Table C -- Eth 1 D -- Eth 2 B – SW3 AFTER SW4 SW3 3 3 C -- SW4 1 2 2 1 C D Useful A INTERNETWORK EXPERT .ORG 5/7/2010

80. Multi Pathing -- Evolved 16-way Equal Cost Multi Path (ECMP) Layer 2 Forwarding SW5 SW1 SW2 SW6 16 SW3 SW4 2 1 3 4 2 1 3 4 MAC Table B -- SW4 A -- Eth5 MAC Table A -- SW3 B -- Eth5 5 5 Per Flow L2-L4 hashing SW Table SW4 -- Eth 1,2,3,4 SW Table SW3 -- Eth 1,2,3,4 A B INTERNETWORK EXPERT .ORG 5/7/2010

81. Loop Free Flooding -- Evolved A unique loop free forwarding topology for Broadcast & Unknown UnicastONLY SW5 SW1 SW2 SW6 16 SW3 2 1 3 4 5 A B INTERNETWORK EXPERT .ORG 5/7/2010 SW4 2 1 3 4 Does not punish path availability for known unicast & multicast conversations 5

82. Multicast -- Evolved More Bandwidth for Multicast – All possible Topologies Used SW5 SW1 SW2 SW6 16 SW3 SW4 Per (S,G) Topologies S1 S2 R1 R2 INTERNETWORK EXPERT .ORG 5/7/2010

84. Larger Tier 2 Scalability

85. Smaller Tier 1 switches

86. Spread Risk (RAID)

87. Pay as you grow

89. No Spanning Tree

90. No complicated MPLS overlay

91. Simple Configuration

92. Flexible any-to-any designINTERNETWORK EXPERT .ORG 5/7/2010

93. Scaling Tier 2 with TRILL Flexible design can scale both Bandwidth and Size SCALING BANDWITH SCALING SIZE AND SW2 SW1 SW2 SW3 SW4 SW1 Tier 1 SW4 SW3 Tier 2 SW5 SW8 SW7 SW6 Minimal Trade-off between Bandwidth or Size Tier 1 switch density less of a scaling factor INTERNETWORK EXPERT .ORG 5/7/2010

99. Stay tuned for more details… http://www.internetworkexpert.org/topics/trill/ SW1 SW2 SW3 INTERNETWORK EXPERT .ORG 5/7/2010