100Gbps Core Network Deployment in an African Network

1. COMMERCIAL–IN-CONFIDENCECOMMERCIAL–IN-CONFIDENCE
100Gbps
Core Network Deployment

2. COMMERCIAL–IN-CONFIDENCE
Overall Network
2

3. COMMERCIAL–IN-CONFIDENCE
Where Did The Upgrade Take Place?
• The upgrades were done in our Johannesburg and Cape Town PoP’s.
• South Africa is our fastest growing market.
• South Africa is a transit point for the WACS cable system.
3

4. COMMERCIAL–IN-CONFIDENCE
Standard Core PoP Infrastructure
4
Cisco CRS Router
8-slot Chassis
Cisco C6880-X
Ethernet Switch

5. COMMERCIAL–IN-CONFIDENCE
Why This Design?
• Core router ports are expensive.
• More efficient use of IP address space (LAN, not point-to-point links).
• The only traffic hitting the core routers is exiting the PoP.
• Hands-off the core routers.
• Easy to add edge devices within the PoP.
• Device upgrades can be detached/independent from/of each other.
• Avoid stranding bandwidth.
5

6. COMMERCIAL–IN-CONFIDENCE
Previous Connectivity Compliment
6
14-port 10Gbps Line Card
Cisco CRS-3 Router
140Gbps/slot
16-port 10Gbps Line Card
C6880-X Switch
80Gbps/slot
- Each PoP was deployed with 80Gbps router ó switch in the core.
- Capacity provisioned as 2x 40Gbps LAG’s.
- Each LAG comprises 4x 10Gbps ports.
- Each router has links to each of the 2 switches.
- 2 core routers + 2 core switches per PoP.

7. COMMERCIAL–IN-CONFIDENCE
Previous Connectivity Compliment
• Generally, 80% of the core PoP’s are fine with this design.
• Johannesburg and Cape Town were starting to become a concern.
• Large EoMPLS flows that don’t load balance well.
• Transit traffic from other markets during a SEACOM cable outage.
• Overall network growth in Johannesburg and Cape Town.
7

8. COMMERCIAL–IN-CONFIDENCE
New Connectivity Compliment
8
4-port 100Gbps Line Card
Cisco CRS-X Router
400Gbps/slot
12-port 100Gbps Line Card
Arista 7508E Switch
1.2Tbps/slot
Arista 7508E
Ethernet Switch
48-port 1Gbps/10Gbps Line Card
Arista 7508E Switch
480Gbps/slot

9. COMMERCIAL–IN-CONFIDENCE
Why Change To Arista?
• Cheapest 100Gbps/port core switch on the market (as at 2017).
• The Cisco Nexus 7710 was too expensive.
• The Juniper QFX10008 was too expensive.
• Arista supported all the basic features we need for a core switch:
• 802.1Q.
• 802.1AX (a.k.a 802.3ad).
• Load balancing both IP and non-IP traffic flows.
9

10. COMMERCIAL–IN-CONFIDENCE
Why Keep The Cisco CRS?
• The Cisco CRS has been our standard core router since 2014.
• Options like the Cisco NCS and Juniper PTX were not available then.
• Nokia’s (ALU’s) 7950 XRS router would have been a new vendor.
• Although remote with each year, non-Ethernet requirements remain.
• The issue with the CRS:
• The 4-port 100Gbps line cards are too pricey.
• The CPAK optics are too costly.
• But no immediate demand needed in other PoP’s.
• Cisco claim maximum future per-slot throughput of 800Gbps.
• Engineering support in the future is “unknown”.
• Focus is on the NCS6000.
• But overall, the Cisco CRS is more than enough for our needs…
10

11. COMMERCIAL–IN-CONFIDENCE
The Challenges
• Optics Incompatibility:
• Initial order was for the Nexus 7710 switch.
• The Cisco CPAK optics were based on the SR4 spec.
• The Arisa optics were based on the SR10 spec.
• We waited a month to get the right SR4 optics for the Arista.
• Poor 100Gbps experience in Africa:
• No one in the industry could assist with our challenges.
• Little to no experience in the African market.
• Lack of 100Gbps accessories in Africa:
• We were unable to source MPO cables locally in-continent.
• Our African suppliers had no knowledge of MPO cables.
• Resorted to our off-continent suppliers for the support.
• Some MPO cables shipped with the wrong termination.
11

12. COMMERCIAL–IN-CONFIDENCE
The Challenges
• Logistics:
• 100Gbps line cards are new to African Customs departments.
• Customs clearance of these was a major problem.
• Cisco CRS fabric upgrade:
• Pre-100Gbps setup was CRS-3, i.e., 140Gbps/slot.
• 100Gbps required a fabric upgrade to CRS-X, i.e., 400Gbps/slot.
• This was known prior to the deployment.
• A change in in-house teams meant this was forgotten.
• Avenue of pleasure when the 100Gbps line card won’t boot J.
• Training on Arista:
• We have generally been a 2-vendor network.
• Cisco and Juniper only.
• Arista is new, so training was needed.
• But this wasn’t difficult. EOS looks like IOS.
12

13. COMMERCIAL–IN-CONFIDENCE
The Challenges
• Vendor optics lock-in:
• Arista tried to force us to use their own optics.
• We insisted on using our own from 3rd party suppliers.
• Arista provided an unlock code to allow these on their platform.
• The cost savings for re-using the optics is over 95%.
• Only need to buy the 100Gbps optics.
• Data centre changes:
• The Cisco 6800-X used 10A power sources.
• The Arista 7508E needed 16A power sources.
• We also needed to obtain additional rack space for the Arista’s.
13

14. COMMERCIAL–IN-CONFIDENCE
Final Result
14

15. COMMERCIAL–IN-CONFIDENCE
Final Result
15

16. COMMERCIAL–IN-CONFIDENCE
EOS vs. IOS
• In short, no real difference in the CLI.
• In fact, Cisco decided to sue Arista for “slavishly” copying IOS:
http://bit.ly/eosvios
• Only need to learn about Arista-specific things.
• You can switch from Cisco to Arista in a matter of hours.
16

17. COMMERCIAL–IN-CONFIDENCE
Too Many Vendors?
• SEACOM have always been a 2-vendor state:
• Cisco
• Juniper
• We did not see introducing Arista as a 3rd vendor as a problem:
• Very specific, simple version – Layer 2 Ethernet core switching.
• Low-touch requirements.
17

18. COMMERCIAL–IN-CONFIDENCE
Thank You
Q&A
mark.tinka@seacom.mu
18