This classic talk from 2002-03, captures some of the key traffic engineering and core network design strategies deployed by carriers from the early 1990's to early 2000's, and (now, in 2011!) provides a great historical perspective on how network cores have evolved. It will prove valuable for those looking to understand network evolution, and the operational principles and considerations behind it...

1. Metanoia, Inc.
Critical Systems Thinking™
Modern Carrier Strategies for
Traffic Engineering
Dr. Vishal Sharma
Principal Consultant
Metanoia, Inc.
Voice: +1 408 394 6321
Email: v.sharma@ieee.org
Web: http://www.metanoia-
© Copyright 2002
All Rights Reserved inc.com

2. Metanoia, Inc.
Critical Systems Thinking™
Basic Service Provider Goals
The two fundamental tasks before any service provider:
 Deploy a physical topology that meets customers’ needs
 Map customer traffic flows on to the physical topology
 Earlier (1990s) the mapping task was uncontrolled!
 By-product of shortest-path IGP routing
 Often handled by over-provisioning
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 2

3. Metanoia, Inc.
The Early Years (< 1994-95): Routed Critical Systems Thinking™
Network Topology
IP Router
IP Router
IP Router
Network Cloud
Router
Interconnections
IP Router
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 3

4. Metanoia, Inc.
The Early Years (< 1994-95): A Critical Systems Thinking™
Stacked View
IP Routers
Router
-- Service creation
-- Pkt. switching
-- Stat muxing
FDDI rings -- Connectivity
(100 Mb/s)
TDM over
copper (T1/T3) MUX
-- Speed match I/Fs
DCS/DXC
SDH/SONET
-- TDM transport
-- Fault isolation
-- Restoration
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 4

5. Metanoia, Inc.
TE in Carrier Networks: Traditional Critical Systems Thinking™
Routed Core (pre 1994-95)
 Prior to advent of ATM ...
... IP metrics were the only means available to control traffic
distribution through IP networks
 Approach was ad-hoc
 Observe traffic flow through network
 Adjust weight of links with load lower/higher than desired
 Overprovision network as, needed
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 5

6. Metanoia, Inc.
Critical Systems Thinking™
Two Paths to TE in IP Networks
 With increase in traffic, emergence of ATM, and higher-speed
SONET, two approaches emerged
Use a Layer 2 (ATM) network Use only Layer 3 (IP) network
 Build ATM backbone  Build SONET infrastructure
 Deploy complete PVC mesh,  Rely on SONET for resilience
bypass use of IP metrics  Run IP directly on SONET (POS)
 TE at ATM layer  Use metrics (systematically) to
 With time, evolve ATM to MPLS- control flow of traffic (more on
based backbone this later)
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 6

7. Metanoia, Inc.
Critical Systems Thinking™
Genesis of the ATM-Core
 Growth in traffic ⇒ needed faster backbones (> T3/45 Mb/s)
 Denser backbones ⇒ metric manipulation impractical
 IP routers lagged: offered only DS3 I/Fs & s/w forwarding
 ATM emerged, was designed for WAN from start
⇒ In 1994-95 had OC-3, and later OC-12 I/Fs available
⇒ Allowed carriers to redesign their networks for high-speeds
⇒ As an evolutionary step, SPs moved to a switched ATM core
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 7

8. Metanoia, Inc.
Critical Systems Thinking™
ATM-based Cores (mid to late 1990s)
Router
-- Service creation
-- Pkt. switching
-- Stat muxing
-- Connectivity
ATM
-- Traffic engg.
-- Hardware fwding
MUX
-- Speed match I/Fs
SDH/SONET
-- TDM transport
-- Fault isolation
-- Restoration
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 8

9. Metanoia, Inc.
Critical Systems Thinking™
Physical Topology with an ATM Core
OC-3
Router
POP 1 OC-3 POP i+1
POP j
POP 2 OC-12
OC-3 ATM
SAR I/F
POP i ATM Switch POP N
ATM Core
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 9

10. Metanoia, Inc.
Critical Systems Thinking™
Logical Topology with an ATM Core
PVC 1' Primary PVC Mesh
A PVC 1 C A C
PVC 4' PVC 4
PVC 2
PVC 3'
B PVC 5 D
PVC 3 B D
PVC 5'
Secondary PVC Mesh
PVC 2'
ATM PVC Layout L3 Logical Topology
 ATM-core (usually) fully owned by SP
 Dedicated (usually) to supporting IP backbone
 Utilized ATM UBR or ATM VBR-rt/CBR, depending on classes of traffic
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 10

11. Metanoia, Inc.
Genesis of the IP-over-SONET/SDH Critical Systems Thinking™
Approach
 Desire to minimize # of network layers
 Easier management
 Simpler operation
 Potentially scalable
 Belief that high-speed SONET/SDH I/Fs would become
available with advances in components (vindicated with time)
 Dictated partly by how (in time) a carrier’s network evolved
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 11

12. Metanoia, Inc.
SONET/SDH-based Cores (mid-to- Critical Systems Thinking™
late 1990s and beyond)
Router
-- Service creation
-- Pkt. switching
-- Stat muxing
-- TE w/ metrics or
MPLS
MUX
-- Speed match I/Fs
SDH/SONET
-- Framing
-- Fault isolation
-- Restoration
(moving away)
DWDM
-- B/w on existing
plant
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 12

13. Metanoia, Inc.
Critical Systems Thinking™
Physical Topology with SONET/SDH Core
OC-3/12 or
STM-1/4
Router
POP 1 POP i+1
OC-48 BLSR/ POP j
MS-SPRing
POP 2
Point-to-point SONET/
SDH framed link
OC-3 SONET/
STM-1 SDH I/F
POP i OC-3/13 UPSR/ POP N
DXC SNCP Ring
SONET/SDH Core
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 13

14. Metanoia, Inc.
Critical Systems Thinking™
Logical Topology with SONET/SDH Core
Parallel Logial Links for
Load Sharing
Ckt. 1
A C A C
Ckt. 5
Ckt. 4
Ckt. 3
B D B D
Ckt. 2
Each logical link provisioned
for 2X the bandwidth
SONET/SDH Circuit Layout
L3 Logical Topology
 SONET/SDH infrastructure (usually) owned by SP
 Logical links between POP routers realized over a physical SONET/SDH
circuit going over a fiber path
 Parallel logical links (physically disjoint) provisioned b/w each router pair
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 14

15. Metanoia, Inc.
Critical Systems Thinking™
Global Crossing IP Backbone Network
Courtesy: Thomas Telkamp, GBLX
100,000 route miles 200+ POPs 5 continents 27 countries 250 major cities
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 15

16. Metanoia, Inc.
Critical Systems Thinking™
Global Crossing (GBLX): A Bit of History
 First independent global fiber network
 Launched operations -- March 1997
 First segment turned on -- May 1998
 Expanded network & svcs. by acquisitions & JVs
 Frontier Telecommunications, Sept 1999
 Racal Telecom, Nov 1999
 Hutchison Global Crossing, Jan 2000
 IXNET/IPC, June 2000
 International network, worldwide reach
 100,000 route miles, 27 countries, 250 major cities
 195 POPs (mid 2001)
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 16

17. Metanoia, Inc.
Critical Systems Thinking™
Global Crossing IP Network
 OC-48c/STM-16c (2.5Gbps) IP backbone
 Selected 10Gbps links operational (e.g. Atlantic)
Edge Equipment
Cisco 7500/7200 ESR, OSR
Juniper M10/20/40
Core Equipment
Cisco GSR 12000/12400
[12.0(17) SI]
 Services offered
 Internet access & Transit services
 IP VPNs -- Layer 3 and Layer 2
 MPLS and DiffServ deployed globally
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 17

18. Metanoia, Inc.
Global Crossing: Critical Systems Thinking™
Network Design Philosophy
 Ensure there are no bottlenecks in normal state
 On handling congestion
 Prevent via MPLS-TE
 Manage via Diffserv
 Over-provisioning
 Well traffic engineered network can handle all traffic
 Can withstand failure of even the most critical link(s)
 Avoid excessive complexity & features
 Makes the network unreliable/unstable
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 18

19. Metanoia, Inc.
Critical Systems Thinking™
Global Crossing’s Approach: Big Picture
Modem
Bank POP2
Web To other ISPs
Server HR BR
To Customers AR
DR
DR
BR CR
HR AR
WR
POP1 CR
WR
POP3
OC-3/OC-12
AR = Access Router WR OC-12/OC-48
BR = Border Router OC-48/OC-192
CR
DR = DSL Aggregation
DR AR
CR = Core Router
HR = Hosting Router HR BR
WR = WAN Router Ethernet
Switch
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 19

20. Metanoia, Inc.
TE in the US IP Network: Critical Systems Thinking™
Deployment Strategy
 Decision to adopt MPLS for traffic engineering & VPNs
 Y2000: 50+ POPs, 300 routers; Y2002: 200+ POPs
 Initially, hierarchical MPLS system  2 levels of LSPs
 Later, a flat MPLS LSP full mesh only between core routers
 Started w/ 9 regions -- 10-50 LSRs/region ⇒ 100-2500 LSPs/region
 Within regions: Routers fully-meshed
 Across regions: Core routers fully-meshed
 Intra-region traffic ~Mb/s to Gb/s, Inter-region traffic ~ Gb/s
Source [Xiao00]
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 20

21. Metanoia, Inc.
Critical Systems Thinking™
Design Principles: Statistics Collection
25 Mb/s
A
25 Mb/s
C
B
Using packets, we do not know
traffic individually to B & C
A LSP1 = 15 Mb/s
C
LSP2 = 10 Mb/s
B
LSP3 = 10 Mb/s
Statistics on individual LSPs can
be used to build matrices
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 21

22. Metanoia, Inc.
Design Principles: LSP Control & Critical Systems Thinking™
Management
New LSP takes
New Request
longer path
A to D = 2.2 Gb/s
B Links utilization is
Manually move traffic away from OC-192 more balanced
A
potential congestion via ERO
OC-48
D
10% in use
before new req.
B
D
B
Lowered load, greater
OC-192 headroom to grow
A LSPs split across
alternate routes B
2 LSPs of 1.2 D A
Gb/s each
OC-48 Load splitting
B D
D ratio = 0.5 each
Adding new LSPs with a
B
configured load splitting ratio D
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 22

23. Metanoia, Inc.
Global Crossing’s Current LSP Critical Systems Thinking™
Layout and Traffic Routing
Region 1 Region 2
Source POP3
POP1 Full LSP Mesh
in Core
POP4
Core LSP between
WRs in POPs 1 & 5
POP2
POP5
Region 3 Destination
Region 4
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 23

24. Metanoia, Inc.
Critical Systems Thinking™
Sprint (FON): A Bit of History
 A century of evolution ...
1899: Brown Telephone Co., Abilene, KS
1976: United Telephone, Kansas City, MS, 3.5M customers
1984: Began building first, all-digital, US-wide, fiber-optic network
1986: United + GTE merge LD subsidiaries  US Sprint
1992: United buys GTE’s stake, renaming co. to Sprint Corp.
2002: ~$23B revenue, 23M customers, 70 countries, 80,000 employees
 110,000+ route miles in the long distance (LD) network
 34,000+ in US, 78,000+ in rest of the world
 Transport infrastructure common to voice, ATM, & IP network
 Provides considerable leverage, as we’ll see later
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 24

25. Metanoia, Inc.
Critical Systems Thinking™
Sprint (FON): IP Network Timeline
Service via native IP bbone
OC-12 4F-BLSR deployed OC-192 TAT links
Cisco GSR tested/deployed OC-192 in Europe
All bbone routers
GSR12416s
All DS3 IP network
DEC FDDI Gigaswitch POP Deploy OC-48
POS over DWDM
Work with Cisco for next
router for OC-3 backbone
GigaPOP bbone
First IXC w/ Internet GSR in POP Deploy GSR12016 Expand to Asia,
svc. on T1 network OC-3 WAN South America
'92 '93 '95 '96 '97 '98 '99 '01 '02
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 25

26. Metanoia, Inc.
Critical Systems Thinking™
SprintLinkTM IP Backbone Network
Represents connectivity
only (not to scale)
110,000+ route miles
(common with Sprint LD network)
400+ POPs 19+ countries
5 continents 30+ major intl. cities
(reach S. America as well) Courtesy: Jeff Chaltas
©Copyright 2002, Sprint Public Relations 26
All Rights Reserved Modern Carrier Strategies for TE

27. Metanoia, Inc.
Critical Systems Thinking™
SprintLinkTM IP Network
 Tier-1 Internet backbone
 Customers: corporations, Tier-2 ISPs, univs., ...
 Native IP-over-DWDM using SONET framing
 4F-BLSR infrastructure (425 SONET rings in network)
 Backbone
 US: OC-48/STM-16 (2.5 Gb/s) links
 Europe: OC-192/STM-64 (10 Gb/s) links
 DWDM with 8-40 λ’s/fiber
 Equipment
 Core: Cisco GSR 12000/12416 (bbone), 10720 metro edge router
 Edge: Cisco 75xxx series
 Optical: Ciena Sentry 4000, Ciena CoreDirector
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 27

28. Metanoia, Inc.
Critical Systems Thinking™
SprintLinkTM IP Design Philosophy
 Large networks exhibit arch., design & engg. (ADE) non-linearities not
seen at smaller scales
⇒ Even small things can & do cause huge effects (amplification)
⇒ More simultaneous events mean greater likelihood of interaction (coupling)
∴ Simplicity Principle: simple n/wks are easier to operate & scale
⇒ Complexity prohibits efficient scaling, driving up CAPEX and OPEX!
 Confine intelligence at edges
 No state in the network core/backbone
 Fastest forwarding of packets in core
⇒ Ensure packets encounter minimal queueing
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 28

29. Metanoia, Inc.
Critical Systems Thinking™
SprintLinkTM Deployment Strategy
L2 failure detection triggers
switchover before L3 converges
Parallel links 50% utilization
under normal state
A 4
Z
1
3
2 SONET framing for
error detection
Line Line
Card Card
SONET
IP Data
Overhead
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 29

30. Metanoia, Inc.
Critical Systems Thinking™
SprintLinkTM Design Principles
 Great value on traffic measurement & monitoring
 Use it for
 Design, operations, management
 Dimensioning, provisioning
 SLAs, pricing
 Minimizing the extent of complex TE & QoS in the core
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 30

31. Metanoia, Inc.
Critical Systems Thinking™
Sprint’s Approach to Monitoring
Backbone
Links
Backbone
Peering Links
Router
Probe
Access Access Access
Router Router Router
Analysis platform located at
Sprint ATL, Burlingame, CA Customers Customers Customers
Adapted from [Diot99]
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 31

32. Metanoia, Inc.
Critical Systems Thinking™
Sprint Approach to TE
Aim: Thoroughly understand backbone traffic dynamics
Answer questions such as:
 Composition of traffic? Origin of traffic?
 Between any pair of POPs
 What is the traffic demand?
 Volume of traffic?
 Traffic patterns? (In time? In space?)
 How is this demand routed?
 How does one design traffic matrics optimally?
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 32

33. Metanoia, Inc.
Critical Systems Thinking™
Obtaining Traffic Matrices between POPs
DA SA POP1 B
1.1.1.1 POP2
A
IP Packet Destination
Subnet
D
POP3 POP4 1.1.1/24
C
Traffic
Exit
DA Exit POP POP
# pkts Protocol Matrices
Build
Table POP1
Combine data,
1.1.1.1 POP4 POP2 Obtain matrix
POP3 By
Protocol City A City B City C City D
POP4
City ACity A City B City C City D
By Time City A B
City
of Day City B C
City
City C D
City
©Copyright 2002,
City D
All Rights Reserved Modern Carrier Strategies for TE 33

34. Metanoia, Inc.
Critical Systems Thinking™
A Peek at a Row of a Traffic Matrix
To Backbone
Peer 2
Peer 1
ISP
Web 2
Web 1
Sprint POP under study
Distribution of aggregate access
traffic across other POPs in the Sprint
backbone
Adapted from [Bhattacharya02]
Summary of Data Collected
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 34

35. Metanoia, Inc.
Critical Systems Thinking™
Applications of Traffic Matrices
 Traffic engineering
 Verify BGP peering
 Intra-domain routing
 SLA drafting
 Customer reports
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 35

36. Metanoia, Inc.
Critical Systems Thinking™
Acknowledgements
 Thomas Telkamp, Global Crossing
 Robert J. Rockell, Jeff Chaltas, Ananth Nagarajan, Sprint
 Steve Gordon, Cable and Wireless
 Jennifer Rexford, Albert Greenberg, Carsten Lund, AT&T Research
 Wai-Sum Lai, AT&T
 Fang Wu, NTT America
 Arman Maghbouleh, Alan Gous, Cariden Technologies
 Yufei Wang, VPI Systems
©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 36