Internet2 will be aggressive in its deployment of its Innovation Platform in order to allow its members to capitalize on the groundswell of support for high-speed software defined networking - summer of networking, Chris Robb, Indiana University/Internet2

1. Chris Robb, Indiana University/Internet2
HISTORY OF 100G AND INTERNET2

2. Topics
• Introduction
• Technical Background
• History of Internet2’s 100G Deployment
• Current Progress
• Lessons Learned

3. Who’s this guy?
• Why, I’m Chris Robb!
• Director of Engineering and Operations for Internet2
• But I work at IU. It’s all horribly confusing.
• Started in the trenches of UITS computer lab support
• Joined campus networking division as a programmer in 1998
• GlobalNOC engineer in 2001. First assignment: Hawaii!
• Managed the 2006 Internet2 transition to Level3
• Subcontracted to Internet2 in 2008
• My day consists of video conferencing, lunch, then more video
conferencing- usually in that order
• I also occasionally haunt the halls of the CIB

4. What’s Internet2?
• “Internet2 is an advanced networking consortium led by the
research and education community”
• Formed in 1996
• More than just a network
– Middleware
– Net+
– Security
– Research
– Measurement
• Based in Ann Arbor with offices in Washington DC, New York and
(soon) California

5. Technical Background

6. National Backbone Organization
• Fiber
• Optical Equipment
• Routers and Switches
• Network Operations Center

7. Dark Fiber
• Someone has to lay fiber in the ground
• Commonly laid along railway lines and interstate roads
• A lot of investment in the late 90s
• Multiple conduits installed
– Ability to blow fiber later
• Multi-strand cable (144ct is common, per conduit)
• Hut spacing defined at installation time
• Huge investment
– Nationally cost prohibitive
– Regionally expensive, but doable
• Sometimes not very well documented

8. Cable conduit is laid

9. Fiber is blown through empty conduit
Photos courtesy of Steve Wallace

10. Access to fiber via handholes

11. Fiber is terminated in huts to
house optical amplification
equipment

12. Who can buy fiber?
• Depends on when you’re asking, who you’re asking, and who you are
• Early 2000s: plentiful supply of fiber (and companies that are no longer
around)
• Now: not so much
– Consolidation
– Not a lot of investment in fiber (exception: high frequency trading)
• Mostly carrier swaps and internal use
• Sold as an Indefeasible Right to Use (IRU), typically on a 20 year term
• Expensive!
• Activate Google Earth

13. OK, we have some fiber….
• What do you put on the fiber once it’s laid?
– Too expensive to just put a single signal on each fiber
• Client interfaces can only go so far
– 10GBase-SR – up to 300m (depending on fiber type)
– 10Gbase-LR – 1310nm – 10km
– 10GBase-ER – 1550nm – 40km
– 10GBase-ZR – 1550nm – 80km
• Put a switch every 80km?
– You saw those huts! Too little space.
– Who wants to manage that many devices? Too complex.
– There are only so many coal plants in the US. Too much power.

14. DWDM Optical Equipment
• Multiplex!
• Convert signals from client equipment (routers & switches) to specific
wavelengths of light (ITU grid)
• Combine the different wavelengths onto a common fiber on one end,
and separate them on the other end
• Uh oh! Light doesn’t travel far enough!
– Amplify!
– Periodically shoot the signal through erbium doped fiber spools that
amplify the signal
• The noise! The noise! Amplification also amplifies noise.
• Periodically regenerate the signal
• Break down the analog signal back into its digital form, then send it
out again as a newly cleaned up analog signal

15. DWDM History
• Early WDM (late 80s)
–Two widely separated wavelengths (1310, 1550nm)
• “Second generation” WDM (early 90s)
–Two to eight channels in 1550 nm window
–400+ GHz spacing
• DWDM systems (mid 90s)
–16 to 40 channels in 1550 nm window
–100 to 200 GHz spacing
• Next generation DWDM systems
–64 to 160 channels in 1550 nm window
–50 and 25 GHz spacing
15

16. ITU Wavelength Grid
1530.33 nm 1553.86 nm
0.80 nm
195.9 THz 193.0 THz
100 GHz
• ITU-T grid is based on 191.7 THz + 100 GHz
• It is a standard for laser in DWDM systems
Freq (THz) ITU Ch Wave (nm) 15201/252 15216 15800 15540 15454
192.90 29 1554.13 x x x x x
192.85 1554.54
192.80 28 1554.94 x x x x x
192.75 1555.34
192.70 27 1555.75 x x x x x
192.65 1556.15
192.60 26 1556.55 x x x x x

17. Modern Optical Components
• Fiber
• Erbium Doped Fiber Amplifier (EDFA) (“Amp”)
• Optical Add/Drop Multiplexor (“OADM”)
• Reconfigurable Optical Add/Drop Multiplexor (“ROADM”)
• Tributary (“Trib”) card
• And a few more concepts
– Directionless
– Colorless
• To the whiteboard!!

18. DWDM Components
1
15xx 1...n
2
3
Transponder
Optical Multiplexer
1
1
2 1...n
2
3
3
Optical De-multiplexer
Optical Add/Drop Multiplexer
(OADM)

19. More DWDM Components
Optical Amplifier Optical Attenuator
(EDFA) Variable Optical Attenuator
Dispersion Compensator (DCM / DCU)

20. Optical Amplifier
Optical Add/Drop

21. Transmission Effects
• Attenuation:
– Reduces power level with distance
• Dispersion and nonlinear effects:
– Erodes clarity with distance and speed
• Noise and Jitter:
Leading to a blurred image

22. National Routed Network
• How do you get millions of people connected to this capacity?
• Routers direct your data transfers around the country
• Generally large boxes at the national level ($500K-$1M each)
– Higher interface speeds (10Gbps+)
– Not a lot of edge features (traffic shaping, policing, etc.)

23. Internet2 Network Topology
• 1998-2003
Evolution
– Cisco GSR routers
– Qwest OC-12/OC-48 backbone
– Average Connector Speed: 450-600Mbps
• 2003-2007
– Juniper T640 routers
– Qwest OC-192 backbone
– Average Connector Speed: 1-4Gbps
• 2007-2009
– Juniper T640/T1600 routers
– Level(3) OC-192/10GigE backbone
– Average Connector Speed: 8.3Gbps
• 2009-2011
– Juniper MX960 Routers
– Level3 Multiple 10GigE Backbone
– Average Connectors Speed: 10+ Gbps
• 2011 and beyond
– Juniper T-1600 Routers
– Owned Infrastructure with 100GigE Backbone

24. Potential Edge Bandwidth Growth
260
2 4 8 Gbs
Total Potential Gbps from Connectors and Peers
220Gbps
200
150
Cisco GSR Juniper T640 Juniper T640
Qwest Footprint Qwest Footprint 124Gbps Level(3) Footprint
2.5G Backbone 10G Backbone 10G+ Backbone
100
50 50Gbps
1.2Mbps
4.8Gbps
3.4Gbps
14Gbps
0
Jan- 02 Jul- 02 Jan- 03 Jul- 03 Jan- 04 Jul- 04 Jan- 05 Jul- 05 Jan- 06 Jul- 06 Jan- 07 Jul- 07 Jan- 08 Jul- 08 Jan- 09
Time

25. Measured Perabytes into Internet2 per
Month
12 11.2
10.3
10
8
6
4
2
0

26. Internet2 Network Backbone Speeds (in Gbps) 100
100
90
80
70
60
50
40
30
30
20
20
10
10
0.6 2.4
0
1997 1998 2002 2008 2010 2011+

27. Internet2’s McLean, VA POP

28. Internet2’s McLean, VA POP

29. History of 100G Deployment

30. A Community Defines Its Future
• In June 2009, Internet2 and the R&E community
author the “Internet2 Architectural Directions”
Document….
• Multiples of 10GigE will be the primary transport to
Regional and State Networks over the next 3-5 years.
– 10G cost low compared to the cost of 40 or 100G
– Multiple large sub-10G flows the norm
• Internet2 Network access will be divorced from
physical interface speeds and available for
apportionment across the network
– Flexibility for connectors an important success factor.

31. Architectural Principles
• Native 100GigE at the optical layer is an
important technology to adopt today
– Take advantage of current opportunities to lay the
groundwork for future expansion.
• Collapsing Layer2 and Layer3 services onto a
single delivery platform is an important step
toward the hybridization of the network
– Reduce overall operating expenses to the Connectors
– Candidate technologies include MPLS L2
VPNs, Layer2 Ethernet VLANs and Virtual Private
LAN Service (VPLS).

32. Architectural Principles
• The Internet2 IP and Layer2 Networks
need a migration path to 40G and 100G in
the next few years
– Backbone must be able to efficiently handle
multiple simultaneous 7-10 G flows and individual
flows >10Gbps
• The Internet2 Network emphasis should
be on additional services and
technologies that will drive transport
bandwidth requirements
– The use case for the network drives the
technology of the network.

33. Architectural Principles
• Internet2 will coordinate with the Regional and
State Network partners to determine the most
optimal node quantity and locations
– Offer a flexible partnership with the connectors.
– Create more options for connections.
• As mission-critical applications become more
integral to the Regional cost-recovery
model, the Internet2 Network must focus on
enhanced redundancy where needed
– Many recent services and uses of the network require
increasingly reliable/redundant/resilient connectivity

34. Architectural Principles
• The Internet2 Network will continue to
be instrumented and operated in a
transparent fashion that supports the
end-to-end model
– The more information that is available about
the network the better everyone understands
the need for and requirements of the network.

35. Why is 100G So Important?
• Edge speeds are outpacing backbone speeds
– mid 2009 potential edge bandwidth was 17x the edge
bandwidth of 2001
• Traffic is steadily growing
• Large flows are becoming increasingly
commonplace
• 100Gbps will scale and become economically
cost effective
• Drive to innovate!

36. History of 100G and Internet2
• June 2009: Internet2 staff began polling the
community for set of technical principles to guide
staff efforts over the next several years
– Resulted in the “Internet2 Architectural Directions”
document that specifically called out 100Gbps
networking as a strategic direction
– AOAC approval of document on October 5th, 2009
• August 2009 Internet2 released an RFI to the
industry that sought industry feedback on a
100Gbps optical platform

37. Funding History
• In late summer of 2009, the NTIA released a
Notice of Funding Availability (NOFA) for
broadband funding
– Broadband Technologies Opportunities Program
(BTOP)
– $7.2B in funding
• Internet2 decided not to respond to the first
round of funding so as not to compete with the
regional networks that were applying for funds

38. Funding
• Second round of BTOP funding in early 2010
• Internet2 submits a joint proposal with NLR, IU and the
Northern Tier Networking Consortium (NTNC)
– Obtain a partial national footprint and equip it with an
optical network
– Leverage the existing NLR and Internet2 optical paths
– Upgrade the NTNC network between Chicago and Seattle
– Upgrade all Internet2 routers to be 100G capable
– Upgrade the TR-CPS routers to newer hardware
– Create a “low-latency” layer of small routers at 27 sites

40. Award and Change of Plans
• Internet2/NLR/NTNC were selected as the only
“national middle mile” network in the BTOP
program in late spring 2010
• During the due diligence phase, NLR opted to
step out of the partnership
– Network topology was refactored to a national
footprint to cover those sections of the country that
had been covered by NLR’s in-kind contribution
• Project officially started in July, 2010

41. Seattle
Spokane
Bozeman
Miles City
Portland Missoula Bismarck
Billings Fargo
Eugene Dickinson Albany
Boise
Minneapolis Boston
Madison Buffalo
Eau Claire Chicago Detroit
New York
Milwaukee
Salt Lake City
Reno Denver Cleveland Philadelphia
Sacramento Pittsburgh
Indianapolis Cincinatti
Sunnyvale Ashburn Washington DC
St. Louis
Kansas City
Las Vegas Louisville
Raleigh
Nashville
Charlotte
Memphis Chattanooga
Albuquerque Tulsa
Los Angeles San Diego
Phoenix
Jackson Atlanta
El Paso
Dallas
Jacksonville
Optical Add/Drop Facility
San Antonio
Optical Regeneration Facility Baton Rouge
Houston

42. Preliminary Optical RFI
• February 2010: meetings were scheduled with several
companies to review their offerings
– Evaluation was based on a number of different factors:
• Optical properties
• Control plane development
• Physical properties
• Road map for product availability and others
– From this review it was determined that there were optical
vendors with viable options for building the U.S. UCAN
network
– Accordingly they were given the option to respond to the
RFP for optical equipment in September 2010

43. RFP Issuance
• Based on this research potential vendors were identified:
– Cisco
– Infinera
– Ciena
• Internet2 released the U.S. UCAN Optical Network Request
for Proposal (RFP) on September 13, 2010, in collaboration
with Indiana University
• Vendor responses were due on September 22
– The response due date was extended three times:
• on September 17 Internet2 granted an extension to September 29
• on September 23 Internet2 granted an extension to October 5th
• on October 5 Internet2 granted an extension to October 9th

44. Review Team
• At the September NTAC meeting, Internet2
solicited volunteers from the community to
review the RFP responses
• A team of community members, Internet2
staff and IU NOC staff was assembled to
evaluate the responses

45. Team Outcome
• All three responses were outstanding and each vendor
included compelling and creative support for this
community effort.
– Infinera will provide an upgrade path for the Northern Tier
Networking Consortium on their next generation platform
– Ciena will provide the national footprint on its Activeflex
(formerly OME) 6500 platform
– Cisco is being engaged on a variety of fronts for parallel
efforts
• Recommendation presented to the AOAC on
November 10th
– AOAC endorsement and forward to Internet2 senior
management for final approval

46. Ciena Platform
• 100Gbps capable 80-channel DWDM system
– 100G cards shipping today
• ROADM-based solution at most or all
add/drop facilities
• Directionless capability in metro areas
• Non-Dispersion-Shifted approach provides
economical approach that reduces CAPEX
• Compact, scalable footprint that adapts to the
changing needs of our community

47. Current Progress

48. Upgraded Internet2 Infrastructure benefits
• Network Infrastructure
– Dark Fiber on a national footprint
– Wave Capacity
• Optical equipment to light new fiber with 100Gbps-capable equipment
• Upgrade of Northern Tier Networking Consortium Infinera equipment
to be 100G-capable
– 100Gbps IP Backbone
• 100G and 10G routers for R&E network
• 10G routers for TR/CPS
• Enhanced Services
– Increased connectivity to Commercial Exchange Points
– Regional interconnects
• Research Opportunity
– Stronger collaboration
– Enhanced capability
48 – 6/17/2012, © 2009 Internet2

49. Internet2 Optical Network Topology
Seattle
Spokane
Bozeman
Miles City
Portland Missoula Bismarck
Billings Fargo
Eugene Dickinson Albany
Boise
Minneapolis Boston
Madison Buffalo
Eau Claire Chicago Detroit
New York
Milwaukee
Salt Lake City
Reno Denver Cleveland Philadelphia
Sacramento Pittsburgh
Indianapolis Cincinatti
Sunnyvale Ashburn Washington DC
St. Louis
Kansas City
Las Vegas Louisville
Raleigh
Nashville
Charlotte
Memphis Chattanooga
Albuquerque Tulsa
Los Angeles San Diego
Phoenix
Jackson Atlanta
El Paso
Dallas
Jacksonville
Optical Add/Drop Facility
San Antonio
Optical Regeneration Facility Baton Rouge
Houston

50. Phase 1 Optical Build - COMPLETE

51. Phase 1 Progress – 100% Complete
• All fiber acquired and accepted
• All Optical Equipment Installed
• BER Testing complete
• System Commissioning
– EMS system installed and populated with nodes
– Internet2 NOC database population nearly complete
• First 100GigE IP circuit between New York and Washington DC
live and integrated into the Internet2 IP Network
• First Trans-Continental 100G configured and passing pings for
Internet2 Fall Member Meeting in early October

52. Phase 2 – Complete
52 – 6/17/2012, © 2009 Internet2

53. Phase 3 – January 2012 – July 2013
53 – 6/17/2012, © 2009 Internet2

54. Current Status
Seattle
Spokane
Olympia Bozeman
Miles City
Portland Missoula Bismarck
Billings Fargo
Eugene Dickinson Albany
Boise
Minneapolis Boston
Madison Buffalo
Eau Claire Chicago Detroit
New York
Milwaukee
Salt Lake City
Reno Denver Cleveland Philadelphia
Sacramento Pittsburgh
Indianapolis Cincinatti
Sunnyvale Ashburn Washington DC
St. Louis
Pueblo Kansas City
Las Vegas Louisville
Raleigh
San Luis Obispo
Nashville
Charlotte
Memphis Chattanooga
Albuquerque Tulsa
Los Angeles San Diego
Phoenix
Tucson
Jackson Atlanta
El Paso
Dallas
Jacksonville
Optical Add/Drop Facility
San Antonio
Baton Rouge
Route Fully Operational
Route Installed, Tested, Not Operational Houston
Route Pending

55. Will someone please think of the routers?
• Optical is great, but what about the routers?
• Well, that’s a tricky story.
• We’re at a crossroads
• Routers are expensive, big, power hungry monsters
• Did I mention expensive?
• We want them everywhere
• We want them cheap
• We want them small
• We want control
• We want them powerful
• Is there a white knight?

56. Openflow ?

57. History of implementation
• Faster
– 2002: One of the first 10G IP national backbones deployed on Juniper T640
platform
– Mid-2009: Architectural Directions document cites need for 100G IP backbone
– Mid-2010: $96M Broadband Technologies Opportunities Program (BTOP)
National Middle Mile Award
– Late-2011: First national 100GigE circuit lit between New York and Sunnyvale
• Smarter
– 2005: Hybrid Optical Packet Infrastructure (HOPI) network deployed in first
national trial of reconfigurable network
– 2007: Interoperable On-demand Network (ION) deployed at 21 locations around
the US
– 2011: DYNES campus deployment project begins pushing dynamic deployment
to the edge
– Mid-2011: Open Networking Foundation formed by a coalition of research and
industry members
– Late-2011: 5-node NDDI deployment and development of OSE software
57 – 6/17/2012, © 2011 Internet2

58. Why now?
• Demand from Internet2 Members to Innovate
• Equipment Refresh Cycle
• Available Funding
58 – 6/17/2012, © 2011 Internet2

59. Engagement Plan
• This is a very high-level restructure of the Internet2 network, so
it’s important that stakeholders at all levels participate in the
discussion
– Researchers
– Campuses
– Regionals
– International Peers
– Funding Agencies
– Governance Councils
– Technical Bodies
• Internet2 will be aggressive in its deployment of its Innovation
Platform in order to allow its members to capitalize on the
groundswell of support for high-speed software defined
networking
59 – 6/17/2012, © 2011 Internet2

60. Implementation possibilities
• Goal: Integrate SDN functionality into the Internet2 Network
– Preferred Goal: platform integration between two sets of services to
provide initial network effect and migration path
• A few different models to consider
• Nothing is set in stone, but we do have to move aggressively
60 – 6/17/2012, © 2011 Internet2

61. Separate Infrastructure: What we do today
61 – 6/17/2012, © 2011 Internet2

62. Combined Infrastructure: What we might do tomorrow
62 – 6/17/2012, © 2011 Internet2

63. Upgraded Layer2 Network
I2 Draft Topology
gcb 5/8/12
v.1
Seattle
Portland Missoula
Albany
Minneapolis
Boise
Boston
Sacramento
Salt Lake City Chicago New York 1
Cleveland
Pittsburgh New York 2
Sunnyvale Kansas City
Indy
Cincy
Denver
St Louis Washington DC
Tulsa
Nashville Raleigh
Phoenix Albuquerque
Los Angeles
El Paso Atlanta
Jackson
Dallas
Jacksonville
Baton Rouge
Houston
NDDI Node

64. Lessons Learned
• Pre-project
– Know your goals
– Know your limitations, then ignore them
– Community Input
• Negotiations
– Have a good lawyer
– Understand what you’re buying
– Budget a lot of time
• Remember: this is for 20+ years
• Implementation
– Documentation is critical
– Inventory management needs to be worked out ahead of time
– It isn’t plug and play
– Budget a lot for space and power

65. QUESTIONS?