Why Not 100GbE as Service
~ JPIX Perspective ~
Yoshiki Ishida, Takejiro Takabayashi
JPIX
(Japan Internet eXchange Co., Ltd.)

What Motivates 100GbE Service?
• Customer (Member) Requirement
• Traffic
• Technology & Operation
• Cost

Traffic Background(1) ~ Europe ~
• IX points Traffic are
keep growing
• The peak traffic of three
global IX points are over
2 tera bit per second.
– AMS-IX(NL)
– DE-CIX(DE)
– LINX(UK)
https://ams-ix.net/technical/statistics
https://www.de-cix.net/about/statistics/
https://www.linx.net/pubtools/trafficstats.html
Source:
Captured date: Jan. 28, 2015

100GbE in Europe ~ No Surprise ~
• 100G services are available at these major IX
Points
– AMS-IX and Limelight Networks Field-Test Brocade
100 Gigabit Ethernet Technology to Stay Ahead of
Internet Growth - June 2011 (*1)
– DE-CIX Completes 100G Optical Layer Build for New
Flagship Apollon Platform - May 2013 (*2)
– BT expands its 21st Century Network as the London
Internet Exchange deploys 100 Gigabit Ethernet across
their leading edge IP/MPLS Internet Exchange - July
2012 (*3)

Traffic Background(2) ~ ASIA ~
http://www.hkix.net/hkix/stat/aggt/hkix-aggregate.html
http://www.mfeed.co.jp/jpnap-tokyo-i/traffic.html
Source:
• HKIX ≈400Gbps peak
– 100GbE available
• JPNAP ≈400Gbps peak
– 100GbE available
Captured date: Jan. 28, 2015

Traffic Background(3) JPIX
• JPIX’s traffic is keep
growing ≈300Gbps
• Some our customers,
especially Hypergiants
and FTTH providers, use
multiple 10G Interfaces.
And finally…
http://www.jpix.ad.jp/en/technical/traffic.html
Source:
Captured date: Jan. 30, 2015

Press Release - July 9, 2014

Press Release - July 9, 2014
• Japan Internet Exchange Co., Ltd., (hereafter referred to as “JPIX”) an IX
provider that has a largest number of customers in Japan, will launch a new IX
port service, 100 Gigabit Ethernet Port, in July. Brocade Communications
Systems, K.K., (hereafter referred to as Brocade) announces that it provides the
Brocade MLXe Series, deployed as switch to support the new service.
• Dwango Co., Ltd., a company that provides Japan’s largest video sharing service,
“niconico”, and a wide range of entertainment contents and distributes vast
amounts of content, will become the first user of the service.
• The equipment for the new IX port service, 100 Gigabit Ethernet Port, are the
Brocade MLXe Series provided by Brocade.
• Brocade MLXe Series, which include a 100 Gigabit Ethernet (GbE)-enabled
model released for the first time in the industry, were selected to equip the
100 GbE Port Service. As a result of Brocade’s feasibility testing for various
interfaces, it was proven that the 100 GbE Port Service can be provided
through main interfaces, including 10×10, 100G Base-LR4, and 100G Base-ER4.
• JPIX continues to strive to provide effective services and quality leading to
stable operation, which can satisfy customers, such as ISPs and content
providers, in order to respond to continuously increasing Internet traffic.

100GbE motives PROS & CONS
• Higher Speed
– Faster the better
• Less operation
– Less LAG, Simplex
• Less consideration
– Once you upgrade, you
are free from concerning
upgrading the
bandwidth
• Opens Opportunity
• Stability/Reliability/RISK
– 100G affects too much
traffic
• Need experience…
• COST
– CAPEX/OPEX may higher
than 10x 10G
• Opportunity Loss
PROS CONS

Issues for deploying 100G Service
~ burdens ~
• Upgrading the IX Architecture
– Needed to upgrade our BIG 100GbE supported switch act
as spine and 40GbE supported switch act as leaf switch
• Verifying 100GbE Interface
– Optical Switch testing needed
• What if one wavelength drops optical power…
• Selection of the Optical Interface and the Optical
Module
– Shall we wait or go?
• 100G Ethernet Tester
– What can we do about it…? Is it CFP or Backplane…?

Upgrading IX Architecture
• Distributed Architecture
• Spine and Leaf Architecture

Verifying 100G Interface
100G Ethernet uses Multiple Waves on a Fiber
Verifying 100G Interface for our Architecture
( eg. Optical A/B Switch)
*4

Optics and Optical Module
• Optics
– LR4(IEEE)
• 10Km SMF/4x25Gbps
– ER4(IEEE)
• 40Km SMF/4x25Gbps
– 10x10(10x10MSA)
• 2KmSMF/10x10Gbps
• 10KmSMF/10x10Gbps
• Optical Module
– CFP
– CFP2/CFP4
– CPAK
*5
*6
CFP image courtesy of Finisar

100G Ethernet Tester
• Purpose for 100G
Ethernet Tester
– Isolate the cause of
failure on a 100GbE port
– Sanity check of a CFP
– Health Check of IX
Switch
– Throughput
Test(RFC2544)
• Specification
– Hardware
• Throughput Test with
Multiple 100GbE (at least
2) Ports
• No limitation for CFP
vendors
• Portability and Battery-
powered
– Software
• Multiple Stream
• CFP Check
• ReportingTester IX Switch
CFP
CFP
CFP
CFP
courtesy of EXFO

Technology Myth & Truth
• 10x10 MSA is very attractive; cost effective
– Really?
• With CFP2 10x10MSA is unsupported… can be dead-stock
• Big Chassis that supports 100GbE can be in the
middle
– Can’t live with old GbE interface… will drop packets… not
enough buffer…, can’t be in the middle
• Vendor lock-in…
– As always
• Timing
– If we wait for another year, we could have install double
or quadruple capacity line-card…

Cost Myth & Truth
• 100GbE is getting less expensive
– Yes it has. In our case, we see the CFP cost dropped by
50%...
• Comparison year 2012 and 2015
• But Still:
– For us, it cost 21 times higher cost…. 21X
• 21x SFP+ = 1x 100G CFP PRICE…
• Is it really less expensive?

Why not 100G now?
Shall we
deploy 100G
Now?
New modules
coming
There may be
less
expensive
better
product in
market
Expensive
interface you
buy today
can be legacy
Let’s just wait
Vicious Circle

Summary
• 100G Ethernet is still “on going” technology.
– Indecisiveness of making investment decision
• Technology(Optics/Wave Multiplexity)
• Cost…
• Cutting Edge Players have already started
adopting to 100G because of the necessity
– Hypergiant, IX, R&E Community
– They needed it, we needed it
• It works fine
• When will do?
＞ Timing is everything ＜ ~ mark your calendar ~

Reference
*1
http://newsroom.brocade.com/press-releases/ams-ix-and-limelight-
networks-field-test-brocade-1-nasdaq-llnw-0765435#.U-rzNGMTNW_
*2
http://press.de-cix.net/press-releases/pressemitteilung/article/de-cix-
completes-100g-optical-layer-build-for-new-flagship-apollon-platform/
*3
https://www.linx.net/publicity/2012releases/pr2012-04.html
*4
http://www.ntt-at.com/product/optical-switch/
*5
http://www.cisco.com/web/JP/product/hs/optical/ons15454/prodlit/whit
e_paper_c11-727398_031813.html
*6
http://www.epsglobal.com/downloads/Finisar/Finisar40G_100G_Presenta
tion.pdf