As a global organization, Fastly carefully selects and deploys POP locations to service the greater audience of the Internet. Fastly currently has 52 global POPs across the Internet, 13 of which are located in the Southern Hemisphere. Another 3 are outside North America, Europe, and Asia. During this talk, VP of Infrastructure Tom Daly will share our experience in building Fastly's network of POPs south of the equator, where, in some cases, the Internet we know here in San Francisco, is much different. Tom will explore the physical datacenter infrastructure, network topology, and network policy that pose of unique challenges when operating in these parts of the world.
3. BUILDING THE SOUTHERN HEMISPHERE OF THE INTERNET
13x POPs; 2Tbps Edge; 21x IXPs
● Chile x1
● Argentina x1
● Brazil x3
● South Africa x2 ● Australia x4
● New Zealand x2
5. • IP Transit
• Interconnection
• Colocation
• Power
• Cooling
Infrastructure
Availability
6. BUILDING THE SOUTHERN HEMISPHERE OF THE INTERNET
Networks, Datacenters & Exchanges
Country Population HE.net ASNs PeeringDB DCs PeeringDB IXPs People/DC
Argentina 44,688,864 914 14 25 3,192,062
Australia 24,772,247 2,494 84 21 294,908
Brazil 210,867,954 6,188 122 33 1,728,426
Chile 18,197,209 301 22 9 827,146
New Zealand 4,749,598 617 31 8 153,213
South Africa 57,398,421 527 18 6 3,188,801
United States 326,766,748 27,162 250 117 1,307,067
Sources:
• Retrieved from http://worldpopulationreview.com/
• Retrieved from http://bgp.he.net/
• Retrieved from http://www.peeringdb.com/
7. BUILDING THE SOUTHERN HEMISPHERE OF THE INTERNET
● Chile, Argentina, Brazil
○ Fledgling Carrier Neutral Ecosystems; otherwise Carrier-centric
● South Africa
○ Carrier Neutral; yet few choices
● Australia
○ Diverse Carrier Neutral Ecosystems
● New Zealand
○ Carrier-centric options.
Datacenter Models
8.
9.
10.
11.
12.
13.
14. BUILDING THE SOUTHERN HEMISPHERE OF THE INTERNET
● Chile, Argentina, Brazil
○ Broadly distributed national or regional exchanges
○ Layer 2 (optional MLPE) and Layer 3 (forced MLPE) models
○ Government subsidized; partial divestitures
● South Africa
○ Limited Local Datacenter or Metro Exchanges
○ Private Operators
● Australia / New Zealand
○ Diverse Local Datacenter or Metro Exchanges
○ Private Operators
IXP Models
15.
16. • Cost of Space / Power
• Cost of IP Transit
• Cost of IXP
Economics
22. Building in South America
• Start in Miami
• Backup in New York
• São Paulo
23. Building in South America
• Start in Miami
• Backup in New York
• São Paulo
• Diversity in Rio de Janeiro
24. Building in South America
• São Paulo
• Diversity in Rio de Janeiro
• Santiago / Buenos Aires
• Rio de Janeiro now provides
provides performance, not
diversity
• Miami and New York become
become irrelevant backup
paths
25. Source City São Paulo
New York 140ms
Miami 110ms
Buenos Aires 51ms
Santiago 32ms
São Paulo ~10ms
29. BUILDING THE SOUTHERN HEMISPHERE OF THE INTERNET
● Purchase a 10GE Port
Burst Capped Capacity
10G
30. BUILDING THE SOUTHERN HEMISPHERE OF THE INTERNET
● Purchase a 10GE Port
○ Provide required 20% commit to cap
Burst Capped Capacity
2G
31. BUILDING THE SOUTHERN HEMISPHERE OF THE INTERNET
● Purchase a 10GE Port
○ Provide required 20% commit to cap
○ Carrier also forces 3:1 burst cap
Burst Capped Capacity
2G
4G
32. BUILDING THE SOUTHERN HEMISPHERE OF THE INTERNET
● Purchase a 10GE Port
○ Provide required 20% commit to cap
○ Carrier also forces 2:1 burst cap
○ Effectively rate limiting port to 4GE
● Circuit now limited to 6GE
○ Additional capacity planning attention required
Burst Capped Capacity
2G
4G
4G
34. Common Fates
• Shared submarine
capacity between IP
providers
• Double/triple fault
conditions result in
segmentation
• Mitigate through review
and audit
35. *** Dates, Locations, Capacity subject to change.
Outlook
• Invest in performance
• Enable growth through capacity
• Respond to customer needs
Zoom down to South America, Africa, Australia, and New Zealand.
13x POPs; 2T edge; 21 IXPs.
25% of POPs and ~1/3rd of IXPs
Take Aways:
Understand Fastly’s approach to deploying infrastructure
Be able to identify who some of the key players outside the US to which traffic flows.
Understand the topology of the region, constraints we encounter, and how we overcome those constraints to maintain quality.
Available infrastructure is different, when compared to North America, Europe, or Asia.
Mapping: Population, Networks, Datacenters, and IXPs.
Where there are many options, there are clusters of viable offerings. Finding the right mix of connectivity at a common datacenter is critical.
These are the methods we whittle down choices.
AR, CL, BR: Limited choices: Fledgling Carrier Neutral Ecosystems; otherwise Carrier-centric
ZA, AU: Broad options by percentage: ZA and AU have carrier neutral options; though sometimes limited by local market.
NZ: Carrier centric options.
Pick DCs with carrier density.
Numbers ultimately reflect the market behaviors.
Methodology:
Gather flows to top 100 destination ASNs sorted by descending volume; sourced by geolocation data.
Normalize data for common ASNs; e.g. AT&T and AT&T Mobile; Charter/TWC/Brighthouse Merger.
Identify count of networks absorbing 80% of the traffic.
Large operators acquiring regional brands into national backbones.
Brazil: Claro, Telefonica, Telemar, Telecom Italia Mobile
Chile: Claro, Telefonica, Entel, VTR
Argentina: Telecom Argentina (Telecom Italia), Telefonica, Claro, Giga
Large operators acquiring regional brands into national backbones.
Brazil: Claro, Telefonica, Telemar, Telecom Italia Mobile
Chile: Claro, Telefonica, Entel, VTR
Argentina: Telecom Argentina (Telecom Italia), Telefonica, Claro, Giga
South Africa: Large incumbents, but with strong regional competition.
Australia: Band of 4x: Telstra, Optus, TPG, and unknown seat
New Zealand: Band of 2x: Spark and Vodafone, but Callplus is a large destination. Band of 3x?
Look for valid in country IXPs - multiple participants.
Traffic gravity shifts, users centrally served.
Miami and New York catch load and network events.
Rio creates local diversity.
NY and MIA fade.
Local traffic ecosystem.
Role of Rio changes as Bnuenos Aires comes online – Rio now performance overlay, GRU and EZE form core.
Ultimately – we do this to impact throughput.
Van Jacobsen and Bob Braden work in 1988 in RFC1072 defining characteristics and mitigation strategies of Long Fat Networks.
As much as 10x more candidate throughput (depending on packet loss) by lowering latency.
Not as simple as planning a move of servers from Miami to Sao Paulo.
That opportunity for more throughput means we need to be highly cognizant of the networks around us – e.g. bursting caps.
Not able to run at line speed.
Must read the fine print.
Often, burst caps come into play on submarine systems where order constraints exist.
In addition, you have to look at the common fates.
Overcome the constraints:
Careful research
Understand capacity and growth
Deploy to grow alongside our customers.
2018 activities south of the equator to date; more to come.