Not merely landlocked, but often doubly landlocked Reliance on neighbors’ terrestrial connectivity is unavoidable The challenge: increase Internet diversity, reduce latency and cost

1. Central Asia: Internet Structure and Trends
Jim Cowie, Chief Scientist
@jimcowie / @DynResearch
Bishkek, Kyrgyz Republic
10 December 2014

2. / 2 @jimcowie / @DynResearch
NOTE: Some cities host multiple collectors. Cable Map credit: Telegeography
Active Measurement Infrastructure

3. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 3 @jimcowie / @DynResearch
Central Asia
Not merely landlocked,
but often doubly
landlocked
Reliance on neighbors’
terrestrial connectivity
is unavoidable
The challenge:
increase Internet
diversity, reduce
latency and cost

4. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 4 @jimcowie / @DynResearch
Top Sites: Kyrgyz Republic
1. Google.com 58ms, service from GGC in Voronezh
2. Youtube.com 65ms, service from Google Moscow
3. Elcat.kg <5ms, Bishkek
4. Mail.ru 66ms, Moscow
5. Facebook.com 204ms, Washington, DC
6. Odnoklassniki.ru 66ms, Moscow
7. Yandex.ru 67ms, Moscow
8. Vk.com 76ms, St Petersburg
9. Ts.kg <5ms, Bishkek
10. Ok.ru 66ms, Moscow

5. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 5 @jimcowie / @DynResearch
Top Sites: Kyrgyz Republic
1. Google.com
2. Youtube.com
3. Elcat.kg
4. Mail.ru
5. Facebook.com
6. Odnoklassniki.ru
7. Yandex.ru
8. Vk.com
9. Ts.kg
10. Ok.ru
11. Namba.kg
12. Wikipedia.org
13. Google.ru
14. Blive.kg
15. Torrent.kg
16. Google.kg
17. Amazon.com
18. Akipress.org
19. Adme.ru
20. Aliexpress.com
21. Vb.kg
22. Instagram.com
23. Alibaba.com
24. Yahoo.com
25. Bolshoyvopros.ru
26. Ebay.com
27. Twitter.com
28. Super.kg
29. Rutracker.org
30. Blogspot.com

6. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 6 @jimcowie / @DynResearch
Supply and Demand
Central Asian consumers,
businesses need to reach
content markets in
Russia, the United States,
and China.
How does traffic flow
today?

7. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 7 @jimcowie / @DynResearch
Kazakhstan exports
Russian transit to all
neighbors
Tajikistan and
Turkmenistan each have
two choices.
Afghanistan has 3.
China is still not a visible
participant
Choice is Important

8. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 8 @jimcowie / @DynResearch
‘Watersheds’
Central Asian consumers,
businesses need to reach
content markets in
Russia, the United States,
and China.
How might traffic flow
tomorrow?

9. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 9 @jimcowie / @DynResearch
‘Watersheds’
Russian connectivity:
most traffic heads
through Moscow en
route to Frankfurt,
London, Stockholm
Southern routes
through Ukraine are
also available (for the
moment)

10. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 10 @jimcowie / @DynResearch
‘Alternative Watersheds’ 1
Balkan connectivity:
Azeri, Armenian, Iranian
paths join east-west
Turkish pipeline routes for
fiber via Ankara, Istanbul
Black Sea routes from
Baku-Tbilisi-Poti-Varna-
Sofia avoid Istanbul

11. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 11 @jimcowie / @DynResearch
‘Watersheds’
Submarine connectivity:
major cables transiting
the Suez en route to
European markets
Iran, Pakistan have
excellent connectivity
to UAE, Oman (SMW3,
SMW4, FLAG, etc.)
‘Alternative Watersheds’ 2

12. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 12 @jimcowie / @DynResearch
‘Watersheds’
Chinese connectivity:
challenging from both a
latency, reliability
perspective
China has its own problems
reaching world markets
reliably
Still, a potential source of
diversity
‘Alternative Watersheds’ 3

13. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 13 @jimcowie / @DynResearch
‘Watersheds’
Combine these 4
watersheds and
simplify. What
emerges, strategically?

14. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 14 @jimcowie / @DynResearch
‘Watersheds’
Key transit
cities:
• Moscow
• Istanbul
• Dubai
• Beijing

15. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 15 @jimcowie / @DynResearch
‘Watersheds’
Key transit
cities:
• Moscow
• Istanbul
• Dubai
• Beijing
Baku and Tehran
are the natural
hingepoint

16. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 16 @jimcowie / @DynResearch
Four Paths to Diversity
We still need to incorporate either
• The Iranian terrestrial network, or
• A route crossing the Caspian Sea,or
• An Afghan-Pakistani alternative
path to the sea
Baku and Tehran
are the natural
hingepoint
Key transit
cities:
• Moscow
• Istanbul
• Dubai
• Beijing

17. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 17 @jimcowie / @DynResearch
Iran’s Terrestrial Net
Iran already provides
considerable Internet
support to its neighbors,
including:
• Turkmenistan
• Iraqi Kurdistan
• Western Afghanistan
Iran can access all three
western watersheds:
Russia, Turkey, submarine

18. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 18 @jimcowie / @DynResearch
Crossing the Caspian Sea
Azerbaijan,
Kazakhstan plan
to build the
transcaspian wet
segment.
Only a few
formalities remain
before construction
can begin.

19. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 19 @jimcowie / @DynResearch
Iran’s Terrestrial Net
Iran already provides
considerable Internet
support to its neighbors,
including:
• Turkmenistan
• Iraqi Kurdistan
• Western Afghanistan
Iran can access all three
western watersheds:
Russia, Turkey, submarine

20. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 20 @jimcowie / @DynResearch
Iran’s Terrestrial Net
Iran already provides
considerable Internet
support to its neighbors,
including:
• Turkmenistan
• Iraqi Kurdistan
• Western Afghanistan
Iran can access all three
western watersheds:
Russia, Turkey, submarine

21. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 21 @jimcowie / @DynResearch
A Novel Path To The Sea
CASA-1000 project could
interconnect electrical grids
from Kyrgyz Republic to
Pakistan
Includes 24 strands of single-
mode fiber, of which 100Mbps is
committed for grid operations
Bishkek could be 40ms from
Persian Gulf submarine
connectivity!

22. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 22 @jimcowie / @DynResearch
In Summary….
The Internet Improves When Diversity Increases
Diversity drives higher reliability, lower prices
Central Asia is gaining new routes that send traffic
through several different “watersheds”
Later today we’ll explore these changes in more detail!

23. The Importance of Internet Measurement
Jim Cowie, Chief Scientist
@jimcowie / @DynResearch
Bishkek, Kyrgyz Republic
10 December 2014

24. / 24 @jimcowie / @DynResearch
NOTE: Some cities host multiple collectors. Cable Map credit: Telegeography
Active Measurement Infrastructure

25. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 25 @jimcowie / @DynResearch
Why We Measure, and What We Measure
1. The relationships
Who participates in each market: retail, wholesale
Nature of their bilateral agreement: peering or transit
2. The paths
There are many, many ways traffic can flow between two
providers in two cities
3. Performance and Reliability
Reachability, Round-trip latency, Variance over time

26. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 26 @jimcowie / @DynResearch
Vocabulary of Interconnection
“Routed Network”
or “Routed Prefix”
Range of IP Addresses used for a particular
function (here, the World Bank’s office in
Bishkek)
=

27. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 27 @jimcowie / @DynResearch
Vocabulary of Interconnection
“Routed Network”
or “Routed Prefix”
“Autonomous System”
Any organization that
requests an AS Number
can route its own prefixes
on the global Internet.

28. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 28 @jimcowie / @DynResearch
Vocabulary of Interconnection
“Routed Network”
or “Routed Prefix”
“Autonomous System”
$€
“Transit”
When one ASN purchases
transit from another ASN,
the provider agrees to
carry the customer’s
traffic anywhere on Earth
it needs to go.

29. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 29 @jimcowie / @DynResearch
Vocabulary of Interconnection
“Routed Network”
or “Routed Prefix”
“Autonomous System”
$€
“Peering”“Transit”
When two ASNs agree to
peer together, they are
agreeing to exchange
traffic between their
customers.
This saves money on
transit and can improve
performance.

30. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 30 @jimcowie / @DynResearch
Connectivity Example
Kazakhstan’s largest
bank buys Internet
transit from two
providers
satellite

31. / 31 @jimcowie / @DynResearch
Transit Strategies for ISPs
Kyrgyz ISPs have different strategies for
interconnection.
We can look over a period of months or
years at the percentage of BGP paths
inbound through each transit provider.
More providers == more diversity ==
lower risks of disconnection

32. / 32 @jimcowie / @DynResearch
Primary/Secondary
ElCat maintains BGP
visibility through a
single provider,
Vimpelcom, very
consistently over months
of observation
…But on certain days,
Orbita Plus is brought in
for satellite transit.

33. / 33 @jimcowie / @DynResearch
Diversity Strategy
AKNET splits virtually
50-50 in terms of BGP
path selection
between IPNET and
Kaztranscom.

34. / 34 @jimcowie / @DynResearch
Mega-Line
Phased out Kaztranscom
transit in June
Became single-homed to
IPNET this summer
Brought in Vimpelcom
transit briefly in
September

35. / 35 @jimcowie / @DynResearch
Kyrgyztelecom
Kyrgyztelecom splits
transit between
Kazakhtelecom and
Kvant Telekom
More address space
appears each day
through Kvant paths
than through
Kazakhtelecom paths

36. / 36 @jimcowie / @DynResearch
Much Can Be Learned from Simply Counting
Let’s take a look at various populations of Autonomous
Systems (ASNs) by country, across Central Asia and in
nearby markets.

37. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 37 @jimcowie / @DynResearch
Domestic ASNs
The routed Internet
consists of 49,108 ASNs
worldwide (29 Nov 2014)
A domestic ASN has at
least 80% of its routed
customers in a single
country
Compare:
• Ukraine: 1,708
• Germany: 1,358
• UK: 1,390
• Netherlands: 616
3 11
30
284
310 35
718
73
102
4,527
38
33

38. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 38 @jimcowie / @DynResearch
ASNs per capita
Domestic ASNs per
million population, 2014
Compare:
• Ukraine: 39.7
• Germany: 16.8
• UK: 21.4
• Netherlands: 36.2
• USA: 48.0
Worldwide: 7.10
0.50 1.38
5.0
0.21
3.97 1.35
0.57
0.39
6.0
31.0
1.35
3.3

39. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 39 @jimcowie / @DynResearch
Frontier ASNs
A frontier ASN has at
least one direct
interconnection with a
foreign ASN
Countries with more
frontier ASNs have more
diversity of
interconnection
Fewer than 5:
vulnerable to national
disconnection
(Syria, Yemen, N. Korea)
2 of 3 8 of 11
7 of 30
52 of 284
9 of 310 15 of 35
42 of 718
6 of 73
21 of 102
339 of 4,527
1 of 38
3
of 33

40. / 40 @jimcowie / @DynResearch
Percent of National Market On-Net
Germany 9% EUNetworks AS13237
Czech Republic 12% Telefonica CZ AS5610
Poland 22% Netia AS12741
Romania 29% Romtelecom AS9050
Ukraine 29% Datagroup AS21219
Bulgaria 33% BTC AS8866
Armenia 36% GNC-ALFA AS6682
Georgia 37% Caucasus Online AS16010
Kyrgyz Republic 43% KyrgyzTelecom AS12997
Tajikistan 45% Babilon-T AS24722
Russia 49% Rostelecom AS12389
Belarus 68% Beltelecom AS6697
Turkey 83% Turk Telekom AS9121
Kazakhstan 87% Kazakhtelecom AS9198
Azerbaijan 90% Delta Telecom AS29049
Turkmenistan 97% Turkmen Telecom AS20661
Low largest-
provider share
suggests strong
market
competition…
Among 112
countries
worldwide:
• Median 42%
• Average 45%

41. / 41 @jimcowie / @DynResearch
Internationally Connected (Cross-Border) ASNs
Germany 9% EUNetworks AS13237 894 of 1,363
Czech Republic 12% Telefonica CZ AS5610 189 of 368
Poland 22% Netia AS12741 583 of 1,700
Romania 29% Romtelecom AS9050 494 of 1,121
Ukraine 29% Datagroup AS21219 208 of 1,654
Bulgaria 33% BTC AS8866 167 of 477
Armenia 36% GNC-ALFA AS6682 11 of 58
Georgia 37% Caucasus Online AS16010 25 of 50
Kyrgyz Republic 43% KyrgyzTelecom AS12997 7 of 30
Tajikistan 45% Babilon-T AS24722 9 of 12
Russia 49% Rostelecom AS12389 356 of 4,514
Belarus 68% Beltelecom AS6697 4 of 87
Turkey 83% Turk Telekom AS9121 27 of 329
Kazakhstan 87% Kazakhtelecom AS9198 23 of 102
Azerbaijan 90% Delta Telecom AS29049 3 of 33
Turkmenistan 97% Turkmen Telecom AS20661 2 of 3

42. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 42 @jimcowie / @DynResearch
Bishkek
latencies
are
typically
100-150ms
for Europe
Note slow
speeds to
China,
Middle East
RTTs

43. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 43 @jimcowie / @DynResearch
Multiple Paths
Amsterdam to
Kyrgyztelecom
Purple band:
100-180ms via Kvant
Telecom
Red bands: 190ms/
220ms via TTK (two
physical paths)

44. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 44 @jimcowie / @DynResearch
Measuring
Failover
Kyrgyz example
from 9 July 2014
ElCat (AS8449) loses
connectivity through
Vimpelcom (AS3216)
and fails over to
TNS-Plus (AS197556)

45. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 45 @jimcowie / @DynResearch
Watersheds Again…
How do neighbors
cooperate to increase
the choices available to
everyone?
Can we measure that
process?

46. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 46 @jimcowie / @DynResearch
Kyrgyzstan Exports Internet Connectivity
Tajikistan relies critically on
Kyrgyz connectivity for
Internet and mobile service
in Dushanbe

47. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 47 @jimcowie / @DynResearch
Measuring a Market Shift: First Signs
October 2014: Tajik provider Saturn-Online stops buying
transit from Kyrgyz Telecom – why?

48. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 48 @jimcowie / @DynResearch
Pakistani Transit for Tajikistan via Afghanistan
Just appeared on Friday,
5 December
‘Special’ transit for the
small network that hosts
vkontakte.tj

49. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
/ 49 @jimcowie / @DynResearch
Summary: Internet Measurement Matters
Traditionally regulators and policymakers have waited for ISPs to
self-report interconnection and performance changes.
The most important metrics (latency, reliability,
interconnections) can now be tracked daily as the local and
regional market evolves.
The Central Asian regional market has several plausible strategies
for expanding diversity and improving access to international
markets.
It’s exciting to see regional ISPs cooperate and interconnect!

50. / 50 @jimcowie / @DynResearch

51. The Importance of Internet Measurement
Jim Cowie, Chief Scientist
@jimcowie / @DynResearch
Bishkek, Kyrgyz Republic
10 December 2014