Information Technology Center, Fukuoka University, Japan
Sho FUJIMURA
fujimura@fukuoka-u.ac.jp
NIPPON TELEGRAPH AND TELEPH...
2
1 Fukuoka University introduction
1 Objectives
2 Background
2 System failure issues
3 Network configuration diagram
3 Tr...
Fukuoka University introduction
n Private university
¡ 83rd anniversary in May 2017
¡ Connected to internet in 1993
n Loca...
Objectives
n Determine cause of NTP traffic
n Reduce NTP traffic
4
Background
n Commenced a public NTP service in October
1993 at Fukuoka University
n First public NTP service using GPS in ...
Network configuration diagram
n Until August, 2015
n NTP servers were
located in
laboratory
¡ Edge of campus
network
¡ Tra...
Campus Network
AS4713AS2907
Internet
・・・ ・・・
・・・ ・・・
BGP
router
BGP
router
FireWall
(Active)
FireWall
(Standby)
Router
(L3...
Campus Network
AS4713AS2907
Internet
・・・ ・・・
・・・ ・・・
BGP
router
BGP
router
FireWall
(Active)
FireWall
(Standby)
Router
(L3...
Traffic during network failure
n Traffic
through
AS2907 to
AS18148
increased to
approximately
135Mbps
9
02/17/2014
135Mbps...
Traffic during network failure
n Traffic
through
AS4713 to
AS18148
increased to
approximately
900Mbps
10
02/18/2014
900Mbp...
Summary until August, 2015
n NTP service failures cause a huge amount
of retry packets, and that causes firewall
failures
...
Current network diagram
n Changed on
September, 2015
n Operating NTP
servers in Information
Technology Center
¡ To avoid h...
NTP Network configuration diagram
n load distribution by
load balancers
n Increased public NTP
servers from 2 to 4 in
cons...
133.100.11.8 Traffic
14
※ AS18148 … Fukuoka University
※ AS2907… Science Information NETwork (SINET) operated by National ...
133.100.9.2 Traffic
15
※ AS18148 … Fukuoka University
※ AS2907… Science Information NETwork (SINET) operated by National I...
16
Approximately
190,000
Packet / s
Current traffic (Number of packets)
Total Throughput:
342,539,104 bits / s
L4 Connecti...
Analyze using ntopng
n Capturing
data from
one of the 4
public NTP
servers
n Real-time
analysis at
a dedicated
server by
“...
Why is it so popular in the world?
n written in manual as setting example
¡ Network devices such as L2, L3 switch
¡ Multif...
n It’s embedded as default setting
n TL-WR740N(TP-LINK) is one of devices
19
Why is it so popular? (2)
n was in source codes of
OpenWRT (2005)
¡ It’s fixed now
[0-3].openwrt.pool.ntp.org
¡ Cannot connect to two
other NTP serv...
Summary
n Statistics of our public NTP servers
¡ Approximately 190,000 requests per second
¡ Presently statistics shows gr...
Request: Please do not use our NTP servers
n To firmware developers
¡ Please confirm you do not have 133.100.9.2 nor
133.1...
Conclusion
n Would like to determine cause of
NTP traffic
n Because of the concentrated nature of
NTP traffic we would lik...
Thank you very much for your kind attention.
Contact information: Sho FUJIMURA (ntp-admin@fukuoka-u.ac.jp)
Prochain SlideShare
Chargement dans…5
×

Fukuoka University Public NTP Service Deployment Use case

348 vues

Publié le

Presentation by Sho Fujimura at APRICOT 2017 on Monday, 27 February 2017.

Publié dans : Internet
  • Soyez le premier à commenter

Fukuoka University Public NTP Service Deployment Use case

  1. 1. Information Technology Center, Fukuoka University, Japan Sho FUJIMURA fujimura@fukuoka-u.ac.jp NIPPON TELEGRAPH AND TELEPHONE WEST CORPORATION Fuminori -Tany- Tanizaki fuminori.tanizaki@west.ntt.co.jp Fukuoka University Public NTP Service Deployment Use Case
  2. 2. 2 1 Fukuoka University introduction 1 Objectives 2 Background 2 System failure issues 3 Network configuration diagram 3 Traffic and Analysis 4 Summary 4 Conclusion Table of Contents
  3. 3. Fukuoka University introduction n Private university ¡ 83rd anniversary in May 2017 ¡ Connected to internet in 1993 n Location: Fukuoka City, Fukuoka Prefecture, JAPAN ¡ The city we had APRICOT2015 n 9 faculties (31 departments) n 10 graduate courses (33 specialties) n Approximately 21,000 students n Attached facilities ¡ Hospital: 2 ¡ High school: 2 ¡ Junior high school: 1 3 AS: 18148 Prefix: 133.100.0.0/16 2405:be00::/32
  4. 4. Objectives n Determine cause of NTP traffic n Reduce NTP traffic 4
  5. 5. Background n Commenced a public NTP service in October 1993 at Fukuoka University n First public NTP service using GPS in Japan ¡ 133.100.9.2 ¡ 133.100.11.8 n Posted “Request of NTP traffic dispersion" to bulletin board named 2channel (Ni-channel: Japanese online forum) on January 20th 2005 ¡ Approximately 900 NTP requests per second ¡ Bandwidth approximately 2Mbps 5
  6. 6. Network configuration diagram n Until August, 2015 n NTP servers were located in laboratory ¡ Edge of campus network ¡ Traffic increases momentarily every hour on the hour 6 Campus Network AS4713AS2907 Internet ・・・ ・・・ ・・・ ・・・ NTP Servers BGP router BGP router FireWall (Active) FireWall (Standby) Router (L3 switch) Router (L3 switch) Each building L2 switch Each building L2 switch Edge switch Edge switch Edge switch AS18148 ※ AS18148 … Fukuoka University ※ AS2907… Science Information NETwork (SINET) operated by National Institute of Informatics ※ AS4713 … Open Computer Network(OCN) operated by NTT Communications Corporation
  7. 7. Campus Network AS4713AS2907 Internet ・・・ ・・・ ・・・ ・・・ BGP router BGP router FireWall (Active) FireWall (Standby) Router (L3 switch) Router (L3 switch) Each building L2 switch Each building L2 switch Edge switch Edge switch Edge switch AS18148 NTP Servers Incident case n 8Mbps rate-limiting for NTP was already configured at the BGP router connecting to AS4713 ¡ To address an issue of high CPU load on firewalls due to a huge number of NTP retry packets from clients while NTP servers were stopped for maintenance ¡ No rate-limit at the BGP router connecting to AS2907 n Friday, February 14, 2014 ¡ Third incident related to the NTP service happened (total 4 troubles) n NTP traffic through AS2907 was increased, and caused high CPU load on firewalls ¡ Introduced 8Mbps rate-limiting at the BGP router connecting to AS2907 ¡ Internet connectivity was restored even though it’s a bit slower than usual 7 ※ AS18148 … Fukuoka University ※ AS2907… Science Information NETwork (SINET) operated by National Institute of Informatics ※ AS4713 … Open Computer Network(OCN) operated by NTT Communications Corporation QoS:8Mbps High load QoS:noneQoS:8Mbps
  8. 8. Campus Network AS4713AS2907 Internet ・・・ ・・・ ・・・ ・・・ BGP router BGP router FireWall (Active) FireWall (Standby) Router (L3 switch) Router (L3 switch) Each building L2 switch Each building L2 switch Edge switch Edge switch Edge switch AS18148 NTP Servers L2 switch Incident case (2) n Saturday, February 15 (the next day) n The BGP router connecting to AS4713 went down ¡ QoS handling on the router was software-based, caused high CPU load on the router n Installed a new L2 switch to perform hardware-based QoS ¡ restored the router without QoS n Set 8Mbps rate-limiting for NTP traffic on both links 8 QoS:8Mbps QoS:8Mbps ↓ Stop QoS:8Mbps ※ AS18148 … Fukuoka University ※ AS2907… Science Information NETwork (SINET) operated by National Institute of Informatics ※ AS4713 … Open Computer Network(OCN) operated by NTT Communications Corporation
  9. 9. Traffic during network failure n Traffic through AS2907 to AS18148 increased to approximately 135Mbps 9 02/17/2014 135Mbps ※ AS18148 … Fukuoka University ※ AS2907… Science Information NETwork (SINET) operated by National Institute of Informatics ※ AS4713 … Open Computer Network(OCN) operated by NTT Communications Corporation
  10. 10. Traffic during network failure n Traffic through AS4713 to AS18148 increased to approximately 900Mbps 10 02/18/2014 900Mbps ※ AS18148 … Fukuoka University ※ AS2907… Science Information NETwork (SINET) operated by National Institute of Informatics ※ AS4713 … Open Computer Network(OCN) operated by NTT Communications Corporation
  11. 11. Summary until August, 2015 n NTP service failures cause a huge amount of retry packets, and that causes firewall failures ¡ Must continue to reply NTP packets n 8Mbps bandwidth limit for NTP traffic on both links to upstreams ¡ The average NTP traffic subsequently exceeded 8Mbps n At that time, we were unable to ascertain what the bandwidth would be ¡ Drop NTP packets or change bandwidth limit level, when trouble occurs 11 ※ AS18148 … Fukuoka University ※ AS2907… Science Information NETwork (SINET) operated by National Institute of Informatics ※ AS4713 … Open Computer Network(OCN) operated by NTT Communications Corporation
  12. 12. Current network diagram n Changed on September, 2015 n Operating NTP servers in Information Technology Center ¡ To avoid high CPU load on firewalls, we moved NTP servers outside of the firewalls 12 Campus Network AS4713AS2907 Internet ・・・ ・・・ ・・・ ・・・ BGP router BGP router FireWall FireWall Router (L3 switch) Router (L3 switch) Each building L2 switch Each building L2 switch Edge switch Edge switch Edge switch AS18148 L2 switch for NTP L2 switch for NTP NTP Servers NTP Servers ※ AS18148 … Fukuoka University ※ AS2907… Science Information NETwork (SINET) operated by National Institute of Informatics ※ AS4713 … Open Computer Network(OCN) operated by NTT Communications Corporation
  13. 13. NTP Network configuration diagram n load distribution by load balancers n Increased public NTP servers from 2 to 4 in consideration of load and redundancy n 2 ‘stratum 1’ servers ¡ These are not open to public, serving for clients in the campus only AS4713AS2907 Internet Public NTP Servers (Stratum2) L2 switch for NTP BGP router BGP router L2 switch for NTP NTP Server Stratum1 NTP Server Stratum1 Load distribution Public NTP Servers (Stratum2) AS18148 ※ AS18148 … Fukuoka University ※ AS2907… Science Information NETwork (SINET) operated by National Institute of Informatics ※ AS4713 … Open Computer Network(OCN) operated by NTT Communications Corporation
  14. 14. 133.100.11.8 Traffic 14 ※ AS18148 … Fukuoka University ※ AS2907… Science Information NETwork (SINET) operated by National Institute of Informatics ※ AS4713 … Open Computer Network(OCN) operated by NTT Communications Corporation
  15. 15. 133.100.9.2 Traffic 15 ※ AS18148 … Fukuoka University ※ AS2907… Science Information NETwork (SINET) operated by National Institute of Informatics ※ AS4713 … Open Computer Network(OCN) operated by NTT Communications Corporation
  16. 16. 16 Approximately 190,000 Packet / s Current traffic (Number of packets) Total Throughput: 342,539,104 bits / s L4 Connections: 221,250 Packet / s !!
  17. 17. Analyze using ntopng n Capturing data from one of the 4 public NTP servers n Real-time analysis at a dedicated server by “ntopng” 17
  18. 18. Why is it so popular in the world? n written in manual as setting example ¡ Network devices such as L2, L3 switch ¡ Multifunction device, etc. 18
  19. 19. n It’s embedded as default setting n TL-WR740N(TP-LINK) is one of devices 19 Why is it so popular? (2)
  20. 20. n was in source codes of OpenWRT (2005) ¡ It’s fixed now [0-3].openwrt.pool.ntp.org ¡ Cannot connect to two other NTP servers ¡ Other vendors might reuse the code and there might be commercial products that are embedded ‘default NTP setting’ 20 { “ntp_server” , “192.5.41.40 192.5.41.41 133.100.9.2” , 0} Why is it so popular? (3) https://dev.openwrt.org/browser/ trunk/package/nvram/src/defaults.c?rev=5461 Copyright 2004, Broadcom Corporation All Rights Reserved.
  21. 21. Summary n Statistics of our public NTP servers ¡ Approximately 190,000 requests per second ¡ Presently statistics shows gradual increase n Origin of the NTP clients ¡ Throughout the world n Implications for the Fukuoka University network... ¡ Further increasing is not desirable n What happens if we stop the NTP service now... ¡ Retry packets will naturally DoS to our network ¡ At this moment, there is no way to terminate the service 21
  22. 22. Request: Please do not use our NTP servers n To firmware developers ¡ Please confirm you do not have 133.100.9.2 nor 133.100.11.8 as default NTP servers ¡ If you do, please change them n To manual authors ¡ Please do not list 133.100.9.2 and 133.100.11.8 as NTP servers n If you have contacts of them ¡ Please pass the above information n We would like to take measures by determining the cause of NTP traffic. So if you know particular product or site which uses our NTP servers, please introduce the contact to us. 22 Contact information: Sho FUJIMURA (ntp-admin@fukuoka-u.ac.jp)
  23. 23. Conclusion n Would like to determine cause of NTP traffic n Because of the concentrated nature of NTP traffic we would like to reduce it. 23 We sincerely appreciate your cooperation. Contact information: Sho FUJIMURA (ntp-admin@fukuoka-u.ac.jp)
  24. 24. Thank you very much for your kind attention. Contact information: Sho FUJIMURA (ntp-admin@fukuoka-u.ac.jp)

×