Contenu connexe


PLNOG 13: Piotr Głaska: Quality of service monitoring in IP networks

  1. Monitorowanie jakości usług w sieciach IP Piotr Głaska Kraków, 29.09.2014
  2. 1 Agenda  NQA – Network Quality Analysis  iPCA – Packet Conservation Algorithm for Internet  AtomEngine
  3. NQA Working Principle  Client: initiates test, gathers statistics  Server: responds to the test initiated by client  Test results can be viewed through command line, SNMP, can be uploaded by FTP, can generate logs, alarms and actions 2
  4. Types of NQA tests LSP ping, traceroute, jitter MPLS 3 Link Layer Multicast traceroute Multicast ICMP ping, traceroute, jitter IP ARP Ping, MAC ping HTTP Video delay & jitter VOIP delay & jitter DNS DHCP DNS lookup FTP Link, download time TCP Jitter, Echo UDP Three-way handshake
  5. Configuring ICMP Test HuaweiA> system-view [HuaweiA] nqa test-instance admin icmp [HuaweiA-nqa-admin-icmp] test-type icmp [HuaweiA-nqa-admin-icmp] destination-address ipv4 [HuaweiA-nqa-admin-icmp] start now 4
  6. Configuring Jitter Test <HuaweiB> system-view [HuaweiB] nqa-server udpecho 9000 <HuaweiA> system-view [HuaweiA] nqa test-instance admin jitter [HuaweiA-nqa-admin-jitter] test-type jitter [HuaweiA-nqa-admin-jitter] destination-address ipv4 [HuaweiA-nqa-admin-jitter] destination-port 9000 [HuaweiA-nqa-admin-jitter] start now 5
  7. IP SLA and NQA interoperability NQA can work with IP SLA as a responder for UDP Echo and UDP Jitter tests [Huawei] ip nqa-compatible responder [vpn-instance vpn-instance-name] enable [Huawei] ip nqa-compatible auto 6
  8. Interface Backup and NQA Router A 7 GE2/0/0 Router B Router C Router D GE1/0/0 GE1/0/0 GE1/0/0 GE1/0/0 GE2/0/0 GE2/0/0 GE2/0/0 <RouterA> system-view [RouterA] nqa test-instance user test [RouterA-nqa-user-test] test-type icmp [RouterA-nqa-user-test] destination-address ipv4 [RouterA-nqa-user-test] start now [RouterA] interface gigabitethernet2/0/0 [RouterA-GigabitEthernet2/0/0] standby track nqa user test Run the display nqa results test-instance user test command.
  9. VRRP Backup Group with NQA 8 Switch Master Router A Router C GE1/0/0 GE2/0/0 NQA agent GE1/0/0 GE2/0/0 GE1/0/0 GE2/0/0 GE1/0/0 GE2/0/0 Router B Router D Backup VRRP Backup Group Virtual IP Address: Host A This mechanism enables the VRRP backup group to monitor the link connecting the master to the external network. If the link fails, hosts on a LAN cannot access an external network through the master router. NQA detects this fault and notifies VRRP. The VRRP backup group lowers the master router's priority by a configured value. The backup router with the highest priority will become the new master router and take over traffic.
  10. DHCP Pool with NQA [Huawei] ip pool p1 [Huawei-ip-pool-p1] excluded-ip-address [Huawei-ip-pool-p1] lock track nqa admin dhcptest 9
  11. DNS Proxy with NQA [Huawei] dns resolve [Huawei] dns server track nqa admin localdns [Huawei] dns server track nqa admin remotedns [Huawei] dns proxy enable 10
  12. 3G/LTE Modem recovery with NQA [Huawei] interface cellular 0/0/0 [Huawei-Cellular0/0/0] modem auto-recovery track nqa user test [Huawei-Cellular0/0/0] modem auto-recovery track action { plmn-search | modem-reboot } fail-times times 11
  13. Adaptive Traffic Shaping with NQA [Huawei] qos adaptation-profile gts1 [Huawei-qos-adaptation-profile-gts1] rate-range low-threshold 128 high-threshold 512 [Huawei-qos-adaptation-profile-gts1] rate-adjust step 32 [Huawei-qos-adaptation-profile-gts1] rate-adjust loss low-threshold 20 high-threshold 30 [Huawei-qos-adaptation-profile-gts1] track nqa admin jitter1 [Huawei] interface gigabitethernet 1/0/0 [Huawei-GigabitEthernet1/0/0] ip address [Huawei-GigabitEthernet1/0/0] qos gts adaptation-profile gts1 [Huawei-GigabitEthernet1/0/0] traffic-policy p1 outbound When configuring an NQA test instance, ensure that NQA packets enter high-priority queues so that they are treated preferentially when the link is congested. 12
  14. NQA for Static Routes ip route-static track nqa user test ip route-static preference 100 nqa test-instance aa bb test-type icmp destination-address ipv4 frequency 3 probe-count 1 start now 13
  15. Policy Based Routing with NQA acl number 2000 rule 10 permit source traffic classifier vlan10 if-match acl 2000 traffic behavior vlan10 redirect ip-nexthop track nqa admin vlan10 traffic policy vlan10 classifier vlan10 behavior vlan10 interface GigabitEthernet1/0/0 ip address traffic-policy vlan10 inbound 14
  16. Smart Policy Routing with NQA [Huawei] smart-policy-route [Huawei-smart-policy-route] prober ethernet 1/0/0 nqa admin nqa1 [Huawei-smart-policy-route] prober ethernet 2/0/0 nqa admin nqa2 [Huawei-smart-policy-route] link-group group1 [Huawei-smart-policy-route-link-group group1] link-member ethernet 1/0/0 [Huawei-smart-policy-route] link-group group2 [Huawei-smart-policy-route-link-group group2] link-member ethernet 2/0/0 [Huawei-smart-policy-route] service-map map1 [Huawei-smart-policy-route-service-map-map1] match acl 3000 [Huawei-smart-policy-route-service-map-map1] set link-group group1 [Huawei-smart-policy-route-service-map-map1] set link-group group2 backup 15
  17. LTE APN Tracking with NQA Example: DSVPN based on 3G/LTE dialup status DSVPN – Dynamic Smart VPN, dynamic VPN based on NHRP and mGRE 16
  18. 3G/LTE APN Tracking with NQA interface Cellular0/0/0 apn-profile orange priority 200 track nqa admin tunnel0/0/1 admin tunnel0/0/2 apn-profile tmo priority 150 track nqa admin tunnel0/0/3 admin tunnel0/0/4 apn profile orange apn internet sim-id 1 apn profile tmo apn internet sim-id 2 17
  19. Spoke router tunnels configuration interface Tunnel0/0/1 ip address rip metricin 1 tunnel-protocol gre p2mp source Cellular0/0/0 gre key cipher @%@%.'YF3l/T'GtCF,$NT-<$~5U]@%@% nhrp authentication cipher %@%@Z1jU$i^[f:xiYUF|Dhj% nhrp registration interval 20 nhrp entry register track apn orange interface Tunnel0/0/2 ip address rip metricin 7 rip metricout 7 tunnel-protocol gre p2mp source Cellular0/0/0 gre key cipher @%@%f94gE3y!0=%Ba0Y-cSR3~6&<@%@% nhrp authentication cipher %@%@HP>P#8z<G#*9<7A70!YUG~ nhrp registration interval 20 nhrp entry register track apn orange 18 interface Tunnel0/0/3 ip address rip metricin 4 rip metricout 4 tunnel-protocol gre p2mp source Cellular0/0/0 gre key cipher @%@%r*crMiQ/b!gLFF~sj}qO~5@f@%@% nhrp authentication cipher %@%@Q2atQl+%C51rQRSVB nhrp registration interval 20 nhrp entry register track apn tmo interface Tunnel0/0/4 ip address rip metricin 10 rip metricout 10 tunnel-protocol gre p2mp source Cellular0/0/0 gre key cipher @%@%<&-+=09yzL]g'*;V)E|~~7"a@%@% nhrp authentication cipher %@%@oB|n3,7,eP]jh)/KzuN~QOa nhrp registration interval 20 nhrp entry register track apn tmo
  20. IVPN – Intelligent VPN 19
  21. IVPN – Intelligent VPN ivpn-proposal p1 encapsulation gre source Dialer1 destination bandwidth up 1024 down 8192 track nqa admin dsl source Cellular0/0/0 destination bandwidth up 15000 down 30000 track nqa admin lte service youtube id 1 schedule-type priority match app-protocol youtube source Dialer1 source Cellular0/0/0 cmi-method D/2+ J x 2 + L cmi-threshold cmi 8500 delay 1000 jitter 500 loss 20 service exchange id 2 schedule-type overload match app-protocol ms_exchange source Cellular0/0/0 source Dialer1 interface Tunnel0/0/1 ip address tunnel-protocol ivpn p2p ivpn-zone 1 ivpn-proposal p1 20 Hub configuration: ivpn-proposal p1 encapsulation gre service s1 id 1 match app-protocol youtube service exchange id 2 match app-protocol ms_exchange interface Tunnel0/0/1 ip address tunnel-protocol ivpn p2mp ivpn-zone 1 ivpn-proposal p1 Default CMI method: CMI = 9000 - (D + J + L) Default CMI, delay, jitter and packet loss thresholds are 0, 5000 ms, 3000 ms and 1000‰
  22. 21 Scalability  Most of NQA tests are processed by the main core on AR G3 routers. They run with low priority, so there is little impact on CPU  UDP jitter detection is of milliseconds level and there is larger impact on CPU. Forwarding cores can support this test and enhance performance of sending and receiving packets, reducing impact on the main core  UDP Jitter tests can be hardware-based and processed by line cards (LPU)
  23. Hardware-based NQA on AR G3 routers  Reduces the interval for sending packets. The minimum interval for sending packets can be 10 ms.  Increases the number of concurrent test instances (up to 6000) and test packets per second (up to 2000)  Improves the accuracy of delay and jitter calculation From miliseconds to microseconds level [Huawei] nqa test-instance user test [Huawei-nqa-user-test] test-type jitter [Huawei-nqa-user-test] hardware-based enable [Huawei-nqa-user-test] timestamp-unit microsecond 22
  24. Hardware-based IP SLA on ASR1K routers  Use QFP for timestamping 23 ip sla 1 udp-jitter 5000 num-packets 20 request-data-size 160 tos 128 frequency 30 precision microseconds optimize timestamp
  25. 24 iPCA
  26. iPCA Concept Packet Conservation Algorithm for Internet (iPCA) is developed by Huawei. iPCA implements packet loss monitoring and fault location for connectionless IP networks by coloring real service packets and partitioning a network. It allows a network to perceive service quality and quickly locate faults. In addition, iPCA breaks the limitation of traditional measurement technologies. P2P, MP2MP 25 L2+L3 mixed network iPCA Performance monitoring based on real service packets Question iPCA When were the packets lost? Where were the packets lost? Who lost the packets? Monitors real service flows, and sends alarms to the administrator immediately when faults occur. Partitions the network into multiple domains, provides device-level, link-level, and network-level monitoring, and automatically locates faults. Monitors the service flows with five specified attributes based on domains or link segments, and determines the type of services with packets lost.
  27. Topology-Centric Configuration and Monitoring 26
  28. Network-Level Measurement View 27
  29. Hop-by-Hop Measurement on Unicast IP Service Path Select building 6 in Warsaw and building 1 in Krakow in the topology view to create a network-level measurement task. eSight can discover service path, display each agile switch on the path, and show packet loss on the path. 1 Perform hop-by-hop path measurement in the network-level view. 2 Display the service forwarding path and agile switches. 28 3 Show packet loss statistics on egress node. Click a device on the path to show real-time packet loss statistics on the device
  30. Based on Real Traffic Traditional quality measurement technologies send simulated detection packets on the network. The simulated detection packets are different from real service packets in sizes and frequencies. Therefore, simulated detection packets cannot reflect real service quality, and occupy bandwidth. Service packets Sim ulated detection packets 29 iPCA colors real service packets, which do not occupy additional bandwidth. Service packets
  31. Packet Loss Measurement Between Two Points Send: A target flow is divided into consecutive measurement intervals, and the number of packets (TX[i]) sent in each interval is counted. Receive: Identify the measurement intervals and count the number of packets received in an interval (RX[i]). Packet unsequencing issue should be noticed. 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 1 0 0 1 1 1 1 1 1 0 1 0 0 0 0 0 0 1 Interval i+1 Interval i Interval i Interval i+1 Receiver Measurement point in sending direction Measurement point in receiving direction Transmitter Service packet flow  The transmitter colors (sets to 1) and decolors (sets to 0) a characteristics bit in service packets periodically to divide the service flow into 30 multiple intervals.  The counters and measurement points are configured on the transmitter and receiver, and the number of sent and received packets is counted based on intervals.  TX[i] and RX[i] packets are sent to the MCP. The MCP identifies the packets, compares the number of sent packets with the number of received packets in the same interval, and obtains the number of lost packets.  Calculate the measurement result: In the interval i (packet sending interval and receiving interval), the number of lost packets = TX[i] - RX[i].
  32. Color Bit Selection 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 Version IHL Total Length Time to Live Protocol Header Checksum 31 Type of Service Identification Fragment Offset Flags Source Address Destination Address Options Padding The reserved bit in IPv4 packet header can be used as the color bit, for example, bit 0 in the Flags field or one of bits 3-7 in ToS field. In iPCA device-level measurement, bit 0 in the Flags field is used as the color bit by default. In iPCA network-level measurement, bit 6 in the ToS field is used as the color bit by default.
  33. iPCA Logical Topology 32 The iPCA system consists of NMS, MCP, DCPs, and TLPs, which have the following responsibilities: NMS: provides GUI  Issues commands to configure measurement instances.  Obtains real-time statistics and historical data from MCP, and displays measurement results. TLP (Target Logical Port):  Executes iPCA measurement tasks, and corresponds to a logic interface on network device  Colors and measures target service flows periodically.  Reports statistics in each interval to DCPs. DCP (Data Collecting Point):  Manages and controls TLPs (configures and issues ACL rules to TLPs).  Collects statistics from TLPs.  Reports statistics to the MCP. MCP (Measurement Control Point):  Collects statistics from DCPs.  Summarizes statistics and calculate results.  Reports measurement results to the NMS. DCP TLP TLP MCP TLP TLP DCP eSight Management data Measurement data report Real service flow
  34. Measurement Principle Packets arriving at the system 33 Measured system (device/link/carrier's network/service path) System core Internally terminated Internally generated Packets leaving the system iPCA quality measurement mechanism: A measured system is in the normal state if the following condition is met: Number of packets arriving at the system + Number of internally generated packets = Number of packets leaving the system + Number of packets internally terminated by the system If this condition is not met, some packets have been dropped in the system.
  35. Device-Level Packet Loss Measurement Incoming flows Outgoing flows Measurement domain: All ENP cards and SFUs form a packet conservation domain (excluding the CPU and non-ENP cards). Object: All incoming and outgoing IP unicast flows of the measurement domain. Measurement interval: 10 seconds Alarm: When the packet loss ratios in five consecutive intervals exceed 5%, the device sends an alarm to the NMS. When the packet loss ratios in five consecutive intervals fall below 1%, the device sends a clear alarm to the NMS. 34 Chassis CPU Non-ENP SFU 1 Chassis C1-2 C1-3 C1-4 C1-5 C1-6 C1-7 SFU 2 C2-1 C2-3 C2-4 C2-5 C2-6 C2-7 Ingress TLP Egress TLP CPU Non-ENP card ENP card 1 C1-1 C1-8 C1-9 C1-10 ENP card 2 C2-1 C2-8 C2-9 C2-10 Number of packets from other devices to ENP cards: C1-1 and C2-1 Number of packets from CPU to ENP cards: C1-5 and C2-5 Number of packets from non-ENP cards to ENP cards: C1-7 and C2-7 The number of packets entering the measurement domain Cin = C1-1 + C2-1 + C1-5 + C2-5 + C1-7 + C2-7 Number of packets from ENP cards to CPU: C1-2 and C2-2 Number of packets from ENP cards to non-ENP cards: C1-4 and C2-4 Number of packets from CPU to ENP cards and then other devices: C1-8 and C2-8 Number of packets from ENP card to ENP card and then other devices: C1-9 and C2-9 Number of packets from non-ENP card to ENP card and then other devices: C1-10 and C2-10 Number of packets leaving the measurement domain Cout = C1-2 + C2-2 + C1-4 + C2-4 + C1-8 + C2- 8 + C1-9 + C2-9 + C1-10 + C2-10 Number of lost packets = Cin - Cout Meas urement domain
  36. Link-Level Packet Loss Measurement Device 1 Device 2 Micro engine MAC ENP card C2_1 C2_2 Measurement domain: The physical link between directly connected devices is a packet conservation domain. The measurement range contains physical direct links, and TM chips and MAC chips on interfaces. Object: All incoming and outgoing IP unicast flows of the measurement domain. Unidirectional packet loss from device 1 to device 2 = C1_1 - C2_1 Unidirectional packet loss from device 2 to device 1 = C2_2 - C1_2 Note: The TM and MAC chips do not support iPCA. The measurement object is all packets. The measurement interval of TM and MAC chips is not synchronized with that of micro engine. Therefore, the statistics are only used as a reference for fault location. 35 TM C1_1 C1_2 MAC ENP card Micro TM engine Ingress TLP Egress TLP Expected measurement range Actual measurement range
  37. Network-Level Packet Loss Measurement Measurement domain: A domain consisting of non-agile devices (including third-party devices) surrounded by agile devices and the links between agile devices and the measurement domain. Object: All incoming and outgoing IP unicast flows of the measurement domain. (The current version only supports measurement on the service flows with known directions.) 36 Device A Device B Ingress TLP Egress TLP Device C Device E C1 C2 C3 C4 C5 Number of lost packets from devices A/B to devices C/D/E = (C1 + C2) - (C3 + C4 + C5) Incoming packets Outgoing packets Measurement domain Note: The measurement object in this example is a unidirectional service flow.
  38. Service Path Hop-by-Hop Measurement Terminal ACH1 ACH2 ACH3 ACH4 ACH5 ACH6 ACH7 Terminal 1 2 3 4 5 6 7 8 S57 (source gateway) 37 Service packet forwarding path detected by IP Tracert S127 S127 S57 (destination gateway) eSight Service flow characteristics: Service packets must have known source and destination IP addresses. Path tracing: eSight searches for the source gateway according to the source IP address of the service flow. The source gateway performs IP Tracert to the destination IP address of the service flow to trace the forwarding path between source and destination gateways. The gateways deliver service flow characteristics to agile devices. The agile devices returns the service flow inbound interfaces (1, 3, 5, and 7) and outbound interfaces (2, 4, 6, and 8) to eSight. The Layer 3 IP path of the service flow is determined. Measurement method: Each agile device measures service packets on its inbound and outbound interfaces. Two neighboring interfaces can calculate the number of lost service packets on each segment (ACH). Constraint: The current version of iPCA only supports IP networks, but does not support MPLS VPN or GRE network. If load balancing paths or active/standby paths are configured, the measurement result on only the path obtained by IP Tracert is displayed.
  39. Huawei Products Supporting iPCA The device must support iPCA and have an ENP card installed. Model Version Remarks eSight V200R005C00 NMS,SLA S5720HI V200R006C00 Fixed-chassis Agile switch S7700 V200R006C00 Modular Agile switch,ENP S9700 V200R006C00 Modular Agile switch,ENP S12700 V200R006C00 Modular Agile switch,ENP 38
  40. Huawei AtomEngine 39
  41. AtomEngine Solution Solution Architecture 40 BS Mobile Core Meter Meter Controller Manager Meter Meter Meter Enterprise Enterprise Performance Test Hop-by-hop Hop-by-hop Hop-by-hop Network E2E Test 1 2 3 1 2 3 Meter: Atom Meter • Bypass network quality measurement • In-line real time flow quality measurement • Identify, Coloring, Statistics Controller: SNC-A • Atom Meter discovery • Management agent Manager: U2000+uTraffic/U2520 • Performance test visualization • Atom Meter management: configuration , log, alarm CSG ASG RSG
  42. 41 X3/X8/X16 NE40E/CX600/ ME60/PTN6900 SNC-A Board SNC – Smart Network Controller One SNC board can manage 1K Atom Meters, maximum 8K per chassis
  43. Huawei AtomEngine 42
  44. Innovative AtomEngine Technology NP Inside 43
  45. HUAWEI ENTERPRISE ICT SOLUTIONS A BETTER WAY Copyright©2012 Huawei Technologies Co., Ltd. All Rights Reserved. The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time without notice.