Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

如何用k8s打造國產5G NFV平臺? 剖析經濟部5G核網技術的關鍵

如何用k8s打造國產5G NFV平臺? 剖析經濟部5G核網技術的關鍵
How to build up k8s based NFVI?

  • Soyez le premier à commenter

如何用k8s打造國產5G NFV平臺? 剖析經濟部5G核網技術的關鍵

  1. 1. 如何用k8s打造國產5G NFV平臺? 剖析經濟部5G核網技術的關鍵 梁維恩 Jace Liang
  2. 2. About Me 版權宣告©工業技術研究院 2 • 梁維恩 Jace Liang • 工研院資通所/資料中心架構與雲端應用軟體組 Div. X. • What our team does ▪ NFV Performance Lab (cooperate with intel®) ▪ NFVI Architect of MOEA 5G Project e-mail: jace.liang@itri.org.tw Github: mJace
  3. 3. Outline 版權宣告©工業技術研究院 3 • 網路功能虛擬化帶來的衝擊 • VNF部屬於Kubernetes的優勢與挑戰 • 電信業者的5G Kubernetes Solution • 國家5G計畫 NFVI 平台介紹 • 系統整合實例分享
  4. 4. 網路功能虛擬化帶來的衝擊 The impact of NFV
  5. 5. Network Function Virtualization 版權宣告©工業技術研究院 5 CDN Session Border Controller WAN Acceleration DPI Carrier Grade NAT Tester / QoE Monitor SGSN / GGSN PE Router Firewall Typical network appliances General purpose servers Standard storage NFV-based Approach Standard switches NFV is a process of decoupling the network function from proprietary hardware appliances
  6. 6. The Impact of NFV 版權宣告©工業技術研究院 6 • The existing player need to invest and adopt new techs ▪ Help telecom increase service agility and efficiencies • Easier to deploy service • Able to reuse physical resources • The new business potential ▪ Server providers ▪ VNF providers ▪ NFVI providers ▪ Solution providers ▪ Network operators
  7. 7. Evolving from VNFs to CNFs 版權宣告©工業技術研究院 7 Ref : https://events.linuxfoundation.org/wp-content/uploads/2017/11/2018- OSSNA-CNF-Journey-in-Telecom-Seminar.pdf
  8. 8. Performance Issue in NFV 版權宣告©工業技術研究院 8 • Traffic hidden from monitoring ▪ Cisco estimates that about 73% of data-center traffic will come from within the data center by 2019—most of this traffic is virtual machine to virtual machine (VM to VM) communication ▪ The inability to isolate production from monitoring traffic • No Solution to address performance uncertainty in NFV ▪ Where is the major bottleneck point of an NFV system Container / POD VNF Application vCPUs vNICs Container / POD VNF Application vCPUs vNICs Host OS (State transition overhead due to privileged operations and idleness) Hardware Layer (Resource exhaustion, buffer overflow or queue imbalance at NICs) Virtual Switch Layer (Less optimized packet processing engine) Less optimized VNF implementations Inter-Container communication overhead Network I/O Overhead NFV Architecture with Major Bottleneck Points
  9. 9. VNF部署於k8s的優勢與挑戰 The Pros and Cons for VNF on k8s
  10. 10. Pros to run VNF on k8s 版權宣告©工業技術研究院 10 • K8S as container orchestrator ▪ Container grouping using pod ▪ Self-healing ▪ Auto-scalability ▪ DNS management ▪ Load balancing ▪ Rolling update or rollback ▪ Resource monitoring and logging ...... 版權宣告©工業技術研究院
  11. 11. Challenge of Networking 版權宣告©工業技術研究院 11 • Kubernetes native CNI is not design for telco VNF ▪ Decoupling Control/Data plane is not easy in native K8S network. • Only one NIC for each Pod. ▪ The network behavior is not suit for VNF. • Whole traffic must go through Master or Load Balancer.
  12. 12. Challenge of Compute & Management 版權宣告©工業技術研究院 12 • Native K8S does not guarantee performance stability. ▪ It is very critical for VNF to require a stable and predictable performance. • Native Kubernetes does not support NUMA aware and CPU Pinning. • Native k8S is lack of detail hardware spec. info. ▪ When Deploy/Migrate/Scale VNF, must consider its required HW function. • Native K8S don’t know if the machine supports certain CPU instruction and the NIC’s spec.
  13. 13. 電信業者的5G k8s Solution The K8S solution from TelCo.
  14. 14. DANM 版權宣告©工業技術研究院 14 • From Nokia ▪ Support VXLan, IPAM ▪ Allowing pods to attach multiple network interfaces. Calico, Flannel, SR-IOV, etc ▪ a Kubernetes controller capable of centrally managing both VxLAN and VLAN interfaces of all Kubernetes hosts ▪ another Kubernetes controller extending Kubernetes' Service-based service discovery concept to work over all network interfaces of a Pod https://github.com/nokia/danm
  15. 15. SONA 版權宣告©工業技術研究院 15 • Used by SK • Extension work from OpenStack SONA • Pure SDN Based overlay solution ▪ OvS L3 load balancing ▪ OvS L2/L3 connectivity • Support OVS-DPDK, Smart NIC • Support Multi NIC https://github.com/sonaproject/sona-cni
  16. 16. CNI-Genie 版權宣告©工業技術研究院 16 • From Huawei • Support Multi-NIC for Pod https://github.com/huawei-cloudnative/CNI-Genie
  17. 17. CNI-Genie 版權宣告©工業技術研究院 17 • From Huawei • Support Multi-NIC for Pod https://github.com/huawei-cloudnative/CNI-Genie
  18. 18. Akraino 版權宣告©工業技術研究院 18 • Create an open source edge software stack for different scenario ▪ IOT Edge, Edge on prem, AI Edge ▪ Nokia, AT&T, SAMSUNG, etc. ▪ https://www.lfedge.org/projects/akraino/
  19. 19. 國家5G計畫 NFVI 平台介紹 K8S NFVI of MOEA 5G Project
  20. 20. How do we address these problems 版權宣告©工業技術研究院 20 • Kubernetes native CNI is not design for telco VNF ▪ Decoupling Control/Data plane is not easy in native K8S network. • Only one NIC for each Pod. ▪ The network behavior is not suit for VNF. • Whole traffic must go through Master or Load Balancer. • Native K8S can not guarantee performance stability. ▪ It is very critical for VNF to require a stable and predictable performance. • Native Kubernetes does not support NUMA aware and CPU Pinning. • Native k8S does not own detail hardware spec. info. ▪ When Deploy/Migrate/Scale VNF, must consider its required HW function. • Native K8S don’t know if the machine support certain CPU instruction or the NIC’s spec.
  21. 21. ITRI Open Source X-K8S solution 版權宣告©工業技術研究院 21 • Network ▪ Multus CNI ▪ SRIOV ▪ SmartNIC ▪ SRIOV Device Plugin • Compute/Management ▪ CMK – CPU Management for Kubernetes ▪ NFD – Node Feature Discovery ▪ VNF Monitor ▪ Performance Optimizer
  22. 22. Multus CNI 版權宣告©工業技術研究院 22 • Enables attaching multiple network interfaces to pods ▪ So that the VNF pod can separate the network by different usage, for example • flannel as manage • SRIOV VFs as Data in/out ▪ Support any CNI – SRIOV, Host-device, etc • Most Popular ▪ Most used by community ▪ Already included in kubespray
  23. 23. SRIOV 版權宣告©工業技術研究院 23 • Single Root I/O Virtualization ▪ As a PCI-SIG Standard, SRIOV allowing PCIe I/O device to provide multiple virtual function. • SRIOV-CNI ▪ Allowing Pods to use SRIOV VF as it’s CNI • SRIOV Device Plugin ▪ Allowing K8S scheduler to manage SRIOV NIC as device resource. NIC Pod Pod vNIC vNIC
  24. 24. 資通所重要營運資料,禁止複製、轉載、外流 ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE -24- SRIOV vs. OVS-DPDK 64 Bytes 128 Bytes 256 Bytes 512 Bytes 1024 Bytes 1280 Bytes OVS-DPDK-1-Core 2.51 2.46 2.38 2.26 1.74 1.47 OVS-DPDK-2-Cores 6.31 6.1 5.64 4.98 3.71 3.04 OVS-DPDK-4-Cores 12.96 12.5 11.82 9.32 4.77 3.83 SRIOV 59.52 32.89 17.85 9.32 4.77 3.83 BM 59.52 32.89 17.85 9.32 4.77 3.83 0 10 20 30 40 50 60 70 Throughput(Mpps) OVS-DPDK vs SRIOV vs Baremetal 4 Port - Throughput (L2-Fwd) OVS-DPDK performance is similar to SRIOV performance in large packet size scenario.
  25. 25. SRIOV – Example https://asciinema.org/a/263659 版權宣告©工業技術研究院 25
  26. 26. SmartNIC 版權宣告©工業技術研究院 26 • Smart NIC enables the functionality of OVS in a hardware approach ▪ Improve network throughput ▪ Reduce CPU usage NIC Pod Pod vNIC vNIC OVS
  27. 27. Ericsson Cloud SDN & Netronome Agilio CX
  28. 28. The Mapping Between CPU and NIC 版權宣告©工業技術研究院 28 • The CPU core and NIC for packet generator should be in the same CPU socket, or the QPI interface will be the performance bottleneck. ▪ 10GbE line rate (64bytes packet) = 10.00e9 bits / (8 bits * (64 + 20)bytes ) = 14.88e6 packets -> 67.2 nano-sec / pkt ▪ CPU and NIC in same socket, 10G port could generate 14.88 Mpps. ▪ CPU and NIC in different socket, 10G port could only generate 10 Mpps. 18 Core Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz CPU 1 18 Core Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz CPU 2 QPI X710-DA4 adapter GEN QPI interface will be the performance bottleneck.
  29. 29. Result of NUMA misconfiguration 版權宣告©工業技術研究院 29 L2-Fwd Rx L2-Fwd Tx L3-Fwd Rx L3-Fwd Tx ACL Rx ACL Tx MPLS Rx MPLS Tx BM 59.52 59.52 53.29 53.29 37.38 33.82 59.52 58.17 VM w/ NUMA 59.52 59.52 53.29 53.29 37.41 33.78 59.52 58.17 VM w/o NUMA 59.52 59.46 53.36 43.4 37.46 30.66 59.52 56.96 0 10 20 30 40 50 60 70 Throughput(Mpps) Baremetal vs SRIOV 4 Port - Throughput (64Bytes) VM without NUMA-aware configuration may decrease 10 – 20% performance. VM performance with NUMA-aware configuration is similar to Baremetal performance.
  30. 30. CMK & NFD 版權宣告©工業技術研究院 30 • CPU Manager for Kubernetes ▪ Pin Pods on specific CPU core. ▪ https://github.com/intel/CPU-Manager-for-Kubernetes • Node Feature Discovery ▪ Detects hardware feature available on each node in k8s cluster. ▪ https://github.com/kubernetes-sigs/node-feature-discovery *CMK is now a milestone for k8s v1.16 as a alpha version.
  31. 31. Compute / Management 版權宣告©工業技術研究院 31 • Compute / Management – node wise ▪ Support NUMA aware and CPU Pinning ▪ Enhanced Platform Awareness (EPA) VNF management SRIOV NIC SmartNIC
  32. 32. Compute / Management 版權宣告©工業技術研究院 32 • Compute / Management – node wise ▪ Support NUMA aware and CPU Pinning ▪ Enhanced Platform Awareness (EPA) VNF management SRIOV NIC SmartNIC kubelet api-server CoreDns Scheduler kube-proxy controller MME SGW-C SGW-U PGW-C PGW-U
  33. 33. Compute / Management 版權宣告©工業技術研究院 33 • Compute / Management – cluster wise ▪ Support NUMA aware and CPU Pinning ▪ Enhanced Platform Awareness (EPA) VNF management kubelet api-server CoreDns Scheduler kube-proxy controller MME PGW-C SGW-U PGW-C PGW-U SRIOV Network SmartNIC Network SGW-C kubelet api-server CoreDns Scheduler kube-proxy controller SGW-C PGW-USGW-U SGW-U SGW-U SGW-U
  34. 34. VNF Monitor 版權宣告©工業技術研究院 34 • How to get NUMA related info? ▪ Which CPU core that my pod runs on? ▪ NUMA node of NIC in my pod? ▪ Is core-affinity set?
  35. 35. Which CPU core that my container runs on? 版權宣告©工業技術研究院 35 https://github.com/mJace/numacc
  36. 36. Developing feature. 版權宣告©工業技術研究院 36 ▪ APM system for 5G VNF • Customized APM system for 5G VNF ▪ Self performance optimization ▪ Data plane network slicing • Network slicing based on the SRIOV data plane ▪ DNS / Service discovery for extra network interfaces
  37. 37. 系統整合實例分享 End-to-End System Integration
  38. 38. EPC 版權宣告©工業技術研究院 38 • 完成3GPP R14版本 Container Based C/U Split vEPC 雛型系統,支援快速彈性擴充。每秒20個UE連線的情 況下約可乘載50K個使用者終端。 • 於 MME 實作 eMBMS 功能,支援車載網路群播應用。 並與Content Provider和商用E-UTRAN整合測試群播 功能完成 • 支持OVS-DPDK and SRIOV技術 • 透過SmartNIC兩種不同的數據路由,藉以實現大吞吐 量(接近10Gbps),改善虛擬化後造成之網路傳輸延遲, 不需額外佔用伺服器計算資源,延遲時間可縮短至 300us以內。 • 完整利用SmartNIC效能進行GTP封包處理 • Future Work: SBA 5GC X-Kubernetes
  39. 39. iMEC 版權宣告©工業技術研究院 39 發展 MEC 與 4G/5G 接取網路的整合,技術成果為 iMEC iMEC 提供節省後端網路頻寬且低延遲雲端平台之解決方案 Enterprise EPC Cloud Wi-Fi AP 4G/5G RAN FTTx iMEC 企業私有網路終 端 設 備 網路&服務虛擬化 智慧路由 使用者辨識 多邊緣雲管理 即時服務啟動 系統效能水平擴展 一鍵自動部署 支援 OPNFV, Kubernetes及 DC/OS等多種 NFV 平台 支援網路本地分流功能(Local breakout/Traffic Offload) , 流量負載平衡 支援 UE Identify 功能,可辨 別垂直專網之企業用戶 一鍵自動部署iMEC系統與 ME APP 支援多邊緣雲互連架構,進 行服務佈建與管理 Just-in-Time 技術,可於0.2 秒快速啟動服務,並節省 edge cloud資源 可以動態增加User Plane Function數量,使得效能線 性成長
  40. 40. End-to-End System Integration 版權宣告©工業技術研究院 40
  41. 41. 5G Network Architecture on X-K8S 版權宣告©工業技術研究院 41 Key Features & Specification • High Throughput  10 / 25 / 40 / 100 Gbps • CNF – Containerized Network Function / Services • Performance Optimized  SmartNIC, SRIOV, CMK, NFD, Multus • Network Isolation CNF / Services on X-K8S • CNF : iMEC  Data traffic offloading • CNF : III-vEPC  Use P4 SmartNICs as an accelerator for GTPU • CNF : SON (Self-Organizing Network) • APP : VR Community (III)
  42. 42. Q&A Contact us. jace.liang@itri.org.tw Try our x-k8s https://github.com/ITRI-ICL-Peregrine/x-k8s

×