[네이버오픈소스세미나] Maglev Hashing Scheduler in IPVS, Linux Kernel - 송인주
•
9 j'aime•19,640 vues
제3회 네이버 오픈소스 세미나 2018.09.04
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à [네이버오픈소스세미나] Maglev Hashing Scheduler in IPVS, Linux Kernel - 송인주
Similaire à [네이버오픈소스세미나] Maglev Hashing Scheduler in IPVS, Linux Kernel - 송인주 (20)
Plus de NAVER Engineering
Plus de NAVER Engineering (20)
Dernier
Dernier (20)
[네이버오픈소스세미나] Maglev Hashing Scheduler in IPVS, Linux Kernel - 송인주
- 1. Maglev Hashing Scheduler in IPVS, Linux Kernel Inju Song Software Engineer, Container Platform Team NAVER
- 2. CONTENTS Load balancing problems in Container Cluster Maglev Consistent Hashing IPVS Maglev Hashing Scheduler Build A Efficient and Reliable Load Balancer Q & A
- 5. Container Host Container Host Container Host Container Host Container Cluster There are many containers...
- 6. Load balancing Problems in Container Cluster
- 7. Static Routing L4-LB L4-LB L4-LB L4-LB
- 8. Container Host Container Host Container Host Container Host Container Cluster Added / Removed Frequently
- 9. Static Routing Problem Container Host(Past) Container Host(Current) L4-LB L4-LB L4-LB L4-LB
- 10. Container Cluster N Too many containers . . . . . . Container Cluster 5 Container Cluster 4 Container Cluster 3 Container Cluster 2 Container Cluster 1
- 11. Guarantee Load Balancing L4-LB Container Host Container Host Container Host Container Host L4-LB
- 13. Load Balancer Down Container Host Container Host Container Host Container Host L4-LB
- 14. Efficient and Reliable Load Balancing
- 16. Large Traffic Balancing L4-LB Container Host Container Host Container Host Container Host L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB
- 19. ECMP L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB Equal Cost Multi Path
- 20. Hash Based Routing L4-LB L4-LB 1.1.1.1 L4-LB L4-LB 1.1.1.2 PATH= Hashing( IP.Src, IP.Dst, IP.Protocol, Port.Src, Port.Dst ) Mod N
- 21. Efficient and Reliable Load Balancing BGP/ ECMP
- 22. Internet L4-LB L4-LB L4-LB If a load balancer down L4-LB L4-LB L4-LB L4-LB
- 23. Hashing Disruption 192.68.0.1 Hashing(IP) % 2 Container[0] Container[1] 192.68.0.2 192.68.0.3 192.68.0.4 Hashing(IP) % 2 Hashing(IP) % 2 Hashing(IP) % 2
- 24. Hashing Disruption 192.68.0.1 Hashing(IP) % 4 192.68.0.2 192.68.0.3 192.68.0.4 Hashing(IP) % 4 Hashing(IP) % 4 Hashing(IP) % 4 Container[0] Container[1] Container[2] Container[3]
- 25. ECMP is A Good Load Balancing, but.. Not Reliable Not scalable High Availability
- 26. Consistent Hashing for High Availability
- 29. Google’s Network Load Balancer Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf
- 30. Google’s Maglev Features Consistent Hashing Backend Selection Packet Processing Packet Forwarding Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf
- 31. Permutation B0 B1 B2 3 0 3 0 2 4 4 4 5 1 6 6 5 1 0 2 3 1 6 5 2 Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf Backend = 3 Table Size = 7 Permutate Hashing Permutationtable ( 3x 7 ) Preference list of Backends Lookup_table ( size =7,empty )
- 32. Population Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf Population Permutationtable ( 3x 7 ) B1 B0 B1 B0 B2 B2 B0 Lookup_table ( size =7 ) B0 B1 B2 3 0 3 0 2 4 4 4 5 1 6 6 5 1 0 2 3 1 6 5 2 [0] [1] [2] [3] [4] [5] [6] Assign backends by preference list
- 33. Load Balancing Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf B1 B0 B1 B0 B2 B2 B0 each backend will receive an almost equal numberof connections. [0] [1] [2] [3] [4] [5] [6] L4-LB L4-LB B0 =3 * connection B1 =2 * connection B2 =2 * connection
- 34. Minimal disruption Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf Population Permutationtable ( 2x 7 ) B0 B2 3 3 0 4 4 5 1 6 5 0 2 1 6 2 Assign backends by preference list Before B1 B0 B1 B0 B2 B2 B2 B1 is Removed Re-Permutate Hashing Lookup_table ( size =7 ) After B1 -> B0 B0 B1 -> B0 B0 B2 B2 B2 Same Value By hashing Lookup_table ( size =7 )
- 35. Hard to implement Google’s Maglev Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf Backend SelectionPacket ProcessingPacket Forwarding
- 37. Internet L4-LB L4-LB L4-LB L4-LB Network Software L4 Load Balancer
- 38. Using Netfilter in Linux Kernel Nftables – Wikipedia| https://en.wikipedia.org/wiki/Nftables
- 39. Replace Maglev With IPVS Consistent Hashing Backend Selection: FUNCTION Packet Processing: NETFILTER Packet Forwarding: NAT, DR, IP Tunneling
- 40. Pluggable Scheduler Module Weighted Round Robin Round Robin Source Hashing least connection
- 42. Maglev Scheduling Module Maglev Hashing Scheduling Round Robin Source Hashing least connection Weighted Round Robin
- 43. Maglev Hashing Table Size Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf ... Lookup_table Size=251 ... Lookup_table Size=131071 Disruption % VS Memory usage
- 44. Optimize your Memory Usage Kernel Configuration -> Networking support -> Networking options -> Netfilter -> IPVS -> Maglev hashing table size IPVS MaglevHashing SchedulerKConfig | https://github.com/torvalds/linux/blob/master/net/netfilter/ipvs/Kconfig#L290
- 45. L4-LB L4-LB L4-LB L4-LB Canary Release Strategy New Service Old Service 5% 95%
- 46. Weight Based Balancing Container Host Container Host Container Host Container Host L4-LB W: 10 W: 40 W: 80
- 47. Destination Fallback Container Host Container Host Container Host Container Host L4-LB W: 10 W: 40 -> 0 W: 80
- 48. 5-Tuples Hash 192.68.0.2 IPVS IPVS MH IPVS IPVS MH IPVS IPVS MH Container[0] Container[1] Container[2] Container[3] IPVS MH is no need to track connection table for forwarding packets to same destination.
- 50. Internet L4-LB L4-LB L4-LB A load balancer down L4-LB L4-LB L4-LB L4-LB
- 51. ECMP Hashing Disruption L4-LB L4-LB 1.1.1.2 L4-LB L4-LB 1.1.1.3 PATH= Hashing() Mod ( N - 1 ) L4-LB L4-LB 1.1.1.4 L4-LB L4-LB 1.1.1.1
- 52. TCP Connection Loss 192.68.0.2 Container[0] Container[1] Container[2] Container[3] L4-LB 1. Connection Established With Container[2] 2. A Load Balancer Down 3. ECMP Disruption. Forward packet another l4 4. Container[0] has no idea about connection with container[2] 5. Container[0] Send RST. Connection Closed
- 53. TCP Connection Loss 2 192.68.0.3 Container[0] Container[1] Container[2] Container[3] L4-LB L4-LB 1. Connection Established With Container[3] 2. A Load Balancer Down 3. ECMP Disruption. Forward packet another l4 although the serving LB is alive. 4. Container[1] has no idea about connection with container[3] 5. Container[1] Send RST. Connection Closed L4-LB
- 55. High Availability Container Host Container Host Container Host Container Host L4-LB
- 56. Build A Efficient and Reliable Load Balancer
- 57. Efficient and Reliable Load Balancer IPVS MH IPVS MH IPVS MH IPVS MH IPVS MH Add IP of containers dynamically To IPVS Add IP of containers dynamically To IPVS
- 58. No Loss TCP Connection 192.68.0.2 Container[0] Container[1] Container[2] Container[3] L4-LB IPVS 1. Connection Established With Container[2] 2. A Load Balancer Down 3. ECMP Disruption. Forward packet another l4 4. IPVS MH can forward the packet to same destination by hashing. 5. Continue the Established connection. IPVS MH
- 59. No Loss TCP Connection 2 192.68.0.3 Container[0] Container[1] Container[2] Container[3] L4-LB 1. Connection Established With Container[3] 2. A Load Balancer Down 3. ECMP Disruption. Forward packet another l4 although the serving LB is alive. 5. Continue the Established connection. IPVS IPVS IPVS MH 4. IPVS MH can forward the packet to same destination by hashing With no connection info.
- 60. At least 1 Load Balancer up, IPVS MH IPVS MH IPVS MH IPVS MH IPVS MH Add IP of containers dynamically To IPVS Add IP of containers dynamically To IPVS
- 61. Can Hash Consistently With No Connection Loss IPVS MH IPVS MH IPVS MH IPVS MH IPVS MH Add IP of containers dynamically To IPVS Add IP of containers dynamically To IPVS
- 62. Efficient and Reliable Load Balancing BGP/ ECMP IPVS Maglev Hashing Scheduler
- 63. Required Linux Kernel >= 4.18 See Also: https://github.com/torvalds/linux/blob/master/net/netfilter/ipvs Choose M in Kernel menuconfig to use IP_VS_MH
- 64. Required Kernel Parameters echo 1 > /proc/sys/net/ipv4/vs/sloppy_tcp See Also: https://github.com/torvalds/linux/blob/master/Documentation/networking/ipvs-sysctl.txt echo 2 > /proc/sys/net/ipv4/vs/conn_reuse_mode
- 65. Q & A