Extreme HTTP Performance Tuning: 1.2M API req/s on a 4 vCPU EC2 Instance
•
0 j'aime•1,957 vues
In this talk I will walk you through the performance tuning steps that I took to serve 1.2M JSON requests per second from a 4 vCPU c5 instance, using a simple API server written in C. At the start of the journey the server is capable of a very respectable 224k req/s with the default configuration. Along the way I made extensive use of tools like FlameGraph and bpftrace to measure, analyze, and optimize the entire stack, from the application framework, to the network driver, all the way down to the kernel. I began this wild adventure without any prior low-level performance optimization experience; but once I started going down the performance tuning rabbit-hole, there was no turning back. Fueled by my curiosity, willingness to learn, and relentless persistence, I was able to boost performance by over 400% and reduce p99 latency by almost 80%.
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Extreme HTTP Performance Tuning: 1.2M API req/s on a 4 vCPU EC2 Instance
Similaire à Extreme HTTP Performance Tuning: 1.2M API req/s on a 4 vCPU EC2 Instance (20)
Plus de ScyllaDB
Plus de ScyllaDB (20)
Dernier
Dernier (20)
Extreme HTTP Performance Tuning: 1.2M API req/s on a 4 vCPU EC2 Instance
- 1. Brought to you by Extreme HTTP Performance Tuning: 1.2M API req/s on a 4 vCPU EC2 Instance Marc Richards Chief Problem Solver at
- 2. Marc Richards Chief Problem Solver at Talawah Solutions Talawah Solutions ■ Based in Kingston Jamaica ■ Cloud Computing Consultant for almost a decade ■ Solutions Architect / DevOps Engineer / Performance Engineer ■ No low-level systems performance tuning experience before this project!
- 3. Demystifying Systems Performance Tuning ■ You don't need to be a kernel developer or a wizard sysadmin. ■ FlameGraph and bpftrace have changed the game. ■ New ebpf based tools coming out will only make things easier!
- 4. Overview ■ I accidentally fell down this optimization rabbit hole. ■ Started with a simple, high-performance API server written in C. ■ Used FlameGraph and bpftrace to analyze and optimize the entire stack.
- 5. Overview ■ Cloud: AWS ■ Hardware: 4 vCPU c5n.xlarge** (server) / 16 vCPU c5n.4xlarge (client) ■ Benchmark: Techempower JSON Serialization test ■ Server: Techempower libreactor implementation ** In order to minimize inconsistencies at the platform level I did the final benchmark run on a c5n.9xlarge that was restricted to 4 vCPUS using the EC2 CPU Options feature.
- 6. Blog post with even more details https://talawah.io/blog/extreme-http-performance-tuning-one -point-two-million/
- 7. Optimizations Optimization Gain Req/s Ground Zero - 224k Application Optimizations 55% 347k Disabling Speculative Execution Mitigations 28% 446k Disabling Syscall Auditing / Blocking 11% 495k Disabling iptables / netfilter 22% 603k Perfect Locality 38% 834k Interrupt Optimizations 28% 1.06M The Case of the Nosy Neighbor 6% 1.12M The Battle Against the Spin Lock 2% 1.15M This Goes to Twelve 4% 1.20M
- 8. Optimizations Optimization Gain Req/s Ground Zero - 224k Application Optimizations 55% 347k Disabling Speculative Execution Mitigations 28% 446k Disabling Syscall Auditing / Blocking 11% 495k Disabling iptables / netfilter 22% 603k Perfect Locality 38% 834k Interrupt Optimizations 28% 1.06M The Case of the Nosy Neighbor 6% 1.12M The Battle Against the Spin Lock 2% 1.15M This Goes to Twelve 4% 1.20M
- 9. Ground Zero Running 10s test @ http://server.tfb:8080/json 16 threads and 256 connections Latency Distribution 50.00% 1.14ms 90.00% 1.21ms 99.00% 1.26ms 99.99% 1.32ms 2243551 requests in 10.00s, 331.64MB read Requests/sec: 224,353.73
- 10. * I modified nginx.conf to send back a hardcoded JSON response. This is not a part of the Techempower implementation.
- 11. Ground Zero
- 14. Application Optimizations ■ Run on all logical cores/vCPUs: ~25% ■ gcc -O3 and march=native: ~15% ■ send/recv instead of write/read: ~5% ■ Remove pthread overhead: ~3%
- 15. Application Optimizations Running 10s test @ http://server.tfb:8080/json 16 threads and 256 connections Latency Distribution 50.00% 723.00us 90.00% 0.88ms 99.00% 0.94ms 99.99% 1.08ms 3470892 requests in 10.00s, 483.27MB read Requests/sec: 347,087.15
- 17. Disabling... Speculative Execution Mitigations Syscall Auditing / Blocking iptables/netfilter
- 18. Disabling... ■ Speculative Execution Mitigations: 28% ● nospectre_v1 nospectre_v2 pti=off mds=off tsx_async_abort=off ■ Syscall Auditing/Blocking: 11% ● auditctl -a never,task ● docker run -d --security-opt seccomp=unconfined libreactor ■ iptables/netfilter: 22% ● modprobe -rv ip_tables ● ExecStart=/usr/bin/dockerd ---bridge=none --iptables=false --ip-forward=false
- 19. Disabling... Running 10s test @ http://server.tfb:8080/json 16 threads and 256 connections Latency Distribution 50.00% 419.00us 90.00% 479.00us 99.00% 517.00us 99.99% 575.00us 6031161 requests in 10.00s, 839.76MB read Requests/sec: 603,112.18
- 22. Perfect Locality + Interrupt Optimizations ■ Perfect Locality ● Pin processes to CPUs ● Pin network queues to CPUs (RSS + XPS) ● SO_REUSEPORT + SO_ATTACH_REUSEPORT_CBPF ■ Interrupt Moderation ● ethtool -C eth0 adaptive-rx on ■ Busy polling ● net.core.busy_poll=1 ■ Perfect Locality + Interrupt Moderation + Busy Polling = 💯
- 23. Perfect Locality + Interrupt Optimizations Running 10s test @ http://server.tfb:8080/json 16 threads and 256 connections Latency Distribution 50.00% 233.00us 90.00% 263.00us 99.00% 292.00us 99.99% 348.00us 10660410 requests in 10.00s, 1.45GB read Requests/sec: 1,066,034.60
- 24. Perfect Locality + Interrupt Optimizations Before After
- 25. The Case of the Nosy Neighbor + The Battle Against the Spin Lock
- 26. The Case of the Nosy Neighbor Someone, somewhere was spying on all my packets (kinda) ■ dev_queue_xmit_nit() -> packet_rcv() ■ packet_rcv() implicates AF_PACKET ■ sudo ss --packet --processes -> (("dhclient",pid=3191,fd=5)) ■ My (extreme) solution was to disable dhclient after boot
- 27. The Case of the Nosy Neighbor Running 10s test @ http://server.tfb:8080/json 16 threads and 256 connections Latency Distribution 50.00% 218.00us 90.00% 254.00us 99.00% 285.00us 99.99% 341.00us 11279049 requests in 10.00s, 1.53GB read Requests/sec: 1,127,894.86
- 28. The Case of the Nosy Neighbor Before After
- 29. The Battle Against the Spin Lock Running 10s test @ http://server.tfb:8080/json 16 threads and 256 connections Latency Distribution 50.00% 212.00us 90.00% 246.00us 99.00% 276.00us 99.99% 338.00us 11551707 requests in 10.00s, 1.57GB read Requests/sec: 1,155,162.15
- 30. The Battle Against the Spin Lock Before After
- 31. This Goes to Twelve
- 32. This Goes to Twelve ■ Disabling Generic Receive Offload (GRO) ■ TCP Congestion Control: cubic -> reno ■ Static Interrupt Moderation
- 33. This Goes to Twelve Running 10s test @ http://server.tfb:8080/json 16 threads and 256 connections Latency Distribution 50.00% 203.00us 90.00% 236.00us 99.00% 265.00us 99.99% 317.00us 12031718 requests in 10.00s, 1.64GB read Requests/sec: 1,203,164.22
- 34. Conclusion 436% increase in requests per second. 79% reduction in p99 latency. ■ Throughput: 224k req/s -> 1.2M req/s ■ p99 latency: 1.26ms -> 265.00us ■ p99.99 latency: 1.32ms -> 317.00us
- 35. All 11 implementations on a c5n.xlarge using the stock Amazon Linux 2 AMI without any OS/Networking optimizations
- 36. All 11 implementations on a c5n.xlarge with all OS/Networking optimizations applied
- 37. Next Steps ■ Next gen kernel: 5.10 LTS ■ Next gen technologies: io_uring ■ Next gen instances: ARM vs Intel vs AMD ■ Driving performance from the bottom-up using Rust, Java, etc
- 38. Brought to you by Marc Richards https://talawah.io/contact @talawahtech