Tuning your EC2 web server will help you to improve application server throughput and cost-efficiency as well as reduce request latency. In this session we will walk through tactics to identify bottlenecks using tools such as CloudWatch in order to drive the appropriate allocation of EC2 and EBS resources. In addition, we will also be reviewing some performance optimizations and best practices for popular web servers such as Nginx and Apache in order to take advantage of the latest EC2 capabilities.
22. Who am I?
•Senior Web Operations Engineer at Chartbeat
•Previously worked at
–Bitly
–TheStreet.com
–Corsis
@lintzston justin@chartbeat.com
23. Chartbeat measures and monetizes attention on the web. Working with 80% of the top US news sites and global media sites in 50 countries, Chartbeat brings together editors and advertisers to identify in real time the active time an audience consumes articles, videos, paid content, and display advertising.
38. Stream logs via Syslog
•Max 1 KB line length per RFC3164
•Only supported in Nginx 1.7.1+
•Apache supported via CustomLog piping to logger
39. Only log at load balancer
•Only one side of picture
•Can’t log custom headers or format logs
•Logs are delayed
40. Pull node on rotate
•Using prerotate/postrotate in logrotate
–Pull node from ELB via API and place back on completion
•Requires staggering nodes
•Probably not worth the effort?
42. Listen queue backlog
net.core.somaxconn = 128
Apache:ListenBackLog 511
Nginx: listen backlog=511
should be larger
43. man listen(2)
If the backlogargument is greater than the value in /proc/sys/net/core/somaxconn, thenit is silently truncated to that value; the default value in this file is 128. In kernels before 2.4.25, this limit was a hard- coded value, SOMAXCONN, with the value 128.
46. Initial congestion window
TCP congestion window -initcwnd (initial)
Starting in Kernel 2.6.39, set to 10
Previous default was 3!
http://research.google.com/pubs/pub36640.html
Older Kernel?
$ ip route change default via 192.168.1.1 dev eth0 proto static initcwnd 10
49. net.ipv4.tcp_max_tw_buckets
•Max number of sockets in TIME_WAIT. We actually set this very high, because before we moved instances behind a load balancer it was normal to have 200K+ sockets in TIME_WAITstate.
•Exceeding this leads to sockets being torn down until under limit
50. net.ipv4.tcp_fin_timeout
•The time a connection should spend in FIN_WAIT_2state. Default is 60 seconds, lowering this will free memory more quickly and transition the socket to TIME_WAIT.
•This will NOT reduce the time a socket is in TIME_WAITwhich is set to 2 * MSL (max segment lifetime).
51. net.ipv4.tcp_fin_timeout continued...
MSL is hardcoded in the kernel at 60 seconds!
https://github.com/torvalds/linux/blob/master/include/ net/tcp.h#L115
#define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT * state, about 60 seconds*/
52. “If it is on the Internet then it must be true, and you can’t question it” —Abraham Lincoln
53. net.ipv4.tcp_tw_recycle DANGEROUS
•Clients behind NAT/stateful FW will get dropped
•*99.99999999% of time should never be enabled
* Probably 100%, but there may be a valid case out there
57. net.ipv4.tcp_rmem/wmem
Format: min default max(in bytes)
•The kernel will autotune the number of bytes to use for each socket based on these settings. It will start at defaultand work between the minand max
58. net.ipv4.tcp_mem
Format: low pressure max (in pages!)
•Below low, Kernel won’t put pressure on sockets to reduce mem usage. When pressure hits, sockets reduce memory until lowis hit. If maxhits, no new sockets.
62. tcp_defer_accept
Apache: AcceptFilterhttp dataAcceptFilterhttps data
Nginx: listen [deferred]
–Wait till we receive data packet before passing socket to server. Completing TCP handshake won’t trigger an accept()
63. sendfile
Apache: EnableSendfile off
Nginx: sendfile off
–Saves context switching from userspace on read/write
–“zero copy”; happens in kernel space
64. tcp_cork
Apache: Enabled w/ sendfile
Nginx: tcp_nopush off
–aka TCP_CORKsockopt
–allows application to control building of packet; e.g., pack a packet with full HTTP response
–Only works with sendfile
65. tcp_nodelay (Nagle’s algo)
Apache: On
•No ability to turn off
Nginx: tcp_nodelay on
•Only affects keep-alive connections
•Will add latency if turned off in favor of bandwidth
66. HTTP Keep-Alive
Apache: KeepAlive On
KeepAliveTimeout 5
MaxKeepAliveRequests 100
Nginx: keepalive_timeout 75s
keepalive_requests 100
Note: If using ELB you must match the timeout to the the ELB timeout setting
67. HTTP Keep-Alive
•Also enable on upstream proxies
–Available since Nginx 1.1.4
proxy_http_version 1.1;
proxy_set_header Connection "";
upstream foo {
server 10.1.1.1;
keepalive 1024;
}