1. Avoiding Common NGINX
Configuration Mistakes
Robert Haynes, Timo Stark
NGINX

2. ©2022 F5
2
1. Not setting enough file
descriptors

3. ©2022 F5
3
File Descriptors?
• Everything in UNIX/LINUX is a file (sort of)
• Each process (e.g. an NGINX worker process) needs a file descriptor for:
• Standard input and standard error
• Every file that a process opens
• Every network socket
• Some system calls

4. ©2022 F5
4
Example
lrwx------ 1 nginx nginx 64 Jul 11 16:55 0 -> /dev/null
lrwx------ 1 nginx nginx 64 Jul 11 16:55 1 -> /dev/null
lrwx------ 1 nginx nginx 64 Jul 11 16:55 10 -> 'anon_inode:[eventpoll]'
lrwx------ 1 nginx nginx 64 Jul 11 16:55 11 -> 'anon_inode:[eventfd]'
lrwx------ 1 nginx nginx 64 Jul 11 16:55 12 -> 'anon_inode:[eventfd]'
l-wx------ 1 nginx nginx 64 Jul 11 16:55 2 -> /var/log/nginx/error.log
l-wx------ 1 nginx nginx 64 Jul 11 16:55 3 -> /var/log/nginx/error.log
l-wx------ 1 nginx nginx 64 Jul 11 16:55 4 -> /var/log/nginx/access.log
lrwx------ 1 nginx nginx 64 Jul 11 16:55 6 -> 'socket:[52276]'
lrwx------ 1 nginx nginx 64 Jul 11 16:55 7 -> 'socket:[7717086]'
ls –l /proc/<NGINX worker process id>/fd
Will increase with the number of connections

5. ©2022 F5
5
What happens if you run out?
root@ip-10-0-1-4:/proc/6179/fdinfo# tail -f
/var/log/nginx/error.log
2022/07/18 16:22:22 [alert] 1485851#1485851: *183 socket()
failed (24: Too many open files) while connecting to upstream,
client: 71.197.217.65, server: www.snarketing.net, request:
"GET / HTTP/1.1", upstream: "http://10.0.1.199:8082/", host:
"www.snarketing.net"
.net"
Application errors
Error log entries

6. ©2022 F5
6
How many do you need?
NGINX Worker
Client
Connection
Upstream
Servers
Log Files
Number of connections
per worker is set by the
worker_connections
directive (default 512)
but example configs are
set at 1024.
Each connection to an
upstream server needs 1
file descriptor, plus some
for response caching
Writing to log files also
require a file descriptor
A good baseline is to set max file handles to
2 X worker_connections (OS default is
usually 1024)

7. ©2022 F5
7
Increasing file descriptors
Add the worker_rlimit_nofile directive to the main{} context
user nginx;
worker_processes 1024;
worker_rlimit_nofile 2048;
error_log /var/log/nginx/error.log
notice;
pid /var/run/nginx.pid;

8. ©2022 F5
8
Demo
Audience Participation Required!

9. ©2022 F5
9
2. Root Only inside Location
Blocks

10. ©2022 F5
10
Document Roots and Location Blocks
root <path>;
location <url> {
# do something
}
Sets the location to look for files to
serve
Defines some actions to take based on
the url (regex/wildcard ok)

11. ©2022 F5
11
Our Directory Structure
.
└── html
├── 50x.html
├── dashboard.html
├── images
│ ├── cat.jpg
│ ├── credits.txt
│ ├── ostrich.jpg
│ └── potato.jpg
├── index.html
├── mammal
│ └── cat
│ └── index.html
├── nginx-modules-reference.pdf
├── ostrich
│ └── index.html
└── veg
└── potato
└── index.html

12. ©2022 F5
12
Our config file:
server {
listen 80 default_server;
server_name www.snarketing.net;
access_log /var/log/nginx/host.access.log main;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
location /cat {
root /usr/share/nginx/html/mammal;
}
location /potato {
root /usr/share/nginx/html/veg;
}
…
}

13. ©2022 F5
13
Example
location /cat {
root /usr/share/nginx/html/mammal;
}
├── mammal
└── cat
└── index.html
Not Secure | http://www.snarketing.net/cat

14. ©2022 F5
14
What if?
location /ostrich {
proxy_set_header foo "test";
}

15. ©2022 F5
15
Solution
server {
listen 80 default_server;
server_name www.snarketing.net;
access_log /var/log/nginx/host.access.log main;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
location /cat {
root /usr/share/nginx/html/mammal;
}
location /potato {
root /usr/share/nginx/html/veg;
}
…
root /usr/share/nginx/html;

16. ©2022 F5
16
Demo

17. ©2022 F5
17
3. Using ‘if’ in a location
context

18. ©2022 F5
18
“Directive if has problems when used in location
context, in some cases it doesn’t do what you
expect but something completely different instead.
In some cases it even segfaults. It’s generally a
good idea to avoid it if possible.”

19. ©2022 F5
19
• Usually computationally more expensive than an in-built function
• Takes a deep understanding of how if executes in the NGINX rewrite module to
avoid problems (NGINX 301?)
• Can cause an NGINX SIGSEV (bad)
Reasons not to use “if” in a location Context

20. ©2022 F5
20
Example: Don’t use if to check for a file
server {
listen 80 default_server;
server_name www.snarketing.net;
access_log /var/log/nginx/host.access.log main;
index index.html index.htm;
root /usr/share/nginx/html;
location / {
if (!-f $request_filename) {
break;
}
}
…

21. ©2022 F5
21
Use try_files instead
server {
listen 80 default_server;
server_name www.snarketing.net;
access_log /var/log/nginx/host.access.log main;
index index.html index.htm;
root /usr/share/nginx/html;
location / {
try_files $uri $uri/ /index.html;
}
}

22. ©2022 F5
22
Demo

23. ©2022 F5
23
4. Directive Inheritance
Confusion

24. ©2022 F5
24
Directives are inherited ”outside in”
http {
server {
location foo {
root /home/user/public_html;
}
}
}
root /home/user/foo;
Sets directive
Inherits Directive
Overrides directive

25. ©2022 F5
25
Array type directives can have multiple values – the most common example is add_header
Beware of array-type Directives
location / {
add_header My-Header 1;
add_header My-Other-Header 2;
add_header My-Other-Othe-Header 3;
}
You might think that inheritance would work by adding the headers together
server {
…
add_header My-Header 1;
location / {
add_header My-Other-Header 2;
}
But you would be wrong!

26. ©2022 F5
26
Example: add_header
http {
add_header X-HTTP-LEVEL-HEADER 1;
add_header X-ANOTHER-HTTP-LEVEL-HEADER 1;
server {
listen 8080;
location / {
return 200 "OK";
}
}
server {
listen 8081;
add_header X-SERVER-LEVEL-HEADER 1;
location / {
return 200 "OK";
}
location /test {
add_header X-LOCATION-LEVEL-HEADER 1;
return 200 "OK";
}
location /correct {
add_header X-HTTP-LEVEL-HEADER 1;
add_header X-ANOTHER-HTTP-LEVEL-HEADER 1;
add_header X-SERVER-LEVEL-HEADER 1;
add_header X-LOCATION-LEVEL-HEADER 1;
return 200 "OK";
}
}
Inherits
Replaces
Replaces
To get all headers must
duplicate

27. ©2022 F5
27
Demo

28. ©2022 F5
28
5. Not Using Keepalives
for upstreams

29. ©2022 F5
29
What are keepalives and why should you care?
Upstreams
Clients
Without keepalives NGINX
opens a new connection for
each HTTP request
Ephemeral port
exhaustion

30. ©2022 F5
30
What are keepalives and why should you care?
Upstreams
Clients
With keepalives NGINX
reuses connections to
upstreams

31. ©2022 F5
31
Demo

32. ©2022 F5
32
Use the keepalive directive in the upstream{} block
Enabling keepalives
upstream http_backend {
server 127.0.0.1:8080;
keepalive 4;
}
This ensures that the connection uses the correct
HTTP version (HTTP 1 does not support
keepalives)
Add the following directives to the location{} block
location / {
proxy_http_version 1.1;
proxy_set_header "Connection" "";
proxy_pass http://backend;
}
What number to set the keepalive value to?
We recommend at least 4 x the number of servers
listed in the upstream{} block.

33. ©2022 F5
33
Demo

34. ©2022 F5
34
Questions?