Nginx: configuration, reverse proxy, TLS and troubleshooting
sudo nginx -t && sudo systemctl reload nginx
Nginx serves static files, terminates TLS, and sits in front of applications as a reverse proxy and load balancer. Its configuration is short and readable, but a few rules (how a request picks a server and a location, how proxy_pass rewrites paths, which directives inherit) decide whether it behaves the way you expect.
This guide builds up a working setup piece by piece and shows what nginx really does at each step. Every example ran for real against nginx 1.24.0 (the version in Ubuntu 24.04) started as an ordinary user on high ports, in front of two small lab application servers named alpha and beta, so the output is genuine: the responses, the headers, the access and error log lines, and the exact error messages. The only differences from a real server are the ports (19080 and 19443 instead of 80 and 443, so no root is needed) and that the lab uses its own test certificate. Commands that need root or a real machine are shown without output.
How nginx is organised
Nginx runs as one MASTER process (reads the configuration, manages workers) and several WORKER processes (handle the connections). The configuration is a tree of CONTEXTS: the main context, then events, then http, inside it one or more server blocks (virtual hosts), and inside those location blocks (URL paths). Directives are settings that end with a semicolon; blocks use braces. A setting in an outer block is inherited by the inner ones unless they set their own. On Debian and Ubuntu the layout is:
/etc/nginx/nginx.conf the main file (global settings, then includes)
/etc/nginx/conf.d/*.conf small config snippets, included in the http block
/etc/nginx/sites-available/NAME one file per website; sites-enabled/NAME is a symlink that turns it on
/etc/nginx/snippets/*.conf reusable pieces you include from several sites
/var/log/nginx/access.log, error.log
The lab uses the same shape in one folder. This is its main file:
$ cat > ngx/nginx.conf <<'EOF'
> worker_processes auto;
> pid logs/nginx.pid;
> error_log logs/error.log;
>
> events {
> worker_connections 1024;
> }
>
> http {
> include lab-paths.conf; # lab only: temp folders for a normal user
> include mime.types;
> default_type application/octet-stream;
> sendfile on;
>
> log_format main '$remote_addr "$request" $status $body_bytes_sent rt=$request_time up=$upstream_response_time cache=$upstream_cache_status';
> access_log logs/access.log main;
>
> include conf.d/*.conf;
> }
> EOF
A first website: a static folder.
$ echo '<h1>Hello from nginx</h1>' > ngx/html/index.html
$ echo 'plain text file' > ngx/html/notes.txt
$ cat > ngx/conf.d/site.conf <<'EOF'
> server {
> listen 127.0.0.1:19080;
> server_name www.example.test;
> root html;
> index index.html;
> }
> EOF
$ nginx -t
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ nginx
$ curl -si http://127.0.0.1:19080/ | sed -n '1p;/^Server/p;/^Content-Type/p;$p'
HTTP/1.1 200 OK
Server: nginx/1.24.0 (Ubuntu)
Content-Type: text/html
<h1>Hello from nginx</h1>
$ curl -si http://127.0.0.1:19080/notes.txt | sed -n '1p;/^Content-Type/p;$p'
HTTP/1.1 200 OK
Content-Type: text/plain
plain text file
$ curl -sI http://127.0.0.1:19080/missing.html | sed -n '1p'
HTTP/1.1 404 Not Found
$ sudo nginx -t && sudo systemctl reload nginx # on a real server: test, then reload without dropping connections
$ systemctl status nginx --no-pager; ss -ltnp | grep nginx
nginx -t checks the whole configuration (including every included file) without touching the running server: always run it before a reload. "listen" says which address and port, "server_name" which host names this block answers to, "root" the folder that maps to the URL path /, "index" the default file for a folder. The Server header shows nginx/1.24.0 (Ubuntu): the version is a free gift to scanners, and "server_tokens off" (see Security) hides it.
Reloading safely: what happens when the config is wrong
nginx -s reload tells the master to read the new configuration, start new workers with it, and let the old workers finish their requests (zero downtime). If the new configuration is invalid, nothing changes: the old one keeps running. See it for real: break the config on purpose.
$ sed -i 's/index index.html;/index index.html/' ngx/conf.d/site.conf
$ nginx -t
nginx: [emerg] unexpected "}" in /home/demo/ngx/conf.d/site.conf:6
nginx: configuration file /home/demo/ngx/nginx.conf test failed
# (exit status 1)
$ nginx -s reload; sleep 0.7; curl -s -o /dev/null -w 'site still answers: %{http_code}\n' http://127.0.0.1:19080/
nginx: [emerg] unexpected "}" in /home/demo/ngx/conf.d/site.conf:6
site still answers: 200
$ tail -n 2 ngx/logs/error.log | cut -c1-135 | sed -E 's/^[0-9]{4}\/[0-9]{2}\/[0-9]{2} [0-9:]{8} //; s/[0-9]+#[0-9]+: //'
[emerg] unexpected "}" in /home/demo/ngx/conf.d/site.conf:6
[emerg] unexpected "}" in /home/demo/ngx/conf.d/site.conf:6
$ sed -i 's/index index.html$/index index.html;/' ngx/conf.d/site.conf
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
The classic errors, and what they look like:
$ nginx -t
nginx: [emerg] unknown directive "server_nam" in /home/demo/ngx/conf.d/typo.conf:3
nginx: configuration file /home/demo/ngx/nginx.conf test failed
# (exit status 1)
$ nginx -t
nginx: [emerg] host not found in upstream "nosuchhost.invalid:9000" in /home/demo/ngx/conf.d/typo.conf:1
nginx: configuration file /home/demo/ngx/nginx.conf test failed
# (exit status 1)
$ nginx -t
nginx: [emerg] a duplicate default server for 127.0.0.1:19080 in /home/demo/ngx/conf.d/typo.conf:2
nginx: configuration file /home/demo/ngx/nginx.conf test failed
# (exit status 1)
$ rm ngx/conf.d/typo.conf; nginx -t
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
Missing semicolon, unknown directive, an upstream host name that does not resolve at start-up (nginx resolves upstream names ONCE, when it loads the configuration, and refuses to start if it cannot: use IP addresses, or a resolver with a variable, for names that may not exist yet) and duplicate default servers all fail the test and name the file and line. A reload with a bad file prints the error (and logs it) and keeps serving, so a typo never takes the site down, as long as you NOTICE the error: reload, then check the result, not just that the command returned.
$ sudo systemctl reload nginx && sudo journalctl -u nginx -n 5 --no-pager # on a server: read the reload result
$ sudo nginx -T | less # -T prints the ENTIRE effective configuration, includes expanded (what nginx really runs)
$ nginx -T 2>/dev/null | grep -E '^\s*(listen|server_name|root)' | sed 's/^ *//'
listen 127.0.0.1:19080;
server_name www.example.test;
root html;
$ nginx -V 2>&1 | tr ' ' '\n' | grep -- '--with' | head -30 # which modules this build has (Ubuntu's nginx includes ssl, v2, realip, stub_status and more)
Which server block answers? (server_name and default_server)
When several server blocks listen on the same port, nginx picks one using the Host header of the request: first an exact server_name match, then the longest wildcard starting with *., then the longest wildcard ending in .*, then regular expressions, and finally the DEFAULT server for that port (the first one listed, or the one marked default_server).
$ cat > ngx/conf.d/site.conf <<'EOF'
> server {
> listen 127.0.0.1:19080 default_server;
> server_name _;
> return 200 "default server (no name matched)\n";
> }
> server {
> listen 127.0.0.1:19080;
> server_name www.example.test example.test;
> return 200 "www server\n";
> }
> server {
> listen 127.0.0.1:19080;
> server_name *.example.test;
> return 200 "wildcard *.example.test server\n";
> }
> server {
> listen 127.0.0.1:19080;
> server_name ~^api(\d+)\.example\.test$;
> return 200 "regex server (api$1)\n";
> }
> EOF
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ for h in www.example.test example.test shop.example.test api7.example.test unknown.org; do printf '%-22s -> ' $h; curl -s -H "Host: $h" http://127.0.0.1:19080/; done
www.example.test -> www server
example.test -> www server
shop.example.test -> wildcard *.example.test server
api7.example.test -> wildcard *.example.test server
unknown.org -> default server (no name matched)
$ curl -s http://127.0.0.1:19080/
default server (no name matched)
$ curl -s -H 'Host:' http://127.0.0.1:19080/
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.24.0 (Ubuntu)</center>
</body>
</html>
Notice that api7.example.test went to the WILDCARD block, not the regular expression one: wildcards are tried before regular expressions, so put a regex server before a wildcard only by making the wildcard more specific. The default server answers anything that matches nothing else (scanners sending the bare IP address or random host names). Make it explicit, and make it boring: return 444 (close the connection with no reply) or a plain 404, so that your real sites are not leaked to someone who only knows the IP. Test virtual hosts without changing DNS by sending the Host header (curl -H) or with curl --resolve.
location: how nginx chooses a URL block
Inside a server, location blocks match the path of the request. The order they are written in does NOT decide, the TYPE does:
1. location = /path exact match: wins immediately
2. location ^~ /prefix a prefix match that, if it is the longest prefix, stops the search (regular expressions are not tried)
3. location ~ regex regular expression, case-sensitive (~* is case-insensitive): the FIRST one in file order that matches wins
4. location /prefix plain prefix: the LONGEST matching prefix is remembered, and used only if no regex matched
5. location / the catch-all prefix
Here is a test rig where every block just announces itself:
$ cat > ngx/conf.d/site.conf <<'EOF'
> server {
> listen 127.0.0.1:19080;
> location / { return 200 "location / (catch-all)\n"; }
> location = / { return 200 "location = / (exact: the home page)\n"; }
> location /images/ { return 200 "location /images/ (prefix)\n"; }
> location ^~ /static/ { return 200 "location ^~ /static/ (prefix that blocks regex)\n"; }
> location ~ \.(jpg|png)$ { return 200 "location ~ jpg|png (regex)\n"; }
> location ~* \.PDF$ { return 200 "location ~* PDF (case-insensitive regex)\n"; }
> location = /login { return 200 "location = /login (exact)\n"; }
> }
> EOF
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ for u in / /about /images/ /images/cat.jpg /static/logo.png /static/a.txt /photo.png /report.pdf /login /login/ /Login; do printf '%-18s -> ' $u; curl -s http://127.0.0.1:19080$u; done
/ -> location = / (exact: the home page)
/about -> location / (catch-all)
/images/ -> location /images/ (prefix)
/images/cat.jpg -> location ~ jpg|png (regex)
/static/logo.png -> location ^~ /static/ (prefix that blocks regex)
/static/a.txt -> location ^~ /static/ (prefix that blocks regex)
/photo.png -> location ~ jpg|png (regex)
/report.pdf -> location ~* PDF (case-insensitive regex)
/login -> location = /login (exact)
/login/ -> location / (catch-all)
/Login -> location / (catch-all)
Read the results against the rules: /images/cat.jpg matched the REGEX (jpg) even though /images/ is a longer prefix, because a regular expression beats a plain prefix; /static/logo.png matched ^~ /static/ and so the png regex was never tried; /login matched exactly but /login/ and /Login fall through to the catch-all (exact means exact). When a location behaves "wrongly", this table is the answer 90% of the time. Rules of thumb: use exact (=) for single URLs, ^~ for a folder of static files, regexes sparingly (they are matched in order and are slower to reason about), and put a plain "location /" last as the safety net.
$ nginx -T 2>/dev/null | grep -c 'location'
7
root, alias and try_files
Two ways to map a URL to a folder, and one is a trap:
$ mkdir -p ngx/html/docs ngx/other/files && echo 'in html/docs' > ngx/html/docs/readme.txt && echo 'in other/files' > ngx/other/files/readme.txt
$ cat > ngx/conf.d/site.conf <<'EOF'
> server {
> listen 127.0.0.1:19080;
> root html;
>
> location /docs/ { } # uses root: html + /docs/readme.txt
> location /files/ { alias other/files/; } # alias REPLACES the matched prefix: other/files/ + readme.txt
> location /bad/ { alias other/files; } # a classic mistake: missing slash
> }
> EOF
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ curl -s http://127.0.0.1:19080/docs/readme.txt
in html/docs
$ curl -s http://127.0.0.1:19080/files/readme.txt
in other/files
$ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:19080/bad/readme.txt
404
$ tail -n 1 ngx/logs/error.log | sed -E 's/^[0-9\/]+ [0-9:]+ //; s/[0-9]+#[0-9]+: (\*[0-9]+ )?//; s/, client:.*//' | cut -c1-130
[error] open() "/home/demo/ngx/other/filesreadme.txt" failed (2: No such file or directory)
root APPENDS the URL path to the folder (root html + /docs/readme.txt = html/docs/readme.txt). alias SUBSTITUTES the matched part: /files/readme.txt with alias other/files/ becomes other/files/readme.txt. When the location ends with / the alias must end with / too, or the paths run together (the last request looked for "other/filesreadme.txt"). The error log line says exactly which file nginx tried: always read it first when you get a 404.
try_files checks a list of candidates in order and uses the first that exists, with a fallback at the end: the heart of single-page apps and "pretty URLs".
$ cat > ngx/conf.d/site.conf <<'EOF'
> server {
> listen 127.0.0.1:19080;
> root html;
> index index.html;
>
> location / {
> try_files $uri $uri/ /index.html; # file, then folder, then the app shell
> }
> location /api/ {
> try_files $uri =404; # no fallback: a missing API path is a real 404
> }
> error_page 404 /404.html;
> location = /404.html { internal; }
> }
> EOF
$ echo '<h1>custom not found page</h1>' > ngx/html/404.html
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ curl -s http://127.0.0.1:19080/some/client/route
<h1>Hello from nginx</h1>
$ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:19080/api/nothing
404
$ curl -s http://127.0.0.1:19080/api/nothing
<h1>custom not found page</h1>
$ curl -s http://127.0.0.1:19080/notes.txt
plain text file
"internal" means the location can only be reached by nginx itself (an error page or an internal redirect), never directly from a browser. Other useful static-file directives: autoindex on (a directory listing: enable only where you mean it), index (the default file), expires and add_header Cache-Control (browser caching, below), and error_page 500 502 503 504 /50x.html for friendly failure pages.
Redirects and rewrites
$ cat > ngx/conf.d/site.conf <<'EOF'
> server {
> listen 127.0.0.1:19080;
> server_name example.test;
> return 301 http://www.example.test:19080$request_uri; # canonical host: bare name redirects to www, keeping the path and query
> }
> server {
> listen 127.0.0.1:19080;
> server_name www.example.test;
> root html;
>
> location = /old-page { return 301 /new-page; }
> location /blog/ { rewrite "^/blog/(\d{4})/(.*)$" /archive/$1/$2 permanent; }
> location /legacy/ { rewrite ^/legacy/(.*)$ /current/$1 last; }
> location /current/ { return 200 "served /current/ for: $uri\n"; }
> location = /gone { return 410; }
> location = /maintenance { return 503 "back soon\n"; }
> }
> EOF
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ curl -sI -H 'Host: example.test' 'http://127.0.0.1:19080/pricing?plan=pro' | grep -E '^(HTTP|Location)'
HTTP/1.1 301 Moved Permanently
Location: http://www.example.test:19080/pricing?plan=pro
$ curl -sI -H 'Host: www.example.test' http://127.0.0.1:19080/old-page | grep -E '^(HTTP|Location)'
HTTP/1.1 301 Moved Permanently
Location: http://www.example.test:19080/new-page
$ curl -sI -H 'Host: www.example.test' http://127.0.0.1:19080/blog/2026/hello | grep -E '^(HTTP|Location)'
HTTP/1.1 301 Moved Permanently
Location: http://www.example.test:19080/archive/2026/hello
$ curl -s -H 'Host: www.example.test' http://127.0.0.1:19080/legacy/page.html
served /current/ for: /current/page.html
$ curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: www.example.test' http://127.0.0.1:19080/gone
410
When to use which: "return 301 URL" is the simple, fast, explicit way for a redirect (prefer it whenever you can write the destination directly). "rewrite ... permanent" (301) or "redirect" (302) redirects with a regular expression. "rewrite ... last" (without a redirect flag) changes the URL INTERNALLY and searches the locations again (the browser never sees it); "break" stops processing rewrites and serves from the current location. $request_uri is the original path plus query string, $uri the normalized path that may have been rewritten: use $request_uri to keep the query string in a redirect. 301 is cached by browsers (hard to undo while testing): use 302 until you are sure.
Avoid "if" inside location for anything but return and rewrite: "if is evil" is nginx folklore for good reasons (it does not combine with other directives the way it reads). Use map, try_files or separate locations instead:
$ cat > ngx/conf.d/map.conf <<'EOF'
> map $http_user_agent $is_bot {
> default 0;
> ~*(curl|wget|python) 1;
> }
> server {
> listen 127.0.0.1:19081;
> location / { return 200 "bot=$is_bot\n"; }
> }
> EOF
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ curl -s -A 'Mozilla/5.0' http://127.0.0.1:19081/; curl -s http://127.0.0.1:19081/
bot=0
bot=1
Reverse proxy: passing requests to an application
A reverse proxy takes the request from the client and forwards it to an application server on your behalf. The application sees nginx as the client, so you must pass the real client details in headers. The lab backend alpha (port 19101) echoes what it receives:
$ curl -s http://127.0.0.1:19101/hello
backend=alpha
GET /hello
Host: 127.0.0.1:19101
X-Real-IP: -
X-Forwarded-For: -
X-Forwarded-Proto: -
Connection: -
Upgrade: -
$ cat > ngx/conf.d/site.conf <<'EOF'
> server {
> listen 127.0.0.1:19080;
> server_name www.example.test;
>
> location /app/ {
> proxy_pass http://127.0.0.1:19101/; # note the trailing slash (see below)
> proxy_set_header Host $host;
> proxy_set_header X-Real-IP $remote_addr;
> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> proxy_set_header X-Forwarded-Proto $scheme;
> }
> }
> EOF
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ curl -s -H 'Host: www.example.test' http://127.0.0.1:19080/app/hello
backend=alpha
GET /hello
Host: www.example.test
X-Real-IP: 127.0.0.1
X-Forwarded-For: 127.0.0.1
X-Forwarded-Proto: http
Connection: close
Upgrade: -
$ curl -s -H 'Host: www.example.test' -H 'X-Forwarded-For: 203.0.113.50' http://127.0.0.1:19080/app/hello | grep Forwarded-For
X-Forwarded-For: 203.0.113.50, 127.0.0.1
What the backend received: the path WITHOUT /app/ (because of the trailing slash), the original Host, the client address in X-Real-IP and X-Forwarded-For (nginx appends to an existing chain: the second request shows how a client-supplied header is kept, which is why an application must only trust the LAST entries added by its own proxies) and the protocol. Without these headers every request would look like it came from 127.0.0.1 over HTTP.
The trailing slash on proxy_pass changes how the path is passed. This is the single most common reverse-proxy surprise:
$ cat > ngx/conf.d/site.conf <<'EOF'
> server {
> listen 127.0.0.1:19080;
> location /one/ { proxy_pass http://127.0.0.1:19101/; } # URI given ("/"): the matched /one/ is replaced by /
> location /two/ { proxy_pass http://127.0.0.1:19101; } # no URI: the path is passed unchanged
> location /three/ { proxy_pass http://127.0.0.1:19101/api/; } # URI given ("/api/"): /three/ becomes /api/
> }
> EOF
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ for p in one two three; do printf '/%s/page -> backend saw: ' $p; curl -s http://127.0.0.1:19080/$p/page | sed -n '2p'; done
/one/page -> backend saw: GET /page
/two/page -> backend saw: GET /two/page
/three/page -> backend saw: GET /api/page
Rule: if proxy_pass has a URI part (even just "/") nginx REPLACES the matched location prefix with it; if it has no URI part, the original path is sent as it is. Use variables or regular expression locations and the rule changes again (then you must build the whole path yourself). Always check with a request to a backend that echoes the path.
More proxy settings you will want in production (shown, not run):
proxy_http_version 1.1; proxy_set_header Connection ""; keep-alive connections to the backend (needed for upstream keepalive)
proxy_read_timeout 60s; proxy_connect_timeout 5s; proxy_send_timeout 60s; how long to wait (see 504 below)
proxy_buffering on; proxy_buffer_size 8k; proxy_buffers 8 16k; response buffering; "upstream sent too big header" means raise proxy_buffer_size
client_max_body_size 20m; uploads larger than this get 413
WebSockets: proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade";
When the backend is the problem: 502 and 504
Nginx itself is working, and it tells you which part is not. A 502 means "I could not get a valid answer from the backend"; a 504 means "the backend took too long". Reproduce both:
$ cat > ngx/conf.d/site.conf <<'EOF'
> server {
> listen 127.0.0.1:19080;
> location /ok/ { proxy_pass http://127.0.0.1:19101/; }
> location /down/ { proxy_pass http://127.0.0.1:19999/; } # nothing listens here
> location /slow/ { proxy_pass http://127.0.0.1:19101/; proxy_read_timeout 1s; }
> }
> EOF
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ curl -s -o /dev/null -w '/ok/ -> %{http_code}\n' http://127.0.0.1:19080/ok/
/ok/ -> 200
$ curl -s -o /dev/null -w '/down/ -> %{http_code}\n' http://127.0.0.1:19080/down/
/down/ -> 502
$ curl -s -o /dev/null -w '/slow/ -> %{http_code}\n' http://127.0.0.1:19080/slow/slow
/slow/ -> 504
$ curl -s http://127.0.0.1:19080/down/ | grep -oE '<title>.*</title>'
<title>502 Bad Gateway</title>
$ tail -n 4 ngx/logs/error.log | grep -E 'upstream' | sed -E 's/^[0-9\/]+ [0-9:]+ //; s/[0-9]+#[0-9]+: (\*[0-9]+ )?//; s/, client: [0-9.]+, server.*, upstream/, upstream/' | cut -c1-175
[error] connect() failed (111: Connection refused) while connecting to upstream, upstream: "http://127.0.0.1:19999/", host: "127.0.0.1:19080"
[error] upstream timed out (110: Connection timed out) while reading response header from upstream, upstream: "http://127.0.0.1:19101/slow", host: "127.0.0.1:19080"
[error] connect() failed (111: Connection refused) while connecting to upstream, upstream: "http://127.0.0.1:19999/", host: "127.0.0.1:19080"
The error log gives the cause in plain words: "connect() failed (111: Connection refused) while connecting to upstream" means nothing is listening at that address and port (the application is down, crashed, or on another port); "upstream timed out (110: Connection timed out) while reading response header from upstream" means it accepted the request but did not answer within proxy_read_timeout. The access log shows the timings, which is how you spot slow backends:
$ tail -n 3 ngx/logs/access.log | sed -E 's/^[0-9.]+ //'
"GET /down/ HTTP/1.1" 502 166 rt=0.000 up=0.000 cache=-
"GET /slow/slow HTTP/1.1" 504 176 rt=1.001 up=1.001 cache=-
"GET /down/ HTTP/1.1" 502 166 rt=0.001 up=0.001 cache=-
rt= is the total time nginx spent on the request; up= is the time the upstream took ("-" when there was none, as for the refused connection). A big rt with a small up means the client or the network was slow; both big means a slow application. Other 502 causes: the backend closed the connection mid-reply (it crashed on that request), the backend speaks another protocol than you configured (https where nginx expected http), a socket permission problem (nginx cannot open the unix socket of PHP-FPM or gunicorn: 13: Permission denied), SELinux blocking network connections from nginx (setsebool -P httpd_can_network_connect 1), and a backend that sends a response header larger than the buffers.
Several backends: load balancing and failover
An upstream block names a pool of servers; proxy_pass sends requests to the pool and nginx spreads them (round robin by default), retrying another server if one fails.
$ cat > ngx/conf.d/site.conf <<'EOF'
> upstream app_pool {
> zone app_pool 64k; # shared memory: all workers share the balancing state
> server 127.0.0.1:19101; # alpha
> server 127.0.0.1:19102; # beta
> # server 127.0.0.1:19103 backup; # used only when the others are down
> # least_conn; # or: ip_hash; hash $request_uri consistent;
> keepalive 16;
> }
> server {
> listen 127.0.0.1:19080;
> location / {
> proxy_pass http://app_pool;
> proxy_http_version 1.1;
> proxy_set_header Connection "";
> proxy_next_upstream error timeout http_502 http_503; # retry on these failures
> }
> }
> EOF
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ for i in 1 2 3 4 5 6; do curl -s http://127.0.0.1:19080/count | cut -d' ' -f1; done | sort | uniq -c
3 backend=alpha
3 backend=beta
Six requests were shared evenly between alpha and beta. (Note the "zone" line: without it every worker process keeps its OWN round-robin counter and failure memory, so on a multi-core server a handful of test requests can all land on the same backend and a dead server is noticed separately by each worker. Always add a zone to an upstream.) Now kill one backend and watch nginx carry on: it notices the failure, retries the request on the other server and stops sending traffic to the dead one for a while (fail_timeout), so users see no errors.
$ for i in 1 2 3 4 5 6; do curl -s -o /dev/null -w '%{http_code} ' http://127.0.0.1:19080/count; done; echo
200 200 200 200 200 200
$ for i in 1 2 3 4 5 6; do curl -s http://127.0.0.1:19080/count | cut -d' ' -f1; done | sort | uniq -c
6 backend=alpha
$ grep -c 'connect() failed' ngx/logs/error.log
3
All answers were 200 and every one came from alpha; the error log recorded the refused connections to beta (each failure is logged even though the user never saw it, so watch the error log, not just the status codes). Bring it back (after nginx's fail_timeout, 10 seconds by default, it tries the server again and puts it back in the rotation):
$ for i in 1 2 3 4 5 6; do curl -s http://127.0.0.1:19080/count | cut -d' ' -f1; done | sort | uniq -c
4 backend=alpha
2 backend=beta
Tuning the pool: "server ADDR weight=3" (a larger share), "max_fails=3 fail_timeout=30s" (how many failures within how long mark a server as down, and for how long), "backup" (a standby), "down" (taken out of rotation by hand, for maintenance), and "least_conn" (to the server with the fewest active requests, better when request times vary a lot). ip_hash keeps a client on the same server (sticky sessions by address). Open-source nginx checks backends only passively (by failures on real requests); active health checks are a commercial feature, or are done by an external tool.
Caching responses
$ cat > ngx/conf.d/site.conf <<'EOF'
> proxy_cache_path tmp/cache levels=1:2 keys_zone=lab:5m max_size=50m inactive=10m;
>
> server {
> listen 127.0.0.1:19080;
> location / {
> proxy_pass http://127.0.0.1:19101;
> proxy_cache lab;
> proxy_cache_valid 200 10s; # cache good answers for 10 seconds
> proxy_cache_use_stale error timeout updating http_502 http_503; # serve OLD content if the backend fails
> add_header X-Cache-Status $upstream_cache_status always;
> }
> }
> EOF
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ for i in 1 2 3; do curl -si http://127.0.0.1:19080/count | grep -E '^(backend|X-Cache)' | tr '\n' ' '; echo; done
X-Cache-Status: MISS
backend=alpha count=28
X-Cache-Status: HIT
backend=alpha count=28
X-Cache-Status: HIT
backend=alpha count=28
The first request was a MISS (nginx asked the application: count went up) and the next two were HITs served from nginx's disk cache (the same count, the application was not touched). Wait for the cache to expire and the answer is refreshed. And the safety net: if the application dies, nginx can keep serving the stale copy:
$ curl -si http://127.0.0.1:19080/count | grep -E '^(backend|X-Cache)' | tr '\n' ' '; echo
X-Cache-Status: EXPIRED
backend=alpha count=29
$ curl -si http://127.0.0.1:19080/count | grep -E '^(backend|X-Cache)' | tr '\n' ' '; echo
X-Cache-Status: STALE
backend=alpha count=29
$ curl -s -o /dev/null -w 'a path nobody cached yet, with the backend down: %{http_code}\n' http://127.0.0.1:19080/never-cached
a path nobody cached yet, with the backend down: 502
After the pause the first request shows EXPIRED: the copy had aged out, so nginx fetched a fresh one (the count went up). Then the application was stopped and the copy aged out again: the next request shows STALE (the same count, nginx served the old copy instead of an error). A URL that was never cached has nothing to fall back on, so it returns 502. X-Cache-Status values: MISS (fetched), HIT (served from cache), EXPIRED (expired and refreshed), STALE (served old because of proxy_cache_use_stale), UPDATING, BYPASS, REVALIDATED. Things to get right before caching in production: only cache what is the same for everyone (NEVER pages that depend on a cookie or login: proxy_cache_bypass $cookie_session / proxy_no_cache, or a cache key that includes it); the application's own Cache-Control and Set-Cookie headers influence what nginx will cache; and add proxy_cache_lock on so that one expired object does not trigger a stampede of identical requests to the backend. Even a one-second cache ("microcaching") can take a heavy page from hundreds of backend requests per second to one.
Compression and static-file performance
$ for i in $(seq 1 2400); do printf 'line %05d: the quick brown fox jumps over the lazy dog\n' $i; done > ngx/html/big.txt
$ cat > ngx/conf.d/site.conf <<'EOF'
> gzip on;
> gzip_min_length 1024; # tiny responses are not worth compressing
> gzip_comp_level 5; # 1 (fast) to 9 (small); 4-6 is the sweet spot
> gzip_types text/plain text/css application/json application/javascript image/svg+xml;
> gzip_vary on; # tell caches the answer depends on Accept-Encoding
>
> server {
> listen 127.0.0.1:19080;
> root html;
> location /assets/ {
> add_header Cache-Control "public, max-age=604800, immutable";
> }
> }
> EOF
$ mkdir -p ngx/html/assets && echo 'body { color: #333; }' > ngx/html/assets/site.css
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ curl -s -o /dev/null -w 'plain: %{size_download} bytes\n' http://127.0.0.1:19080/big.txt
plain: 134400 bytes
$ curl -s -o /dev/null -w 'gzip: %{size_download} bytes\n' -H 'Accept-Encoding: gzip' http://127.0.0.1:19080/big.txt
gzip: 6234 bytes
$ curl -sI -H 'Accept-Encoding: gzip' http://127.0.0.1:19080/big.txt | grep -E '^(Content-Encoding|Vary)'
Vary: Accept-Encoding
Content-Encoding: gzip
$ curl -sI http://127.0.0.1:19080/assets/site.css | grep -E '^Cache-Control'
Cache-Control: public, max-age=604800, immutable
The text shrank to a small fraction of its size. Compress text-like content (HTML, CSS, JavaScript, JSON, SVG, plain text); do NOT compress images, video, archives or PDFs (already compressed: it wastes CPU). Static assets with a content hash in their name (app.3f9a1c.js) can be cached for a year with "immutable"; HTML pages should not. For big static sites also consider: sendfile on; tcp_nopush on; (efficient file sending), open_file_cache max=1000 inactive=20s; (caches file handles), gzip_static on; (serve pre-compressed .gz files), and http2 (below).
HTTPS: certificates, redirects and a modern configuration
$ cat > ngx/conf.d/site.conf <<'EOF'
> server {
> listen 127.0.0.1:19080;
> server_name www.example.test;
> return 301 https://$host:19443$request_uri; # on a real server: https://$host$request_uri
> }
> server {
> listen 127.0.0.1:19443 ssl http2; # nginx 1.24: http2 goes on the listen line (1.25+ has a separate "http2 on;")
> server_name www.example.test;
>
> ssl_certificate certs/www.pem; # on a real server: the FULL CHAIN file (fullchain.pem)
> ssl_certificate_key certs/www.key;
> ssl_protocols TLSv1.2 TLSv1.3;
> ssl_prefer_server_ciphers off;
> ssl_session_cache shared:SSL:10m;
> ssl_session_timeout 1d;
> ssl_session_tickets off;
>
> add_header Strict-Transport-Security "max-age=31536000" always;
> root html;
> }
> EOF
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ curl -sI -H 'Host: www.example.test' http://127.0.0.1:19080/notes.txt | grep -E '^(HTTP|Location)'
HTTP/1.1 301 Moved Permanently
Location: https://www.example.test:19443/notes.txt
$ curl -s --cacert ngx/certs/ca.pem --resolve www.example.test:19443:127.0.0.1 https://www.example.test:19443/notes.txt
plain text file
$ curl -sI --http2 --cacert ngx/certs/ca.pem --resolve www.example.test:19443:127.0.0.1 https://www.example.test:19443/ | grep -E '^(HTTP|strict)'
HTTP/2 200
strict-transport-security: max-age=31536000
$ echo | openssl s_client -connect 127.0.0.1:19443 -servername www.example.test -CAfile ngx/certs/ca.pem -verify_hostname www.example.test -brief 2>&1 | grep -E 'Protocol version|Verification'
Protocol version: TLSv1.3
Verification: OK
$ echo | openssl s_client -connect 127.0.0.1:19443 -servername www.example.test -tls1_2 -brief 2>&1 | grep -E 'Protocol version'
Protocol version: TLSv1.2
$ echo | openssl s_client -connect 127.0.0.1:19443 -servername www.example.test -tls1_1 -cipher 'DEFAULT:@SECLEVEL=0' 2>&1 | grep -oE 'tlsv1 alert protocol version'
tlsv1 alert protocol version
The HTTP side redirects with a 301 to HTTPS, the HTTPS side answers over TLS 1.3 (and 1.2) and HTTP/2, the HSTS header tells browsers to use HTTPS only for a year (add it only when you are sure, it is hard to undo), and TLS 1.1 is refused. Real certificates:
$ sudo apt install certbot python3-certbot-nginx
$ sudo certbot --nginx -d www.example.com -d example.com # obtains a certificate, edits the server block, sets up renewal
$ sudo certbot renew --dry-run # test the renewal; it reloads nginx by itself through its deploy hook
$ echo | openssl s_client -connect www.example.com:443 -servername www.example.com 2>/dev/null | openssl x509 -noout -dates -issuer # what is actually being served
Use the full chain file (the site certificate followed by the intermediates) as ssl_certificate: with only the leaf, browsers may cope but curl, Java and mobile apps fail with "unable to get local issuer certificate". After a renewal nginx must be reloaded to use the new file (certbot does it). More TLS settings you may add: ssl_stapling on; ssl_stapling_verify on; (OCSP stapling, needs a resolver), ssl_dhparam for old DHE cipher suites, and for the strictest clients the Mozilla "intermediate" cipher list (the TLS guide and the Firewall/hardening pages cover the policy side). Test your deployed site with a public scanner such as SSL Labs, and with openssl s_client as above.
Security and limits
$ cat > ngx/conf.d/site.conf <<'EOF'
> limit_req_zone $binary_remote_addr zone=perip:10m rate=2r/s;
> limit_conn_zone $binary_remote_addr zone=conns:10m;
> limit_req_status 429;
>
> server {
> listen 127.0.0.1:19080;
> server_tokens off;
> root html;
> client_max_body_size 1m;
> client_body_timeout 10s;
> client_header_timeout 10s;
> keepalive_timeout 15s;
>
> add_header X-Content-Type-Options nosniff always;
> add_header X-Frame-Options SAMEORIGIN always;
> add_header Referrer-Policy no-referrer always;
>
> location / { }
> location ~ /\. { deny all; } # .git, .env, .htaccess are never served
> location /login { limit_req zone=perip burst=3 nodelay; limit_conn conns 5; proxy_pass http://127.0.0.1:19101; }
> location /upload { proxy_pass http://127.0.0.1:19101; }
> location /admin/ { allow 10.0.0.0/8; deny all; } # only the 10.x network
> location /office/ { allow 127.0.0.1; deny all; } # only this machine
> location /private/ { auth_basic "Staff only"; auth_basic_user_file htpasswd; }
> }
> EOF
$ mkdir -p ngx/html/admin ngx/html/office ngx/html/private; echo 'admin area' > ngx/html/admin/index.html; echo 'office area' > ngx/html/office/index.html; echo 'welcome, staff' > ngx/html/private/index.html
$ printf 'alice:%s\n' "$(openssl passwd -apr1 'correct-horse')" > ngx/htpasswd
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ curl -sI http://127.0.0.1:19080/ | grep -E '^(Server|X-Content|X-Frame|Referrer)' | tr '\n' ' '; echo
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Referrer-Policy: no-referrer
$ echo 'DB_PASSWORD=hunter2' > ngx/html/.env; curl -s -o /dev/null -w '.env -> %{http_code}\n' http://127.0.0.1:19080/.env
.env -> 403
$ curl -s -o /dev/null -w '/admin/ (only 10.x allowed) -> %{http_code}\n' http://127.0.0.1:19080/admin/; curl -s -w '/office/ (127.0.0.1 allowed) -> %{http_code}\n' -o /dev/null http://127.0.0.1:19080/office/
/admin/ (only 10.x allowed) -> 403
/office/ (127.0.0.1 allowed) -> 200
$ curl -s -o /dev/null -w '/private/ with no login -> %{http_code}\n' http://127.0.0.1:19080/private/
/private/ with no login -> 401
$ curl -s -u alice:wrong -o /dev/null -w '/private/ wrong password -> %{http_code}\n' http://127.0.0.1:19080/private/
/private/ wrong password -> 401
$ curl -s -u alice:correct-horse http://127.0.0.1:19080/private/
welcome, staff
$ head -c 2000000 /dev/zero > /home/demo/big.bin; curl -s -o /dev/null -w 'POST of 2 MB (limit 1m) -> %{http_code}\n' --data-binary @/home/demo/big.bin http://127.0.0.1:19080/upload
POST of 2 MB (limit 1m) -> 413
$ head -c 500000 /dev/zero > /home/demo/ok.bin; curl -s --data-binary @/home/demo/ok.bin http://127.0.0.1:19080/upload
backend=alpha received 500000 bytes
$ for i in $(seq 1 10); do curl -s -o /dev/null -w '%{http_code} ' http://127.0.0.1:19080/login; done; echo
200 200 200 200 429 429 429 429 429 429
The rate limit: 2 requests a second with a burst of 3 means a client's first requests pass, then the rest are refused with 429 until the rate recovers (the lab runs the ten requests in a few milliseconds). Always set limit_req_status 429 (the default is 503, which looks like an outage). Do not test these with "return": return is executed in an earlier phase than allow, deny and auth_basic, so a location that only says return 200 is open to everyone whatever the access rules say (the first version of this very example fell into that trap). Serve a real file or proxy_pass instead. Basic authentication sends the password in every request, so use it only over HTTPS. For anything serious use an application login or an authentication proxy; auth_request can ask another service to approve each request.
Hiding the version and sending security headers costs nothing. More hardening: limit_except GET HEAD { deny all; } (only allow certain methods), return 444 (close without answering) for clearly malicious paths, server blocks that refuse requests for unknown hosts, a low client_max_body_size by default and a higher one only where uploads are expected, and keeping nginx updated (security fixes arrive through normal package updates: apt upgrade, then a reload or restart picks them up).
Behind another proxy or load balancer, $remote_addr is the PROXY's address, which breaks rate limits and logs. The realip module restores the client address, but trust it only from addresses you control:
$ cat > ngx/conf.d/site.conf <<'EOF'
> server {
> listen 127.0.0.1:19080;
> set_real_ip_from 127.0.0.1; # trust X-Forwarded-For only from this proxy
> real_ip_header X-Forwarded-For;
> real_ip_recursive on;
> location / { return 200 "you are $remote_addr\n"; }
> }
> EOF
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ curl -s http://127.0.0.1:19080/; curl -s -H 'X-Forwarded-For: 203.0.113.50' http://127.0.0.1:19080/
you are 127.0.0.1
you are 203.0.113.50
Without set_real_ip_from anyone could put any address in X-Forwarded-For and spoof it. With it, the header is honoured only when the connection comes from your own proxy.
Logs and debugging
Two logs: access.log (one line per request) and error.log (what went wrong, with a level: debug, info, notice, warn, error, crit, alert, emerg). The log_format in the lab's main file added timings and the cache status. Read what a few requests produced:
$ printf 'server {\n listen 127.0.0.1:19080;\n root html;\n}\n' > ngx/conf.d/site.conf; nginx -t 2>/dev/null && nginx -s reload
$ curl -s -o /dev/null http://127.0.0.1:19080/; curl -s -o /dev/null http://127.0.0.1:19080/nothing-here; true
$ tail -n 3 ngx/logs/access.log
203.0.113.50 "GET / HTTP/1.1" 200 21 rt=0.000 up=- cache=-
127.0.0.1 "GET / HTTP/1.1" 200 26 rt=0.000 up=- cache=-
127.0.0.1 "GET /nothing-here HTTP/1.1" 404 162 rt=0.000 up=- cache=-
Turn that file into answers with ordinary tools (the format starts with the client address, then the request in quotes, then the status: field 5 is the status, field 3 the path, field 7 the request time):
$ cat > /home/demo/sample-access.log <<'EOF'
> 203.0.113.9 "GET / HTTP/1.1" 200 612 rt=0.002 up=- cache=-
> 203.0.113.9 "GET /api/items HTTP/1.1" 200 1840 rt=0.140 up=0.139 cache=MISS
> 198.51.100.4 "GET /api/items HTTP/1.1" 200 1840 rt=0.001 up=- cache=HIT
> 198.51.100.4 "GET /login HTTP/1.1" 502 157 rt=0.003 up=0.003 cache=-
> 198.51.100.4 "GET /login HTTP/1.1" 502 157 rt=0.002 up=0.002 cache=-
> 192.0.2.77 "GET /wp-login.php HTTP/1.1" 404 146 rt=0.000 up=- cache=-
> 203.0.113.9 "GET /report HTTP/1.1" 504 160 rt=60.001 up=60.001 cache=-
> EOF
$ awk '{print $5}' /home/demo/sample-access.log | sort | uniq -c | sort -rn
3 200
2 502
1 504
1 404
$ awk '$5 >= 500 {print $5, $3}' /home/demo/sample-access.log | sort | uniq -c
2 502 /login
1 504 /report
$ awk '{split($7, a, "="); if (a[2] > 1) print "slow:", $3, a[2] "s"}' /home/demo/sample-access.log
slow: /report 60.001s
$ awk '{print $1}' /home/demo/sample-access.log | sort | uniq -c | sort -rn | head -3
3 203.0.113.9
3 198.51.100.4
1 192.0.2.77
$ grep -c 'cache=HIT' /home/demo/sample-access.log
1
Handy log recipes on a real server (sudo needed to read the files):
$ sudo tail -f /var/log/nginx/error.log # watch errors live while you test
$ sudo awk '$9 ~ /^5/' /var/log/nginx/access.log | tail # (default "combined" format: field 9 is the status) recent 5xx
$ sudo grep ' 404 ' /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head # the most requested missing URLs
$ sudo journalctl -u nginx --since "10 min ago" # start, reload and crash messages from systemd
$ sudo nginx -T | grep -n 'access_log\|error_log' # where does THIS config log?
Log levels and debugging: error_log /var/log/nginx/error.log warn; is the usual level; for a stubborn problem raise it to info or debug for ONE site or client for a short time (debug is extremely verbose and the packages build includes it: error_log ... debug; with "debug_connection 203.0.113.9;" in the events block limits it to one client). Remember to put it back.
The built-in status page shows live counters (enable it only on a private address):
$ cat > ngx/conf.d/site.conf <<'EOF'
> server {
> listen 127.0.0.1:19080;
> location = /nginx_status { stub_status; allow 127.0.0.1; deny all; }
> }
> EOF
$ nginx -t && nginx -s reload
nginx: the configuration file /home/demo/ngx/nginx.conf syntax is ok
nginx: configuration file /home/demo/ngx/nginx.conf test is successful
$ curl -s http://127.0.0.1:19080/nginx_status | sed -n '1,2p'
Active connections: 1
server accepts handled requests
$ curl -s http://127.0.0.1:19080/nginx_status | awk 'NR==3 {print ($1==$2 ? "accepts == handled: no connection was dropped" : "handled < accepts: connections were dropped!")}'
accepts == handled: no connection was dropped
Active connections is the current load; the three counters (accepts, handled, requests) only grow, and "handled" falling behind "accepts" means nginx ran out of worker_connections or file descriptors. Monitoring tools scrape this page.
Test from the outside the way a client sees it: curl -v shows the whole conversation (connect, TLS, request headers, response headers), curl -w the timings, and the response headers of your own add_header lines prove which config answered:
$ curl -v --resolve www.example.test:443:203.0.113.5 https://www.example.test/ 2>&1 | grep -E '^[*<>] ' | head -30
Running it on a real server
$ sudo apt install nginx # Ubuntu and Debian; sudo dnf install nginx on RHEL (package nginx from the AppStream module)
$ sudo systemctl enable --now nginx
$ sudo nginx -t # ALWAYS before a reload
$ sudo systemctl reload nginx # graceful: new workers, old ones finish their requests
$ sudo systemctl restart nginx # full restart: drops connections; needed only for some changes (e.g. a new binary, changing worker user)
$ sudo ln -s /etc/nginx/sites-available/example /etc/nginx/sites-enabled/example && sudo nginx -t && sudo systemctl reload nginx # enable a site
$ sudo rm /etc/nginx/sites-enabled/example && sudo nginx -t && sudo systemctl reload nginx # disable it again
$ ls -l /etc/nginx/sites-enabled/; sudo nginx -T | grep -E 'server_name|listen' | head -20 # what is enabled, what does nginx listen for?
$ sudo ss -ltnp | grep nginx # which addresses and ports the master really opened
$ ps -eo pid,ppid,user,cmd | grep '[n]ginx' # one master (root) and workers (www-data)
$ sudo kill -HUP "$(cat /run/nginx.pid)" # the same as reload, by signal; -USR1 reopens the log files (logrotate does this)
Permissions: the workers run as an unprivileged user (www-data on Debian, nginx on RHEL) and need read access to the files AND execute (traverse) permission on every folder above them: a 403 on a file that "exists and looks readable" is usually a parent folder (namei -l /var/www/html/index.html shows the permissions of each part; see the Users and Permissions guides). The master needs root only to bind ports below 1024 and to read the private key.
Layout habits that age well: one file per site in sites-available, shared pieces in snippets (include snippets/security-headers.conf;), settings that apply to everything in conf.d, no config edited on the server without being in Git (see the Git for Sysadmins guide), and "nginx -t" in your deployment pipeline before any reload. Rotate logs with logrotate (the package installs it: weekly, 14 copies, reopen on rotation); watch the disk with the usual tools. Update with the normal package manager; nginx.org also publishes newer "mainline" packages if you need a feature earlier than Ubuntu ships it.
Tuning worth knowing (defaults are fine for most sites): worker_processes auto; (one per CPU core), worker_connections 1024; (per worker: connections = workers x this), worker_rlimit_nofile 65535; (file descriptors: each proxied request uses two connections), keepalive_timeout 65 (shorter on busy servers), reuseport on the listen line for very high connection rates, and upstream keepalive (shown earlier) to avoid opening a new backend connection per request.
Common problems
Symptom Cause and where to look
502 Bad Gateway the backend is down, on another port/address, speaks another protocol, or crashed on that request: error.log says "connect() failed (111: Connection refused)" or "upstream prematurely closed connection"
504 Gateway Timeout the backend is too slow: error.log "upstream timed out (110)", raise proxy_read_timeout only if the slowness is legitimate; find the slow query otherwise
403 Forbidden a missing index file with autoindex off, a file or parent folder the nginx user cannot read ("(13: Permission denied)" in error.log), an allow/deny rule, or SELinux
404 Not Found, but the file exists root versus alias confusion (error.log shows the exact path nginx tried), a wrong location match, a missing trailing slash, or a different server block answered (check server_name with curl -H Host)
413 Request Entity Too Large the upload exceeds client_max_body_size (default 1m): raise it for the upload location only
"too many redirects" (browser) a redirect loop: http to https at nginx AND at the application or a CDN, or www versus non-www rules fighting; curl -IL shows the chain
Works on HTTP, fails on HTTPS certificate and key mismatch, a missing intermediate (use fullchain), the listen line lacks "ssl", or port 443 is blocked: openssl s_client -connect host:443 -servername host
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use) another process (an old nginx, Apache) holds the port: sudo ss -ltnp | grep ':80 '
nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied) running without root, or SELinux/AppArmor; ports below 1024 need root at start
could not build the server_names_hash too many or too long server names: raise server_names_hash_bucket_size 128; in http
worker_connections are not enough raise worker_connections and the file descriptor limit; look for slow clients holding connections
upstream sent too big header while reading response header the backend's headers (often cookies) exceed the buffer: proxy_buffer_size 16k; proxy_buffers 8 16k;
Changes have no effect you edited a file that is not included (nginx -T shows what IS), forgot the reload, or the browser cached a 301 (test with curl)
The reload "worked" but nothing changed the new config was invalid and nginx kept the old one: read the error output of nginx -t and the error.log (see "Reloading safely")
Same URL, different answer for different people several server blocks or a cache keyed without the cookie or Accept-Encoding: curl -H and X-Cache-Status
Tips
nginx -t before every reload, and read the result after it. A reload with a typo keeps the old config and says so only in the log.
Remember the three picks: a server by Host and listen, a location by exact, ^~, regex, then longest prefix, and a path rewrite by the trailing slash of proxy_pass. Test them with curl, not by eye.
When a request goes wrong, read the error log first: it names the file, the upstream and the errno.
Pass the real client details to the backend (Host, X-Forwarded-For, X-Forwarded-Proto) and trust them only from your own proxy.
Cache only what is the same for everyone. Use proxy_cache_use_stale so an outage serves old content instead of an error page.
Redirect to HTTPS, use the full chain, reload after renewals, and add HSTS last.
Rate limit logins and expensive endpoints; hide the version; deny dotfiles; keep client_max_body_size small by default.
Keep configuration in Git, one file per site, and the whole thing boring. The best nginx configs are the ones nobody has to think about.
