The Linux kernel's network defaults are sensible for general use, but a web server that handles thousands of concurrent connections can hit limits such as full listen queues, exhausted ephemeral ports or too few file descriptors. In this tutorial you will measure the current state of an Nginx server on Ubuntu 24.04, apply a small, well-understood set of sysctl settings, enable the BBR congestion control algorithm, raise Nginx's file descriptor limit and verify each change with the counters the kernel exposes.
Prerequisites
To follow this guide you need:
- A server running Ubuntu 24.04 LTS (kernel 6.8) with Nginx installed, for example a CubePath VPS.
- A non-root user with
sudoprivileges. - Optionally, a second machine to generate load with
wrkso you can compare before and after.
ImportantKernel tuning is not a substitute for fixing slow application code. Change settings because a counter shows a problem, one group at a time, and measure the result. Copying large lists of sysctl values from the internet often does nothing, and some old values (such as
tcp_tw_recycle) no longer exist.
Step 1 - Measuring the current state
Start by recording the values you are going to change. The kernel reads them from /proc/sys, and sysctl prints them:
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog net.core.netdev_max_backlog \
net.ipv4.ip_local_port_range net.ipv4.tcp_tw_reuse net.ipv4.tcp_fin_timeout \
net.ipv4.tcp_congestion_control net.core.default_qdisc
Typical output on Ubuntu 24.04:
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 4096
net.core.netdev_max_backlog = 1000
net.ipv4.ip_local_port_range = 32768 60999
net.ipv4.tcp_tw_reuse = 2
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_congestion_control = cubic
net.core.default_qdisc = fq_codel
Next, look at the counters that reveal whether these limits are actually being hit. nstat from the iproute2 package prints kernel network statistics; -a shows absolute values and -z includes counters that are zero:
nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtTCPTimeWaitOverflow
#kernel
TcpExtListenOverflows 0 0.0
TcpExtListenDrops 0 0.0
TcpExtTCPTimeWaitOverflow 0 0.0
ListenOverflows and ListenDrops increase when a listening socket's accept queue is full and the kernel drops new connections. On a server with real traffic, run this command again during peak hours: if these counters grow, the tuning in Steps 2 and 4 applies to you.
Check the listen queues of Nginx itself. For listening sockets, Send-Q is the configured backlog and Recv-Q is the number of connections currently waiting to be accepted:
ss -ltn 'sport = :80 or sport = :443'
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 511 0.0.0.0:80 0.0.0.0:*
LISTEN 0 511 0.0.0.0:443 0.0.0.0:*
Nginx uses a backlog of 511 by default on Linux, regardless of somaxconn.
Finally, count connections by state:
ss -s
Total: 1843
TCP: 2210 (estab 1204, closed 861, orphaned 3, timewait 842)
Step 2 - Creating a persistent sysctl configuration
Settings applied with sysctl -w are lost at reboot. Put them in a file under /etc/sysctl.d/ instead, where systemd applies them at every boot. Create the file:
sudo nano /etc/sysctl.d/99-webserver.conf
Add the following settings. Each one is explained below the block:
# Accept queue and SYN backlog for busy listening sockets
net.core.somaxconn = 8192
net.ipv4.tcp_max_syn_backlog = 8192
# Packets queued between the network card and the TCP stack
net.core.netdev_max_backlog = 16384
# More ephemeral ports for outgoing connections (reverse proxy to upstreams)
net.ipv4.ip_local_port_range = 10240 65535
# Reuse TIME_WAIT sockets for new outgoing connections when safe
net.ipv4.tcp_tw_reuse = 1
# Release orphaned connections in FIN_WAIT_2 sooner
net.ipv4.tcp_fin_timeout = 30
# Keep the congestion window on idle keep-alive connections
net.ipv4.tcp_slow_start_after_idle = 0
# Larger maximum socket buffers for high-bandwidth, high-latency clients
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 131072 16777216
net.ipv4.tcp_wmem = 4096 16384 16777216
# BBR congestion control with the fq packet scheduler
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
What these settings do:
somaxconncaps the accept queue of every listening socket.tcp_max_syn_backloglimits half-open connections waiting for the final ACK of the handshake. Raising them only helps if the application also requests a larger backlog (Step 4).netdev_max_backlogis the per-CPU queue of received packets waiting for the kernel to process them. The default of 1000 can overflow during traffic bursts on fast links.ip_local_port_rangesets the source ports used for outgoing connections. When Nginx proxies to backends, each upstream connection uses one. The range starts at 10240 so it does not overlap with services listening on low ports.tcp_tw_reuse = 1lets the kernel reuse a socket inTIME_WAITfor a new outgoing connection when TCP timestamps make it safe. The Ubuntu default of2enables this only for loopback. It has no effect on incoming connections.tcp_fin_timeoutcontrols how long an orphaned socket stays inFIN_WAIT_2. It does not shortenTIME_WAIT, which is fixed at 60 seconds in the kernel.tcp_slow_start_after_idle = 0avoids resetting the congestion window of keep-alive connections that were idle for a moment, which helps HTTP/1.1 and HTTP/2 clients that reuse connections.- The buffer settings raise the maximum TCP buffer size to 16 MB. The kernel still auto-tunes each connection between the minimum and maximum, so memory is only used by connections that need it.
- BBR is a congestion control algorithm that models bandwidth and round-trip time instead of reacting only to packet loss, and usually improves throughput to distant or lossy clients. It is recommended with the
fqqueueing discipline.
Step 3 - Applying and verifying the settings
Ubuntu's kernel ships BBR as the tcp_bbr module. Make sure it loads at boot, before the sysctl file is applied:
echo tcp_bbr | sudo tee /etc/modules-load.d/bbr.conf
sudo modprobe tcp_bbr
Confirm that bbr is now listed as available:
sysctl net.ipv4.tcp_available_congestion_control
net.ipv4.tcp_available_congestion_control = reno cubic bbr
Load all sysctl configuration files, including the new one:
sudo sysctl --system
The output lists every file processed and the values it applied. Look for errors such as sysctl: cannot stat /proc/sys/..., which mean a key is misspelled or does not exist in this kernel.
Verify a few of the new values:
sysctl net.core.somaxconn net.ipv4.tcp_congestion_control net.core.default_qdisc
net.core.somaxconn = 8192
net.ipv4.tcp_congestion_control = bbr
net.core.default_qdisc = fq
The new congestion control applies to new connections. You can confirm that live connections use BBR with ss -ti, which prints bbr in the details of each socket:
ss -ti state established '( sport = :443 )' | grep -o bbr | head -3
The default_qdisc setting only applies to network interfaces brought up after the change. It takes full effect after the next reboot.
Step 4 - Raising Nginx's backlog and file descriptor limits
The kernel limits from Step 2 are only upper bounds. Nginx must also ask for a larger listen backlog and be allowed to open enough files. Every client connection and every upstream connection uses one file descriptor.
First, raise the limit of open files for the Nginx service with a systemd drop-in:
sudo systemctl edit nginx
In the editor that opens, add these lines between the comment markers:
[Service]
LimitNOFILE=65535
Next, edit the main Nginx configuration:
sudo nano /etc/nginx/nginx.conf
Set the number of files each worker may open and the connections per worker. worker_rlimit_nofile goes in the main context, and worker_connections goes in the events block:
worker_processes auto;
worker_rlimit_nofile 65535;
events {
worker_connections 16384;
}
Then add backlog to the listen directives of your site, for example in /etc/nginx/sites-available/default. The option can be set only once per address and port across all server blocks, so add it to the default server:
listen 80 default_server backlog=8192;
listen [::]:80 default_server backlog=8192;
Test the configuration and restart Nginx so the new limits apply:
sudo nginx -t
sudo systemctl restart nginx
Verify the listen backlog:
ss -ltn 'sport = :80'
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 8192 0.0.0.0:80 0.0.0.0:*
Verify the file descriptor limit of a worker process:
grep 'open files' /proc/$(pgrep -f 'nginx: worker' | head -1)/limits
Max open files 65535 65535 files
Step 5 - Checking the connection tracking table
If UFW or any other iptables/nftables firewall is active, the kernel tracks every connection in the conntrack table. When it fills up, new connections are dropped silently. Check its usage and size:
sudo sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_count = 18342
net.netfilter.nf_conntrack_max = 262144
If you see No such file or directory, connection tracking is not loaded and you can skip this step. If the count gets close to the maximum at peak, or the kernel log contains nf_conntrack: table full, dropping packet, raise the limit:
sudo journalctl -k | grep 'table full'
To raise it, add the following line to /etc/sysctl.d/99-webserver.conf and run sudo sysctl --system again. Each entry uses a few hundred bytes of kernel memory:
net.netfilter.nf_conntrack_max = 524288
Step 6 - Comparing before and after under load
Run the same load test before and after tuning to see whether anything changed. From a second machine, install wrk:
sudo apt install wrk
Run a 60-second test with 4 threads and 2,000 open connections against a static page. Raise the shell's file descriptor limit first with ulimit -n 65535, because each connection uses one on the client too:
wrk -t4 -c2000 -d60s --latency http://your_server_ip/
Latency Distribution
50% 18.42ms
75% 24.10ms
90% 33.77ms
99% 61.20ms
1204330 requests in 1.00m, 1.01GB read
Requests/sec: 20066.41
Compare Requests/sec, the latency percentiles and any Socket errors line between the two runs. On the web server, check that the counters from Step 1 no longer grow during the test:
nstat -az TcpExtListenOverflows TcpExtListenDrops
If both values stay the same while the test runs, the listen queues are no longer the bottleneck.
Troubleshooting
sysctl: cannot stat /proc/sys/net/netfilter/nf_conntrack_max: No such file or directory at boot: the conntrack module was not loaded when the file was applied. Remove the line if the server does not use a firewall, or add nf_conntrack to a file in /etc/modules-load.d/ so the module loads before the sysctl settings are applied.
tcp_congestion_control stays at cubic after a reboot: the tcp_bbr module was not loaded. Check that /etc/modules-load.d/bbr.conf contains tcp_bbr and run lsmod | grep bbr.
nginx: [emerg] duplicate listen options for 0.0.0.0:80: backlog is set on more than one listen directive for the same address and port. Keep it on one server block only.
Nginx logs worker_connections are not enough: raise worker_connections, keeping it below worker_rlimit_nofile.
Conclusion
You measured the listen queues and TCP counters of an Nginx server, applied a focused set of persistent sysctl settings, enabled BBR, raised Nginx's backlog and file descriptor limits, checked the conntrack table and compared the results under load. As next steps, load test your real application paths with k6, monitor ListenDrops and connection states continuously in your metrics system, and review Nginx settings such as keepalive_timeout and upstream keepalive to reduce the number of new connections.