Low-Latency Tuning on Ubuntu 24.04: CPU Isolation and tuned

Some workloads care less about average throughput and more about the worst case: trading gateways, real-time audio and game servers, telecom packet processing or control loops. For them, a single 5 ms stall caused by a timer interrupt or another process on the same core matters more than the average response time. In this tutorial you will measure the scheduling latency of an Ubuntu 24.04 server, apply a low-latency tuned profile, dedicate CPU cores to your application, keep interrupts and kernel housekeeping off those cores, and run your service pinned to them.

Prerequisites

To follow this tutorial you need:

• A server running Ubuntu 24.04 LTS with at least 4 CPU cores and the stock Ubuntu kernel.
• A non-root user with sudo privileges.
• Console access in case a boot parameter mistake prevents SSH from starting.

Important

The largest gains come from bare metal. On a virtual machine you control the guest scheduler, but the hypervisor still decides when your vCPUs run, so worst-case latency will stay higher. For microsecond-level requirements, use a dedicated server such as a CubePath bare metal server.

The examples assume 4 CPUs, where CPUs 0-1 stay for the operating system (housekeeping) and CPUs 2-3 are isolated for the application. Adjust the lists to your core count. On servers with Hyper-Threading, isolate both sibling threads of a physical core together; check siblings with lscpu -e.

Step 1 - Measuring a baseline with cyclictest

Always measure before tuning, or you will not know what helped. cyclictest from the rt-tests package repeatedly sleeps for a fixed interval and records how late the kernel wakes it up. The maximum value is the number that matters.

sudo apt update
sudo apt install rt-tests

Run a 60-second test with one real-time thread per CPU, locked memory and a 200 microsecond interval:

sudo cyclictest -m -p 80 -i 200 -t -D 60 -q

T: 0 ( 2201) P:80 I:200 C: 299987 Min:      3 Act:    5 Avg:    6 Max:    1843
T: 1 ( 2202) P:80 I:200 C: 299981 Min:      3 Act:    6 Avg:    6 Max:    2210
T: 2 ( 2203) P:80 I:200 C: 299975 Min:      3 Act:    4 Avg:    5 Max:    1502
T: 3 ( 2204) P:80 I:200 C: 299970 Min:      3 Act:    5 Avg:    5 Max:     987

Values are in microseconds. Note the Max column for later. For a realistic result, run your normal workload or a stress tool at the same time, because latency spikes appear under load.

Step 2 - Applying a low-latency tuned profile

tuned applies a consistent set of kernel and power settings from a named profile. The network-latency profile builds on latency-performance: it sets the CPU frequency governor to performance, keeps CPUs out of deep idle states, disables transparent huge pages and enables socket busy polling.

sudo apt install tuned
sudo systemctl enable --now tuned
sudo tuned-adm profile network-latency

Verify the active profile and that its settings are really applied:

tuned-adm active
sudo tuned-adm verify

Current active profile: network-latency
Verification succeeded, current system settings match the preset profile.

If your workload is not network bound, latency-performance is the lighter choice. List all profiles with tuned-adm list.

Step 3 - Isolating CPU cores with kernel boot parameters

By default the scheduler places any task on any core. Isolating cores removes them from general scheduling, so only tasks you pin explicitly run there. Four boot parameters work together:

Edit the GRUB defaults:

sudo nano /etc/default/grub

Append the parameters to GRUB_CMDLINE_LINUX_DEFAULT, keeping anything already there:

GRUB_CMDLINE_LINUX_DEFAULT="isolcpus=managed_irq,domain,2-3 nohz_full=2-3 rcu_nocbs=2-3 irqaffinity=0-1"

Regenerate the GRUB configuration and reboot:

sudo update-grub
sudo reboot

After the reboot, confirm the kernel received the parameters and applied them:

cat /proc/cmdline
cat /sys/devices/system/cpu/isolated
cat /sys/devices/system/cpu/nohz_full

BOOT_IMAGE=/vmlinuz-6.8.0-45-generic root=... isolcpus=managed_irq,domain,2-3 nohz_full=2-3 rcu_nocbs=2-3 irqaffinity=0-1
2-3
2-3

Check that ordinary processes no longer run on the isolated cores. The PSR column shows the CPU each thread last ran on; apart from per-CPU kernel threads such as ksoftirqd/2, you should not see CPUs 2 or 3:

ps -eLo psr,comm | awk '$1 >= 2' | sort | uniq -c | sort -rn | head

Step 4 - Keeping interrupts away from isolated cores

The irqaffinity parameter sets the default at boot, but the irqbalance daemon moves interrupts around afterwards. Tell it to never use the isolated CPUs:

sudo nano /etc/default/irqbalance

IRQBALANCE_BANNED_CPULIST=2-3

Restart it:

sudo systemctl restart irqbalance

Watch the interrupt counters for the isolated CPUs. The columns for CPU2 and CPU3 should stay flat for device interrupts while the other columns increase:

watch -n 1 -d "grep -E 'CPU|eth|virtio|nvme' /proc/interrupts"

A few per-CPU interrupts (such as LOC local timer or RES rescheduling) still fire on every core. With nohz_full, the local timer count on an isolated core grows very slowly once a single task runs there.

Step 5 - Pinning your application with systemd

Now run your latency-sensitive service on the isolated cores. Using a systemd drop-in keeps the settings with the service instead of in a wrapper script. Replace myapp.service with your unit:

sudo systemctl edit myapp.service

[Service]
CPUAffinity=2 3
CPUSchedulingPolicy=fifo
CPUSchedulingPriority=50
LimitMEMLOCK=infinity
Nice=-10

What each line does:

• CPUAffinity=2 3: the process and its threads may only run on CPUs 2 and 3.
• CPUSchedulingPolicy=fifo with priority 50: real-time scheduling, so ordinary tasks cannot preempt it.
• LimitMEMLOCK=infinity: allows the application to lock its memory with mlockall() and avoid page faults on the hot path.

Warning

A SCHED_FIFO thread that spins in a busy loop never yields the CPU. On an isolated core this is intended, but never give real-time priority to a busy-polling application running on housekeeping CPUs, or you can starve SSH and system services.

Restart and verify the affinity and policy of the main process:

sudo systemctl restart myapp.service
pid=$(systemctl show -p MainPID --value myapp.service)
taskset -cp "$pid"
chrt -p "$pid"

pid 5123's current affinity list: 2,3
pid 5123's current scheduling policy: SCHED_FIFO
pid 5123's current scheduling priority: 50

For a quick test without systemd, taskset and chrt also launch a command directly:

sudo taskset -c 2 chrt -f 50 ./my_benchmark

Step 6 - Tuning the NIC for latency

For network-driven workloads, the NIC delays interrupts on purpose (interrupt coalescing) to save CPU. Check the current settings of your interface:

sudo apt install ethtool
sudo ethtool -c eth0

If the driver supports it, lower the receive coalescing delay so each packet raises an interrupt immediately. This costs CPU time on the housekeeping cores:

sudo ethtool -C eth0 rx-usecs 0

Not every driver supports every option, and virtio NICs on virtual machines often report Operation not supported. That is expected; skip this step in that case. ethtool changes are lost at reboot; make them permanent with a systemd unit or your network configuration tool once you have validated them.

Step 7 - Measuring again

Run cyclictest again, this time only on the isolated cores, and compare the Max column with your baseline:

sudo cyclictest -m -p 80 -i 200 -t -a 2-3 -D 60 -q

T: 0 ( 1840) P:80 I:200 C: 299998 Min:      2 Act:    3 Avg:    3 Max:      21
T: 1 ( 1841) P:80 I:200 C: 299998 Min:      2 Act:    3 Avg:    3 Max:      18

On bare metal, maximums in the low tens of microseconds are typical after these changes. On a virtual machine the improvement is real but smaller. Repeat the test under your production load before drawing conclusions, and keep a copy of the results with each configuration change.

Troubleshooting

/sys/devices/system/cpu/isolated is empty after reboot. The parameters did not reach the kernel. Check that you edited GRUB_CMDLINE_LINUX_DEFAULT, ran sudo update-grub, and that no file in /etc/default/grub.d/ overrides the variable.

Latency spikes remain on isolated cores. Look for other tasks placed there with the ps command from Step 3, check /proc/interrupts for a device still interrupting those CPUs, and confirm tuned-adm verify succeeds. On a VM, compare with the st (steal) column in vmstat 1; steal time means the hypervisor is the limit.

The server feels slow after isolation. Everything except your pinned service now shares CPUs 0-1. Leave enough housekeeping cores for the operating system, logging and monitoring agents.

chrt reports Operation not permitted. Real-time scheduling needs root or the CAP_SYS_NICE capability. Start the service through systemd as shown, which applies the policy before dropping privileges.

Conclusion

You measured scheduling latency, applied a low-latency tuned profile, isolated CPU cores from the scheduler, timers, RCU callbacks and interrupts, and pinned your service to those cores with real-time priority. Next steps are reserving huge pages for your application's memory, aligning cores and memory to the same NUMA node on multi-socket servers (check with lscpu and numactl --hardware), and, for packet processing at line rate, looking at kernel-bypass frameworks such as DPDK.