--- title: "How to Find CPU Bottlenecks with perf on Ubuntu 24.04" description: "Install Linux perf on Ubuntu 24.04, count CPU events, profile programs and running services, and build flame graphs to find the functions that burn CPU time." category: "Performance Optimization" language: en source: https://cubepath.com/docs/performance-optimization/bottleneck-identification-with-perf updated: 2026-09-25 --- # How to Find CPU Bottlenecks with perf on Ubuntu 24.04 `perf` is the profiler built into the Linux kernel. It samples what the CPU is executing many times per second, with very low overhead, and tells you which functions, in your application or in the kernel, consume the most CPU time. It can also count hardware events such as instructions, cache misses and branch mispredictions. In this tutorial you will install `perf` on Ubuntu 24.04, measure a program with `perf stat`, find its hot function with `perf record` and `perf report`, profile a running service, and turn the result into a flame graph. ## Prerequisites To follow this tutorial, you will need: - A server running Ubuntu 24.04 LTS, for example a CubePath VPS. - A non-root user with `sudo` privileges. - About 200 MB of free memory for the example program. - Git, which is installed by default on Ubuntu Server, for the flame graph tools in Step 6. ## Step 1 - Installing perf On Ubuntu, `perf` is shipped in the kernel tools packages, and the version must match the running kernel. Install the common package and the one for your exact kernel: ```bash sudo apt update sudo apt install linux-tools-common linux-tools-$(uname -r) ``` Verify the installation: ```bash perf --version ``` ```output perf version 6.8.12 ``` Ubuntu sets `kernel.perf_event_paranoid` to `4`, which blocks profiling for regular users: ```bash cat /proc/sys/kernel/perf_event_paranoid ``` ```output 4 ``` Run `perf` with `sudo` throughout this tutorial rather than lowering that setting on a shared server. > Note: After a kernel upgrade and reboot, `perf` prints `WARNING: perf not found for kernel ...`. Run the `apt install` command above again to get the tools for the new kernel. ## Step 2 - Building an example program To learn the workflow it helps to profile a program whose bottleneck you know. This one sums the same 128 MB array twice: `strided_sum` jumps 64 bytes (one cache line) between reads, so almost every read misses the CPU cache, while `linear_sum` reads memory in order. ```bash nano hotspot.c ``` ```c #include #include __attribute__((noinline)) static unsigned long strided_sum(const int *a, size_t n) { unsigned long s = 0; for (size_t start = 0; start < 16; start++) for (size_t i = start; i < n; i += 16) s += a[i]; return s; } __attribute__((noinline)) static unsigned long linear_sum(const int *a, size_t n) { unsigned long s = 0; for (size_t i = 0; i < n; i++) s += a[i]; return s; } int main(void) { size_t n = 32UL * 1024 * 1024; /* 32M ints = 128 MB */ int *a = malloc(n * sizeof *a); if (!a) return 1; for (size_t i = 0; i < n; i++) a[i] = (int)(i & 0xff); unsigned long r = 0; for (int k = 0; k < 5; k++) { r += strided_sum(a, n); r += linear_sum(a, n); } printf("%lu\n", r); free(a); return 0; } ``` Compile it with debug symbols (`-g`) and frame pointers, so `perf` can show function names and full call stacks: ```bash sudo apt install build-essential gcc -O2 -g -fno-omit-frame-pointer -o hotspot hotspot.c ``` The `noinline` attributes keep the two functions separate at `-O2`, so they show up individually in the profile. ## Step 3 - Counting events with perf stat `perf stat` runs a command and reports counters for the whole run. It is the quickest way to see whether a program is CPU-bound and how efficiently it uses the CPU: ```bash sudo perf stat ./hotspot ``` ```output Performance counter stats for './hotspot': 2,487.31 msec task-clock # 0.999 CPUs utilized 9 context-switches # 3.618 /sec 0 cpu-migrations # 0.000 /sec 32,832 page-faults # 13.200 K/sec 8,912,405,127 cycles # 3.583 GHz 5,873,112,904 instructions # 0.66 insn per cycle 763,528,410 branches # 306.970 M/sec 342,118 branch-misses # 0.04% of all branches 2.489603847 seconds time elapsed ``` Key lines: - `CPUs utilized` close to 1.0 means the program was busy on one CPU the whole time (CPU-bound). A low value means it was mostly waiting, and `strace` is the better tool. - `insn per cycle` (IPC) below 1 on a modern CPU often means the CPU is stalled waiting on memory. Ask for cache events explicitly to confirm the memory problem: ```bash sudo perf stat -e cache-references,cache-misses ./hotspot ``` ```output 412,907,331 cache-references 291,530,016 cache-misses # 70.60% of all cache refs ``` > Note: Many virtual machines do not expose hardware counters to the guest. If `cycles`, `instructions` or `cache-misses` show ``, the hypervisor is hiding the CPU's performance counters. Software events such as `task-clock`, `context-switches` and `page-faults` still work, and CPU profiling in the next steps works too because `perf record` falls back to the `cpu-clock` timer. ## Step 4 - Finding the hot function with perf record `perf record` samples the call stack at a fixed frequency and writes the samples to `perf.data`. Use `-F 99` (99 samples per second per CPU, which avoids aliasing with timers that fire at round frequencies) and `-g` to record call stacks: ```bash sudo perf record -F 99 -g -- ./hotspot ``` ```output [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.061 MB perf.data (247 samples) ] ``` Show the report as text. `--no-children` attributes each sample only to the function that was actually executing, which is what you want to find hot spots: ```bash sudo perf report --stdio --no-children | grep -v '^$' | head -n 20 ``` ```output # Overhead Command Shared Object Symbol 82.19% hotspot hotspot [.] strided_sum | ---__libc_start_call_main main strided_sum 9.31% hotspot hotspot [.] linear_sum 6.07% hotspot hotspot [.] main 1.62% hotspot [kernel.kallsyms] [k] clear_page_erms ... ``` `strided_sum` takes about 82% of the CPU time even though it does exactly the same additions as `linear_sum`. `[.]` marks user-space code and `[k]` marks kernel code; here the kernel time is page zeroing for the 128 MB allocation (`main` fills the array). Run `sudo perf report` without `--stdio` for an interactive view: use the arrow keys to select a function, `Enter` to expand it, and `a` to annotate it and see which source lines and instructions are hot. Once you have found the hotspot, the fix follows: rewrite the loop to read memory sequentially, and the program runs several times faster. ## Step 5 - Profiling a running service The same technique works on any process that is already running, such as a web server, database or application worker. Find its process ID first, for example for MySQL: ```bash pgrep -x mysqld ``` ```output 1523 ``` Record 30 seconds of samples from that process while it is under its normal (or benchmark) load. `sleep 30` only defines how long the recording lasts: ```bash sudo perf record -F 99 -g -p 1523 -- sleep 30 sudo perf report --stdio --no-children | head -n 40 ``` To see everything the server is doing, including the kernel and all processes, use `-a` (all CPUs) instead of `-p`: ```bash sudo perf record -F 99 -g -a -- sleep 30 ``` For a live view that updates every few seconds, similar to `top` but per function, use `perf top`: ```bash sudo perf top ``` A few things to know when profiling real software: - Ubuntu 24.04 builds its packages with frame pointers enabled, so call stacks of packaged software (Nginx, MySQL, PostgreSQL, Python) are usually complete without extra work. - For your own compiled code, build with `-g -fno-omit-frame-pointer`, or record with `--call-graph dwarf`, which works without frame pointers but produces much larger `perf.data` files. - Interpreted and JIT-compiled languages need help to show their own function names. Python 3.12 supports `python3 -X perf your_script.py`, and Node.js supports `node --perf-basic-prof your_app.js`. Without these, you only see the interpreter's C functions. ## Step 6 - Generating a flame graph A flame graph shows all sampled stacks at once: each box is a function, its width is the share of CPU time, and callers sit below callees. It is the fastest way to understand a profile with many code paths. Clone Brendan Gregg's FlameGraph scripts: ```bash git clone --depth 1 https://github.com/brendangregg/FlameGraph.git ~/FlameGraph ``` From the directory that contains the `perf.data` file from Step 4 (or Step 5), convert it into text stacks, fold them, and render an SVG: ```bash sudo perf script > out.perf ~/FlameGraph/stackcollapse-perf.pl out.perf > out.folded ~/FlameGraph/flamegraph.pl out.folded > flame.svg ``` Copy `flame.svg` to your computer and open it in a web browser. Run this on your local machine, adjusting the path if you did not work in your home directory: ```bash scp your_user@your_server_ip:~/flame.svg . ``` In the browser, click a box to zoom into that part of the stack and use the search box at the top right to highlight a function. Look for wide plateaus at the top of the graph: those are the functions where the CPU actually spends its time. For the example program you will see one wide `strided_sum` tower on top of `main`. Clean up the temporary files when you are done: ```bash sudo rm -f perf.data perf.data.old out.perf out.folded ``` ## Troubleshooting - **`Access to performance monitoring and observability operations is limited`**: `perf` was run without `sudo` while `perf_event_paranoid` is `4`. Run it with `sudo`. - **`WARNING: perf not found for kernel 6.8.0-xx`**: the tools package for the running kernel is missing. Install `linux-tools-$(uname -r)`. - **Functions appear as `[unknown]` or hex addresses**: the binary has no symbols or no frame pointers. Rebuild with `-g -fno-omit-frame-pointer`, record with `--call-graph dwarf`, or install the package's debug symbols from Ubuntu's `ddebs` repository. - **`` for hardware events**: the virtual machine does not expose the CPU's performance counters. Use software events (`task-clock`, `cpu-clock`) for counting; sampling with `perf record` still works. - **The profile is dominated by idle or `sleep`**: the process was not busy while you recorded. Generate load (for example with `wrk`) during the recording, or profile the whole system with `-a`. ## Conclusion You installed `perf`, used `perf stat` to confirm a CPU-bound workload with poor cache behavior, located the exact function responsible with `perf record` and `perf report`, profiled a running service, and visualized the profile as a flame graph. Next, combine `perf` with a load generator such as `wrk` to profile your web stack under realistic traffic, use `strace` when a process is slow but not busy, and compare flame graphs before and after each optimization to confirm it helped.