How to Use the /proc Filesystem to Inspect Linux Systems

/proc is a virtual filesystem that the Linux kernel generates on the fly. Nothing in it is stored on disk: every time you read a file, the kernel builds its content from live data structures. Tools such as ps, top, free and ss are, to a large extent, readers of /proc. In this tutorial you will use /proc directly on Ubuntu 24.04 to inspect a process, read memory, CPU and pressure metrics, look at network state and change kernel parameters safely with sysctl.

Prerequisites

To follow this tutorial you need:

• A server running Ubuntu 24.04 LTS, for example a CubePath VPS. Everything except the file paths under /etc works the same on other distributions.
• A non-root user with sudo privileges. Reading other users' processes and writing kernel parameters requires root.

Step 1 - Exploring the layout of /proc

Confirm that /proc is mounted as proc:

findmnt /proc

TARGET SOURCE FSTYPE OPTIONS
/proc  proc   proc   rw,nosuid,nodev,noexec,relatime

The top level contains two kinds of entries: numeric directories, one per running process ID, and named files with system-wide information. The ones you will use most are:

Most files report a size of 0 because their content does not exist until you read it, so always read them with cat, grep or awk rather than relying on ls -l.

Step 2 - Inspecting a running process

Pick a process to inspect. Asking systemd for the main PID of a service is more reliable than parsing ps output. This example uses the journal daemon, which runs on every Ubuntu system:

PID=$(systemctl show -p MainPID --value systemd-journald)
echo "$PID"

312

The status file is the human-readable summary of a process:

grep -E '^(Name|State|PPid|Threads|VmRSS|VmSize|VmSwap|voluntary_ctxt_switches|nonvoluntary_ctxt_switches)' /proc/$PID/status

Name:	systemd-journal
State:	S (sleeping)
PPid:	1
VmSize:	  48912 kB
VmRSS:	  17344 kB
VmSwap:	      0 kB
Threads:	1
voluntary_ctxt_switches:	9241
nonvoluntary_ctxt_switches:	118

The fields that matter most are:

• State: R running, S sleeping, D uninterruptible sleep (usually waiting for disk or network storage), Z zombie, T stopped.
• VmRSS: resident memory, the RAM the process is actually using. VmSize is virtual address space and is usually much larger.
• nonvoluntary_ctxt_switches: how often the kernel preempted the process. A fast-growing value means the process competes for CPU.

Note that Name is truncated to 15 characters, which is why it reads systemd-journal.

The command line arguments are stored separated by null bytes. Convert them to spaces to read them:

sudo tr '\0' ' ' < /proc/$PID/cmdline; echo

/usr/lib/systemd/systemd-journald

The exe and cwd entries are symbolic links to the binary and the working directory:

sudo readlink /proc/$PID/exe /proc/$PID/cwd

/usr/lib/systemd/systemd-journald
/

The resource limits that apply to the process, including the maximum number of open files, are in limits:

grep 'open files' /proc/$PID/limits

Max open files            524288               524288               files

Every open file descriptor is a link in the fd directory. Counting them tells you how close the process is to that limit:

sudo ls /proc/$PID/fd | wc -l

47

If the count approaches the soft limit, the process will start failing with Too many open files. To see which files and sockets they are, list them with sudo ls -l /proc/$PID/fd.

Finally, smaps_rollup gives an accurate memory breakdown, including the proportional set size (Pss) that divides shared pages between the processes that use them:

sudo grep -E '^(Rss|Pss|Private_Dirty|Swap):' /proc/$PID/smaps_rollup

Rss:               17344 kB
Pss:               12203 kB
Private_Dirty:      6120 kB
Swap:                  0 kB

Pss is the best single number for "how much memory does this process really cost".

Step 3 - Reading memory, CPU and load

/proc/meminfo is where free gets its numbers. Print the key values in megabytes:

awk '/^(MemTotal|MemAvailable|Cached|SwapTotal|SwapFree|Dirty):/ {printf "%-14s %8.0f MB\n", $1, $2/1024}' /proc/meminfo

MemTotal:          3915 MB
MemAvailable:      2986 MB
Cached:            1822 MB
SwapTotal:            0 MB
SwapFree:             0 MB
Dirty:                1 MB

Use MemAvailable, not MemFree, to judge how much memory is left. MemFree excludes the page cache, which the kernel reclaims whenever applications need memory, so it is always low on a healthy server.

Count the logical CPUs and read the model:

grep -c '^processor' /proc/cpuinfo
grep -m1 'model name' /proc/cpuinfo

4
model name	: AMD EPYC 9454P 48-Core Processor

Check that a CPU feature is available, for example AES instructions for fast TLS and disk encryption:

grep -qw aes /proc/cpuinfo && echo "AES-NI available"

AES-NI available

/proc/loadavg holds the 1, 5 and 15 minute load averages, the number of runnable and total tasks, and the most recent PID:

cat /proc/loadavg

0.42 0.35 0.30 2/287 48121

Load average counts tasks that are running or in D state, so a high load can come from slow storage and not only from the CPU. Pressure stall information (PSI) tells you which resource is the bottleneck. It reports the share of time in which at least one task (some) or all tasks (full) were stalled waiting for a resource:

cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io

some avg10=0.00 avg60=0.12 avg300=0.08 total=18523411
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
some avg10=0.00 avg60=0.00 avg300=0.00 total=402117
full avg10=0.00 avg60=0.00 avg300=0.00 total=301542
some avg10=4.31 avg60=2.87 avg300=1.02 total=95510284
full avg10=3.95 avg60=2.54 avg300=0.88 total=87219034

The lines are in the order CPU, memory, I/O. The avg10, avg60 and avg300 values are percentages over 10 seconds, 1 minute and 5 minutes. In this example the I/O line is the only one above zero: tasks spent about 4% of the last 10 seconds waiting for disk, so storage, not CPU, is what slows this server down.

To find processes currently stuck in uninterruptible sleep, filter on the D state:

ps -eo pid,stat,comm | awk '$2 ~ /^D/'

Step 4 - Looking at network state

/proc/net exposes per-namespace network counters. Traffic totals per interface are in /proc/net/dev; the first data column is received bytes and the ninth is transmitted bytes:

awk 'NR > 2 {sub(":", "", $1); printf "%-10s rx=%d MB tx=%d MB\n", $1, $2/1048576, $10/1048576}' /proc/net/dev

lo         rx=12 MB tx=12 MB
ens3       rx=8421 MB tx=1936 MB

Socket usage for the whole system is summarised in sockstat:

cat /proc/net/sockstat

sockets: used 412
TCP: inuse 18 orphan 0 tw 37 alloc 24 mem 3
UDP: inuse 4 mem 2
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

tw is the number of sockets in TIME_WAIT. A value in the tens of thousands on a busy proxy is expected; a steady rise of orphan sockets is worth investigating.

/proc/net/tcp and /proc/net/tcp6 list every TCP socket, but addresses and ports are in hexadecimal and states are numeric codes. Use ss, which reads the same kernel data and decodes it:

sudo ss -tlnp

State  Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0      4096         0.0.0.0:22        0.0.0.0:*     users:(("sshd",pid=1021,fd=3))
LISTEN 0      511          0.0.0.0:80        0.0.0.0:*     users:(("nginx",pid=1340,fd=6))

Step 5 - Reading and changing kernel parameters

Files under /proc/sys are kernel parameters. The path maps to a sysctl name by replacing slashes with dots, so /proc/sys/vm/swappiness is vm.swappiness. Read a value both ways:

cat /proc/sys/vm/swappiness
sysctl vm.swappiness

60
vm.swappiness = 60

Change it at runtime with sysctl -w. The change takes effect immediately but is lost on reboot:

sudo sysctl -w vm.swappiness=10

vm.swappiness = 10

To make a setting permanent, put it in a file under /etc/sysctl.d/. Files are applied in lexical order at boot, so a high number such as 99- wins over the defaults shipped by packages. Create the file:

sudo nano /etc/sysctl.d/99-local.conf

Add the settings you want to keep. This example lowers the tendency to swap on a server with enough RAM and raises the inotify watch limit that file-watching tools and some applications run out of:

vm.swappiness = 10
fs.inotify.max_user_watches = 524288

Load all sysctl files exactly as the boot process does:

sudo sysctl --system

* Applying /usr/lib/sysctl.d/10-apparmor.conf ...
...
* Applying /etc/sysctl.d/99-local.conf ...
vm.swappiness = 10
fs.inotify.max_user_watches = 524288

Verify the running values:

sysctl vm.swappiness fs.inotify.max_user_watches

vm.swappiness = 10
fs.inotify.max_user_watches = 524288

Warning

change one parameter at a time and only when you can explain why. Copying large "performance" lists from the internet often sets values that are already the default, obsolete, or harmful for your workload, such as vm.overcommit_memory = 1 on a database server.

Troubleshooting

Permission denied when reading /proc/<PID>/fd, environ or smaps_rollup. These files are only readable by the process owner and root. Use sudo. Files such as status and cmdline are readable by everyone unless /proc is mounted with the hidepid option.

A sysctl change is gone after a reboot. Values written with sysctl -w or echo > /proc/sys/... are runtime only. Save them in /etc/sysctl.d/*.conf, then run sudo sysctl --system and check that no file applied later in the output sets the same key to another value.

sysctl: cannot stat /proc/sys/...: No such file or directory. The parameter does not exist in your kernel, or it belongs to a module that is not loaded yet. For example, net.bridge.* keys only appear after the br_netfilter module is loaded.

Too many open files errors. Check the system-wide counters in /proc/sys/fs/file-nr (allocated, unused, maximum). If the first number is far below the third, the system limit is fine and the per-process limit is the problem: raise LimitNOFILE= in the service's systemd unit rather than changing fs.file-max.

Conclusion

You used /proc to inspect a process's state, memory and file descriptors, read system-wide memory, CPU, load and pressure metrics, checked network counters, and changed kernel parameters in a way that survives reboots. As next steps, explore /sys, which exposes devices and drivers in a similar way, set up monitoring that collects PSI metrics, and learn which sysctl settings your databases and web servers actually document as recommended.