proc Filesystem Deep Dive

The /proc filesystem is a virtual filesystem in Linux that exposes kernel data structures, process information, hardware details, and network statistics as readable files in a live interface. This guide covers navigating /proc for system monitoring, reading process details, tuning kernel parameters via sysctl, analyzing network statistics, and using /proc for runtime system configuration.

Prerequisites

  • Any Linux system (Ubuntu, CentOS, Rocky, Debian)
  • Basic command-line knowledge
  • Root access for some /proc/sys writes and certain process directories

Understanding the proc Filesystem

/proc is mounted as a procfs virtual filesystem—no data is stored on disk. Every file read triggers a kernel function that generates the content dynamically.

# Verify procfs is mounted
mount | grep proc
# proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

# Top-level structure
ls /proc/
# Numbered directories = PID of running processes
# Named files/dirs = system-wide kernel info

Key top-level entries:

PathContents
/proc/[PID]/Per-process information
/proc/cpuinfoCPU details
/proc/meminfoMemory usage
/proc/net/Network statistics
/proc/sys/Kernel parameters (writable)
/proc/filesystemsSupported filesystems
/proc/mountsCurrently mounted filesystems
/proc/loadavgLoad average
/proc/uptimeSystem uptime
/proc/versionKernel version
/proc/cmdlineKernel boot parameters

Process Information in /proc/PID

Every running process has a directory at /proc/[PID]/:

# Find PID of a process
PID=$(pgrep nginx | head -1)

# Overview of process directory
ls /proc/$PID/

Key files per process:

# Command line arguments (null-separated)
cat /proc/$PID/cmdline | tr '\0' ' '; echo

# Current working directory
ls -la /proc/$PID/cwd

# Executable path
ls -la /proc/$PID/exe

# Environment variables
cat /proc/$PID/environ | tr '\0' '\n' | grep -v "^$"

# Open file descriptors
ls -la /proc/$PID/fd

# File descriptor limits and counts
cat /proc/$PID/limits

# Memory maps (loaded libraries and mappings)
cat /proc/$PID/maps

# Memory statistics (in pages)
cat /proc/$PID/statm

# Process status (human-readable)
cat /proc/$PID/status

Process status fields:

cat /proc/$PID/status
# Name: nginx
# State: S (sleeping)    # R=running, D=disk wait, Z=zombie, T=stopped
# Pid: 1234
# VmRSS: 12345 kB        # Resident Set Size (actual RAM used)
# VmVirt: 67890 kB       # Virtual memory size
# Threads: 4             # Thread count
# voluntary_ctxt_switches: 1234
# nonvoluntary_ctxt_switches: 56

Process statistics (raw):

# /proc/[PID]/stat - machine-readable process stats
# Fields: pid, comm, state, ppid, pgrp, session, tty_nr, ...
# Field 14 = utime (user mode CPU ticks)
# Field 15 = stime (kernel mode CPU ticks)
awk '{print "user:"$14, "sys:"$15}' /proc/$PID/stat

# Memory breakdown
cat /proc/$PID/smaps | grep -E "^(Shared_Clean|Private_Clean|Private_Dirty|Swap):" | \
  awk '{sum[$1]+=$2} END {for(k in sum) print k, sum[k], "kB"}' | sort

Open files and network connections:

# Open file descriptors with targets
ls -la /proc/$PID/fd | awk '{print $NF}' | grep -v "^$" | sort

# Network connections via fdinfo
ls /proc/$PID/fd | while read fd; do
    target=$(readlink /proc/$PID/fd/$fd 2>/dev/null)
    [[ "$target" == socket:* ]] && echo "Socket fd=$fd $target"
done

# Simpler: use ss with PID filter
ss -p | grep "pid=$PID"

System-Wide Kernel Information

# Kernel version
cat /proc/version

# Boot command line parameters
cat /proc/cmdline

# System uptime (seconds) and idle time
cat /proc/uptime
# 86400.12 172345.89   <- uptime, idle (cumulative across CPUs)

# Load averages (1, 5, 15 min) + running/total threads + last PID
cat /proc/loadavg
# 0.45 0.32 0.28 2/450 12345

# Kernel statistics (CPU times, context switches, boots)
cat /proc/stat | head -20

# Running processes count
awk '/^procs_running/ {print "Running:", $2}' /proc/stat
awk '/^procs_blocked/ {print "Blocked:", $2}' /proc/stat

Filesystem and module information:

# Supported filesystems (nodev = virtual, not backed by device)
cat /proc/filesystems

# Loaded kernel modules
cat /proc/modules | awk '{print $1}' | head -20

# Block devices
cat /proc/partitions
cat /proc/diskstats   # Disk I/O statistics

# IRQ assignments
cat /proc/interrupts

# DMA channels
cat /proc/dma

# I/O ports
cat /proc/ioports

Network Statistics in /proc/net

# All network-related proc files
ls /proc/net/

# TCP connections (hex format)
# Local addr:port, Remote addr:port, state (0A=LISTEN, 01=ESTABLISHED)
cat /proc/net/tcp

# UDP sockets
cat /proc/net/udp

# Network interface statistics
cat /proc/net/dev
# Inter-|   Receive                                 |   Transmit
# face  | bytes packets errs drop fifo frame multi overruns...

# Parse interface stats
awk 'NR>2 {print $1, "rx_bytes="$2, "tx_bytes="$10}' /proc/net/dev

# ARP table
cat /proc/net/arp

# Routing table
cat /proc/net/route   # Hex format
route -n              # Human-readable equivalent

# DNS resolution stats (if using systemd-resolved)
cat /proc/net/if_inet6   # IPv6 interfaces

# Socket statistics
cat /proc/net/sockstat

Decode hex network addresses:

# /proc/net/tcp uses hex: local_address = HEXIP:HEXPORT
# Convert hex IP (little-endian) to dotted decimal:
python3 -c "
import socket, struct
hex_addr = '0100007F'   # 127.0.0.1 in little-endian hex
packed = bytes.fromhex(hex_addr)
print(socket.inet_ntoa(packed[::-1]))  # Reverse for big-endian
"

Hardware and Memory Information

# Detailed CPU information
cat /proc/cpuinfo

# CPU count
grep -c "^processor" /proc/cpuinfo

# CPU model
grep "model name" /proc/cpuinfo | head -1

# CPU flags (features)
grep "flags" /proc/cpuinfo | head -1 | tr ' ' '\n' | sort | head -20

# Memory information (all values in kB)
cat /proc/meminfo

# Key memory metrics
awk '/MemTotal|MemFree|MemAvailable|Buffers|Cached|SwapTotal|SwapFree|Dirty/ {printf "%-20s %10.0f MB\n", $1, $2/1024}' /proc/meminfo

# NUMA nodes (for multi-socket servers)
ls /proc/sys/vm/numa_stat 2>/dev/null && cat /proc/sys/vm/numa_stat

# Huge pages
grep -i "huge" /proc/meminfo

# PCI devices (basic)
cat /proc/bus/pci/devices | head -10

# SCSI devices
cat /proc/scsi/scsi 2>/dev/null

Kernel Tuning with sysctl and /proc/sys

/proc/sys/ is the writable portion of /proc. Changes here affect the running kernel immediately.

# List all sysctl parameters
sysctl -a 2>/dev/null | wc -l

# Read a parameter two ways:
cat /proc/sys/net/ipv4/ip_forward
sysctl net.ipv4.ip_forward

# Write a parameter two ways (temporary, lost on reboot):
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
sudo sysctl -w net.ipv4.ip_forward=1

# Make permanent:
echo "net.ipv4.ip_forward = 1" | sudo tee /etc/sysctl.d/99-custom.conf
sudo sysctl -p /etc/sysctl.d/99-custom.conf

Common performance tunings:

sudo tee /etc/sysctl.d/99-performance.conf << 'EOF'
# Networking
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.core.netdev_max_backlog = 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65535

# Virtual memory
vm.swappiness = 10
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5
vm.overcommit_memory = 1

# File system
fs.file-max = 2097152
fs.inotify.max_user_watches = 524288
EOF

sudo sysctl -p /etc/sysctl.d/99-performance.conf

Monitoring with /proc

Build a custom monitoring script using /proc:

cat << 'EOF' > /usr/local/bin/proc-monitor.sh
#!/bin/bash
# System monitoring via /proc filesystem

while true; do
    echo "=== $(date) ==="
    
    # Load average
    read la1 la5 la15 threads lastpid < /proc/loadavg
    echo "Load: $la1 $la5 $la15 | Threads: $threads"
    
    # Memory (in MB)
    memtotal=$(awk '/MemTotal/ {printf "%.0f", $2/1024}' /proc/meminfo)
    memavail=$(awk '/MemAvailable/ {printf "%.0f", $2/1024}' /proc/meminfo)
    swap=$(awk '/SwapFree/ {printf "%.0f", $2/1024}' /proc/meminfo)
    echo "Memory: ${memavail}MB free of ${memtotal}MB | Swap free: ${swap}MB"
    
    # CPU usage (delta between reads)
    read cpu user nice sys idle iowait irq < <(grep "^cpu " /proc/stat)
    total=$((user + nice + sys + idle + iowait + irq))
    echo "CPU total ticks: $total (idle: $idle)"
    
    # Network
    echo "Network stats:"
    awk 'NR>2 && $1 !~ /^lo/ {printf "  %s: rx=%s tx=%s\n", $1, $2, $10}' /proc/net/dev
    
    # Top memory processes
    echo "Top memory processes:"
    for pid in $(ls /proc/ | grep -E '^[0-9]+$'); do
        rss=$(awk '/VmRSS/ {print $2}' /proc/$pid/status 2>/dev/null)
        comm=$(cat /proc/$pid/comm 2>/dev/null)
        echo "$rss $comm"
    done | sort -rn | head -5 | awk '{printf "  %s: %s MB\n", $2, $1/1024}'
    
    echo ""
    sleep 5
done
EOF
chmod +x /usr/local/bin/proc-monitor.sh

Troubleshooting

Permission denied reading /proc/[PID]:

# You can only read /proc/[PID] for your own processes
# or as root for any process
sudo cat /proc/1/cmdline | tr '\0' ' '

High iowait in /proc/stat:

# Check which processes are in disk wait state (D state)
awk '/^[0-9]/{if($3=="D") print}' /proc/*/stat 2>/dev/null | head -10
# Or simpler:
ps aux | awk '$8 == "D" {print $0}'

sysctl change not persisting after reboot:

# Ensure you saved to /etc/sysctl.d/ not directly to /proc/sys
ls /etc/sysctl.d/
# Also check /etc/sysctl.conf is processed
sudo sysctl -p /etc/sysctl.conf

"Too many open files" errors:

# Check system-wide open file limit
cat /proc/sys/fs/file-max
# Check current usage
cat /proc/sys/fs/file-nr
# field 1 = open, field 2 = not used, field 3 = max

# Increase limit
echo "fs.file-max = 2097152" | sudo tee /etc/sysctl.d/99-files.conf
sudo sysctl -p /etc/sysctl.d/99-files.conf

Conclusion

The /proc filesystem is an indispensable tool for Linux system monitoring, performance tuning, and troubleshooting. By reading per-process directories for memory maps and file descriptors, using /proc/net for network connection analysis, and writing kernel parameters through /proc/sys (or sysctl), you can inspect and modify nearly every aspect of the running system. All changes to /proc/sys are instantaneous but temporary—always persist important tuning in /etc/sysctl.d/ files.