Linux Boot Process and Troubleshooting

Understanding the Linux boot process—from BIOS/UEFI firmware through GRUB, the kernel, initramfs, and systemd—is essential for diagnosing boot failures and recovering systems. This guide explains each boot stage, GRUB configuration, kernel parameters, initramfs management, rescue mode procedures, and recovery techniques for common boot failures.

Prerequisites

  • Linux system (Ubuntu 20.04+, CentOS/Rocky 8+)
  • Physical or console access to the server
  • A bootable live USB or rescue ISO for recovery scenarios
  • Basic familiarity with the command line

Boot Stages Overview

The Linux boot process follows this sequence:

1. Power on → Firmware (BIOS/UEFI)
2. Firmware → Bootloader (GRUB2)
3. GRUB → Kernel + initramfs loaded into RAM
4. Kernel → Hardware detection, mount initramfs as rootfs
5. initramfs → Mount real root filesystem
6. Kernel → Hand off to /sbin/init (systemd)
7. systemd → Start services, reach target (login prompt)

# View systemd boot timing analysis
systemd-analyze time
systemd-analyze blame | head -20
systemd-analyze critical-chain

# View kernel boot messages
dmesg | head -50
dmesg | grep -E "error|fail|warn" -i

BIOS vs UEFI Boot

BIOS (Legacy):

  • Reads MBR (first 512 bytes of disk)
  • MBR contains GRUB stage 1 bootloader
  • Limited to 4 primary partitions, 2TB max disk size

UEFI:

  • Reads EFI System Partition (ESP, usually /boot/efi)
  • Supports GPT partition tables, 9.4ZB disks
  • Has Secure Boot for verified bootloaders
# Check if system booted via UEFI or BIOS
[ -d /sys/firmware/efi ] && echo "UEFI boot" || echo "BIOS/Legacy boot"

# List EFI boot entries (UEFI only)
sudo efibootmgr -v

# Show ESP contents
ls /boot/efi/EFI/

GRUB Bootloader

GRUB (GRand Unified Bootloader) is the standard Linux bootloader, responsible for loading the kernel.

# GRUB configuration file (generated - do not edit directly)
cat /boot/grub/grub.cfg | head -50

# Edit GRUB settings here:
sudo nano /etc/default/grub

Key GRUB settings:

# Default boot entry (0 = first entry)
GRUB_DEFAULT=0

# Show GRUB menu for N seconds (0 = skip, -1 = wait forever)
GRUB_TIMEOUT=5

# Kernel parameters appended to boot command line
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"

# Enable GRUB menu (comment out to hide)
GRUB_TERMINAL=console

# Apply changes after editing /etc/default/grub
sudo update-grub              # Ubuntu/Debian
sudo grub2-mkconfig -o /boot/grub2/grub.cfg   # CentOS/Rocky

# For UEFI systems (CentOS/Rocky)
sudo grub2-mkconfig -o /boot/efi/EFI/rocky/grub.cfg

Useful kernel boot parameters:

# Add to GRUB_CMDLINE_LINUX in /etc/default/grub

# Debug options
systemd.log_level=debug      # Verbose systemd logging
debug                        # Kernel debug output
ignore_loglevel              # Show all kernel messages

# Recovery options
init=/bin/bash               # Boot to shell without init
single                       # Single-user mode
rd.break                     # Drop to initramfs shell
systemd.unit=rescue.target   # Boot to rescue mode
systemd.unit=emergency.target # Boot to emergency mode

# Hardware options
nomodeset                    # Disable kernel mode setting (GPU issues)
acpi=off                     # Disable ACPI (hardware compatibility)
noapic                       # Disable APIC
mem=4G                       # Limit RAM (testing)

Kernel and initramfs

The kernel and initramfs are loaded by GRUB into RAM before the root filesystem is accessible.

# List installed kernels
ls /boot/vmlinuz-*
ls /boot/initrd.img-*   # Ubuntu
ls /boot/initramfs-*    # CentOS/Rocky

# Current kernel version
uname -r

# Rebuild initramfs (needed after adding kernel modules)
sudo update-initramfs -u -k $(uname -r)   # Ubuntu
sudo dracut --force                        # CentOS/Rocky

# Rebuild for all installed kernels
sudo update-initramfs -u -k all   # Ubuntu
sudo dracut --regenerate-all      # CentOS/Rocky

Inspect initramfs contents:

# Ubuntu (initrd.img is zstd compressed cpio)
mkdir /tmp/initrd-inspect
cd /tmp/initrd-inspect

# Decompress and extract
sudo unmkinitramfs /boot/initrd.img-$(uname -r) .

# Browse extracted content
ls -la /tmp/initrd-inspect/
ls /tmp/initrd-inspect/etc/
ls /tmp/initrd-inspect/scripts/

systemd Initialization

After the kernel mounts the root filesystem, it hands control to systemd (/sbin/init).

# View system boot target
systemctl get-default

# List all targets in boot order
systemctl list-units --type=target

# Check units that failed during boot
systemctl --failed

# View all services that started at boot
systemctl list-units --state=active --type=service

# Detailed boot analysis
systemd-analyze plot > /tmp/boot-chart.svg
# Open boot-chart.svg in browser for visual timeline

Boot sequence for multi-user.target:

sysinit.target
  → local-fs.target (mount filesystems)
  → basic.target (basic system setup)
    → multi-user.target (network services)
      → network.target
      → sshd.service
      → ... other services

Diagnosing Boot Failures

When a system fails to boot, identify the failure point:

# View journal from last boot (run after recovery)
journalctl -b -1   # Previous boot
journalctl -b -1 -p err   # Errors only from previous boot

# View kernel messages from failed boot
journalctl -b -1 -k

# Find units that failed
journalctl -b -1 | grep "Failed to start"
journalctl -b -1 | grep "Job for"

Common failure indicators:

# Filesystem check failure
journalctl -b -1 | grep "fsck"
journalctl -b -1 | grep "mounting"

# Dependency failures
journalctl -b -1 | grep "dependency failed"

# Timeout failures
journalctl -b -1 | grep "timed out"

# Check specific service logs
journalctl -b -1 -u networking.service
journalctl -b -1 -u [email protected]

Rescue and Emergency Mode

Access rescue/emergency mode from the GRUB menu when the system won't boot:

Method 1: GRUB edit at boot time

  1. At GRUB menu, press e to edit the boot entry
  2. Find the linux line and add systemd.unit=rescue.target at the end
  3. Press Ctrl+X to boot

Method 2: Emergency shell in initramfs

  1. Add rd.break to the kernel line in GRUB
  2. Boots to an initramfs shell before mounting root
# In rd.break initramfs shell:
# Mount the real root filesystem
mount -o remount,rw /sysroot
chroot /sysroot

# Now you have full access to fix issues

# If SELinux is enforcing, create relabel file
touch /.autorelabel
exit

In rescue mode:

# Rescue mode gives root shell with most filesystems mounted
# Check what failed
systemctl status
systemctl --failed

# Fix the problem, then continue boot
systemctl default

# Or reboot
systemctl reboot

Recovery Procedures

Reset forgotten root password:

# 1. At GRUB: edit boot entry, add 'init=/bin/bash' to kernel line
# 2. Boot into single-process bash shell

# Remount root as read-write
mount -o remount,rw /

# Reset password
passwd root

# Relabel SELinux if enabled (CentOS/Rocky)
touch /.autorelabel

# Reboot
exec /sbin/init

Repair broken filesystem:

# Boot from live USB
# Mount the drive
sudo blkid   # Find the device

# Run fsck (filesystem must be unmounted)
sudo fsck -y /dev/sda1   # -y = answer yes to all repairs

# For ext4 specifically
sudo e2fsck -f /dev/sda1

# For XFS (CentOS/Rocky default)
sudo xfs_repair /dev/sda1

Reinstall GRUB after corruption:

# Boot from live USB
# Mount the installed system
sudo mount /dev/sda1 /mnt
sudo mount --bind /dev /mnt/dev
sudo mount --bind /proc /mnt/proc
sudo mount --bind /sys /mnt/sys

# For UEFI: also mount ESP
sudo mount /dev/sda2 /mnt/boot/efi

# Chroot into the system
sudo chroot /mnt

# Reinstall GRUB
# BIOS/MBR:
grub-install /dev/sda
update-grub

# UEFI (Ubuntu):
grub-install --target=x86_64-efi --efi-directory=/boot/efi
update-grub

# UEFI (CentOS/Rocky):
grub2-install --target=x86_64-efi --efi-directory=/boot/efi
grub2-mkconfig -o /boot/efi/EFI/rocky/grub.cfg

exit
sudo umount -R /mnt
sudo reboot

Boot with broken /etc/fstab:

# At GRUB, add 'rescue' or 'rd.break' to kernel line
# Or boot to emergency.target

# Fix /etc/fstab
mount -o remount,rw /
nano /etc/fstab

# Test fstab without rebooting
mount -a   # Should produce no errors

Troubleshooting

Kernel panic at boot:

# Read the panic message carefully—it shows the failing function
# Common causes:
# - Missing root device: check GRUB root= parameter matches /etc/fstab
# - Corrupt initramfs: rebuild with update-initramfs / dracut
# - Missing kernel module: add to /etc/initramfs-tools/modules

"A start job is running for..." timeout:

# Long boot wait usually caused by network wait
sudo systemctl disable systemd-networkd-wait-online.service
# Or fix the NetworkManager connection that's failing

Stuck at "Loading initial ramdisk":

# Initramfs is corrupt or incompatible with kernel
# Boot previous kernel from GRUB (Advanced Options)
# Then rebuild: sudo update-initramfs -u -k all

UEFI Secure Boot blocking kernel:

# Check Secure Boot status
mokutil --sb-state

# If blocking your kernel:
# Option 1: Disable Secure Boot in UEFI firmware settings
# Option 2: Enroll MOK (Machine Owner Key)
sudo mokutil --import /path/to/signing-key.der

Conclusion

The Linux boot process is a well-defined sequence of stages, each with specific failure modes and recovery procedures. When a system fails to boot, work backwards from the last successful stage: check GRUB can find the kernel, verify the initramfs is intact, confirm root filesystem mounts correctly, and review systemd unit failures with journalctl -b -1 --failed. With rescue mode and a live USB, most boot failures are recoverable without data loss.