Upgrading Ubuntu LTS to a New Version

Upgrading Ubuntu Long Term Support (LTS) releases is a critical maintenance task for system administrators managing production servers. While Ubuntu's upgrade process is generally reliable, performing a major version upgrade requires careful planning, thorough testing, and systematic execution to avoid downtime, data loss, or system instability.

This comprehensive guide walks you through the complete process of upgrading Ubuntu LTS to a new version, from pre-upgrade planning and backup procedures to the actual upgrade execution and post-upgrade verification. Whether you're upgrading from Ubuntu 20.04 to 22.04, or planning for future LTS transitions, this guide provides the knowledge and procedures needed for a successful upgrade.

Table of Contents

• Introduction
• Understanding Ubuntu LTS Release Cycle
• Pre-Upgrade Planning and Assessment
• Backup Strategies and Disaster Recovery
• Testing the Upgrade Process
• Preparing the System for Upgrade
• Performing the Upgrade (Method 1: do-release-upgrade)
• Alternative Upgrade Methods
• Troubleshooting Common Issues
• Post-Upgrade Verification and Cleanup
• Rolling Back Failed Upgrades
• Best Practices for Production Systems
• Automating Upgrades at Scale
• Conclusion

Introduction

Ubuntu LTS (Long Term Support) releases receive five years of standard support and an additional five years of extended security maintenance through Ubuntu Pro. Major LTS versions are released biennially in April, with recent releases including:

• Ubuntu 20.04 LTS (Focal Fossa) - April 2020
• Ubuntu 22.04 LTS (Jammy Jellyfish) - April 2022
• Ubuntu 24.04 LTS (Noble Numbat) - April 2024

Upgrading between LTS releases brings security improvements, kernel updates, newer software versions, and performance enhancements. However, the upgrade process also introduces risks including:

• Application compatibility issues with newer library versions
• Configuration file conflicts requiring manual resolution
• Kernel changes affecting custom drivers or modules
• Third-party repository conflicts
• Service disruption during the upgrade process

A methodical approach to LTS upgrades minimizes these risks while maximizing the benefits of newer Ubuntu versions. This guide emphasizes preparation, testing, and verification to ensure successful upgrades in production environments.

Understanding Ubuntu LTS Release Cycle

Before planning your upgrade, understand Ubuntu's release and support structure.

LTS Support Timeline

# Check current Ubuntu version
lsb_release -a

# Example output:
# Distributor ID: Ubuntu
# Description:    Ubuntu 22.04.3 LTS
# Release:        22.04
# Codename:       jammy

Ubuntu LTS Support Periods:

LTS to LTS Upgrade Path

Ubuntu officially supports direct upgrades only between consecutive LTS releases:

# Supported upgrade paths:
# 20.04 LTS → 22.04 LTS → 24.04 LTS

# NOT supported (skipping versions):
# 20.04 LTS → 24.04 LTS  # Must go through 22.04

Point Release Considerations

Ubuntu enables LTS-to-LTS upgrades after the first point release (X.04.1) of the new LTS version:

# Check if upgrades to next LTS are offered
/usr/lib/ubuntu-release-upgrader/check-new-release-gtk

# LTS upgrades typically available after:
# - 22.04.1 release (approximately July 2022 for 22.04)
# - System administrators can upgrade immediately after .0 release
# - Desktop users see prompts after .1 release

Pre-Upgrade Planning and Assessment

Thorough planning prevents upgrade failures and reduces downtime.

System Inventory and Documentation

#!/bin/bash
# Create comprehensive system inventory

REPORT_DIR="/root/upgrade-prep-$(date +%Y%m%d)"
mkdir -p "$REPORT_DIR"

echo "Creating system inventory for upgrade planning..."

# Current system information
hostnamectl > "$REPORT_DIR/system-info.txt"
lsb_release -a >> "$REPORT_DIR/system-info.txt"
uname -a >> "$REPORT_DIR/kernel-info.txt"

# Hardware information
lscpu > "$REPORT_DIR/cpu-info.txt"
free -h > "$REPORT_DIR/memory-info.txt"
df -h > "$REPORT_DIR/disk-info.txt"
lsblk >> "$REPORT_DIR/disk-info.txt"

# Installed packages
dpkg -l > "$REPORT_DIR/installed-packages.txt"
apt-mark showmanual > "$REPORT_DIR/manually-installed.txt"

# Running services
systemctl list-units --type=service --state=running > "$REPORT_DIR/running-services.txt"

# Network configuration
ip addr show > "$REPORT_DIR/network-config.txt"
ip route show >> "$REPORT_DIR/network-config.txt"
cat /etc/netplan/*.yaml > "$REPORT_DIR/netplan-config.txt" 2>/dev/null

# Kernel modules
lsmod > "$REPORT_DIR/loaded-modules.txt"

# Third-party repositories
grep -r --include '*.list' '^deb ' /etc/apt/ > "$REPORT_DIR/repositories.txt"

# Custom configurations
find /etc -name '*.conf' -type f -exec ls -la {} \; > "$REPORT_DIR/config-files.txt"

echo "Inventory saved to $REPORT_DIR"

Compatibility Assessment

Check Application Compatibility:

# Identify critical applications
systemctl list-units --type=service --state=running | grep -v '@'

# Check each critical service for compatibility
# Example: Check PHP version compatibility
php -v

# Check database versions
mysql --version
psql --version

# Check web server versions
nginx -v
apache2 -v

Research Known Issues:

# Search Ubuntu release notes
# Visit: https://wiki.ubuntu.com/JammyJellyfish/ReleaseNotes

# Check for specific application compatibility
apt-cache policy application-name

# Review changelog for major packages
apt-get changelog package-name | less

Estimate Downtime Window

# Calculate estimated upgrade time based on:
# - Number of packages to upgrade
# - Network bandwidth
# - System resources

# Check number of packages
apt list --upgradable | wc -l

# Estimate download size
apt-get -s dist-upgrade | grep "Need to get"

# Typical upgrade times:
# - Small system (< 500 packages): 30-60 minutes
# - Medium system (500-1000 packages): 1-2 hours
# - Large system (> 1000 packages): 2-4 hours
# Add buffer time for unexpected issues

Create Upgrade Plan Document

cat << 'EOF' > /root/upgrade-plan.md
## Ubuntu LTS Upgrade Plan

### System Information
- Hostname: production-web-01
- Current Version: Ubuntu 20.04.6 LTS
- Target Version: Ubuntu 22.04.3 LTS
- Role: Web Application Server

### Pre-Upgrade Checklist
- [ ] Full system backup completed
- [ ] Application compatibility verified
- [ ] Third-party repositories documented
- [ ] Maintenance window scheduled
- [ ] Rollback procedure documented
- [ ] Team notified
- [ ] Monitoring alerts configured

### Upgrade Window
- Date: 2024-12-15
- Start Time: 02:00 UTC
- Estimated Duration: 2 hours
- Maximum Duration: 4 hours

### Rollback Plan
- Snapshot available: Yes
- Snapshot location: /backup/snapshots/pre-upgrade-20241215
- Rollback tested: Yes
- Rollback duration: 30 minutes

### Critical Services
1. Nginx web server
2. PostgreSQL database
3. Redis cache
4. Application workers (Celery)

### Success Criteria
- [ ] System boots successfully
- [ ] All critical services running
- [ ] Application responds correctly
- [ ] No critical errors in logs
- [ ] Performance metrics acceptable
EOF

Backup Strategies and Disaster Recovery

Never upgrade without comprehensive backups and tested recovery procedures.

Full System Backup

#!/bin/bash
# Comprehensive system backup before upgrade

BACKUP_DIR="/backup/pre-upgrade-$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"

echo "Starting full system backup..."

# Backup entire system (excluding virtual filesystems)
sudo tar --exclude=/backup \
         --exclude=/proc \
         --exclude=/tmp \
         --exclude=/mnt \
         --exclude=/dev \
         --exclude=/sys \
         --exclude=/run \
         --exclude=/media \
         --exclude=/var/cache \
         --exclude=/var/tmp \
         -czf "$BACKUP_DIR/system-backup.tar.gz" /

echo "System backup completed: $BACKUP_DIR/system-backup.tar.gz"
echo "Backup size: $(du -sh $BACKUP_DIR/system-backup.tar.gz)"

# Backup critical configurations separately
tar -czf "$BACKUP_DIR/etc-backup.tar.gz" /etc
tar -czf "$BACKUP_DIR/home-backup.tar.gz" /home

# Create list of installed packages
dpkg --get-selections > "$BACKUP_DIR/package-selections.txt"

Database Backups

#!/bin/bash
# Backup databases before upgrade

BACKUP_DIR="/backup/databases-$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"

# MySQL/MariaDB backup
if systemctl is-active --quiet mysql; then
    echo "Backing up MySQL databases..."
    mysqldump --all-databases --single-transaction --quick --lock-tables=false \
        > "$BACKUP_DIR/mysql-all-databases.sql"
    gzip "$BACKUP_DIR/mysql-all-databases.sql"
fi

# PostgreSQL backup
if systemctl is-active --quiet postgresql; then
    echo "Backing up PostgreSQL databases..."
    sudo -u postgres pg_dumpall > "$BACKUP_DIR/postgresql-all-databases.sql"
    gzip "$BACKUP_DIR/postgresql-all-databases.sql"
fi

echo "Database backups completed in $BACKUP_DIR"

Snapshot-Based Backups (Virtual Machines)

# For VMware environments
# Take snapshot before upgrade
# Name: "Pre-upgrade to 22.04 - $(date +%Y%m%d)"

# For VirtualBox
VBoxManage snapshot "Ubuntu-Server" take "Pre-Ubuntu-22.04-Upgrade" \
    --description "Snapshot before upgrading from 20.04 to 22.04"

# For LVM-based systems
# Create LVM snapshot
sudo lvcreate -L 20G -s -n root-snapshot /dev/vg0/root

# For cloud instances (AWS)
# Create AMI or EBS snapshot
aws ec2 create-image --instance-id i-1234567890abcdef0 \
    --name "Pre-upgrade-ubuntu-2204-$(date +%Y%m%d)" \
    --description "Backup before Ubuntu 22.04 upgrade"

Verify Backup Integrity

#!/bin/bash
# Verify backup integrity

BACKUP_FILE="/backup/pre-upgrade-20241215/system-backup.tar.gz"

echo "Verifying backup integrity..."

# Test archive integrity
if tar -tzf "$BACKUP_FILE" > /dev/null; then
    echo "✓ Backup archive is valid"
else
    echo "✗ Backup archive is corrupted!"
    exit 1
fi

# Check backup size
SIZE=$(du -sh "$BACKUP_FILE" | cut -f1)
echo "Backup size: $SIZE"

# List backup contents sample
echo "Sample of backup contents:"
tar -tzf "$BACKUP_FILE" | head -20

# Calculate checksum
sha256sum "$BACKUP_FILE" > "$BACKUP_FILE.sha256"
echo "Checksum saved to $BACKUP_FILE.sha256"

Testing the Upgrade Process

Always test the upgrade on a non-production system first.

Create Test Environment

# Option 1: Clone virtual machine
# Create exact copy of production system

# Option 2: Use cloud snapshot
# AWS - Launch instance from snapshot
aws ec2 run-instances --image-id ami-snapshot-id \
    --instance-type t3.medium \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=upgrade-test}]'

# Option 3: LXD container clone
lxc copy production-server upgrade-test
lxc start upgrade-test

# Option 4: Local VM with Vagrant
vagrant init ubuntu/focal64
vagrant up
vagrant ssh

Run Test Upgrade

#!/bin/bash
# Test upgrade procedure

echo "Starting test upgrade on non-production system..."

# Update current system
sudo apt update
sudo apt upgrade -y
sudo apt dist-upgrade -y
sudo apt autoremove -y

# Clean package cache
sudo apt clean

# Reboot if kernel updated
if [ -f /var/run/reboot-required ]; then
    echo "Reboot required, rebooting test system..."
    sudo reboot
    # Wait and reconnect
fi

# Perform upgrade
sudo do-release-upgrade -d  # -d for development release if testing early

# Document any issues encountered
# Note configuration file conflicts
# Record services that failed to start

Document Test Results

cat << 'EOF' > /root/test-upgrade-results.md
## Test Upgrade Results

### Test Environment
- Date: 2024-12-10
- System: Clone of production-web-01
- From: Ubuntu 20.04.6 LTS
- To: Ubuntu 22.04.3 LTS

### Upgrade Process
- Start time: 14:30
- End time: 16:15
- Duration: 1 hour 45 minutes
- Download size: 1.2 GB
- Packages upgraded: 847

### Issues Encountered
1. **PHP version change (7.4 → 8.1)**
   - Action: Updated application code
   - Time: 30 minutes

2. **PostgreSQL configuration conflict**
   - File: /etc/postgresql/14/main/postgresql.conf
   - Action: Kept local version
   - Verified: Working correctly

3. **Custom systemd service failed**
   - Service: custom-app.service
   - Issue: Deprecated directive
   - Fix: Updated service file

### Post-Upgrade Status
- ✓ System boots successfully
- ✓ All services started
- ✓ Application functional
- ✓ Database accessible
- ✓ Performance acceptable

### Recommendations
1. Update PHP code before production upgrade
2. Review PostgreSQL configuration changes
3. Update custom systemd services
4. Allocate 2-hour maintenance window
5. Have database administrator on call
EOF

Preparing the System for Upgrade

Proper preparation prevents most upgrade issues.

Update Current System

#!/bin/bash
# Fully update current system before upgrading

echo "Updating current Ubuntu 20.04 system..."

# Update package lists
sudo apt update

# Upgrade all installed packages
sudo apt upgrade -y

# Perform distribution upgrade (within current version)
sudo apt dist-upgrade -y

# Remove unused packages
sudo apt autoremove -y

# Clean package cache
sudo apt autoclean

echo "Current system fully updated"

Resolve Package Issues

# Check for broken packages
sudo dpkg --configure -a
sudo apt --fix-broken install

# Check for held packages
apt-mark showhold

# If packages are held, investigate why
# Release hold if safe
# sudo apt-mark unhold package-name

# Verify no packages in inconsistent state
sudo dpkg --audit

Disable Third-Party Repositories

#!/bin/bash
# Disable third-party repositories before upgrade

echo "Disabling third-party repositories..."

# Backup repository configuration
sudo cp -r /etc/apt/sources.list.d /etc/apt/sources.list.d.backup

# Disable PPA repositories
sudo mv /etc/apt/sources.list.d/ppa-*.list /etc/apt/sources.list.d.backup/

# Alternative: Comment out third-party sources
sudo sed -i 's/^deb /#deb /g' /etc/apt/sources.list.d/*.list

# Keep only official Ubuntu repositories
sudo apt update

echo "Third-party repositories disabled"
echo "Re-enable after upgrade completion"

Remove Obsolete Packages

# Remove old kernels (keep current and one previous)
sudo apt autoremove --purge

# Remove obsolete packages
sudo apt remove --purge $(dpkg -l | grep '^rc' | awk '{print $2}')

# Check for manually installed packages that might conflict
apt list --installed | grep -v "$(lsb_release -cs)"

Free Up Disk Space

#!/bin/bash
# Ensure sufficient disk space for upgrade

echo "Checking available disk space..."

# Check root partition space
ROOT_AVAILABLE=$(df -h / | tail -1 | awk '{print $4}')
echo "Available space on /: $ROOT_AVAILABLE"

# Upgrade requires at least 5GB free on /
if [ $(df / | tail -1 | awk '{print $4}') -lt 5242880 ]; then
    echo "WARNING: Less than 5GB free on /"
    echo "Cleaning up space..."

    # Clean package cache
    sudo apt clean

    # Remove old log files
    sudo journalctl --vacuum-time=7d

    # Clean temp files
    sudo rm -rf /tmp/*
    sudo rm -rf /var/tmp/*

    # Remove old kernels
    sudo apt autoremove --purge
fi

# Verify space after cleanup
df -h /

Backup and Update Configuration Files

#!/bin/bash
# Backup configuration files before upgrade

CONFIG_BACKUP="/root/config-backup-$(date +%Y%m%d)"
mkdir -p "$CONFIG_BACKUP"

# Backup critical configurations
sudo cp -r /etc/netplan "$CONFIG_BACKUP/"
sudo cp -r /etc/nginx "$CONFIG_BACKUP/"
sudo cp -r /etc/apache2 "$CONFIG_BACKUP/"
sudo cp -r /etc/mysql "$CONFIG_BACKUP/"
sudo cp -r /etc/postgresql "$CONFIG_BACKUP/"
sudo cp -r /etc/ssh "$CONFIG_BACKUP/"
sudo cp /etc/fstab "$CONFIG_BACKUP/"
sudo cp /etc/hosts "$CONFIG_BACKUP/"

# Create tarball
cd /root
tar -czf config-backup-$(date +%Y%m%d).tar.gz config-backup-$(date +%Y%m%d)/

echo "Configuration backup saved to $CONFIG_BACKUP"

Performing the Upgrade (Method 1: do-release-upgrade)

The official and recommended method for upgrading Ubuntu LTS releases.

Update Release Upgrader Configuration

# Configure to allow LTS to LTS upgrades
sudo nano /etc/update-manager/release-upgrades

# Ensure this line is present:
# Prompt=lts

# This ensures only LTS releases are offered for upgrade

Check for Available Upgrades

# Check if new LTS release is available
sudo do-release-upgrade -c

# Example output:
# Checking for a new Ubuntu release
# New release '22.04.3 LTS' available.
# Run 'do-release-upgrade' to upgrade to it.

Start the Upgrade Process

#!/bin/bash
# Perform the actual upgrade

# Ensure you're in a stable terminal session
# Use screen or tmux for SSH connections to prevent disconnection issues

# Start screen session
screen -S upgrade

# Or start tmux session
# tmux new -s upgrade

# Begin upgrade
sudo do-release-upgrade

# The upgrade tool will:
# 1. Check system readiness
# 2. Calculate required changes
# 3. Download packages
# 4. Install new packages
# 5. Handle configuration file conflicts
# 6. Remove obsolete packages
# 7. Prompt for reboot

Handling Configuration File Conflicts

During the upgrade, you'll encounter configuration file conflicts:

# Typical prompt:
# Configuration file '/etc/nginx/nginx.conf'
#  ==> Modified (by you or by a script) since installation.
#  ==> Package distributor has shipped an updated version.
#    What would you like to do about it?
#  Your options are:
#   Y or I  : install the package maintainer's version
#   N or O  : keep your currently-installed version
#   D       : show the differences between the versions
#   Z       : start a shell to examine the situation

# Best practice: Choose 'D' to see differences
# Then decide based on your customizations

# For files you've heavily modified: Choose 'N' (keep)
# For files you haven't modified: Choose 'Y' (install new)

# You can always merge changes later

Monitor Upgrade Progress

# In another terminal (if available)
# Monitor disk I/O
iostat -x 5

# Monitor network activity
iftop

# Watch syslog
tail -f /var/log/syslog

# Monitor process activity
htop

Complete the Upgrade

# After package installation completes
# Remove obsolete packages when prompted
# Restart when prompted

sudo reboot

# After reboot, verify new version
lsb_release -a

# Output should show:
# Description:    Ubuntu 22.04.3 LTS
# Release:        22.04

Alternative Upgrade Methods

While do-release-upgrade is recommended, alternative methods exist.

Manual Upgrade via APT

#!/bin/bash
# Advanced users only - manual upgrade method

# Update sources.list to new release
sudo sed -i 's/focal/jammy/g' /etc/apt/sources.list

# Update package lists
sudo apt update

# Upgrade packages
sudo apt upgrade -y
sudo apt dist-upgrade -y

# Install new Ubuntu base
sudo apt install ubuntu-desktop  # or ubuntu-server

# Reboot
sudo reboot

# Note: This method is NOT officially supported
# Use only if do-release-upgrade fails

Upgrade from Installation Media

# If system upgrade fails, boot from Ubuntu 22.04 USB
# Choose "Upgrade Ubuntu 20.04 to 22.04"
# Existing data and configurations preserved

# This method requires:
# - Physical or virtual media access
# - Potential downtime
# - Manual intervention

Troubleshooting Common Issues

Even well-planned upgrades can encounter issues.

Upgrade Fails to Start

# Issue: do-release-upgrade reports no new release

# Solution 1: Check release upgrade configuration
cat /etc/update-manager/release-upgrades
# Ensure: Prompt=lts

# Solution 2: Force check for development release
sudo do-release-upgrade -d

# Solution 3: Update release upgrader
sudo apt update
sudo apt install ubuntu-release-upgrader-core

# Solution 4: Check if you're on latest point release
sudo apt update && sudo apt dist-upgrade

Package Dependency Conflicts

# Issue: Unresolved dependencies during upgrade

# View detailed dependency information
apt-cache depends package-name
apt-cache rdepends package-name

# Try to fix broken packages
sudo apt --fix-broken install
sudo dpkg --configure -a

# Remove conflicting package temporarily
sudo apt remove conflicting-package

# Retry upgrade
sudo do-release-upgrade

SSH Connection Lost During Upgrade

# Prevention: Use screen or tmux

# Start screen before upgrade
screen -S ubuntu-upgrade

# If disconnected, reconnect
screen -r ubuntu-upgrade

# Recovery if upgrade interrupted:
# Reconnect to server (console access may be required)

# Continue interrupted upgrade
sudo dpkg --configure -a
sudo apt --fix-broken install
sudo apt dist-upgrade

Boot Failure After Upgrade

# Access system via recovery mode or rescue disk

# Mount root filesystem
mount /dev/sda1 /mnt  # adjust device as needed

# Chroot into system
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
chroot /mnt

# Repair GRUB bootloader
update-grub
grub-install /dev/sda

# Fix package issues
dpkg --configure -a
apt --fix-broken install

# Exit chroot and reboot
exit
umount /mnt/dev /mnt/proc /mnt/sys
umount /mnt
reboot

Services Fail to Start

# Check service status
systemctl status service-name

# View service logs
journalctl -u service-name -n 50

# Check for configuration file issues
service-name -t  # test configuration (nginx, apache2)

# Restore configuration from backup if needed
sudo cp /root/config-backup-20241215/nginx/nginx.conf /etc/nginx/

# Restart service
sudo systemctl restart service-name

Kernel Panic or System Instability

# Boot into previous kernel from GRUB menu
# Hold Shift during boot to access GRUB
# Select "Advanced options"
# Choose previous kernel version

# After booting into old kernel
# Remove problematic new kernel
sudo apt remove linux-image-5.15.0-*

# Hold kernel version temporarily
sudo apt-mark hold linux-image-generic

# Investigate kernel issues
dmesg | less
journalctl -k

# When resolved, unhold kernel
sudo apt-mark unhold linux-image-generic

Post-Upgrade Verification and Cleanup

Thorough verification ensures the upgrade succeeded completely.

System Verification Checklist

#!/bin/bash
# Comprehensive post-upgrade verification

echo "Post-Upgrade Verification Checklist"
echo "==================================="

# Verify Ubuntu version
echo -n "Ubuntu version: "
lsb_release -rs

# Verify kernel version
echo -n "Kernel version: "
uname -r

# Check system status
echo -n "System degraded: "
systemctl is-system-running

# List failed services
echo "Failed services:"
systemctl list-units --state=failed

# Check disk space
echo "Disk space:"
df -h / | tail -1

# Verify network connectivity
echo -n "Internet connectivity: "
ping -c 1 google.com > /dev/null 2>&1 && echo "OK" || echo "FAILED"

# Check DNS resolution
echo -n "DNS resolution: "
nslookup ubuntu.com > /dev/null 2>&1 && echo "OK" || echo "FAILED"

# List package issues
echo "Package issues:"
dpkg --audit

echo "Verification complete"

Application Testing

#!/bin/bash
# Test critical applications

# Web server
if systemctl is-active --quiet nginx; then
    curl -I http://localhost
    echo "✓ Nginx responding"
fi

# Database
if systemctl is-active --quiet postgresql; then
    sudo -u postgres psql -c "SELECT version();"
    echo "✓ PostgreSQL responding"
fi

# Application-specific tests
# Add your application health checks here
curl http://localhost:8000/health
systemctl status custom-app.service

Review Log Files

# Check for errors in system logs
sudo journalctl -p err -b

# Check specific service logs
sudo journalctl -u nginx.service -b
sudo journalctl -u postgresql.service -b

# Check kernel messages
sudo dmesg | grep -i error

# Review auth log
sudo tail -100 /var/log/auth.log

# Check for security issues
sudo grep -i "failure" /var/log/auth.log

Re-enable Third-Party Repositories

#!/bin/bash
# Re-enable third-party repositories for new release

echo "Re-enabling third-party repositories..."

# Update repository release names
# focal → jammy for Ubuntu 22.04

sudo sed -i 's/focal/jammy/g' /etc/apt/sources.list.d/*.list

# Or restore and update from backup
# sudo cp -r /etc/apt/sources.list.d.backup/* /etc/apt/sources.list.d/
# sudo sed -i 's/focal/jammy/g' /etc/apt/sources.list.d/*.list

# Update package lists
sudo apt update

# Fix any repository issues
# Some PPAs may not support new release yet

echo "Third-party repositories updated"

Clean Up Obsolete Packages

# Remove old kernel versions (keep current and one previous)
sudo apt autoremove --purge -y

# Remove obsolete packages
sudo apt autopurge -y

# Clean package cache
sudo apt clean

# Remove configuration files from removed packages
sudo dpkg --purge $(dpkg -l | grep '^rc' | awk '{print $2}')

# Check for residual issues
sudo deborphan  # install: sudo apt install deborphan

Update Monitoring and Alerts

# Update monitoring configurations for new OS version
# Update alert thresholds if needed
# Verify monitoring agents still functioning

# Example: Update node_exporter for Prometheus
systemctl status node_exporter

# Example: Update monitoring dashboards
# Update OS version tags in Grafana
# Update alert rules for new kernel version

Rolling Back Failed Upgrades

If the upgrade fails critically, rollback procedures restore the previous state.

Restore from Snapshot (Virtual Machines)

# VMware: Revert to snapshot
# 1. Power off VM
# 2. Right-click VM → Snapshot → Revert to snapshot
# 3. Select pre-upgrade snapshot
# 4. Power on VM

# VirtualBox: Restore snapshot
VBoxManage controlvm "Ubuntu-Server" poweroff
VBoxManage snapshot "Ubuntu-Server" restore "Pre-Ubuntu-22.04-Upgrade"
VBoxManage startvm "Ubuntu-Server" --type headless

# Cloud instance (AWS): Launch from AMI
aws ec2 run-instances --image-id ami-backup-id \
    --instance-type t3.medium \
    --subnet-id subnet-xxxxx \
    --security-group-ids sg-xxxxx

Restore from LVM Snapshot

# Merge LVM snapshot back to original volume
sudo lvconvert --merge /dev/vg0/root-snapshot

# Reboot to complete merge
sudo reboot

# After reboot, snapshot volume is removed
# Original volume restored to pre-upgrade state

Restore from Tar Backup

#!/bin/bash
# Restore from tar backup (last resort)

# Boot from live USB/rescue mode
# Mount root filesystem
mount /dev/sda1 /mnt

# Clear current system
rm -rf /mnt/*

# Restore from backup
cd /mnt
tar -xzf /backup/pre-upgrade-20241215/system-backup.tar.gz

# Reinstall GRUB
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
chroot /mnt
grub-install /dev/sda
update-grub

# Exit and reboot
exit
umount /mnt/dev /mnt/proc /mnt/sys
umount /mnt
reboot

Downgrade Specific Packages

# If only specific packages are problematic
# Downgrade to previous version

# List available package versions
apt-cache policy package-name

# Install specific version
sudo apt install package-name=version

# Hold package at current version
sudo apt-mark hold package-name

# Example: Downgrade PHP
sudo apt install php=7.4* -y
sudo apt-mark hold php*

Best Practices for Production Systems

Minimize risk and downtime in production environments.

Staged Rollout Strategy

# 1. Test environment: Week 1
# 2. Development environment: Week 2
# 3. Staging environment: Week 3
# 4. Production (1 server): Week 4
# 5. Production (remaining servers): Week 5-8

# Monitor each stage for issues before proceeding

Blue-Green Deployment

# Maintain parallel infrastructure
# Blue: Current Ubuntu 20.04 servers
# Green: New Ubuntu 22.04 servers

# 1. Deploy application on new 22.04 servers
# 2. Test thoroughly
# 3. Switch traffic to new servers
# 4. Monitor for issues
# 5. Keep old servers for quick rollback
# 6. Decommission old servers after stability confirmed

# Load balancer switch
# Update DNS or load balancer to point to new servers

Automate with Configuration Management

# Ansible playbook for Ubuntu upgrade
---
- name: Upgrade Ubuntu LTS
  hosts: ubuntu_servers
  serial: 1  # Upgrade one server at a time

  tasks:
    - name: Update all packages
      apt:
        update_cache: yes
        upgrade: dist

    - name: Check if reboot required
      stat:
        path: /var/run/reboot-required
      register: reboot_required

    - name: Reboot if necessary
      reboot:
        reboot_timeout: 300
      when: reboot_required.stat.exists

    - name: Perform release upgrade
      command: do-release-upgrade -f DistUpgradeViewNonInteractive
      register: upgrade_result

    - name: Reboot after upgrade
      reboot:
        reboot_timeout: 600

Maintenance Window Planning

# Schedule maintenance window
cat << 'EOF'
Maintenance Window Notification

Service: Production Web Servers
Date: December 15, 2024
Time: 02:00 - 06:00 UTC (4-hour window)
Expected Impact: Service interruption
Expected Duration: 2 hours
Reason: Ubuntu LTS upgrade (20.04 → 22.04)

Rollback Plan: Available within 30 minutes if issues occur
Support Contact: [email protected]
Status Page: https://status.example.com
EOF

Automating Upgrades at Scale

Managing dozens or hundreds of servers requires automation.

Unattended Upgrades

# Configure unattended-upgrades for security updates
sudo apt install unattended-upgrades

# Configure automatic upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades

# Edit configuration
sudo nano /etc/apt/apt.conf.d/50unattended-upgrades

# Enable automatic reboot if needed
# Unattended-Upgrade::Automatic-Reboot "true";
# Unattended-Upgrade::Automatic-Reboot-Time "03:00";

Landscape (Ubuntu Management Tool)

# Canonical's Landscape for managing multiple Ubuntu systems
# Commercial tool with free tier for limited servers

# Install landscape client
sudo apt install landscape-client

# Configure landscape
sudo landscape-config --account-name ACCOUNT --computer-title "Server01"

# Manage upgrades through web interface

Custom Automation Script

#!/bin/bash
# Automated LTS upgrade script with safety checks

set -e  # Exit on error

LOG_FILE="/var/log/ubuntu-upgrade-$(date +%Y%m%d).log"

log() {
    echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}

# Pre-flight checks
log "Starting pre-flight checks..."

# Check internet connectivity
if ! ping -c 1 google.com > /dev/null 2>&1; then
    log "ERROR: No internet connectivity"
    exit 1
fi

# Check disk space (need at least 5GB free)
FREE_SPACE=$(df / | tail -1 | awk '{print $4}')
if [ $FREE_SPACE -lt 5242880 ]; then
    log "ERROR: Insufficient disk space"
    exit 1
fi

# Create backup
log "Creating backup..."
BACKUP_DIR="/backup/pre-upgrade-$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"
tar -czf "$BACKUP_DIR/etc-backup.tar.gz" /etc
dpkg --get-selections > "$BACKUP_DIR/package-selections.txt"

# Update current system
log "Updating current system..."
apt update
apt upgrade -y
apt dist-upgrade -y

# Perform upgrade
log "Starting release upgrade..."
DEBIAN_FRONTEND=noninteractive do-release-upgrade -f DistUpgradeViewNonInteractive

log "Upgrade completed successfully"

# Reboot
log "Rebooting system..."
reboot

Conclusion

Upgrading Ubuntu LTS releases is a manageable process when approached methodically with proper planning, testing, and backup procedures. While Ubuntu's do-release-upgrade tool handles most of the complexity, successful upgrades in production environments require careful preparation and verification.

Key Takeaways:

1. Never Skip Planning: Document your system, identify dependencies, and create detailed upgrade plans before beginning.

2. Always Backup: Comprehensive backups are mandatory. Test restore procedures before the upgrade.

3. Test First: Never upgrade production without testing the exact procedure on a replica system first.

4. Use Official Tools: The do-release-upgrade utility is the supported and most reliable upgrade method.

5. Staged Rollouts: Upgrade production systems incrementally, monitoring each stage before proceeding.

6. Verify Thoroughly: Post-upgrade verification is as important as the upgrade itself. Test all critical services and applications.

7. Have a Rollback Plan: Know your recovery options and practice them before the upgrade window.

8. Monitor and Document: Log everything during the upgrade and document any issues for future reference.

Recommended Timeline for Production Upgrades:

• Month 1: Test upgrade on replica system, document issues
• Month 2: Update procedures, test again, train team
• Month 3: Upgrade non-critical production systems
• Month 4-6: Staged rollout to critical production systems

When to Upgrade:

Ubuntu LTS upgrades should be performed:

• Before current LTS reaches end-of-life
• When applications require newer dependencies
• To access security features in newer releases
• Following thorough testing and validation

When to Delay:

Consider delaying upgrades when:

• Critical applications lack compatibility testing
• Major project deadlines approach
• Team lacks bandwidth for potential issues
• Current LTS still has years of support remaining

By following the procedures outlined in this guide, you can confidently upgrade Ubuntu LTS systems while minimizing risk and downtime. Remember that patience and thoroughness prevent far more problems than they cause.

Next Steps

After a successful upgrade:

1. Document lessons learned and update procedures
2. Train team members on new features and changes
3. Update disaster recovery documentation
4. Review and optimize system configurations for new release
5. Plan for the next LTS upgrade cycle
6. Subscribe to Ubuntu security announcements
7. Join Ubuntu community channels for ongoing support

Ubuntu's LTS releases provide a stable, well-supported foundation for production systems. With proper upgrade procedures, you can maintain that stability while benefiting from the security, performance, and feature improvements in newer releases.