Image Optimization Pipeline for Web Servers

An image optimization pipeline reduces bandwidth and improves page load times by converting images to modern formats (WebP, AVIF), compressing them, and resizing on demand. This guide covers building an automated image optimization pipeline on Linux web servers using Nginx, on-the-fly processing, CDN integration, and batch conversion scripts.

Prerequisites

Ubuntu 20.04+/Debian 11+ or CentOS 8+/Rocky Linux 8+
Nginx installed
Root or sudo access
At least 1 GB RAM for image processing operations

Installing Image Processing Tools

# Install core image tools
sudo apt update && sudo apt install -y \
  imagemagick \
  libvips-tools \
  webp \
  libjpeg-turbo-progs \
  pngquant \
  optipng \
  gifsicle

# Install AVIF support (via libavif)
sudo apt install -y libavif-bin  # Ubuntu 22.04+

# Install sqip/sharp via Node.js for advanced processing
npm install -g sharp-cli

# Verify installations
cwebp --version
convert --version | head -1
vips --version

For CentOS/Rocky:

sudo dnf install -y ImageMagick libwebp-tools pngquant gifsicle

WebP and AVIF Conversion Scripts

Create a script to convert images to WebP:

cat > /usr/local/bin/convert-to-webp.sh << 'SCRIPT'
#!/bin/bash
# Convert JPEG/PNG images to WebP format
# Usage: convert-to-webp.sh [directory] [quality]

DIR="${1:-.}"
QUALITY="${2:-80}"
CONVERTED=0
SKIPPED=0

find "$DIR" -type f \( -iname "*.jpg" -o -iname "*.jpeg" -o -iname "*.png" \) | \
while read -r FILE; do
  WEBP="${FILE%.*}.webp"
  if [ ! -f "$WEBP" ] || [ "$FILE" -nt "$WEBP" ]; then
    if cwebp -q "$QUALITY" "$FILE" -o "$WEBP" 2>/dev/null; then
      ORIG_SIZE=$(stat -c%s "$FILE")
      NEW_SIZE=$(stat -c%s "$WEBP")
      SAVING=$(( (ORIG_SIZE - NEW_SIZE) * 100 / ORIG_SIZE ))
      echo "Converted: $FILE → $WEBP (${SAVING}% smaller)"
      ((CONVERTED++))
    fi
  else
    ((SKIPPED++))
  fi
done

echo "Done. Converted: $CONVERTED, Skipped (up-to-date): $SKIPPED"
SCRIPT

chmod +x /usr/local/bin/convert-to-webp.sh

Create an AVIF conversion script:

cat > /usr/local/bin/convert-to-avif.sh << 'SCRIPT'
#!/bin/bash
# Convert images to AVIF format
# Requires: avifenc (libavif-bin) or ImageMagick with AVIF support

DIR="${1:-.}"
QUALITY="${2:-60}"  # AVIF quality 0-63, lower = better quality

find "$DIR" -type f \( -iname "*.jpg" -o -iname "*.jpeg" -o -iname "*.png" \) | \
while read -r FILE; do
  AVIF="${FILE%.*}.avif"
  if [ ! -f "$AVIF" ] || [ "$FILE" -nt "$AVIF" ]; then
    # Try avifenc first (better quality), fall back to ImageMagick
    if command -v avifenc &>/dev/null; then
      avifenc -q "$QUALITY" "$FILE" "$AVIF" 2>/dev/null && \
        echo "Converted: $FILE → $AVIF"
    else
      convert "$FILE" -quality "$QUALITY" "$AVIF" 2>/dev/null && \
        echo "Converted: $FILE → $AVIF"
    fi
  fi
done
SCRIPT

chmod +x /usr/local/bin/convert-to-avif.sh

Nginx On-the-Fly Image Serving

Serve WebP/AVIF to supporting browsers automatically via Nginx:

# Create a map for WebP support detection
# Add to the http {} block in /etc/nginx/nginx.conf

# In http {} block
map $http_accept $webp_suffix {
    default   "";
    "~*webp"  ".webp";
}

map $http_accept $avif_suffix {
    default   "";
    "~*avif"  ".avif";
}

In your server block:

server {
    listen 443 ssl;
    server_name www.yourdomain.com;
    root /var/www/html;

    # Try AVIF first, then WebP, then original
    location ~* \.(jpe?g|png)$ {
        # Check for AVIF support
        set $img_uri $uri;
        
        # Try serving AVIF for browsers that support it
        location ~* \.(jpe?g|png)$ {
            add_header Vary Accept;
            try_files
                $uri$avif_suffix
                $uri$webp_suffix
                $uri =404;
        }
    }

    # Serve pre-generated WebP files
    location / {
        add_header Vary Accept;
        try_files $uri$webp_suffix $uri $uri/ =404;
    }

    # Set long cache for optimized images
    location ~* \.(webp|avif)$ {
        expires 1y;
        add_header Cache-Control "public, immutable";
    }
}

A cleaner approach with a dedicated image server location:

server {
    root /var/www/html;

    location ~* \.(jpg|jpeg|png)$ {
        add_header Vary Accept;
        expires 30d;

        # Serve WebP to supporting browsers
        if ($http_accept ~* "webp") {
            set $webp_path $document_root$uri.webp;
            if (-f $webp_path) {
                rewrite ^ $uri.webp last;
            }
        }
    }

    location ~* \.webp$ {
        add_header Content-Type image/webp;
        expires 30d;
        add_header Cache-Control "public";
    }
}

On-the-Fly Resizing with Nginx Image Filter

The ngx_http_image_filter_module resizes images on the fly:

# Install the module (usually included in nginx-extras)
sudo apt install -y nginx-extras
nginx -V 2>&1 | grep image_filter  # Verify

server {
    # Image resizing endpoint
    # Usage: /resize/300x200/path/to/image.jpg
    location ~ ^/resize/(\d+)x(\d+)/(.+)$ {
        alias /var/www/html/$3;
        image_filter resize $1 $2;
        image_filter_jpeg_quality 85;
        image_filter_webp_quality 80;
        image_filter_buffer 20M;
        
        add_header Cache-Control "public, max-age=86400";
    }

    # Thumbnail generation
    location ~ ^/thumb/(\d+)/(.+)$ {
        alias /var/www/html/$2;
        image_filter resize $1 -;  # Resize width, auto height
        image_filter_jpeg_quality 80;
        image_filter_buffer 10M;
    }
}

For more advanced resizing, use ngx_small_light or proxy to a dedicated image server like Thumbor or imgproxy:

# Deploy imgproxy for production-grade on-the-fly processing
docker run -d \
  --name imgproxy \
  -p 8080:8080 \
  -e IMGPROXY_KEY=your_hex_key \
  -e IMGPROXY_SALT=your_hex_salt \
  -e IMGPROXY_QUALITY=80 \
  -e IMGPROXY_WEBP_COMPRESSION=80 \
  -e IMGPROXY_AVIF_SPEED=8 \
  darthsim/imgproxy:latest

Automated Batch Processing

Automate image optimization as part of your deployment pipeline:

cat > /usr/local/bin/optimize-web-images.sh << 'SCRIPT'
#!/bin/bash
# Full image optimization pipeline
DIR="${1:-/var/www/html}"

echo "=== Optimizing JPEG files ==="
find "$DIR" -name "*.jpg" -o -name "*.jpeg" | while read f; do
    jpegoptim --max=85 --strip-all "$f"
done

echo "=== Optimizing PNG files ==="
find "$DIR" -name "*.png" | while read f; do
    pngquant --force --quality=65-80 --output "$f" "$f"
done

echo "=== Generating WebP variants ==="
/usr/local/bin/convert-to-webp.sh "$DIR" 80

echo "=== Generating AVIF variants ==="
/usr/local/bin/convert-to-avif.sh "$DIR" 60

echo "=== Done ==="
du -sh "$DIR"
SCRIPT

chmod +x /usr/local/bin/optimize-web-images.sh

Set up inotify to process new uploads automatically:

sudo apt install -y inotify-tools

cat > /usr/local/bin/watch-uploads.sh << 'SCRIPT'
#!/bin/bash
WATCH_DIR="/var/www/html/uploads"

inotifywait -m -e close_write -e moved_to "$WATCH_DIR" | \
while read DIR EVENT FILE; do
  case "${FILE##*.}" in
    jpg|jpeg|png)
      echo "Processing: $DIR$FILE"
      cwebp -q 80 "$DIR$FILE" -o "${DIR}${FILE%.*}.webp"
      jpegoptim --max=85 "$DIR$FILE" 2>/dev/null || true
      ;;
  esac
done
SCRIPT

chmod +x /usr/local/bin/watch-uploads.sh

# Run as a systemd service
sudo tee /etc/systemd/system/image-watcher.service << 'EOF'
[Unit]
Description=Image Optimization Watcher
After=network.target

[Service]
ExecStart=/usr/local/bin/watch-uploads.sh
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now image-watcher

Lazy Loading Configuration

Nginx can inject lazy loading attributes into HTML responses:

# Use sub_filter to add lazy loading to images
location ~* \.html$ {
    sub_filter '<img ' '<img loading="lazy" ';
    sub_filter_once off;
    sub_filter_types text/html;
}

Or configure it at the application level. For static sites, process with a script:

# Add loading="lazy" to all img tags in HTML files
find /var/www/html -name "*.html" -exec sed -i \
  's/<img\([^>]*\) \(src\)/<img\1 loading="lazy" \2/g' {} \;

CDN Integration

After converting images locally, sync to a CDN origin:

# Sync optimized images to S3-compatible storage (CDN origin)
aws s3 sync /var/www/html/images/ s3://my-cdn-bucket/images/ \
  --exclude "*" \
  --include "*.webp" \
  --include "*.avif" \
  --include "*.jpg" \
  --cache-control "max-age=31536000,public" \
  --content-type "image/webp" \
  --metadata-directive REPLACE

# Or use rclone for non-AWS CDNs
rclone sync /var/www/html/images/ mycdnremote:bucket/images/ \
  --header-upload "Cache-Control: max-age=31536000"

Troubleshooting

WebP files not being served to Chrome:

# Verify Accept header detection in Nginx
curl -H "Accept: image/webp,*/*" -I https://yourdomain.com/image.jpg
# Should show: Content-Type: image/webp

Image filter module returning 415:

The image_filter_buffer may be too small for large images. Increase it:

image_filter_buffer 50M;

Batch conversion running out of memory:

Process in smaller batches and limit ImageMagick memory:

export MAGICK_MEMORY_LIMIT=256MB
export MAGICK_MAP_LIMIT=512MB

AVIF conversion very slow:

AVIF encoding is CPU-intensive. Use parallel processing:

find /var/www/html -name "*.jpg" | \
  parallel -j4 'avifenc -q 60 {} {.}.avif'

Conclusion

A well-designed image optimization pipeline combining WebP/AVIF conversion, on-the-fly resizing, and lazy loading can reduce image payload by 50-80% compared to unoptimized JPEGs. Start with static WebP conversion and the try_files Nginx pattern for immediate gains, then add on-the-fly resizing via imgproxy or Nginx image filter as your needs grow. Automate the pipeline with inotify watchers to handle new uploads without manual intervention.

Image Optimization Pipeline for Web Servers

On this page

On this page

Image Optimization Pipeline for Web Servers

Prerequisites

Installing Image Processing Tools

WebP and AVIF Conversion Scripts

Nginx On-the-Fly Image Serving

On-the-Fly Resizing with Nginx Image Filter

Automated Batch Processing

Lazy Loading Configuration

CDN Integration

Troubleshooting

Conclusion

Latest Video

Get $20 Free Credit