FFmpeg Transcoding Server Configuration

FFmpeg is the industry-standard tool for video and audio encoding, transcoding, and streaming on Linux servers, capable of converting between virtually any format using CPU or hardware-accelerated encoders like Intel VAAPI and NVIDIA NVENC. Configuring FFmpeg correctly for server-side transcoding requires understanding codec selection, rate control, hardware acceleration passthrough, and pipeline optimization for throughput.

Prerequisites

Ubuntu 20.04+, Debian 11+, or CentOS/Rocky 8+
Root or sudo access
For hardware transcoding: Intel GPU (VAAPI) or NVIDIA GPU (NVENC)
Sufficient storage for input/output files
FFmpeg 4.0+ (FFmpeg 6.0+ recommended)

Installing FFmpeg

Ubuntu/Debian:

# Install from system repositories
sudo apt update && sudo apt install -y ffmpeg

# For a more up-to-date version, use the Jellyfin/deb-multimedia repo or compile from source
# Check version
ffmpeg -version

# Install additional codec libraries
sudo apt install -y \
  libavcodec-extra \
  libx264-dev \
  libx265-dev \
  libvpx-dev \
  libopus-dev

CentOS/Rocky (with RPM Fusion):

# Enable RPM Fusion
sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
sudo dnf install -y https://mirrors.rpmfusion.org/free/el/rpmfusion-free-release-9.noarch.rpm

# Install FFmpeg
sudo dnf install -y ffmpeg ffmpeg-devel

Compile from source (latest features):

# Install build dependencies
sudo apt install -y \
  build-essential nasm yasm \
  libx264-dev libx265-dev libvpx-dev \
  libopus-dev libfdk-aac-dev \
  libass-dev libfreetype6-dev

# Download and compile
wget https://ffmpeg.org/releases/ffmpeg-7.0.tar.xz
tar -xf ffmpeg-7.0.tar.xz && cd ffmpeg-7.0

./configure \
  --enable-gpl \
  --enable-libx264 \
  --enable-libx265 \
  --enable-libvpx \
  --enable-libopus \
  --enable-libfdk-aac \
  --enable-nonfree \
  --enable-vaapi \
  --enable-nvenc

make -j$(nproc)
sudo make install

Basic Transcoding Commands

# Convert video to H.264 MP4 (most compatible)
ffmpeg -i input.mkv -c:v libx264 -c:a aac -b:a 192k output.mp4

# Convert to H.265/HEVC (50% smaller, less compatible)
ffmpeg -i input.mp4 -c:v libx265 -c:a copy output.mkv

# Extract audio only
ffmpeg -i video.mp4 -vn -c:a libopus -b:a 192k audio.opus

# Change resolution (scale down to 1080p)
ffmpeg -i 4k_input.mkv -vf "scale=1920:1080" -c:v libx264 -c:a copy output_1080p.mp4

# Trim video (start at 00:01:00, duration 30 seconds)
ffmpeg -ss 00:01:00 -i input.mp4 -t 30 -c copy trimmed.mp4

# Create HLS segments for streaming
ffmpeg -i input.mp4 \
  -c:v libx264 -c:a aac \
  -hls_time 6 \
  -hls_list_size 0 \
  -f hls output.m3u8

Codec Selection and Quality Settings

H.264 (libx264) — best compatibility:

# Constant Rate Factor (CRF) - lower = better quality, larger file
# CRF 18-28 is the typical range; 23 is default
ffmpeg -i input.mkv \
  -c:v libx264 \
  -crf 22 \
  -preset medium \     # ultrafast, superfast, veryfast, faster, fast, medium, slow, veryslow
  -profile:v high \
  -level 4.1 \
  -c:a aac -b:a 192k \
  output.mp4

# For streaming (CBR with buffer)
ffmpeg -i input.mkv \
  -c:v libx264 \
  -b:v 2500k \
  -maxrate 2500k \
  -bufsize 5000k \
  -c:a aac -b:a 192k \
  output_stream.mp4

H.265 (libx265) — better compression:

ffmpeg -i input.mkv \
  -c:v libx265 \
  -crf 28 \
  -preset slow \
  -c:a copy \
  output_hevc.mkv

VP9 (libvpx-vp9) — open source, good for web:

ffmpeg -i input.mp4 \
  -c:v libvpx-vp9 \
  -crf 33 \
  -b:v 0 \
  -c:a libopus -b:a 128k \
  output.webm

Hardware Acceleration with VAAPI

Intel VAAPI uses the integrated graphics chip for encoding and decoding:

# Check VAAPI support
vainfo
ls /dev/dri/

# H.264 encoding with VAAPI
ffmpeg -hwaccel vaapi \
  -hwaccel_device /dev/dri/renderD128 \
  -hwaccel_output_format vaapi \
  -i input.mp4 \
  -vf "format=nv12|vaapi,hwupload" \
  -c:v h264_vaapi \
  -qp 22 \
  -c:a copy \
  output_vaapi.mp4

# H.265 encoding with VAAPI
ffmpeg -hwaccel vaapi \
  -hwaccel_device /dev/dri/renderD128 \
  -hwaccel_output_format vaapi \
  -i input.mkv \
  -vf "format=nv12|vaapi,hwupload" \
  -c:v hevc_vaapi \
  -qp 28 \
  output_hevc_vaapi.mkv

# Verify hardware is being used (look for h264_vaapi in output)
ffmpeg -hwaccels

VAAPI with scaling (reduce resolution on GPU):

ffmpeg -hwaccel vaapi \
  -hwaccel_device /dev/dri/renderD128 \
  -hwaccel_output_format vaapi \
  -i input_4k.mkv \
  -vf "format=nv12|vaapi,hwupload,scale_vaapi=1920:1080" \
  -c:v h264_vaapi \
  -qp 24 \
  -c:a copy \
  output_1080p.mp4

Hardware Acceleration with NVENC

NVIDIA NVENC uses the GPU's dedicated hardware encoder:

# Verify NVENC support
ffmpeg -encoders 2>/dev/null | grep -E "nvenc|hevc_nv|h264_nv"
nvidia-smi

# H.264 encoding with NVENC
ffmpeg -hwaccel cuda \
  -hwaccel_output_format cuda \
  -i input.mp4 \
  -c:v h264_nvenc \
  -preset p4 \          # p1 (fastest) to p7 (slowest/best quality)
  -rc vbr \
  -b:v 3M \
  -maxrate 5M \
  -bufsize 6M \
  -c:a copy \
  output_nvenc.mp4

# H.265 with NVENC
ffmpeg -hwaccel cuda \
  -hwaccel_output_format cuda \
  -i input.mkv \
  -c:v hevc_nvenc \
  -preset p5 \
  -rc vbr \
  -cq 28 \
  -c:a copy \
  output_hevc_nvenc.mkv

# NVENC with GPU scaling
ffmpeg -hwaccel cuda \
  -hwaccel_output_format cuda \
  -i input_4k.mkv \
  -vf "scale_cuda=1920:1080" \
  -c:v h264_nvenc \
  -preset p4 \
  output_1080p_nvenc.mp4

Batch Processing

# Convert all MKV files in a directory to MP4
for file in /media/movies/**/*.mkv; do
  output="${file%.mkv}.mp4"
  if [[ ! -f "$output" ]]; then
    echo "Converting: $file"
    ffmpeg -i "$file" \
      -c:v libx264 -crf 22 -preset fast \
      -c:a aac -b:a 192k \
      "$output" \
      && echo "Done: $output"
  fi
done

# Parallel batch processing (4 jobs at a time)
find /media/ -name "*.mkv" | \
  xargs -P 4 -I{} bash -c \
  'ffmpeg -i "{}" -c:v libx264 -crf 22 -c:a aac "${{}%.mkv}.mp4" 2>/dev/null'

# Create a conversion queue script
tee /opt/transcode-queue.sh <<'EOF'
#!/bin/bash
INPUT_DIR="/media/incoming"
OUTPUT_DIR="/media/processed"
LOG="/var/log/transcode.log"

find "$INPUT_DIR" -name "*.mkv" -newer "$LOG" | while read -r file; do
  basename=$(basename "$file" .mkv)
  ffmpeg -i "$file" \
    -c:v libx264 -crf 22 -preset medium \
    -c:a aac -b:a 192k \
    "${OUTPUT_DIR}/${basename}.mp4" \
    2>> "$LOG"
  touch "$LOG"
done
EOF
chmod +x /opt/transcode-queue.sh

Streaming Pipelines

RTMP live stream with FFmpeg:

# Stream a file to an RTMP endpoint (Nginx-RTMP, Owncast, etc.)
ffmpeg -re \
  -i /media/source.mp4 \
  -c:v libx264 -preset veryfast \
  -b:v 2500k -maxrate 2500k -bufsize 5000k \
  -c:a aac -b:a 160k \
  -f flv rtmp://your-server/live/streamkey

# Stream webcam + microphone
ffmpeg \
  -f v4l2 -i /dev/video0 \
  -f alsa -i default \
  -c:v libx264 -preset veryfast -b:v 2000k \
  -c:a aac -b:a 128k \
  -f flv rtmp://your-server/live/streamkey

Generate adaptive HLS with multiple bitrates:

ffmpeg -i input.mp4 \
  -filter_complex \
    "[v:0]split=3[v1][v2][v3]; \
     [v1]scale=1920:1080[v1out]; \
     [v2]scale=1280:720[v2out]; \
     [v3]scale=854:480[v3out]" \
  -map "[v1out]" -c:v:0 libx264 -b:v:0 5000k \
  -map "[v2out]" -c:v:1 libx264 -b:v:1 2500k \
  -map "[v3out]" -c:v:2 libx264 -b:v:2 1000k \
  -map a:0 -map a:0 -map a:0 \
  -c:a aac -b:a 192k \
  -f hls -hls_time 4 -hls_list_size 0 \
  -master_pl_name master.m3u8 \
  -hls_segment_filename "stream_%v_%03d.ts" \
  -var_stream_map "v:0,a:0 v:1,a:1 v:2,a:2" \
  stream_%v.m3u8

Troubleshooting

No hardware acceleration available:

# Check available hardware encoders
ffmpeg -encoders 2>/dev/null | grep -E "vaapi|nvenc|qsv"

# Check device permissions
ls -la /dev/dri/
sudo usermod -aG video,render $USER

# Test VAAPI
ffmpeg -hwaccels
vainfo --display drm --device /dev/dri/renderD128

Transcoding too slow:

# Check CPU usage
top -d 1 -p $(pgrep ffmpeg)

# Use faster preset (trades quality for speed)
# -preset ultrafast  (much faster, larger output)

# Enable multithreading
ffmpeg -i input.mp4 -threads 0 -c:v libx264 output.mp4
# -threads 0 = use all available cores

# Check if audio encoding is a bottleneck
ffmpeg -i input.mp4 -c:v libx264 -c:a copy output.mp4

Output file has no sound:

# List all audio streams
ffprobe -v quiet -print_format json -show_streams input.mkv | \
  jq '.streams[] | select(.codec_type == "audio") | .codec_name, .index'

# Select a specific audio stream
ffmpeg -i input.mkv -map 0:v:0 -map 0:a:1 -c:v copy -c:a aac output.mp4

Conclusion

FFmpeg is an essential tool for any server-side media pipeline, offering unparalleled flexibility for transcoding, streaming, and format conversion across virtually any codec and container. By leveraging hardware acceleration through Intel VAAPI or NVIDIA NVENC, you can transcode multiple simultaneous streams with a fraction of the CPU overhead of software encoding, making it practical to run transcoding workloads on VPS hardware without dedicated media processing appliances.

FFmpeg Transcoding Server Configuration

On this page

On this page

FFmpeg Transcoding Server Configuration

Prerequisites

Installing FFmpeg

Basic Transcoding Commands

Codec Selection and Quality Settings

Hardware Acceleration with VAAPI

Hardware Acceleration with NVENC

Batch Processing

Streaming Pipelines

Troubleshooting

Conclusion

Latest Video

Get $20 Free Credit