PCI passthrough gives a KVM virtual machine direct, exclusive access to a physical PCIe device such as a GPU, a network card or an NVMe controller. The guest loads the device's native driver and gets near bare-metal performance, while the host no longer uses the device at all. In this tutorial you will enable the IOMMU on an Ubuntu 24.04 host, reserve a GPU for the vfio-pci driver, create a libvirt virtual machine that owns it, and verify the device inside the guest.

Prerequisites

To follow this guide you need:

  • A bare-metal server running Ubuntu 24.04 LTS, for example a CubePath dedicated server. Passthrough does not work inside a VPS, because you need control over the physical hardware and firmware.
  • A CPU and motherboard with IOMMU support: Intel VT-d or AMD-Vi, enabled in the BIOS/UEFI setup together with VT-x or AMD-V.
  • The PCIe device you want to pass through. This guide uses a GPU at address 03:00.0 with its HDMI audio function at 03:00.1; your addresses and IDs will differ.
  • A non-root user with sudo privileges and out-of-band console access (IPMI, iKVM) in case the host loses its display or network after a reboot.
  • An installation ISO for the guest operating system.

Step 1 - Enabling the IOMMU

The IOMMU lets the host map a device's DMA directly into a guest's memory safely. First check that the CPU exposes hardware virtualization:

grep -Ec '(vmx|svm)' /proc/cpuinfo

Any number greater than 0 means VT-x or AMD-V is available. Next, add the kernel parameters. Open the GRUB defaults file:

sudo nano /etc/default/grub

On an Intel host, set the default command line like this, keeping any options that were already there:

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt"

On an AMD host the IOMMU is enabled automatically when AMD-Vi is on in the firmware, so only iommu=pt is needed:

GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt"

iommu=pt puts devices that stay on the host in passthrough mode, which avoids translation overhead for them. Regenerate the GRUB configuration and reboot:

sudo update-grub
sudo reboot

After the reboot, confirm the parameters were applied and the IOMMU is active:

cat /proc/cmdline
sudo dmesg | grep -iE 'DMAR|AMD-Vi|IOMMU' | head

On Intel you should see a line like the following; on AMD look for AMD-Vi lines mentioning interrupt remapping:

DMAR: IOMMU enabled

Finally, check that the kernel created IOMMU groups:

ls /sys/kernel/iommu_groups/ | wc -l

A value of 0 means the IOMMU is still disabled; recheck the firmware settings.

Step 2 - Identifying the device and its IOMMU group

The IOMMU group is the smallest set of devices the kernel can isolate. Every device in a group must be given to the same VM (PCIe bridges are the exception), so check the group before going further. Print all groups with this loop:

for g in /sys/kernel/iommu_groups/*; do
  echo "IOMMU group ${g##*/}:"
  for d in "$g"/devices/*; do
    echo "    $(lspci -nns "${d##*/}")"
  done
done

Find your device in the output. A well isolated GPU looks like this:

IOMMU group 14:
    03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
    03:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)

Write down two things for each function of the device:

  • The PCI address (03:00.0 and 03:00.1), used in the VM definition.
  • The vendor:device ID in brackets (10de:2206 and 10de:1aef), used to bind the driver.

If the group also contains unrelated devices such as a USB controller or a SATA controller, move the card to another slot (slots wired to the CPU usually have their own group) or check for a firmware update. Avoid the unofficial ACS override patch on production hosts: it hides the isolation problem instead of solving it.

Step 3 - Binding the device to vfio-pci

libvirt can detach a device from its host driver automatically when a VM starts. That works well for NICs and NVMe drives, so if that is your use case you can skip to Step 4. GPUs are different: nouveau, nvidia or amdgpu grab the card at boot and often cannot release it cleanly, so reserve the GPU for vfio-pci before any other driver loads.

Add the IDs from Step 2 to the kernel command line. Edit /etc/default/grub again:

sudo nano /etc/default/grub

Append vfio-pci.ids with both IDs, separated by a comma:

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt vfio-pci.ids=10de:2206,10de:1aef"

Then make sure vfio-pci loads before the GPU and audio drivers. Create a modprobe configuration file:

sudo nano /etc/modprobe.d/vfio.conf

Add these soft dependencies (use amdgpu instead of nouveau and nvidia for an AMD card):

softdep nouveau pre: vfio-pci
softdep nvidia pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci

Include the VFIO modules in the initramfs so they are available early in the boot process:

printf '%s\n' vfio vfio_iommu_type1 vfio_pci | sudo tee -a /etc/initramfs-tools/modules

Rebuild the initramfs and GRUB configuration, then reboot:

sudo update-initramfs -u
sudo update-grub
sudo reboot

Check which driver owns each function now:

lspci -nnk -s 03:00

Both functions must report vfio-pci:

03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
	Kernel driver in use: vfio-pci
	Kernel modules: nvidiafb, nouveau
03:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

Step 4 - Installing KVM and libvirt

Install QEMU, libvirt, the virt-install tool and the OVMF UEFI firmware. Modern GPUs expect a UEFI guest, so OVMF is required:

sudo apt update
sudo apt install qemu-system-x86 libvirt-daemon-system libvirt-clients virtinst ovmf

Add your user to the libvirt group so you can manage system VMs without sudo, then log out and back in for the change to take effect:

sudo usermod -aG libvirt "$USER"

Confirm the daemon is running and that the host passes libvirt's checks:

systemctl status libvirtd --no-pager
virt-host-validate qemu

The IOMMU lines should report PASS:

  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : PASS

Step 5 - Creating a VM with the device attached

libvirt names PCI devices pci_DOMAIN_BUS_SLOT_FUNCTION. List them to confirm the names of your device:

virsh -c qemu:///system nodedev-list --cap pci | grep 03_00

pci_0000_03_00_0
pci_0000_03_00_1

Copy the ISO to libvirt's image directory so the libvirt-qemu user can read it, replacing your_image.iso with your file:

sudo cp your_image.iso /var/lib/libvirt/images/

Create the VM with a Q35 machine type, UEFI firmware and both GPU functions attached. Adjust memory, CPUs and disk size to your needs:

virt-install --connect qemu:///system \
  --name gpu-vm \
  --memory 16384 \
  --vcpus 8 \
  --machine q35 \
  --boot uefi \
  --osinfo linux2022 \
  --disk size=80 \
  --cdrom /var/lib/libvirt/images/your_image.iso \
  --network network=default \
  --graphics vnc \
  --hostdev pci_0000_03_00_0 \
  --hostdev pci_0000_03_00_1

The VNC display gives you a console for the installation. Connect to it through an SSH tunnel with a VNC viewer, or with virt-manager from your workstation. Once the guest OS is installed and has the GPU driver, you can remove the virtual display if you only use the GPU's physical output or remote desktop.

To add a device to an existing VM instead, create an XML file such as gpu.xml:

nano gpu.xml

<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
  </source>
</hostdev>

Attach it to the VM definition; it will be present from the next VM start:

virsh -c qemu:///system attach-device gpu-vm gpu.xml --config

Repeat with function='0x1' for the audio function.

Step 6 - Verifying the device inside the guest

Start the VM if it is not running and log in to the guest:

virsh -c qemu:///system start gpu-vm

Inside the guest, the GPU appears as a normal PCIe device at a different address:

lspci -nnk | grep -A3 -i nvidia

05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
	Subsystem: ...
	Kernel driver in use: nvidia

Install the vendor driver in the guest exactly as on a physical machine. For an NVIDIA card on an Ubuntu guest, sudo ubuntu-drivers install followed by a reboot and nvidia-smi is the quickest check. Current NVIDIA drivers no longer refuse to run in a VM, so hiding the hypervisor from the guest is not needed.

Step 7 - Tuning performance (optional)

Two changes make the biggest difference for GPU and low-latency workloads: pinning vCPUs to dedicated host cores and backing guest memory with huge pages.

Check the host topology first, so each vCPU is pinned to a physical core on the same NUMA node as the device:

lscpu -e
cat /sys/bus/pci/devices/0000:03:00.0/numa_node

Reserve huge pages for the guest. With the default 2 MiB page size, 16 GiB of guest memory needs 8192 pages. Create a sysctl file:

sudo nano /etc/sysctl.d/80-hugepages.conf

vm.nr_hugepages = 8192

Apply it and check how many pages the kernel could allocate:

sudo sysctl --system
grep HugePages_Total /proc/meminfo

If the total is lower than requested, memory is fragmented; reboot so the pages are reserved at boot. Now edit the VM definition:

virsh -c qemu:///system edit gpu-vm

Add or adjust these elements inside <domain>. The example pins 8 vCPUs to host CPUs 4-11 and keeps QEMU's own threads on CPUs 2-3:

<memoryBacking>
  <hugepages/>
</memoryBacking>
<vcpu placement='static'>8</vcpu>
<cputune>
  <vcpupin vcpu='0' cpuset='4'/>
  <vcpupin vcpu='1' cpuset='5'/>
  <vcpupin vcpu='2' cpuset='6'/>
  <vcpupin vcpu='3' cpuset='7'/>
  <vcpupin vcpu='4' cpuset='8'/>
  <vcpupin vcpu='5' cpuset='9'/>
  <vcpupin vcpu='6' cpuset='10'/>
  <vcpupin vcpu='7' cpuset='11'/>
  <emulatorpin cpuset='2-3'/>
</cputune>
<cpu mode='host-passthrough' check='none'/>

Restart the VM (a full shutdown, not a reboot from inside the guest) and confirm the pinning:

virsh -c qemu:///system shutdown gpu-vm
virsh -c qemu:///system start gpu-vm
virsh -c qemu:///system vcpupin gpu-vm

Troubleshooting

virt-host-validate reports that the IOMMU is not enabled. Check cat /proc/cmdline. If intel_iommu=on is missing, update-grub was not run or another file in /etc/default/grub.d/ overrides the command line. If the parameter is present, VT-d or AMD-Vi is disabled in the firmware.

The device is still bound to nouveau, nvidia or amdgpu. Confirm the IDs in vfio-pci.ids match lspci -nn exactly, that /etc/modprobe.d/vfio.conf contains the right driver names, and that you ran sudo update-initramfs -u after editing it. lsinitramfs /boot/initrd.img-$(uname -r) | grep vfio shows whether the modules made it into the initramfs.

The VM fails with "group N is not viable". Another device in the same IOMMU group is still in use by the host. List the group with the loop from Step 2 and either pass through every non-bridge device in it or move the card to another slot.

The VM starts but the GPU shows error 43 on Windows or a black screen. Make sure the VM uses UEFI (--boot uefi) and the Q35 machine type, and that both the video and audio functions are attached. Some older cards also need their VBIOS dumped and supplied with a <rom file='...'/> element.

Messages about the device not resetting. Some GPUs do not support a clean function-level reset, so they fail on the second VM start until the host reboots. Check the host log for details:

sudo journalctl -k | grep -i vfio

Conclusion

Your Ubuntu 24.04 host now isolates a PCIe device with the IOMMU, reserves it for vfio-pci at boot and hands it to a libvirt VM that uses it with its native driver. From here you can build a CUDA or inference VM on top of it, pass through a high-speed NIC or NVMe drive with the same hostdev mechanism, or back up the VM definition with virsh dumpxml gpu-vm so you can recreate it on another host.