Step By Step Guide for Proxmox GPU Passthrough!

5 min readMay 24, 2024

In this write-up, I will walk you through the steps to enable GPU passthrough on a Windows 11 machine using the NVIDIA GeForce GTX 1650 SUPER. This guide will cover the necessary steps and configurations to achieve GPU passthrough using the vfio-pci driver and ensure the GPU is correctly bound to the virtual machine.

Step 1: Verify PCI Devices

First, let’s verify the PCI devices using the `lspci` command.

lspci -v

# NOTE: Initially, it might not show the kernel driver vfio-pci, please check the kernel driver.

01:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] (rev a1) (prog-if 00 [VGA controller])
 Subsystem: ZOTAC International (MCO) Ltd. TU116 [GeForce GTX 1650 SUPER]
 Flags: bus master, fast devsel, latency 0, IRQ 11, IOMMU group 1
 Memory at de000000 (32-bit, non-prefetchable) [size=16M]
 Memory at c0000000 (64-bit, prefetchable) [size=256M]
 Memory at d0000000 (64-bit, prefetchable) [size=32M]
 I/O ports at e000 [size=128]
 Expansion ROM at 000c0000 [disabled] [size=128K]
 Capabilities: [60] Power Management version 3
 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
 Capabilities: [78] Express Legacy Endpoint, MSI 00
 Capabilities: [100] Virtual Channel
 Capabilities: [250] Latency Tolerance Reporting
 Capabilities: [258] L1 PM Substates
 Capabilities: [128] Power Budgeting <?>
 Capabilities: [420] Advanced Error Reporting
 Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
 Capabilities: [900] Secondary PCI Express
 Capabilities: [bb0] Physical Resizable BAR
 Kernel driver in use: vfio-pci
 Kernel modules: nvidiafb, nouveau

01:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1)
 Subsystem: ZOTAC International (MCO) Ltd. TU116 High Definition Audio Controller
 Flags: bus master, fast devsel, latency 0, IRQ 10, IOMMU group 1
 Memory at df080000 (32-bit, non-prefetchable) [size=16K]
 Capabilities: [60] Power Management version 3
 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
 Capabilities: [78] Express Endpoint, MSI 00
 Capabilities: [100] Advanced Error Reporting
 Kernel driver in use: vfio-pci
 Kernel modules: snd_hda_intel

Step 2: Check NVIDIA Devices

Use the following commands to identify NVIDIA devices.

lspci | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] (rev a1)
01:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1)
01:00.2 USB controller: NVIDIA Corporation TU116 USB 3.1 Host Controller (rev a1)
01:00.3 Serial bus controller: NVIDIA Corporation TU116 USB Type-C UCSI Controller (rev a1)

# To get the IDs of Devices
lspci -nn | grep -i nvidia
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] [10de:2187] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1)
01:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1)
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller [10de:1aed] (rev a1)

Step 3: Update GRUB Configuration

Add the necessary parameters to the GRUB configuration.

vim /etc/default/grub
...
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init"
# Alternative configuration
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream initcall_blacklist=sysfb_init nofb video=vesafb:off,simplefb:off,efifb:off rd.driver.blacklist=snd_hda_intel,nvidia,nvidiafb,nouveau module.blacklist=snd_hda_intel,nvidia,nvidiafb,nouveau vfio-pci.ids=10de:2187,10de:1aeb"
...

update-grub

Step 4: Load VFIO Modules

Add vfio modules to be loaded at boot.

# On the Proxmox host, add the following to /etc/modules:
vim /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
vfio_nvidia

Step 5: Configure Module Blacklist

Blacklist the default NVIDIA drivers to ensure vfio-pci is used.

# On the Proxmox host, add the following to /etc/modprobe.d/blacklist.conf:
vim /etc/modprobe.d/blacklist.conf

blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist snd_hda_intel

Step 6: Configure VFIO

Add the GPU and audio device IDs to the vfio configuration.

# Add the following to /etc/modprobe.d/vfio.conf:
# Get the IDs of GPU and Audio from the previous `lspci -nn | grep -i nvidia` command
vim /etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:2187,10de:1aeb disable_vga=1 disable_idle_d3=1 initcall_blacklist=sysfb_init

Step 7: Update Initramfs and Reboot

Update initramfs and reboot the machine to apply changes.

update-initramfs -u -k all
reboot

Step 8: Verify VFIO Binding

After reboot, verify that the vfio-pci driver is being used.

lsmod | grep vfio
vfio_pci               16384  0
vfio_pci_core          86016  1 vfio_pci
irqbypass              12288  2 vfio_pci_core,kvm
vfio_iommu_type1       49152  0
vfio                   69632  3 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd                98304  1 vfio

# Run lspci -v again to confirm both video and audio components are bound to vfio-pci
lspci -v

Step 9: Verify IOMMU Groups

Create a script to verify IOMMU groups.

vim iommu_group.sh 
#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

chmod +x iommu_group.sh 

./iommu_group.sh 

...
        01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] [10de:2187] (rev a1)
        01:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1)
        01:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1)
        01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller [10de:1aed] (rev a1)
...

Step 10: Create a Windows VM

I’m using Windows 11 Pro. Make sure to mount the Virtio driver ISO to add the missing drivers.

Everything else will proceed as usual during the installation. To add GPU passthrough, add a PCI device and select your GPU with the All Functions, ROM-Bar and PCI-Express options checked.