ProxMenux/web/messages/en/docs/hardware/gpu-vm-passthrough.json

{
  "meta": {
    "title": "Add GPU to VM (Passthrough) | ProxMenux Documentation",
    "description": "Pass an Intel, AMD or NVIDIA GPU through to a Proxmox VM with near-native performance. ProxMenux handles host preparation (VFIO modules, driver blacklist, kernel cmdline), VM configuration (hostpci, audio function, IOMMU group siblings), vendor-specific workarounds (NVIDIA Code 43, AMD reset bug, ROM dump) and switch-mode conflicts with LXCs."
  },
  "header": {
    "title": "Add GPU to VM (Passthrough)",
    "description": "Give one of your GPUs to a virtual machine with near-native performance. ProxMenux detects Intel / AMD / NVIDIA, validates IOMMU, analyses the GPU's IOMMU group to pass every sibling device together, configures VFIO on the host, writes the right hostpci lines into the VM config, and applies vendor-specific fixes where needed.",
    "section": "Hardware: GPUs and Coral-TPU"
  },
  "intro": {
    "title": "What this does",
    "body": "Automates the relevant host and <code>hostpci</code> configuration steps described in the <pveLink>Proxmox PCI passthrough documentation</pveLink>. It checks assignments in <em>other</em> guests and offers conflict-handling choices. These checks do not guarantee support for every GPU, reset method or guest driver."
  },
  "who": {
    "heading": "Who is this for?",
    "body": "You have a physical GPU in your Proxmox host and you want a <strong>virtual machine</strong> (Windows gaming, macOS, a headless GPU compute node, a VM-based media server) to use it directly. Passing a GPU to a VM is not the same as passing it to an LXC — VMs need the kernel to treat the GPU as a VFIO device (essentially \"the host won't touch it\"), which means the host cannot use that GPU for anything else while the VM is running. For <em>LXC</em> transcoding / compute, use <lxcLink>Add GPU to LXC</lxcLink> instead — it shares the GPU and does not need VFIO."
  },
  "prereqs": {
    "title": "Before you start",
    "gpu": "<strong>A supported GPU</strong> physically installed. The script detects Intel, AMD and NVIDIA via <code>lspci</code>.",
    "gpuCheck": "lspci | grep -iE 'VGA|3D|Display'",
    "iommu": "<strong>IOMMU support</strong> must be available in hardware and enabled in firmware where required (Intel VT-d or AMD-Vi). The script checks the running system and can configure Linux boot parameters; it cannot change BIOS/UEFI settings or prove that every inactive IOMMU is disabled in firmware.",
    "q35": "This assistant requires the <strong>q35</strong> machine type and refuses <code>i440fx</code>. Machine type is separate from BIOS/UEFI firmware; changing an existing VM chipset requires planning and is not performed automatically.",
    "q35Check": "qm config VMID | grep machine",
    "moreGpus": "Preferably <strong>more than one GPU</strong> in the host, or console access on another output (IPMI, KVM-over-IP, serial). Once you pass the only GPU to a VM, the host console goes dark. With two NVIDIA GPUs you can pass one to a VM and keep the other on the host — the script handles this per-BDF (see <em>NVIDIA</em> in the vendor notes below).",
    "nvidiaInstalled": "If you installed <nvidiaLink>NVIDIA Drivers on the Host</nvidiaLink>, selected functions are reserved for <code>vfio-pci</code> rather than <code>nvidia</code>. The <code>nvidia</code> module can remain available for <strong>other</strong> cards outside the affected IOMMU group; verify them with <code>nvidia-smi</code>."
  },
  "pickOne": {
    "title": "VM passthrough vs LXC sharing — pick one per GPU",
    "body": "A GPU bound to <code>vfio-pci</code> for VM passthrough cannot simultaneously be used by the host or an LXC. If you have two GPUs, you can dedicate one to each path. If you have only one, choose:",
    "vmItem": "<strong>VM route (this page):</strong> full hardware access, but exclusively for the VM that owns the GPU while it's running.",
    "lxcItem": "<strong>LXC route (<lxcLink>Add GPU to LXC</lxcLink>):</strong> shared with the host and other containers, great for transcoding, no VFIO magic needed."
  },
  "running": {
    "heading": "Running the installer",
    "body": "Open ProxMenux on the host, go to <strong>Hardware: GPUs and Coral-TPU → Add GPU to VM</strong>.",
    "imageAlt": "Menu entry for 'Add GPU to VM' inside Hardware: GPUs and Coral-TPU"
  },
  "howRuns": {
    "heading": "How the script runs",
    "body": "Detection and selection precede the main host/VM changes and a final summary confirms that plan. Enabling IOMMU has its own earlier confirmation and can already edit the boot configuration. Later associated-audio cleanup may also ask for a choice."
  },
  "walkthrough": {
    "heading": "Walking through the flow",
    "detect": {
      "title": "Detect GPUs and check IOMMU",
      "body": "Treats IOMMU as active only when the running kernel has <code>intel_iommu=on</code> or <code>amd_iommu=on</code> and IOMMU groups exist. If the parameter is already saved for the next boot, it continues in pending-reboot mode; otherwise a separately confirmed step can add it with <code>iommu=pt</code>. Its boot-file heuristic uses <code>/etc/kernel/cmdline</code> when a ZFS-root marker such as <code>root=ZFS=</code> exists there; otherwise it uses <code>/etc/default/grub</code>. That heuristic is not proof of the actual bootloader. Newly configured IOMMU takes effect after reboot; inspect the real groups afterwards.",
      "tipTitle": "Already ran post-install?",
      "tipBody": "If <postLink>VFIO IOMMU support</postLink> was previously configured and is active in the running kernel, this check passes. A saved next-boot setting alone does not prove it is active.",
      "imageAlt": "List of detected GPUs with vendor and PCI address"
    },
    "preflight": {
      "title": "Pick a GPU and run pre-flight checks",
      "intro": "Once you pick the GPU, the script runs a series of checks that can each block further progress:",
      "items": [
        "<strong>SR-IOV guard.</strong> This assistant refuses VFs and PFs with active VFs because its full-device workflow is not an SR-IOV configuration flow. SR-IOV passthrough must be configured separately on supported hardware.",
        "<strong>Single-GPU warning.</strong> If this is the only GPU in the host, you get a scary dialog reminding you that after reboot the console goes dark — make sure you have SSH or web-UI access from another machine.",
        "<strong>AMD reset policy.</strong> Reads <code>/sys/bus/pci/devices/&lt;pci&gt;/reset_method</code>. An APU or dedicated card without FLR is <em>blocked by this assistant’s policy</em>; an unknown subtype without FLR requires an explicit override. This does not establish universal hardware impossibility.",
        "<strong>Not in D3cold.</strong> Some AMD cards report <code>D3cold</code> power state while idle, which makes them invisible during VM startup. Blocked until you wake the GPU.",
        "<strong>IOMMU group analysis.</strong> Reads <code>/sys/kernel/iommu_groups/</code> and includes <em>all non-bridge members</em> in the VM plan, possibly network or storage controllers. When the group contains devices from another slot, the script lists them before the summary; check that the host does not need any of them. If IOMMU is pending reboot, real groups are unavailable and the plan initially contains only the selected GPU function; verify isolation after reboot."
      ],
      "audioIntro": "<strong>Audio companion.</strong> Two paths, depending on where the audio lives:",
      "audioDgpu": "<strong>Discrete GPU:</strong> HDMI audio often occupies function <code>.1</code> of the same PCI slot and is automatically included when detected. If the host was using that audio device, it loses access when the device is reserved for the VM.",
      "audioIgpu": "<strong>Intel iGPU (or any GPU without a <code>.1</code> sibling):</strong> the HDMI / analog audio lives on the chipset at a different slot (<code>00:1f.3</code> typically). The script scans the host, lists every PCI audio controller with its current driver (<code>snd_hda_intel</code>, etc.), and asks you which one(s) to pass through. Default is <strong>none</strong> — you explicitly opt in."
    },
    "pickVm": {
      "title": "Pick the target VM",
      "body": "Choose a VM from the list. The assistant requires q35 and reuses existing GPU entries where possible. BIOS/UEFI firmware is a separate setting, not a machine type; this check does not automatically change the installed guest’s chipset or boot mode.",
      "imageAlt": "VM list with name, ID and status shown as a picker"
    },
    "switchMode": {
      "title": "Switch mode — handling the GPU already being elsewhere",
      "intro": "The script scans every VM config and every LXC config on the host looking for the GPU you picked. Three possible outcomes:",
      "items": [
        "<strong>GPU is free.</strong> Nothing to do, continue.",
        "<strong>GPU is in a different VM.</strong> If the source VM is currently running, the script aborts — two VMs can't share an exclusive VFIO assignment, and the source VM has to be stopped first. If the source VM is stopped, you get two options: <em>Keep GPU in the source VM's config but disable Start on boot</em>, or <em>Remove the GPU lines from the source VM's config and keep Start on boot</em>. A fast-path also exists: if the GPU is already bound to <code>vfio-pci</code> and the source VM is a plain VM→VM swap, no host reconfiguration is done and no reboot is needed.",
        "<strong>GPU used by LXC.</strong> For ordinary containers, choose <em>keep the GPU configuration and disable Start on boot</em> or <em>remove GPU device configuration and keep Start on boot</em>. Removal deletes the <code>devN</code> entries and, for AMD/Intel, also legacy <code>/dev/dri</code> <code>lxc.mount.entry</code> and <code>lxc.cgroup2.devices.allow</code> rules. OCI containers retain their configuration and are stopped with Start on boot disabled; remove their GPU through OCI manager Apps. Conflict detection is by GPU device family and may include other cards of that family."
      ],
      "imageAlt": "Dialog offering to remove the GPU from an LXC before assigning it to the VM",
      "smartTitle": "Audio siblings are cleaned up smartly too",
      "smartBody": "After removing GPU assignments from the source VM, the script can offer a checklist of <strong>potentially orphaned audio devices</strong>. Review it and uncheck anything the source still needs. For example, dedicated-card audio at <code>02:00.1</code> may remain while chipset audio at <code>00:1f.3</code> is offered for removal. The checklist, not a universal display/audio pairing guarantee, determines the extra removals."
    },
    "audioPick": {
      "title": "(If no .1 sibling) Pick which audio controllers to include",
      "body": "Only happens for Intel iGPU and similar split-audio setups. A checklist shows every PCI audio controller on the host (excluding any already in the GPU's IOMMU group), labelled with its current driver. Select the ones you want — or none, if the VM doesn't need audio from the host hardware.",
      "imageAlt": "Checklist dialog with every host PCI audio controller (BDF + driver) when the GPU has no .1 sibling audio",
      "warnTitle": "Don't tick audio the host relies on",
      "warnBody": "Assigning an audio controller to the VM can remove host access; it cannot simultaneously belong to another running VM. Leave optional audio unselected when unnecessary, and verify ownership before moving it. PCI audio is not necessarily the source of the host console beep."
    },
    "summary": {
      "title": "Review the confirmation summary",
      "body": "Review the final host and VM plan and cancel if it is wrong. It shows the IOMMU group and its device count; members from another slot are listed in the previous step. Cancelling stops the main changes; it does not undo an earlier separately confirmed IOMMU boot-configuration change.",
      "imageAlt": "Summary dialog listing host changes (VFIO/blacklist files) and VM config changes (hostpci lines) before applying"
    },
    "hostApply": {
      "title": "Host changes are applied",
      "intro": "The apply phase performs the host and VM changes. Associated-audio cleanup may still prompt for a selection. Host changes can include:",
      "items": [
        "<code>/etc/modules</code> — adds <code>vfio</code>, <code>vfio_iommu_type1</code>, <code>vfio_pci</code> (plus <code>vfio_virqfd</code> on kernels &lt; 6.2).",
        "<code>/etc/modprobe.d/vfio.conf</code> — for AMD / Intel, sets <code>options vfio-pci ids=&lt;vendor:device,...&gt; disable_vga=1</code> so VFIO claims the GPU early at boot. For NVIDIA the file only adds <code>softdep nvidia pre: vfio-pci</code> (plus <code>_drm</code>/<code>_modeset</code>/<code>_uvm</code>) — actual binding is per-BDF via the udev rule below. On AMD, also adds <code>softdep</code> lines forcing <code>vfio-pci</code> to load before <code>radeon</code> / <code>amdgpu</code>.",
        "Writes <code>/etc/modprobe.d/iommu_unsafe_interrupts.conf</code> and <code>kvm.conf</code> with <code>allow_unsafe_interrupts=1</code> and <code>ignore_msrs=1</code>. These are compatibility settings applied by the script, not universal Windows/macOS requirements; unsafe interrupts relax interrupt-remapping protection.",
        "<code>/etc/modprobe.d/blacklist.conf</code>: NVIDIA — <code>nouveau</code>/<code>lbm-nouveau</code>; AMD — <code>radeon</code> + <code>amdgpu</code>; Intel — <code>i915</code>. These vendor-wide entries may affect other cards using the same driver. The proprietary <code>nvidia</code> module is blocked <strong>only</strong> through <code>/etc/modprobe.d/proxmenux-nvidia-vfio-blacklist.conf</code> when <em>all</em> host NVIDIA cards are tracked for VFIO.",
        "<code>/etc/udev/rules.d/10-proxmenux-vfio-bind.rules</code> + <code>/etc/proxmenux/vfio-bind.bdfs</code> — <strong>NVIDIA only.</strong> The udev rule sets <code>ATTR'{'driver_override'}'=\"vfio-pci\"</code> at PCI ADD for tracked BDFs, including affected IOMMU-group functions. Cards outside those groups can retain the <code>nvidia</code> driver. Per-BDF policy does not split a hardware IOMMU group.",
        "<strong>AMD only.</strong> Attempts a ROM dump from <code>/sys/bus/pci/.../rom</code> or ACPI VFCT into <code>/usr/share/kvm/vbios_&lt;vendor&gt;_&lt;device&gt;.bin</code>. If successful, the VM uses <code>romfile=</code>. A valid dump does not guarantee successful GPU initialization.",
        "<strong>NVIDIA only.</strong> Host NVIDIA services (<code>nvidia-persistenced</code>, <code>nvidia-powerd</code>, <code>nvidia-fabricmanager</code>) are stopped and disabled only when every NVIDIA GPU is assigned to VFIO. With a mixed host they remain active, together with the <code>nvidia</code> module, for the GPU(s) that stay native.",
        "<code>update-initramfs -u -k all</code> — only runs if any of the above actually changed."
      ]
    },
    "vmApply": {
      "title": "VM config is applied via qm set",
      "body": "The VM config at <code>/etc/pve/qemu-server/&lt;vmid&gt;.conf</code> is updated via <code>qm set</code> (never by direct <code>sed</code>):",
      "after1": "The script adds <code>x-vga=1</code> for AMD and NVIDIA, <strong>not</strong> Intel. This describes its configuration policy; it does not establish that every Intel iGPU lacks firmware or that the flag universally causes hangs.",
      "after2": "Additional <code>hostpciN</code> lines are appended if the GPU's IOMMU group contains other devices you need to pass together."
    },
    "reboot": {
      "title": "Reboot if host config was touched",
      "body": "Follow the reboot prompt when host boot or driver settings change, and verify the GPU’s active binding before starting the VM. Saved configuration does not prove it is active. A VM-to-VM reassignment with the correct VFIO state can avoid a host reboot.",
      "imageAlt": "Summary screen showing what was changed, followed by a reboot prompt"
    }
  },
  "vendors": {
    "heading": "Vendor-specific notes",
    "nvidiaHeading": "NVIDIA",
    "nvidiaBody": "The script retains a legacy <em>Code 43</em> workaround: <code>hidden=1</code>, vendor ID <code>NV43FIX</code> in <code>args</code>, and <code>kvm=off</code>. It is not a universal requirement for current GeForce drivers: NVIDIA enabled Windows-VM passthrough support in R465. Code 43 can have other causes; verify the actual GPU and driver combination.",
    "nvidiaMultiHeading": "Multi-GPU NVIDIA support",
    "nvidiaMultiBody": "Mixed NVIDIA use is supported <strong>per BDF</strong>. Functions assigned to <code>vfio-pci</code> are tracked by PCI address, not merely a shared <code>vendor:device</code> ID. Cards outside the affected IOMMU groups can keep <code>nvidia</code> and be checked with <code>nvidia-smi</code>. Two cards in the same group are not independently isolated by this policy.",
    "amdHeading": "AMD",
    "amdBody": "Some AMD GPUs fail to reset after a VM stops. The assistant checks reported capabilities, not every real restart cycle. <vendorResetLink>vendor-reset</vendorResetLink> supports a limited device list and requires compatible kernel support and early loading; the script does not install it. The Windows <em>RadeonResetBugFix</em> project is another hardware-dependent workaround, not a guaranteed fix for all AMD GPUs.",
    "intelHeading": "Intel iGPU",
    "intelBody": "Full Intel iGPU passthrough and SR-IOV are separate workflows. This assistant reserves the full device for one VM and omits <code>x-vga=1</code>. <sriovLink>i915-sriov-dkms</sriovLink> applies only to supported SR-IOV hardware/kernel combinations; a “UHD 630+” label is not a compatibility rule. Plan alternate host-console access."
  },
  "verification": {
    "heading": "Verification"
  },
  "troubleshoot": {
    "heading": "Troubleshooting",
    "code43Title": "Code 43 in Windows (NVIDIA)",
    "code43Body": "Code 43 is not proof of failed KVM hiding. Inspect <code>qm config VMID | grep -E \"cpu|args\"</code> (replace VMID) for the script’s <code>hidden=1</code> and <code>hv_vendor_id=NV43FIX</code> settings, but also check the guest driver, GPU firmware and assignment. Re-running may exit early if the GPU is already assigned.",
    "amdResetTitle": "AMD GPU works once, fails on VM restart",
    "amdResetBody": "A reset failure is one possible cause. A host reboot or power cycle may restore the device temporarily. Check whether <code>vendor-reset</code> supports this exact GPU and kernel and load it early as its documentation requires; <code>softdep amdgpu pre: vendor-reset</code> alone does not guarantee early loading before VFIO. The Windows <em>RadeonResetBugFix</em> workaround also requires a compatible GPU.",
    "stuckBootTitle": "VM stuck booting / GPU not detected",
    "stuckBootBody": "Inspect the selected PCI address with <code>lspci -nnk -s PCI_ADDRESS</code>; expect <code>Kernel driver in use: vfio-pci</code>. Replace PCI_ADDRESS with the real BDF. If the native driver remains, inspect the appropriate binding policy, not only <code>/etc/modprobe.d/blacklist.conf</code>; NVIDIA also uses per-BDF state. Read <code>dmesg | grep vfio</code>. After correcting persistent configuration, rebuild with <code>update-initramfs -u -k all</code> and follow the reboot plan.",
    "darkTitle": "Host console goes dark after reboot and I can't SSH in",
    "darkBody": "A dark display does not prove the host is down. If SSH also fails, check network connectivity and whether a network/storage controller was included in the IOMMU group. Recover through IPMI, serial or a recovery environment; inspect the VM config <code>/etc/pve/qemu-server/&lt;vmid&gt;.conf</code> and only revert the relevant assignments and host binding rules. Do not delete unrelated blacklist or VFIO configuration.",
    "logTitle": "Check the install log",
    "logBody": "Every run writes to <code>/tmp/add_gpu_vm.log</code>. Attach it when asking for help on GitHub."
  },
  "revert": {
    "heading": "Reverting manually",
    "intro": "Stop the VM and remove only its relevant PCI assignments in Proxmox Hardware, or with the explicit index shown below. Detaching from the VM does not return the device to a native host driver. Use Switch GPU Mode for that separate transition and inspect the affected guests before confirming."
  },
  "related": {
    "heading": "Related",
    "items": [
      {
        "label": "Install NVIDIA Drivers (Host)",
        "href": "/docs/hardware/nvidia-host",
        "tail": " — prepare host-driver access for cards kept native; a card bound to VFIO cannot be used by the host driver simultaneously."
      },
      {
        "label": "Add GPU to LXC",
        "href": "/docs/hardware/igpu-acceleration-lxc",
        "tail": " — alternative model: share the GPU with multiple containers instead of dedicating it to a VM."
      },
      {
        "label": "Switch GPU Mode (VM ↔ LXC)",
        "href": "/docs/hardware/switch-gpu-mode",
        "tail": " — flip the same GPU between modes without re-doing all the wiring."
      },
      {
        "label": "GPU Passthrough commands",
        "href": "/docs/help-info/gpu-commands",
        "tail": " — lspci, IOMMU verification, qm set hostpci reference."
      }
    ]
  }
}