"title":"Cross-kernel restore — direction detection, safe-subset filter, hydration | ProxMenux",
"description":"How ProxMenux restores a host backup on a target running a different major kernel version. Documents how the direction of the mismatch is detected, the safe-subset filter that skips boot-critical paths when the target runs a newer kernel than the backup, and the four-phase kernel-agnostic hydration that reapplies user tuning like IOMMU, VFIO IDs and custom cmdline tokens without copying kernel-tied files verbatim.",
"ogTitle":"ProxMenux Backup — cross-kernel restore and hydration",
"ogDescription":"Direction-aware cross-kernel restore with a safe-subset filter and kernel-agnostic hydration.",
"twitterDescription":"How ProxMenux handles a restore when the target kernel differs from the backup's."
},
"header":{
"title":"Cross-kernel restore",
"description":"How ProxMenux handles a restore when the target host runs a different kernel from the one recorded in the backup — especially when the target runs a newer kernel. Documents how the difference is detected, how boot-critical paths that could break the target are filtered, and how the user's own configuration is reapplied without copying kernel-tied files verbatim.",
"section":"Backup & Restore"
},
"intro":{
"title":"Every kernel jump is handled differently",
"body":"A Proxmox host's kernel is identified by a version string like <code>6.17.13-2-pve</code>. The first two numbers (<strong>6.17</strong> in this example) define the major version: when they change, some configuration files written for the previous kernel may not be valid on the new one. When the restore detects that the backup and the target have different major kernel versions, the flow branches based on the direction of the jump — whether the backup is older or newer than the target — because the two cases have opposite failure modes. Backups newer than the target restore cleanly as-is (bench-verified across multiple kernel jumps). Backups older than the target need a filter to avoid breaking the target's boot with configuration written for a kernel that has since changed."
},
"directionCheck":{
"heading":"The direction check",
"intro":"During the compatibility check, <code>hb_compat_check</code> compares the kernel recorded in the backup's manifest against the target's current kernel (<code>uname -r</code>) and classifies the restore into one of three cases, stored in the internal variable <code>HB_COMPAT_KERNEL_DIRECTION</code>:",
"rows":[
{"direction":"same","condition":"The major kernel version matches between backup and target.","behavior":"The full restore flow runs unchanged. No extra filter, no hydration, no special notice in the UI."},
{"direction":"bk_newer","condition":"Backup kernel is NEWER than the target's (for example: the backup was made on kernel 7.0 and the target runs kernel 6.17).","behavior":"The full restore flow runs unchanged, exactly like <code>same</code>. No filter is applied. Verified empirically: drivers auto-reinstall against the target's kernel, IOMMU config applies cleanly, VMs with GPU passthrough come up without issue."},
{"direction":"bk_older","condition":"Backup kernel is OLDER than the target's (for example: the backup was made on kernel 6.17 and the target runs kernel 7.0).","behavior":"The safe-subset filter and the four-phase hydration below are activated. The restore proceeds — the target reproduces the source, but boot-critical files are not copied verbatim."}
]
},
"whyBkNewerIsSafe":{
"heading":"Why a backup with a newer kernel than the target restores unchanged",
"body":"When the restore runs against a target with an <em>older</em> kernel than the one recorded in the backup, every mechanism the restore depends on is independent of the kernel version. GPU driver installers and other PCI-device installers detect the running kernel with <code>uname -r</code> and compile DKMS against whatever the target has. The APT package install pulls binaries built for the target's distribution. IOMMU cmdline tokens like <code>intel_iommu=on</code> are stable across major kernel versions. The backup carries paths written under a newer kernel, but those paths (module blacklists, GRUB defaults, initramfs config) are still valid syntax on the older one — kernels ignore tokens they do not recognise instead of failing. This case is equivalent in practice to a same-kernel restore, and ProxMenux treats it as such."
},
"safeSubsetFilter":{
"heading":"The safe-subset filter (only when the target kernel is newer)",
"intro":"When the target kernel is newer than the backup's, the compatibility check adds 16 boot-critical paths from <code>hb_unsafe_paths_cross_version</code> to <code>RS_SKIP_PATHS</code>. These paths are excluded from the restore because writing a version of them from an older kernel over a target running a newer one has caused kernel panics in bench tests. The paths cover four categories:",
"categoryRows":[
{"category":"Bootloader","paths":"<code>/etc/default/grub</code>, <code>/etc/kernel</code>","reason":"GRUB defaults tied to the prior kernel order, and proxmox-boot-tool state (cmdline, ESP UUIDs, hooks) that references paths and identifiers from the older install."},
{"category":"Kernel modules and boot artifacts","paths":"<code>/etc/modules-load.d</code>, <code>/etc/modprobe.d</code>, <code>/etc/initramfs-tools</code>","reason":"Autoload lists that may reference modules renamed between kernel majors, module options that may not apply, initramfs hooks written for the older kernel."},
{"category":"Storage stack and filesystem identity","paths":"<code>/etc/fstab</code>, <code>/etc/multipath</code>, <code>/etc/iscsi</code>, <code>/etc/udev/rules.d</code>, <code>/etc/zfs</code>","reason":"UUIDs that may not exist on this install, multipath drivers that change between kernels, iSCSI parameters that evolve, udev rules that may bind to nonexistent subsystems, ZFS state (<code>zpool.cache</code> + <code>hostid</code>) that can lock the pool as foreign."},
{"category":"APT sources","paths":"<code>/etc/apt</code>","reason":"APT source suites may trigger a downgrade of critical packages on the next upgrade."},
{"category":"systemd","paths":"<code>/etc/systemd/system</code>, <code>/etc/systemd/journald.conf</code>, <code>/etc/systemd/logind.conf</code>, <code>/etc/systemd/system.conf</code>, <code>/etc/systemd/user.conf</code>","reason":"Unit overrides and .wants tied to the older systemd major; configuration keys that may not parse on a newer systemd."}
],
"outroBody":"The filter runs BEFORE the confirmation dialog so the user sees the exact list of paths that will be skipped, categorised by reason. The restore still proceeds — everything else (VMs, LXCs, network, /etc/pve, users, cron, ProxMenux state, packages, drivers, /root) is restored normally."
},
"hydration":{
"heading":"Kernel-agnostic hydration",
"intro":"The safe-subset filter alone would leave the target without the tuning the user had inside those boot-critical files: IOMMU cmdline for GPU passthrough, VFIO device IDs, custom <code>GRUB_TIMEOUT</code>, nvidia blacklists. The hydration pass re-applies those bits kernel-agnostically. Four phases run when the target kernel is newer than the backup's, each additive (never overwrites a value the target already carries) and idempotent (running twice is a no-op).",
"phaseRows":[
{"phase":"1a — GRUB path","detail":"For hosts using GRUB (ext4/lvm installs). <code>_rs_hyd_grub</code> merges every token from the backup's <code>manifest.kernel_params.cmdline_extra</code> into the target's live <code>GRUB_CMDLINE_LINUX_DEFAULT</code>, skipping tokens whose key the target already carries. Then merges whitelisted <code>GRUB_*</code> keys (<code>GRUB_TIMEOUT</code>, <code>GRUB_TIMEOUT_STYLE</code>, <code>GRUB_DEFAULT</code>, <code>GRUB_TERMINAL</code>, <code>GRUB_DISABLE_OS_PROBER</code>, <code>GRUB_SERIAL_COMMAND</code>, <code>GRUB_GFXMODE</code>, <code>GRUB_GFXPAYLOAD_LINUX</code>) from the backup's <code>/etc/default/grub</code> if they differ from the target's."},
{"phase":"1b — systemd-boot / ZFS path","detail":"For hosts using systemd-boot (typically ZFS-on-root). <code>_rs_hyd_kernel_cmdline</code> merges user tokens from <code>cmdline_extra</code> into the target's <code>/etc/kernel/cmdline</code>, keeping the target's own <code>root=</code>, <code>boot=</code> and <code>rootflags=</code> boilerplate intact."},
{"phase":"2 — /etc/modules merge","detail":"<code>_rs_hyd_modules</code> appends modules from <code>manifest.kernel_params.modules_loaded_at_boot</code> that are in the whitelist (<code>vfio</code>, <code>vfio_pci</code>, <code>vfio_iommu_type1</code>, <code>vfio_virqfd</code>, <code>kvm</code>, <code>kvm_intel</code>, <code>kvm_amd</code>, <code>nvidia</code>, <code>nvidia_drm</code>, <code>nvidia_modeset</code>, <code>nvidia_uvm</code>, <code>i915</code>, <code>xe</code>) AND not already present in the target's <code>/etc/modules</code>."},
{"phase":"3 — Whitelisted files copy","detail":"<code>_rs_hyd_files</code> copies user-authored files from the staging rootfs to the live target when the content differs. Whitelist covers VFIO/nvidia/blacklist files under <code>/etc/modprobe.d</code>, <code>/etc/modules-load.d</code>, and the ProxMenux VFIO bind rule + nvidia udev rules under <code>/etc/udev/rules.d</code>. Distro-owned files (<code>pve-blacklist.conf</code>, <code>mdadm.conf</code>, <code>nvme.conf</code>) are intentionally excluded — their contents evolve between releases."},
{"phase":"4 — Force post-boot reflows","detail":"The four phases write directly to the live target OUTSIDE the normal restore pipeline. To make the merged tokens/modules/files take effect on the next boot, <code>HB_HYDRATION_APPLIED=1</code> propagates through <code>plan.env</code> to <code>apply_pending_restore.sh</code>, which forces <code>NEEDS_INITRAMFS=1</code> and <code>NEEDS_GRUB=1</code> regardless of what was in the apply list. The post-boot dispatcher then regenerates initramfs and refreshes the bootloader."}
"body":"The hydration runs in two modes. Before the confirmation dialog, ProxMenux runs <code>_rs_apply_bk_older_hydration</code> in <code>plan</code> mode: it computes exactly what would be merged, populates <code>RS_HYDRATION_SUMMARY</code> with a green block listing each action, and returns without writing. The confirmation dialog shows that green block alongside the amber safe-subset skip list, so the user sees UPFRONT what will be re-applied automatically. After the user confirms, ProxMenux re-runs the same helper in <code>commit</code> mode — same phases, same logic, but this time each phase writes to the live target. Cancelling the confirmation dialog leaves the target untouched."
},
"flowDiagram":{
"heading":"The flow when the target kernel is newer than the backup's",
"intro":"The steps added when the target kernel is newer sit inside the normal restore flow. Everything else — hot apply, pending prepare, package install, post-boot dispatcher — runs identically to a same-kernel restore.",
