"description":"The instance contract of an OCI container and the operations that use it: update, modify, remove, recovery of an interrupted operation and recovery after a restore on this or another host."
"description":"Each instance keeps a reproducible contract, so its rootfs can be replaced without losing configuration or persistent data.",
"section":"OCI manager Apps"
},
"sections":[
{
"id":"contract",
"title":"The instance contract",
"blocks":[
{
"p":"After an installation, the effective configuration is stored in <code>/usr/local/share/proxmenux/oci/instances/VMID/oci-compose.json</code>. It is not a copy of the original Compose file: it is the reproducible contract of the LXC that exists on this host."
},
{
"code":{
"code":"instances/\n└── 105/\n └── oci-compose.json\n ├── image and resolved digest\n ├── resources and network\n ├── environment (secrets protected)\n ├── container disks and host directories\n ├── hardware profile and devices\n ├── console log and terminal mode\n └── stack membership and lifecycle"
"p":"A copy of the contract travels inside the container, in <code>/.proxmenux/oci-record.json</code>, readable only by root of the container, so a backup of the container always carries the contract it had at that moment. A second copy is kept in <code>/etc/pve/priv/proxmenux/oci</code>, which every node of a cluster shares and only root of the host reads. Both are written after every installation, update and change."
"intro":"This option of the main screen lists the registered instances with their application and image. Each one is checked against its contract before any action; a CT that no longer matches its record is neither modified nor deleted. The same update and the same editor are offered in the Updates tab of <monitorLink>ProxMenux Monitor</monitorLink>.",
"p":"For a multi-container application the menu offers <strong>Update every container of the application</strong>, <strong>Modify extra paths and devices</strong>, <strong>Recreate every container with its saved configuration</strong>, the watchdog and the removal. Modify adds or removes the extra paths and devices of the application container without rebuilding anything and, in Immich, changes what runs recognition. Recreate rebuilds every container from the image it was installed with, without looking for a newer one."
"chartCode":"sequenceDiagram\n participant U as {{user}}\n participant P as ProxMenux\n participant R as {{registry}}\n participant X as Proxmox VE\n U->>P: {{update}}\n P->>R: {{resolve}}\n R-->>P: digest\n P->>P: {{verify}}\n P->>X: {{backup}}\n P->>X: {{import}}\n P->>X: {{reapply}}\n X-->>P: healthcheck\n alt {{healthy}}\n P-->>U: {{commit}}\n else {{failure}}\n P->>X: Rollback\n P-->>U: {{restored}}\n end",
"labels":{
"user":"User",
"registry":"OCI registry",
"update":"Update instance",
"resolve":"Resolve the tag",
"verify":"Verify archive and preflight",
"backup":"Stop and back up the CT",
"import":"Import the new rootfs",
"reapply":"Apply the contract again",
"healthy":"healthy",
"failure":"failure",
"commit":"Contract published with the new digest",
"restored":"Previous instance restored"
}
}
},
{
"list":{
"items":[
"When the registry still serves the installed digest, nothing is downloaded and the instance is not touched.",
"The new image is verified layer by layer before the CT stops. A download that arrives damaged is fetched a second time; an image already cached that fails the check is downloaded again.",
"Settings changed in Proxmox VE after the installation (memory, swap, cores, CPU limit, CPU priority, start with the node) are kept and carried into the new contract.",
"The LAN interface that the Arr suite and multi-container applications add (<code>net1</code>) and the start order (<code>startup</code>) are given back to the new container as they are.",
"intro":"An update and a change use a persistent transaction. If the process, the terminal or the node is interrupted after the CT stops, the operation is not taken as finished.",
"title":"Watchdog: restarting an application that stops",
"intro":"Proxmox has no restart policy: when the process of an application container ends, the container stays stopped. With the watchdog on, an application that stops on its own is started again. It does for an OCI application what <code>restart: unless-stopped</code> does in a Compose file.",
"blocks":[
{
"table":{
"headers":[
"What happens",
"With the watchdog on"
],
"rows":[
[
"The application crashes or its process ends",
"The container is started again within seconds"
],
[
"It is stopped or shut down from Proxmox, from the Monitor or with <code>pct</code>",
"It stays stopped until somebody starts it"
],
[
"A backup, a migration or a ProxMenux operation is working on it",
"Nothing is done until that ends"
],
[
"It was already stopped when the host started",
"It stays stopped; <strong>Start with Proxmox</strong> decides that"
],
[
"It keeps stopping right after starting",
"An application that stops again within seconds of starting is tried five times, waiting 30 seconds, 1, 2 and 5 minutes between them. Then it is left stopped and a notification is sent"
]
]
}
},
{
"p":"The installer asks it in both installation modes and proposes what the recipe of the application declares. It is changed later from <strong>Manage installed OCI applications</strong> or with the <strong>Watchdog</strong> switch of the container in <monitorLink>ProxMenux Monitor</monitorLink>, and it applies to every container of a multi-container application. One service of the host, <code>proxmenux-oci-watchdog.service</code>, watches every application that asked for it; its installation is listed in the Changes tab of Audit & Report."
},
{
"calloutInfo":{
"title":"What the host cannot tell",
"body":"The exit code of the application does not reach the host, so a clean exit and a crash look the same and both are restarted, as <code>always</code> and <code>unless-stopped</code> do in Docker. A stop asked outside Proxmox, with <code>lxc-stop</code> or <code>systemctl</code>, leaves no stop task and is taken as a stop of its own."
"intro":"A backup made with vzdump or Proxmox Backup Server includes the container, its disks and the copy of its contract. What the installation keeps on the host is not part of it. <strong>Manage installed OCI applications</strong> detects the containers restored on a host that has no contract for them, a new Proxmox installation or another host, and offers to register them again.",
"blocks":[
{
"steps":{
"items":[
{
"title":"Restore the containers in Proxmox",
"body":"From the backup storage, with the original ID or with any free one. A multi-container application needs every one of its containers."
},
{
"title":"Open Manage installed OCI applications",
"body":"The restored containers are listed and the recovery is offered. The <strong>Recover</strong> button of the Updates tab of <monitorLink>ProxMenux Monitor</monitorLink> opens the same recovery."
},
{
"title":"Check before changing",
"body":"The recovery checks that the host has everything each application needs. An application that cannot be recovered whole is left as it was found, with the reason."
},
{
"title":"Register and start",
"body":"The contract is registered on this host and what the installation kept on it is written again. Starting the applications is a separate question, answered with No when the original containers are still running on another host."
}
]
}
},
{
"figure":{
"src":"/oci-manager/restore-offer.png",
"alt":"Dialog that lists the restored containers and offers to recover them",
"caption":"The recovery offered when Manage installed OCI applications opens."
}
},
{
"table":{
"headers":[
"What the host kept",
"After the recovery"
],
"rows":[
[
"Instance contract",
"Registered from the copy the container carries, checked against the configuration Proxmox restored"
],
[
"Private network of a multi-container application",
"Created again with the same bridge and subnet; the fixed addresses of the containers do not change"
],
[
"Start order of a multi-container application",
"The dependency hookscript and its contract are installed again; snippets are enabled on the <code>local</code> storage when no storage accepts them"
],
[
"Network sysctls and host-monitor file",
"Written again in <code>/etc/pve/proxmenux</code>"
],
[
"NVIDIA runtime",
"The hook of an unprivileged container is installed again. A privileged container gets the driver files of this host instead of those of the host it comes from"
],
[
"Rclone mount",
"The hookscript and the programs that publish the mount are written again, with the same views on the host"
],
[
"Host firewall rule of a host monitor",
"Asked again, for its web port and the subnet of the bridge on this host"
],
[
"<code>lost+found</code> of each restored disk",
"Removed when empty; a restore creates it and some applications cannot start with it in their data"
],
[
"Disks restored on another storage",
"The contract is updated to the storage they are on now"
]
]
}
},
{
"p":"A container restored with another ID keeps its application. The contract, its console log, its network sysctls, the hookscript of an Rclone mount and the start order of a multi-container application are registered with the IDs the containers have on this host."
},
{
"table":{
"headers":[
"What stops a recovery",
"What to do"
],
"rows":[
[
"A container of a multi-container application is missing",
"Restore it too; the application is recovered whole"
],
[
"The private subnet is already used on this host",
"The recovery is cancelled and nothing is changed: the addresses of the containers are fixed and are not moved to another subnet"
],
[
"A host directory does not exist",
"Mount or create it with its data; a backup of the container does not include host directories"
],
[
"A device does not exist on this host",
"Connect it, or remove it from the container in Proxmox"
],
[
"An application with NVIDIA on a host without the driver",
"Install the NVIDIA driver and the Container Toolkit"
],
[
"The backup was made before the copy of the contract existed",
"The container stays as an ordinary LXC and is not offered again"
]
]
}
},
{
"calloutInfo":{
"title":"A container that comes back",
"body":"A container restored over itself from an older backup, rolled back to a snapshot or returned from another node carries the contract of the state it is in. When it is selected for an operation and its contract differs from the one of this host, the operation does not start and the container is offered for recovery."
}
},
{
"figure":{
"src":"/oci-manager/restore-result.png",
"alt":"Result of the recovery with the private network, the start order and the registered containers",
"caption":"The result of a recovery, step by step."
"p":"Before the confirmation, a summary is built from the real configuration: the containers that are removed, the data deleted with them, the private network that is released and the host directories that are kept. The confirmation defaults to <strong>No</strong>, since the data of the deleted disks cannot be recovered afterwards."
"alt":"Removal summary with the containers, data and host directories affected",
"caption":"The removal summary, built from the real configuration."
}
}
]
},
{
"id":"archives",
"title":"Downloaded images and host space",
"blocks":[
{
"p":"The OCI archive is used to build the rootfs; the running CT does not read it. After an installation or an update, the downloaded archives of that operation are listed with their size and their deletion is offered. Deleting them frees the space without affecting the container or its data."
},
{
"calloutInfo":{
"title":"Updates do not need the archive",
"body":"Without the archive, an update resolves the saved channel and downloads the new digest. With it, an archive is reused only when its reference and integrity match what is requested."
"p":"When <strong>Manage installed OCI applications</strong> opens, what is left of containers that exist on no node of the cluster is removed first: the saved record of a container deleted from the Proxmox interface, which is kept as history, and the host files a removal left behind. A record with an operation left halfway is kept, because its backup may still be needed. Volumes and host directories are never deleted by inference."
}
]
},
{
"id":"cluster",
"title":"In a cluster: migration and high availability",
"intro":"An OCI container is an ordinary Proxmox LXC: it can be migrated or managed by HA like any other, within the same limits. What it needs to start is kept where every node of the cluster finds it.",
"blocks":[
{
"table":{
"headers":[
"Part",
"On another node of the cluster"
],
"rows":[
[
"Console log",
"The container creates <code>/var/log/proxmenux/oci</code> before it starts, on whichever node runs it. Each node keeps the log of the starts it ran."
],
[
"Network sysctls and host monitor",
"Kept in <code>/etc/pve/proxmenux</code>, which every node of the cluster shares, so a migrated container finds them."
],
[
"Container disks",
"Proxmox moves them with the container. HA needs them on shared storage."
],
[
"Host directories and devices",
"The same rules as any LXC: a host directory must exist on the target node and be marked as shared, and a GPU, Coral or NPU must be present there."
],
[
"ProxMenux record",
"The contract stays on the node where the application was installed, and every node reads the copy kept in <code>/etc/pve/priv/proxmenux/oci</code>. A single-container application that migrates is registered on the node it arrives at when <strong>Manage installed OCI applications</strong> opens or an operation is launched for it. A multi-container application is offered for recovery there, since its private network has to be created on that node."
]
]
}
},
{
"calloutWarning":{
"title":"Not for high availability",
"body":"A multi-container application reaches its members through a private bridge of the node it was installed on, so its members stay on that node. A host monitor, such as Glances in host mode or Netdata, monitors the node it runs on; moving it would monitor another node."
}
},
{
"p":"A backup restored outside the cluster gets the files of <code>/etc/pve/proxmenux</code> the container uses when the application is recovered."
"title":"A rolling tag is not an unattended update",
"blocks":[
{
"p":"The catalog installs the rolling tag its maintainer publishes, but the effective digest is resolved, recorded and changed only through an explicit update with preflight and rollback. <monitorLink>ProxMenux Monitor</monitorLink> compares the installed digest with the one the registry publishes and shows, and notifies, when a new image is available."
