install from a hand-written oci-compose; Log2RAM journal fix

OCI manager Apps
- "Install an image that is not in the catalog" reads an oci-compose from a file of the host or from an address.
- A file written by hand states the image, its ports, variables and volumes, and what the installation asks; the rest is completed when it is read. A complete file, or the record of an installation, is installed as it is, and of a record only the template is used.
- Errors name the field they are in, also on a host without python3-jsonschema.
- oci/README.md describes the engine and the oci-compose block by block, with two examples in oci/examples/.
- The documentation page of the option covers it, in English and Spanish.

Log2RAM (reported by @f3rs3n)
- The sync removes the old active journal file from the disk copy, so a boot no longer restores it and journald deletes its oldest archives first. Log2RAM optimization 1.7; it takes effect from the next reboot.

Translations
- The bot keeps "oci-compose" as written.
- Spanish texts for the new option and for the Log2RAM notice.
This commit is contained in:
MacRimi
2026-10-12 22:29:08 +02:00
parent 9089f79342
commit cc12c625a6
15 changed files with 1560 additions and 45 deletions
@@ -102,6 +102,9 @@ PROTECTED_TECHNICAL_TERMS = (
# Open Container Initiative. Read as a word it becomes "BEC" in French,
# which named nothing, across thirty-one strings of the OCI section.
"OCI manager Apps",
# The file an application is installed from. Its name is the one of the
# file on disk; translated, it names something the menu does not offer.
"oci-compose",
"OCI",
"beta",
# Debian release names. "Bookworm" is a bird and a reader of books, and
@@ -0,0 +1,93 @@
"""Log2RAM syncs only the archived journal files, so the active file it found
on disk when it was installed stays there and is restored at every boot.
journald then continues an old sequence and deletes its newest archives first.
The post-install makes the sync remove that file from the disk copy."""
from pathlib import Path
import shutil
import stat
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[3]
SHARED = ROOT / "scripts/global/utils-install-functions.sh"
FLOWS = {"auto_post_install.sh": "install_log2ram_auto", "customizable_post_install.sh": "configure_log2ram"}
# The sync of Log2RAM in its journald-aware mode, as its project ships it.
JOURNALD_AWARE = '''#!/usr/bin/env bash
sync_to_disk() {
is_safe
local optional_params=()
if journald_logrotate; then
optional_params+=("--include=journal/*/*@*.journal")
optional_params+=("--exclude=journal/*/*")
fi
rsync -aXv --no-acls --delete-after "${optional_params[@]}" "${RAM_LOG}"/ "${HDD_LOG}"/
}
'''
NOT_AWARE = '''#!/usr/bin/env bash
sync_to_disk() {
rsync -aXv --delete-after "${RAM_LOG}"/ "${HDD_LOG}"/
}
'''
def gnu_sed():
return shutil.which("bash") and subprocess.run(["sed", "--version"], capture_output=True).returncode == 0
@unittest.skipUnless(gnu_sed(), "bash and GNU sed are needed")
class Log2ramJournalSyncTests(unittest.TestCase):
def correct(self, folder, content, times=1):
script = Path(folder) / "log2ram"
script.write_text(content)
script.chmod(0o755)
calls = "; ".join(['pmx_log2ram_drop_stale_journal "$1"; echo "rc=$?"'] * times)
# As the change journal does, the new content is read whole before it is written.
write = 'pmx_write_file() { local content; content="$(cat)"; printf "%s\\n" "$content" > "$1"; }'
run = subprocess.run(["bash", "-c", f'source "{SHARED}"; {write}; {calls}',
"bash", str(script)], capture_output=True, text=True, check=False)
return script, run.stdout.split()
def test_the_sync_removes_what_it_does_not_copy(self):
with tempfile.TemporaryDirectory() as folder:
script, results = self.correct(folder, JOURNALD_AWARE)
lines = script.read_text().splitlines()
position = lines.index(' optional_params+=("--exclude=journal/*/*")')
self.assertEqual(lines[position + 1], ' optional_params+=("--delete-excluded")')
self.assertEqual(lines[position + 2], " fi")
self.assertEqual(results, ["rc=0"])
self.assertEqual(subprocess.run(["bash", "-n", str(script)]).returncode, 0)
self.assertTrue(script.stat().st_mode & stat.S_IXUSR)
self.assertEqual(Path(str(script) + ".proxmenux.bak").read_text(), JOURNALD_AWARE)
def test_it_is_applied_once(self):
with tempfile.TemporaryDirectory() as folder:
script, results = self.correct(folder, JOURNALD_AWARE, times=2)
self.assertEqual(results, ["rc=0", "rc=1"])
self.assertEqual(script.read_text().count("--delete-excluded"), 1)
def test_a_sync_that_copies_every_file_is_left_as_it_is(self):
with tempfile.TemporaryDirectory() as folder:
script, results = self.correct(folder, NOT_AWARE)
self.assertEqual(results, ["rc=1"])
self.assertEqual(script.read_text(), NOT_AWARE)
self.assertFalse(Path(str(script) + ".proxmenux.bak").exists())
def test_both_flows_apply_it_to_new_and_existing_installations(self):
for name, function in FLOWS.items():
with self.subTest(flow=name):
text = (ROOT / "scripts/post_install" / name).read_text()
self.assertEqual(text.count("pmx_log2ram_drop_stale_journal"), 2)
# An existing installation is told that it needs a reboot.
self.assertIn('if pmx_log2ram_drop_stale_journal "$log2ram_bin"; then\n NECESSARY_REBOOT=1', text)
# The version is what offers the update to existing installations.
self.assertIn(f'{function}() {{\n local FUNC_VERSION="1.7"', text)
if __name__ == "__main__":
unittest.main()
+14
View File
@@ -292,6 +292,7 @@
"Address of the PostgreSQL server": "Dirección del servidor PostgreSQL",
"Address of the Redis or Valkey server": "Dirección del servidor Redis o Valkey",
"Address of the database server (mysql and pgsql only)": "Dirección del servidor de base de datos (solo mysql y pgsql)",
"Address of the oci-compose file": "Dirección del archivo oci-compose",
"Address to reach Pydio Cells (https://domain or https://IP:8080)": "Dirección para acceder a Pydio Cells (https://dominio o https://IP:8080)",
"Address used to reach wallabag (http://IP or https://wallabag.example.com)": "Dirección usada para acceder a wallabag (http://IP o https://wallabag.example.com)",
"Addresses to update: ipv4, ipv6 or both (uses an external service)": "Direcciones que se actualizan: ipv4, ipv6 o both (usa un servicio externo)",
@@ -1177,6 +1178,7 @@
"Containers created from an OCI image are not listed.": "Los contenedores creados desde una imagen OCI no aparecen en la lista.",
"Containers created from an OCI image are not listed: their devices are changed in OCI manager Apps.": "Los contenedores creados desde una imagen OCI no aparecen: sus dispositivos se cambian en OCI manager Apps.",
"Containers created from an OCI image are not listed: their paths are changed in OCI manager Apps.": "Los contenedores creados desde una imagen OCI no aparecen: sus rutas se cambian en OCI manager Apps.",
"Containers of the application:": "Contenedores de la aplicación:",
"Containers returned to their previous state": "Los contenedores regresaron a su estado anterior",
"Containers targeted for removal:": "Contenedores que se eliminarán:",
"Containers that are removed:": "Contenedores que se eliminan:",
@@ -1836,6 +1838,7 @@
"Doplarr_rs is a Discord bot for requesting media through *arr backends, written in Rust.": "Doplarr_rs es un bot de Discord, escrito en Rust, para solicitar contenido a través de backends *arr.",
"Double Commander is a free cross platform open source file manager with two panels side by side. It is inspired by Total Commander and features some new ideas.": "Double Commander es un gestor de archivos gratuito, multiplataforma y de código abierto con dos paneles en paralelo. Se inspira en Total Commander y aporta algunas ideas nuevas.",
"Download Spotify music with album art and metadata": "Descarga música de Spotify con carátulas y metadatos",
"Download an oci-compose from an address": "Descargar un oci-compose desde una dirección",
"Download failed for all attempted URLs": "La descarga falló en todos los intentos de URL",
"Download its Compose file from an address": "Descargar su archivo Compose desde una dirección",
"Download keyfile": "Descargar archivo de claves",
@@ -2382,6 +2385,7 @@
"FileDrop is a free, open source file sharing service": "FileDrop es un servicio gratuito y de código abierto para compartir archivos",
"Files & Downloads": "Files & Downloads",
"Files & Sharing": "Files & Sharing",
"Files it writes in the container:": "Archivos que escribe en el contenedor:",
"Files volume size in GB": "Tamaño del volumen de archivos en GB",
"Filesystem": "Sistema de archivos",
"Filesystem Tools Required": "Herramientas del sistema de archivos necesarias",
@@ -3504,6 +3508,7 @@
"Log2RAM installation cancelled by user": "Instalación de Log2RAM cancelada por el usuario",
"Log2RAM installation verification failed. Check /tmp/log2ram_install.log": "Error en la verificación de la instalación de Log2RAM. Compruebe /tmp/log2ram_install.log",
"Log2RAM installed successfully": "Log2RAM se instaló correctamente",
"Log2RAM keeps the journal in order from the next reboot": "Log2RAM mantiene el diario del sistema en orden a partir del próximo reinicio",
"Log2RAM size set to:": "Tamaño de Log2RAM establecido en:",
"Log2RAM write scheduled every": "Escritura de Log2RAM programada cada",
"Log:": "Log:",
@@ -4550,6 +4555,7 @@
"Path must be absolute (start with /).": "La ruta debe ser absoluta (comenzar con /).",
"Path not found": "Ruta no encontrada",
"Path of the Compose file": "Ruta del archivo Compose",
"Path of the oci-compose file": "Ruta del archivo oci-compose",
"Path removed:": "Ruta eliminada:",
"Path:": "Ruta:",
"Paths applied:": "Rutas aplicadas:",
@@ -4866,6 +4872,7 @@
"Re-enter the BORG REPOKEY passphrase to confirm:": "Vuelva a ingresar la frase de contraseña de BORG REPOKEY para confirmar:",
"Re-running pre-check after repairs...": "Volviendo a ejecutar la verificación previa después de las reparaciones...",
"Reachable": "Accesible",
"Read an oci-compose from a file of this host": "Leer un oci-compose desde un archivo de este host",
"Read its Compose file from a file of this host": "Leer su archivo Compose desde un archivo de este host",
"Read the link with pct console CTID on the Proxmox host, or from the Console panel of the container in the Proxmox web interface, then open the https://playit.gg/claim/ address it shows in a browser and sign in to playit.gg. Ctrl+a q leaves pct console.": "Lee el enlace con pct console CTID en el host Proxmox, o desde el panel Console del contenedor en la interfaz web de Proxmox; después abre en un navegador la dirección https://playit.gg/claim/ que muestra e inicia sesión en playit.gg. Ctrl+a q sale de pct console.",
"Read-Only": "Solo lectura",
@@ -6294,7 +6301,9 @@
"The dynamic profile does not support static driver mounts": "El perfil dinámico no admite montajes de controlador estático",
"The file does not exist, is empty or is not readable.": "El archivo no existe, está vacío o no es legible.",
"The file does not exist:": "El archivo no existe:",
"The file is not a ProxMenux oci-compose.": "El archivo no es un oci-compose de ProxMenux.",
"The file is too large to be a Compose file": "El archivo es demasiado grande para ser un archivo Compose",
"The file is too large to be an oci-compose": "El archivo es demasiado grande para ser un oci-compose",
"The filesystem": "El sistema de archivos",
"The final cleanup did not complete:": "La limpieza final no terminó:",
"The following DKMS-managed drivers will now be rebuilt against it so they keep working after reboot:": "Los siguientes controladores administrados por DKMS ahora se reconstruirán para que sigan funcionando después del reinicio:",
@@ -6413,6 +6422,9 @@
"The new prompt will be used in new terminal sessions.": "El nuevo prompt se usará en las nuevas sesiones de terminal.",
"The next visit to the dashboard will show the initial setup wizard.": "La próxima visita al panel mostrará el asistente de configuración inicial.",
"The observed inventory differs from the validated runtime": "El inventario observado difiere del tiempo de funcionamiento validado",
"The oci-compose has errors:": "El oci-compose tiene errores:",
"The oci-compose is incomplete: it does not say which image to install.": "El oci-compose está incompleto: no dice qué imagen instalar.",
"The oci-compose is not valid JSON:": "El oci-compose no es JSON válido:",
"The official Tandoor startup executable is missing": "Falta el ejecutable oficial de arranque de Tandoor",
"The official inventory contains no NVIDIA devices": "El inventario oficial no contiene dispositivos NVIDIA",
"The official inventory contains no NVIDIA driver components": "El inventario oficial no contiene componentes de controlador NVIDIA",
@@ -6664,6 +6676,7 @@
"This directory is already shared:": "Este directorio ya está compartido:",
"This erases existing metadata.": "Esto borra los metadatos existentes.",
"This explicitly marks the container as privileged": "Esto marca explícitamente el contenedor como privilegiado",
"This file decides what is mounted and with which privileges. Install it only if it comes from a source you trust.": "Este archivo decide qué se monta y con qué privilegios. Instálalo solo si viene de una fuente de confianza.",
"This guarantees that device nodes are available before applying LXC GPU config.": "Esto garantiza que los nodos del dispositivo estén disponibles antes de aplicar la configuración de GPU LXC.",
"This host has no active subscription, switch to the no-subscription repository? The inaccessible Enterprise PVE source will be disabled. Enterprise Ceph sources, if present, will be disabled without choosing a replacement Ceph channel; configure Ceph separately if needed.": "Este host no tiene una suscripción activa, ¿cambiar al repositorio sin suscripción? La fuente Enterprise de PVE, que no es accesible, se desactivará. Las fuentes Enterprise de Ceph, si existen, se desactivarán sin elegir un canal de Ceph que las sustituya; configura Ceph por separado si lo necesitas.",
"This image cannot be installed as it is described:": "Esta imagen no se puede instalar como se describe:",
@@ -6682,6 +6695,7 @@
"This is not a valid image reference:": "Esta no es una referencia de imagen válida:",
"This is unexpected since credentials were validated.": "Esto es inesperado ya que se validaron las credenciales.",
"This is what ProxMenux understood from the": "Esto es lo que ProxMenux entendió del",
"This is what the oci-compose installs:": "Esto es lo que instala el oci-compose:",
"This job will no longer start these host backups:": "Este trabajo dejará de lanzar estos backups del host:",
"This marks the container as unprivileged": "Esto marca el contenedor como sin privilegios",
"This may be normal for a fresh installation": "Esto puede ser normal para una instalación nueva",
+486 -10
View File
@@ -1,12 +1,487 @@
# ProxMenux OCI laboratory
# OCI manager Apps
Prototype that converts official image documentation and discovered Docker
Compose definitions into one ProxMenux JSON template per distribution or
deployment profile, then optionally installs reviewed images as native Proxmox
VE OCI LXC containers.
OCI manager Apps runs official OCI images, the ones published for Docker, as native Proxmox VE LXC containers. No Docker engine is installed inside them: Proxmox VE imports the filesystem and the process metadata of the image, and the application becomes the main process of an LXC that is managed like any other container of the node.
This repository is a laboratory. Generated templates are not considered
compatible merely because conversion succeeded.
The image stays the one its maintainer publishes. What ProxMenux adds is the environment Docker Compose would have created around it: persistent data, network, variables, devices and start order, written with Proxmox VE primitives.
This document describes the engine in this directory: how it is organised, how a container is created and updated, how a Compose file or a `docker run` command becomes an `oci-compose`, and how to write an `oci-compose` by hand and install it. The user documentation, with screenshots, is the [OCI manager Apps section](https://proxmenux.com/en/docs/oci-manager) of the ProxMenux documentation.
- [The model](#the-model)
- [Structure](#structure)
- [How a container is created](#how-a-container-is-created)
- [Update, modify, recreate and remove](#update-modify-recreate-and-remove)
- [From a Compose file or a docker run command to an oci-compose](#from-a-compose-file-or-a-docker-run-command-to-an-oci-compose)
- [The oci-compose, block by block](#the-oci-compose-block-by-block)
- [Writing and testing an oci-compose](#writing-and-testing-an-oci-compose)
- [Adding an application to the catalog](#adding-an-application-to-the-catalog)
- [Design and laboratory notes](#design)
## The model
One OCI image becomes one native LXC.
| Requirement | Docker expresses it as | OCI manager Apps turns it into |
|---|---|---|
| Run the application | `image`, `entrypoint`, `command` | The rootfs built from the image and the main process of the LXC |
| Keep data | `volumes` | A container disk (`mpN`) included in the backup, or a host directory |
| Publish the service | `ports` | An address of its own for the LXC; the service answers on its original port |
| Use hardware | `devices`, `group_add`, `runtime: nvidia` | `devN` entries with the group of the host device, and a reviewed profile |
| Connect dependencies | `networks`, `depends_on` | One LXC per service, a private bridge, fixed addresses, start order and healthchecks |
| Update | `pull` and recreate | A new rootfs with the same data, network and devices |
A Compose key with no safe equivalent is not dropped: it is listed as a blocker and the application is not offered for installation until it is resolved.
## Structure
```
oci/
├── catalog/
│ ├── index.json list of applications shown by the menu
│ ├── apps/<app>.json the oci-compose generated for each image
│ ├── curated/<app>.json an oci-compose written and reviewed by hand
│ ├── overlays/<app>.json adjustments laid over a generated oci-compose
│ ├── categories.json category of each application
│ ├── volume-policy.json which paths hold content of the user
│ └── verification.json applications tested for real
├── examples/ oci-compose files written by hand
├── schemas/ the JSON schema every oci-compose follows
├── src/proxmenux_oci/ menu, converter and deployment plan
├── remote/ what creates and maintains the LXC on the node
└── tests/
```
An **oci-compose** is the JSON file an application is installed from. Its `kind` is `proxmenux.oci-template`. It says which image runs, which ports, variables and volumes it has, what the installation asks, and how each thing is done in Proxmox VE.
There are three places an oci-compose comes from:
- **The catalog.** The converter writes `catalog/apps/<app>.json` from the Compose file the project of the image publishes. When that file needs something Compose cannot say, such as a question, a generated password or a GPU profile, it goes in `catalog/overlays/<app>.json` and is laid over the generated file each time it is read. An application whose definition is written whole by hand lives in `catalog/curated/`.
- **A definition given at the moment.** The option *Install an image that is not in the catalog* translates a Compose file, a `docker run` command or an image reference with the same converter.
- **A file written by hand**, read by the same option. It is described in [Writing and testing an oci-compose](#writing-and-testing-an-oci-compose).
On a node, the engine is installed in `/usr/local/share/proxmenux/oci/engine/`. Every installation keeps its record in `/usr/local/share/proxmenux/oci/instances/<VMID>/oci-compose.json`: the oci-compose it was installed from, under `template`, together with the answers given during the installation and the configuration Proxmox VE ended with. Updates, changes and removals read that record.
## How a container is created
Selecting an application, or giving a definition, opens the same sequence.
1. **Read.** The oci-compose is composed and checked against the schema. One with blockers is not offered.
2. **Ask.** *Install with default settings* asks what every installation decides: where the data of the user goes, the address, required values with no default, and whether it starts with the node. *Install with advanced settings* opens every decision: VMID, cores, memory, storage of each path, bridge, optional variables and devices.
3. **Review.** A summary shows image, resources, storage, each path and where it goes, network, acceleration, security and variables, with secrets masked. Nothing has been created yet.
4. **Resolve.** The registry is queried, the architecture of the host is selected and the digest the tag points to at that moment is fixed.
5. **Download and verify.** Skopeo downloads the image as an OCI archive. Every layer is checked against its digest before anything is created.
6. **Build.** `pct create` builds the rootfs from the archive and imports `Entrypoint`, `Cmd`, `Env`, `User`, `WorkingDir` and `StopSignal` from the image.
7. **Connect.** The volumes are added as mount points, then the devices, the environment, the network and the security settings the oci-compose declares.
8. **Prepare.** Each new volume is made ready for the first start: `lost+found` is removed and the directory is given to the user the application writes with. The files the oci-compose asks to write are written.
9. **Start and check.** The container starts, an address is waited for, the settings that need the running application are applied, and the service is checked. A failed first start is reported as a failed installation.
10. **Register.** The record of the installation is written, and a copy travels inside the container so that a backup carries it.
The result is an ordinary LXC:
```
entrypoint: /jellyfin/jellyfin
hostname: jellyfin
cores: 2
memory: 2048
unprivileged: 1
rootfs: local-zfs:subvol-126-disk-0,size=8G
mp0: local-zfs:subvol-126-disk-1,mp=/config,backup=1,size=16G
mp1: local-zfs:subvol-126-disk-2,mp=/cache,backup=1,size=16G
mp2: local-zfs:subvol-126-disk-3,mp=/media,backup=1,size=32G
dev0: path=/dev/dri/renderD128,mode=0660,deny-write=0,gid=993
net0: name=eth0,bridge=vmbr0,firewall=1,host-managed=1,ip=dhcp,type=veth
tags: media;oci;proxmenux
```
An application made of several images is one entry of the catalog and one installation. It creates one LXC per service on a private bridge, with fixed addresses and the names the services use to find each other, and starts them in order.
## Update, modify, recreate and remove
*Manage installed OCI applications* lists the registered installations. Before any action a container is compared with its record; one that no longer matches is neither modified nor deleted.
| Operation | What changes | What is kept |
|---|---|---|
| **Update** | The rootfs is replaced by the image the saved tag publishes today | Record, container disks, host directories, network and devices |
| **Modify** | An editor opens with the current resources, network, paths and GPU; the container is rebuilt with the changes | The data of container disks and host directories |
| **Recreate** (multi-container) | Every container is rebuilt from the image it was installed with | Data, network and devices |
| **Watchdog** | Whether the application is started again when it stops on its own | Everything; only the choice is saved |
| **Remove** | The LXC, or every container of the application, its disks and its record | Host directories, with their content |
An update is a transaction:
1. The tag is resolved. When the registry still serves the installed digest, nothing is downloaded and the container is not touched.
2. The new image is downloaded and verified, and the operation checks that the backup fits, before the container stops.
3. The container is stopped and backed up.
4. The new rootfs is imported and the record is applied again: mount points, environment, network, devices and security.
5. The application is checked. When it answers, the record is published with the new digest. When it does not, the previous container is restored from the backup.
Settings changed in Proxmox VE after the installation, such as memory, cores or start with the node, are carried into the new record. An operation interrupted by a closed terminal or a reboot is detected the next time the menu opens, and the previous installation can be recovered. A container restored from a backup on another host is registered again from the copy of the record it carries.
A rolling tag is not an unattended update: the digest changes only through an explicit update.
## From a Compose file or a docker run command to an oci-compose
The converter reads a Compose file with one service. A `docker run` command and an image reference alone are first written as a Compose file and follow the same path, so the three produce the same kind of oci-compose.
Starting from this definition:
```yaml
services:
app:
image: ghcr.io/example/app:latest
ports:
- 8080:8080
volumes:
- ./config:/config
environment:
- TZ=Europe/Madrid
- API_KEY=
restart: unless-stopped
```
each key becomes a field of the oci-compose:
| Compose key | Field of the oci-compose | Result in Proxmox VE |
|---|---|---|
| `image` | `container_contract.image` | The image that is downloaded; its tag is resolved to a digest at each installation |
| `ports` | `container_contract.ports`, `first_run.endpoints`, `catalog_ui.launch` | The LXC has its own address, so the published port is not used: the service answers on the container port. The first TCP port is taken as the web address |
| `volumes` | `container_contract.volumes`, `proxmox.installer_profile.volume_preparations` | A container disk or a host directory for each path |
| `environment` | `container_contract.environment` | Variables of the LXC. A name that reads as a password, token or key is asked as a secret |
| `restart` | `container_contract.restart` | The watchdog is proposed by default |
| `mem_limit` | `proxmox.installer_profile.resources.memory_default_mb` | The memory proposed for the LXC |
| `cpu_shares` | `proxmox.installer_profile.resources.cpu_shares` | `cpuunits` of the LXC |
| `ulimits` | `proxmox.installer_profile.resources.rlimits` | `lxc.prlimit.*` |
| `shm_size` | `proxmox.installer_profile.tmpfs_mounts` | A tmpfs at `/dev/shm` of that size |
| `user`, `working_dir`, `entrypoint`, `command`, `hostname`, `group_add` | `proxmox.installer_profile.runtime` | The process the LXC starts, its user, directory and groups |
| `extra_hosts` | `proxmox.installer_profile.extra_hosts` | Entries of `/etc/hosts` in the container |
| `devices` | `proxmox.installer_profile.device_requests` | A `devN` entry for each device, confirmed during the installation |
| `runtime: nvidia` | `proxmox.installer_profile.device_requests` | The NVIDIA devices and the driver files of the host |
| `healthcheck` | `proxmox.installer_profile.startup_healthcheck` | The check of the first start. It runs during installations and updates, not continuously |
| `cap_add`, `sysctls`, `security_opt` | `proxmox.installer_profile.security` | Capabilities checked against the LXC model, `lxc.sysctl` entries and AppArmor or seccomp settings, each confirmed |
| `privileged` | `proxmox.security_profile` | The LXC stays unprivileged; the privileged mode is offered as an explicit choice |
| `network_mode` | `proxmox.installer_profile.network` | `host`, `bridge` and `default` all mean the network of the LXC, never the one of the Proxmox host |
| `stop_grace_period` | `proxmox.defaults.shutdown_timeout_seconds` | The time a shutdown waits |
| `labels`, `logging` | Kept in `container_contract.original_compose` only | Not applied: they are read by Docker tools |
What has no translation stops the installation and is named: `pid: host`, a Compose file with several services given to this option, `configs`, external secrets, a `build` section without a published image, and shell substitutions such as `$(command)`.
Besides those fields, the converter writes the blocks that describe the translation itself: where the definition came from, the keys it understood, the adaptations it applied and the state of each validation. They are listed in [Blocks ProxMenux writes](#blocks-proxmenux-writes).
## The oci-compose, block by block
A complete oci-compose has around 550 lines, and most of them are written by the converter. One written by hand contains only what its author decides; ProxMenux completes the rest when it reads the file.
A file written this way describes one image. An application made of several images is read from its complete file, which carries the `compose_stack` block with its services.
Two files of [`examples/`](examples/) are used below. Both install the same as the entry of the catalog with the same name:
- [`jellyfin-official.oci-compose.json`](examples/jellyfin-official.oci-compose.json): two ports, an optional variable, three volumes with different owners, a GPU profile, a setting applied after the first start and a check of the service.
- [`duplicati-official.oci-compose.json`](examples/duplicati-official.oci-compose.json): two secrets generated during the installation and the credentials shown at the end.
The smallest valid file names the image and nothing else:
```json
{
"kind": "proxmenux.oci-template",
"container_contract": {
"image": { "reference": "traefik/whoami:latest" }
}
}
```
### `kind`
Always `"proxmenux.oci-template"`. It is how the file is recognised.
### `catalog_ui`: how the application is presented
```json
"catalog_ui": {
"title": { "en_US": "Jellyfin Official" },
"tagline": { "en_US": "Official Jellyfin image with optional VA-API or NVIDIA acceleration." },
"category": "media",
"icon": "https://raw.githubusercontent.com/linuxserver/docker-templates/master/linuxserver.io/img/jellyfin-icon.png",
"website": "https://jellyfin.org/",
"documentation": "https://jellyfin.org/docs/general/installation/container/",
"repository": "https://github.com/jellyfin/jellyfin-packaging"
}
```
| Field | What it is |
|---|---|
| `title` | The name shown in the menu, in the notes of the container and in ProxMenux Monitor. The name of the installation is taken from it |
| `tagline`, `description` | One line and a longer text, shown on the page of the application |
| `category` | One of the keys of `catalog/categories.json`: `media`, `backup`, `network`, `databases`, `documents`, `downloads`, `sharing`, `monitoring`, `smarthome`, `security`, `ai`, `tools`, `misc` and the others listed there |
| `icon` | Address of the logo, used in the notes of the container and in the Monitor |
| `website`, `documentation`, `repository` | Links of the project. The installation shows the documentation link when it asks about an optional variable or names a service the application needs |
| `launch` | `scheme`, `port` and `path` of the web interface. `"port": null` declares a service with no web interface, together with an empty `first_run.endpoints` |
Without this block the title is taken from the name of the image and the category is `misc`.
### `container_contract`: what the image needs
This block is the equivalent of the service of a Compose file.
```json
"container_contract": {
"service_name": "jellyfin",
"container_name": "jellyfin",
"image": { "reference": "jellyfin/jellyfin:latest" },
"ports": [
{ "container_port": 8096 },
{ "container_port": 7359, "protocol": "udp", "required": false }
],
"environment": [
{ "name": "JELLYFIN_PublishedServerUrl", "required": false,
"prompt": "Optional published URL for Jellyfin" }
],
"volumes": [
{ "container_path": "/config" },
{ "container_path": "/cache" },
{ "container_path": "/media", "default": "host-bind", "managed_volume": { "default_size_gb": 32 } }
],
"restart": "unless-stopped"
}
```
**`image.reference`** is the only required field of the file: the image as it would be written after `docker pull`, with its tag. The registry, the repository and the tag are worked out from it.
**`service_name`** and **`container_name`** are the name of the service and the hostname proposed for the LXC. Without them both come from the title.
**`ports`**. One entry for each port the application listens on.
| Field | Default | What it is |
|---|---|---|
| `container_port` | — | The port inside the container. It is the one the service answers on |
| `protocol` | `tcp` | `tcp` or `udp` |
| `required` | `true` | `false` marks a port the application works without; the summary shows it as optional |
The first TCP port is taken as the web address unless `first_run.endpoints` says otherwise.
**`environment`**. One entry for each variable the installation sets.
| Field | Default | What it is |
|---|---|---|
| `name` | — | The name of the variable |
| `example` | empty | The value proposed. A variable with a value is installed with it in default mode and can be changed in advanced mode |
| `required` | `true` | A required variable with no value is asked in both modes. An optional one is offered only in advanced mode |
| `sensitive` | `false` | `true` asks the value hidden, asks it twice and masks it in the summary |
| `prompt` | the name | The text of the question |
| `container_address` | — | A value built from the address the container gets, written with the mark `@CONTAINER_ADDRESS@`, for example `"http://@CONTAINER_ADDRESS@"`. With a static address it is known before the installation; with DHCP it is resolved when the container gets its lease |
The variables the image itself defines are imported with it and are not written here.
**`volumes`**. One entry for each path that has to survive an update.
| Field | Default | What it is |
|---|---|---|
| `container_path` | — | The path inside the container |
| `installation_choice` | see below | Where the path may go: `managed-volume` (a container disk), `host-bind` (a host directory) |
| `default` | `managed-volume` | The choice proposed |
| `managed_volume.default_size_gb` | 16, or 8 for a path with content of the user | The size proposed for the container disk |
| `managed_volume.backup` | `true` | Whether the disk is part of the backup of the container |
| `read_only` | `false` | Mounts the path read-only |
Where a path goes follows one rule. A path that holds content of the user, such as `/media`, `/downloads`, `/music` or `/backups`, is offered as a container disk or as a host directory, and the installation asks which. The list is in `catalog/volume-policy.json`. Every other path is state of the application and goes to a container disk without asking. Writing `"installation_choice": ["managed-volume", "host-bind"]` or `"default": "host-bind"` on a volume offers the host directory for a path that is not in that list.
**`restart`**. With `unless-stopped`, `always` or `on-failure`, the watchdog is proposed by default: an application that stops on its own is started again.
### `first_run`: what is shown when the installation ends
```json
"first_run": {
"endpoints": [
{ "label": "Jellyfin WebUI", "scheme": "http", "port": 8096, "path": "/",
"source": "official-container-documentation" }
]
}
```
**`endpoints`** are the addresses printed at the end, written in the notes of the container and opened from the Monitor. Each one needs `label`, `scheme`, `port`, `path` and `source`, a free text that says where the address is documented. An empty list declares a service with no web interface: its ports are then shown as `IP:port` text and no link is offered.
**`credentials`** are the logins printed at the end. From the Duplicati example:
```json
"credentials": [
{ "label": "Duplicati web interface (password only)",
"type": "configured-or-installer-generated",
"username": "Not required (password only)", "password": null,
"password_environment": "DUPLICATI__WEBSERVICE_PASSWORD",
"change_required": false, "source": "duplicati-documentation", "retrieval": null }
]
```
| `type` | Meaning |
|---|---|
| `static-default` | A login the image documents, written in `username` and `password` |
| `configured-or-installer-generated` | The value of the variable named in `password_environment`, typed or generated during the installation |
| `runtime-generated` | A password the application prints at its first start; `retrieval` says how it is read |
### `proxmox.defaults`: the container that is proposed
```json
"proxmox": {
"defaults": { "memory_mb": 2048 }
}
```
Only what differs from the defaults is written. The defaults are 2 cores, 1024 MB of memory, 512 MB of swap, an 8 GB rootfs, an unprivileged LXC with `nesting=1`, DHCP on `vmbr0`, the firewall on, and no start with the node. The fields are `cores`, `memory_mb`, `swap_mb`, `rootfs_size_gb`, `unprivileged`, `features`, `onboot` and `shutdown_timeout_seconds`. The storage and the bridge proposed are the ones of the node.
### `proxmox.installer_profile`: what Compose cannot say
This block holds the decisions of the installation. Each part is optional.
**`generated_sensitive_environment`**: secrets the installation generates.
```json
"generated_sensitive_environment": {
"SETTINGS_ENCRYPTION_KEY": { "strategy": "token-hex", "bytes": 16, "prompt": false },
"DUPLICATI__WEBSERVICE_PASSWORD": { "strategy": "token-hex", "bytes": 12, "prompt": true }
}
```
The variable is declared in `environment` with `"sensitive": true`. In default mode its value is generated. In advanced mode a variable with `"prompt": true` is asked, and left empty it is generated. The generated value is shown at the end when `first_run.credentials` names the variable.
**`volume_preparations`**: who owns each new volume.
```json
"volume_preparations": [
{ "container_path": "/config", "remove_lost_found": true,
"owner_strategy": "mapped-root", "only_when_mount_type": "managed-volume" },
{ "container_path": "/media", "remove_lost_found": true,
"owner_strategy": "mapped-application-user", "only_when_mount_type": "managed-volume" }
]
```
A new container disk is an empty filesystem owned by root that carries a `lost+found` directory. `owner_strategy` gives it to `mapped-root`, the root of the container, for an image that starts as root and takes ownership of its directories itself, or to `mapped-application-user`, the user 1000 the application writes with. Without this list every volume is prepared for `mapped-application-user`.
**`startup_healthcheck`**: how the first start is checked.
```json
"startup_healthcheck": {
"scheme": "http", "port": 8096, "path": "/System/Info/Public",
"timeout_seconds": 180, "request_timeout_seconds": 10,
"stability_seconds": 4, "verify_tls": false
}
```
The installation and every update wait up to `timeout_seconds` for that address to answer, and then `stability_seconds` more to see that it keeps answering. An update that does not pass the check is undone. Without this block the web address is checked, or that the container keeps running when there is no web address.
**`hardware_acceleration`**: the choice of GPU.
```json
"hardware_acceleration": {
"prompt": "Hardware acceleration for Jellyfin",
"default": "none",
"profiles": [
{ "id": "none", "label": "No acceleration (CPU)", "device_requests": [] },
{ "id": "vaapi", "label": "Intel/AMD (VA-API)",
"device_requests": [
{ "id": "vaapi-render", "kind": "character-device", "purpose": "vaapi",
"path_prompt": "GPU render device", "host_path_default": "/dev/dri/renderD128",
"container_path_strategy": "same-as-host", "mode": "0660", "deny_write": false,
"gid_strategy": "host-device-gid",
"environment": [
{ "name": "LIBVA_DRIVER_NAME", "prompt": "VA-API driver",
"choices": ["auto", "radeonsi", "iHD", "i965"], "default": "auto", "omit_values": ["auto"] }
] }
] },
{ "id": "nvidia", "label": "NVIDIA (NVENC/NVDEC)",
"device_requests": [
{ "id": "nvidia-runtime", "kind": "nvidia-runtime", "purpose": "nvidia-cuda-nvenc-nvdec",
"device_selection": "all-requested-by-compose" }
],
"environment": [ { "name": "NVIDIA_VISIBLE_DEVICES", "value": "all" } ] }
]
}
```
The installation shows one option for each profile the host can serve: a profile whose device is not on the host is not listed. Each profile declares the devices it passes and the variables it sets.
| Field of a device request | What it is |
|---|---|
| `kind` | `character-device` for a device node such as a render node, or `nvidia-runtime` for an NVIDIA GPU with the driver files of the host |
| `host_path_default`, `path_prompt` | The device proposed and the text that asks for it |
| `container_path_strategy` | `same-as-host` keeps the path of the host inside the container |
| `mode`, `gid_strategy` | The permissions of the device. `host-device-gid` reads the group from the device of the host, so no fixed `video` or `render` number is assumed |
| `environment` | Variables asked with the device, with `choices` and the values that are left unset (`omit_values`) |
A profile may also change the image, with `image`, for projects that publish one tag for each GPU vendor, and apply settings once the application is running, with `post_start_configurations`. In the Jellyfin example the VA-API and NVIDIA profiles write the transcoding and tone mapping settings to `encoding.xml` after Jellyfin creates it.
**Other parts of the profile.**
| Part | What it does |
|---|---|
| `completion_notes` | A list of texts printed when the installation ends, for what is left to do by hand |
| `external_services` | Names of services the application needs already running, such as `["MariaDB"]`. They are announced before the installation asks how to reach them |
| `generated_files` | Files written in the container before its first start, each with `container_path`, `content`, `owner` and `mode`. With `"only_if_missing": true` a file that already exists in a kept volume is not replaced |
| `volume_owner` | `uid` and `gid` of the user the application writes with, when it is not 1000 |
| `non_persistent_image_volumes` | Paths the image declares as volumes that are left inside the rootfs |
| `runtime`, `resources`, `security`, `device_requests`, `tmpfs_mounts`, `extra_hosts` | What the Compose keys of the [translation table](#from-a-compose-file-or-a-docker-run-command-to-an-oci-compose) become |
### `proxmox.security_profile`: privileges
Without this block the LXC is unprivileged with the default AppArmor profile. `"optional_privileged_lxc": true` offers the privileged mode as a choice that defaults to no; `"requires_privileged_lxc": true` asks for it with a confirmation that states the risk. A GPU, a USB device or a serial port are passed as individual devices and need neither.
### Blocks ProxMenux writes
These blocks are part of every complete oci-compose. They are not written by hand: when a file does not bring them, they are added as the converter writes them for the same image, ports, variables and volumes.
| Block | What it holds |
|---|---|
| `schema_version`, `id`, `status` | The version of the format, the identifier and the state of the review. `laboratory-validated` is given only by `catalog/verification.json`: a file that declares itself verified is read as not verified |
| `source` | Where the definition came from and when it was generated |
| `container_contract.original_compose` | The Compose file the oci-compose was translated from |
| `compose_stack` | The services of the application, their order and their private network. For one image it describes a single service |
| `compatibility` | The Compose keys that were understood and the blockers left |
| `proxmox.adaptations` | Each difference with Docker, its reason and its effect |
| `validation`, `lifecycle` | The state of each test and how updates are detected |
What the author writes always wins over what is added.
## Writing and testing an oci-compose
1. **Write the file.** Start from one of the [`examples/`](examples/) or from the smallest file above, and add the blocks the application needs. Any file of `catalog/apps/` or `catalog/curated/` is also a valid starting point.
2. **Copy it to the node**, for example to `/root/oci-compose.json`, or publish it where the node can download it.
3. **Open OCI manager Apps** from the ProxMenux menu and choose *Install an image that is not in the catalog*.
4. **Choose how it is given**: *Read an oci-compose from a file of this host* or *Download an oci-compose from an address*.
5. **Read the result.** A file with errors is not installed, and each error names the field:
```
The oci-compose has errors:
container_contract/ports: array
container_contract/volumes/3/container_path: required
first_run/endpoints/0: 'label' is a required property
```
A file that is not valid JSON reports the line and the column of the fault.
6. **Review.** A valid file shows what it installs: image, web address, ports, persistent paths, the variables the installation asks for and the files it writes. The image is looked up in its registry, and its architectures are listed. Answering no leaves the node untouched.
7. **Install** with default or advanced settings. The installation is the one of any application of the catalog.
8. **Adjust.** Remove the test installation from *Manage installed OCI applications*, edit the file and read it again.
An oci-compose decides what is mounted and with which privileges, so the review says it before installing. Install only files that come from a source you trust.
The record of an installation, `/usr/local/share/proxmenux/oci/instances/<VMID>/oci-compose.json`, can be read the same way: only its `template` block is used, as it was installed, and none of the answers of that installation. That record also holds the values typed during the installation, passwords included, so it is not a file to publish as it is.
The checks of the engine run with:
```bash
cd oci
python3 -m unittest discover -s tests -p 'test_*.py'
```
`tests/test_install_from_oci_compose.py` checks, among other things, that the two files of `examples/` install what the catalog installs.
## Adding an application to the catalog
An application reaches the catalog in three steps.
1. **Suggest it** in [Discussions](https://github.com/MacRimi/ProxMenux/discussions) with the image, the registry it is published in and the category.
2. **Try it.** Installing it from its Compose file or its `docker run` command shows whether the image runs as a native LXC. When it needs something Compose cannot say, an oci-compose written by hand lets that be tested as well. The file that worked, attached to the suggestion, is what the entry of the catalog is written from.
3. **Validate it.** An application tested for real is recorded in `catalog/verification.json` and shown as verified in the installer. [VALIDATION.md](VALIDATION.md) lists the verified applications and describes a test: install, check that it works, restart, update and remove.
---
The sections that follow are the design and laboratory notes of the engine.
## Design
@@ -106,9 +581,10 @@ assuming that every Compose `privileged: true` is an image requirement:
blocked until a safe native LXC translation is validated; it must never be
mislabeled as ordinary GPU access or silently promoted to privileged mode.
Before creating an LXC, the tool prints a redacted deployment plan and requires
the exact confirmation `INSTALAR`. Remote credentials are handled by normal SSH;
they are never placed in a template or command argument.
Before creating an LXC, the installer shows the installation summary with every
secret masked and asks for confirmation: `Install with this configuration?` in
the menu and `Install? [y/N]` in the `install` command. Remote credentials are
handled by normal SSH; they are never placed in a template or command argument.
Image downloads run in a pseudo-terminal when `script` is available, allowing
Skopeo layer progress to remain visible. The final result lists the detected IP,
@@ -0,0 +1,83 @@
{
"kind": "proxmenux.oci-template",
"catalog_ui": {
"title": {
"en_US": "Duplicati Official"
},
"tagline": {
"en_US": "Official Duplicati image: encrypted, incremental backups to local, network and cloud storage."
},
"category": "backup",
"icon": "https://cdn.jsdelivr.net/gh/selfhst/icons@main/webp/duplicati.webp",
"website": "https://www.duplicati.com/",
"documentation": "https://docs.duplicati.com/",
"repository": "https://github.com/duplicati/duplicati"
},
"container_contract": {
"service_name": "duplicati",
"container_name": "duplicati",
"image": {
"reference": "duplicati/duplicati:latest"
},
"ports": [
{
"container_port": 8200
}
],
"environment": [
{
"name": "SETTINGS_ENCRYPTION_KEY",
"required": true,
"sensitive": true
},
{
"name": "DUPLICATI__WEBSERVICE_PASSWORD",
"required": true,
"sensitive": true,
"prompt": "Password of the Duplicati web interface"
}
],
"volumes": [
{
"container_path": "/data"
},
{
"container_path": "/backups"
},
{
"container_path": "/source"
}
],
"restart": "unless-stopped"
},
"first_run": {
"credentials": [
{
"label": "Duplicati web interface (password only)",
"type": "configured-or-installer-generated",
"username": "Not required (password only)",
"password": null,
"password_environment": "DUPLICATI__WEBSERVICE_PASSWORD",
"change_required": false,
"source": "duplicati-documentation",
"retrieval": null
}
]
},
"proxmox": {
"installer_profile": {
"generated_sensitive_environment": {
"SETTINGS_ENCRYPTION_KEY": {
"strategy": "token-hex",
"bytes": 16,
"prompt": false
},
"DUPLICATI__WEBSERVICE_PASSWORD": {
"strategy": "token-hex",
"bytes": 12,
"prompt": true
}
}
}
}
}
@@ -0,0 +1,234 @@
{
"kind": "proxmenux.oci-template",
"catalog_ui": {
"title": {
"en_US": "Jellyfin Official"
},
"tagline": {
"en_US": "Official Jellyfin image with optional VA-API or NVIDIA acceleration."
},
"description": {
"en_US": "Jellyfin is a Free Software Media System that puts you in control of managing and streaming your media."
},
"category": "media",
"icon": "https://raw.githubusercontent.com/linuxserver/docker-templates/master/linuxserver.io/img/jellyfin-icon.png",
"website": "https://jellyfin.org/",
"documentation": "https://jellyfin.org/docs/general/installation/container/",
"repository": "https://github.com/jellyfin/jellyfin-packaging"
},
"container_contract": {
"service_name": "jellyfin",
"container_name": "jellyfin",
"image": {
"reference": "jellyfin/jellyfin:latest"
},
"ports": [
{
"container_port": 8096
},
{
"container_port": 7359,
"protocol": "udp",
"required": false
}
],
"environment": [
{
"name": "JELLYFIN_PublishedServerUrl",
"required": false,
"prompt": "Optional published URL for Jellyfin"
}
],
"volumes": [
{
"container_path": "/config"
},
{
"container_path": "/cache"
},
{
"container_path": "/media",
"default": "host-bind",
"managed_volume": {
"default_size_gb": 32
}
}
],
"restart": "unless-stopped"
},
"first_run": {
"endpoints": [
{
"label": "Jellyfin WebUI",
"scheme": "http",
"port": 8096,
"path": "/",
"source": "official-container-documentation"
}
]
},
"proxmox": {
"defaults": {
"memory_mb": 2048
},
"installer_profile": {
"hardware_acceleration": {
"prompt": "Hardware acceleration for Jellyfin",
"default": "none",
"profiles": [
{
"id": "none",
"label": "No acceleration (CPU)",
"device_requests": []
},
{
"id": "vaapi",
"label": "Intel/AMD (VA-API)",
"device_requests": [
{
"id": "vaapi-render",
"kind": "character-device",
"purpose": "vaapi",
"path_prompt": "GPU render device",
"host_path_default": "/dev/dri/renderD128",
"container_path_strategy": "same-as-host",
"mode": "0660",
"deny_write": false,
"gid_strategy": "host-device-gid",
"environment": [
{
"name": "LIBVA_DRIVER_NAME",
"prompt": "VA-API driver",
"choices": [
"auto",
"radeonsi",
"iHD",
"i965"
],
"default": "auto",
"omit_values": [
"auto"
]
}
]
}
],
"post_start_configurations": [
{
"id": "jellyfin-vaapi-hdr-tone-mapping",
"type": "jellyfin-encoding-xml",
"enable_prompt": "Enable VA-API transcoding and HDR10/Dolby Vision to SDR tone mapping",
"enabled_default": true,
"required": true,
"timeout_seconds": 120,
"candidate_paths": [
"/config/encoding.xml",
"/config/config/encoding.xml"
],
"settings": {
"HardwareAccelerationType": "vaapi",
"VaapiDevice": {
"device_path_from": "vaapi-render"
},
"EnableDecodingColorDepth10Hevc": "true",
"EnableHardwareEncoding": "true",
"EnableTonemapping": "true",
"EnableVppTonemapping": "false",
"TonemappingAlgorithm": "bt2390",
"TonemappingMode": "auto",
"TonemappingRange": "auto"
},
"lists": {
"HardwareDecodingCodecs": [
"h264",
"hevc",
"vc1"
]
}
}
]
},
{
"id": "nvidia",
"label": "NVIDIA (NVENC/NVDEC)",
"device_requests": [
{
"id": "nvidia-runtime",
"kind": "nvidia-runtime",
"purpose": "nvidia-cuda-nvenc-nvdec",
"device_selection": "all-requested-by-compose"
}
],
"environment": [
{
"name": "NVIDIA_VISIBLE_DEVICES",
"value": "all"
}
],
"post_start_configurations": [
{
"id": "jellyfin-nvidia-hdr-tone-mapping",
"type": "jellyfin-encoding-xml",
"enable_prompt": "Enable NVIDIA transcoding and HDR10/Dolby Vision to SDR tone mapping",
"enabled_default": true,
"required": true,
"timeout_seconds": 120,
"candidate_paths": [
"/config/encoding.xml",
"/config/config/encoding.xml"
],
"settings": {
"HardwareAccelerationType": "nvenc",
"EnableDecodingColorDepth10Hevc": "true",
"EnableEnhancedNvdecDecoder": "true",
"EnableHardwareEncoding": "true",
"EnableTonemapping": "true",
"EnableVppTonemapping": "false",
"TonemappingAlgorithm": "bt2390",
"TonemappingMode": "auto",
"TonemappingRange": "auto"
},
"lists": {
"HardwareDecodingCodecs": [
"h264",
"hevc",
"vc1"
]
}
}
]
}
]
},
"startup_healthcheck": {
"scheme": "http",
"port": 8096,
"path": "/System/Info/Public",
"timeout_seconds": 180,
"request_timeout_seconds": 10,
"stability_seconds": 4,
"verify_tls": false
},
"volume_preparations": [
{
"container_path": "/config",
"remove_lost_found": true,
"owner_strategy": "mapped-root",
"only_when_mount_type": "managed-volume"
},
{
"container_path": "/cache",
"remove_lost_found": true,
"owner_strategy": "mapped-root",
"only_when_mount_type": "managed-volume"
},
{
"container_path": "/media",
"remove_lost_found": true,
"owner_strategy": "mapped-root",
"only_when_mount_type": "managed-volume"
}
]
}
}
}
+78 -9
View File
@@ -376,7 +376,8 @@ class Catalog:
return self._volume_policy
def _apply_volume_policy(self, app_id: str, contract: dict[str, Any],
profile: dict[str, Any] | None = None) -> None:
profile: dict[str, Any] | None = None,
offered: frozenset[str] = frozenset()) -> None:
"""A path that holds content of the user is offered as a container
volume or as a host directory, and the installation asks which one; the
state of the application stays in a container volume without asking.
@@ -396,7 +397,8 @@ class Catalog:
profile["volume_preparations"] = [item for item in profile["volume_preparations"]
if item.get("container_path") not in dropped]
policy = self.volume_policy()
shared = policy["shared_paths"] | set(policy["applications"].get(app_id, []))
# `offered` are the paths an oci-compose gives as a host directory.
shared = policy["shared_paths"] | set(policy["applications"].get(app_id, [])) | offered
# A volume known to stay small is created at its stated size.
sizes = policy.get("sizes", {}).get(app_id, {})
for volume in contract.get("volumes", []):
@@ -727,20 +729,61 @@ class Catalog:
self._write_json(self.catalog_dir / report_name, report)
return report
def validate(self, template: dict[str, Any], required: bool = True) -> None:
# Templates ship already validated; on a node without python3-jsonschema
# the installer skips the check instead of adding a package to the host.
def schema_errors(self, template: dict[str, Any]) -> list[str] | None:
"""What a template breaks of the schema, or None on a node without
python3-jsonschema, where it cannot be checked."""
try:
from jsonschema import Draft202012Validator
except ImportError:
return None
schema = json.loads(self.schema_path.read_text(encoding="utf-8"))
errors = sorted(Draft202012Validator(schema).iter_errors(template), key=lambda error: list(error.path))
return [f"{'/'.join(map(str, error.path))}: {error.message}" for error in errors]
def shape_errors(self, template: dict[str, Any]) -> list[str]:
"""The fields a template lacks or gives with another type, read from
the schema without python3-jsonschema: what the installer relies on."""
schema = json.loads(self.schema_path.read_text(encoding="utf-8"))
kinds = {"object": dict, "array": list, "string": str, "boolean": bool,
"integer": int, "number": (int, float), "null": type(None)}
errors: list[str] = []
def check(value: Any, rule: dict[str, Any], path: str) -> None:
if "$ref" in rule:
target: Any = schema
for part in rule["$ref"].lstrip("#/").split("/"):
target = target.get(part, {})
rule = target
wanted = rule.get("type")
wanted = [wanted] if isinstance(wanted, str) else wanted or []
if wanted and not any(isinstance(value, kinds[kind]) and not (kind in ("integer", "number")
and isinstance(value, bool))
for kind in wanted if kind in kinds):
errors.append(f"{path}: {' / '.join(wanted)}")
return
if isinstance(value, dict):
errors.extend(f"{path}/{name}".lstrip("/") + ": required" for name in rule.get("required", [])
if name not in value)
for name, inner in rule.get("properties", {}).items():
if name in value:
check(value[name], inner, f"{path}/{name}".lstrip("/"))
elif isinstance(value, list) and isinstance(rule.get("items"), dict):
for position, item in enumerate(value):
check(item, rule["items"], f"{path}/{position}")
check(template, schema, "")
return errors
def validate(self, template: dict[str, Any], required: bool = True) -> None:
# Templates ship already validated; on a node without python3-jsonschema
# the installer skips the check instead of adding a package to the host.
errors = self.schema_errors(template)
if errors is None:
if required:
raise ConversionError("python3-jsonschema is required to generate templates")
return
schema = json.loads(self.schema_path.read_text(encoding="utf-8"))
errors = sorted(Draft202012Validator(schema).iter_errors(template), key=lambda error: list(error.path))
if errors:
details = "; ".join(f"{'/'.join(map(str, error.path))}: {error.message}" for error in errors)
raise ConversionError(f"La plantilla generada no cumple el esquema: {details}")
raise ConversionError(f"La plantilla generada no cumple el esquema: {'; '.join(errors)}")
def compose(self, app_id: str) -> dict[str, Any]:
"""The installable template, built only from the files shipped with
@@ -761,6 +804,32 @@ class Catalog:
self.validate(template, required=False)
return template
def finish(self, template: dict[str, Any], offered: frozenset[str] | None = None) -> dict[str, Any]:
"""The last steps of compose() for an oci-compose that does not come
from the files of the catalog: one somebody wrote or shared. The
overlay, the category and the verification the catalog keeps under a
name are not given to it: it is not verified because it says so.
`offered` are the paths its author gave as a host directory; without
it, the ones the file already offers as one."""
from .stack import apply_stack_support
apply_stack_support(template)
from .gpu import apply_gpu_contract, apply_nginx_runtime_contract
apply_gpu_contract(template)
apply_nginx_runtime_contract(template)
if template.get("status") == "laboratory-validated":
template["status"] = "generated-unvalidated"
template.pop("community_tested", None)
contract = template["container_contract"]
# The policy of an application is kept under its name in the catalog,
# which a file of the catalog is recognised by.
name = next((item["id"] for item in self.load_index().get("applications", [])
if template.get("id") and item.get("template_id") == template.get("id")), "")
if offered is None:
offered = frozenset(volume["container_path"] for volume in contract.get("volumes", [])
if "host-bind" in volume.get("installation_choice", []))
self._apply_volume_policy(name, contract, template.get("proxmox", {}).get("installer_profile"), offered)
return template
def load_template(self, app_id: str, generate_if_missing: bool = True) -> dict[str, Any]:
path = self.apps_dir / f"{app_id}.json"
if not path.exists() and generate_if_missing:
+182 -16
View File
@@ -25,6 +25,15 @@ from .ui import UserCancelled
IMAGE_REFERENCE = re.compile(r'^[A-Za-z0-9][A-Za-z0-9._/-]*(?::[A-Za-z0-9._-]+)?(?:@sha256:[a-f0-9]{64})?$')
MAX_COMPOSE_BYTES = 256 * 1024
# An oci-compose: the file ProxMenux installs an application from. The catalog
# keeps one for each of its applications and every installation records its own.
MAX_OCI_COMPOSE_BYTES = 2 * 1024 * 1024
OCI_COMPOSE_KIND = 'proxmenux.oci-template'
# What read_definition() names an oci-compose with, in the place of the origin
# of a Compose file: it is installed as it is written and not translated.
OCI_COMPOSE = 'oci-compose'
# The lists of an oci-compose are completed item by item, each found by this field.
CONTRACT_LISTS = {'environment': 'name', 'volumes': 'container_path', 'ports': 'container_port'}
class _Dumper(yaml.SafeDumper):
@@ -292,22 +301,159 @@ def compose_from_docker_run(command: str, title: str | None = None) -> str:
return '\n'.join(lines) + '\n'
def _read_url(url: str) -> str:
def _read_url(url: str, limit: int = MAX_COMPOSE_BYTES) -> str:
if not url.startswith(('http://', 'https://')):
raise ConversionError(translate('The address must start with http:// or https://'))
with urllib.request.urlopen(url, timeout=60) as response: # noqa: S310 - the user gives the address
return response.read(MAX_COMPOSE_BYTES + 1).decode('utf-8', errors='replace')
return response.read(limit + 1).decode('utf-8', errors='replace')
def _read_file(path: str) -> str:
def _read_file(path: str, limit: int = MAX_COMPOSE_BYTES) -> str:
file = Path(path).expanduser()
if not file.is_file():
raise ConversionError(f"{translate('The file does not exist:')} {file}")
if file.stat().st_size > MAX_COMPOSE_BYTES:
raise ConversionError(translate('The file is too large to be a Compose file'))
if file.stat().st_size > limit:
raise ConversionError(translate('The file is too large to be a Compose file') if limit == MAX_COMPOSE_BYTES
else translate('The file is too large to be an oci-compose'))
return file.read_text(encoding='utf-8', errors='replace')
def _catalog():
from .catalog import Catalog
from .cli import PROJECT_ROOT
return Catalog(PROJECT_ROOT)
def _errors(template: dict[str, Any]) -> list[str]:
"""What an oci-compose lacks or gives in another form than the installer reads."""
catalog = _catalog()
errors = catalog.schema_errors(template)
return catalog.shape_errors(template) if errors is None else errors
def _lay(target: dict[str, Any], written: dict[str, Any], contract: bool = False) -> None:
"""Lays what the author wrote on what ProxMenux writes for the same image."""
for key, value in written.items():
if isinstance(value, dict) and isinstance(target.get(key), dict):
_lay(target[key], value, contract or key == 'container_contract')
elif contract and key in CONTRACT_LISTS and isinstance(value, list) and isinstance(target.get(key), list):
field = CONTRACT_LISTS[key]
known = {item[field]: item for item in target[key]}
target[key] = []
for item in value:
complete = json.loads(json.dumps(known.get(item[field], {})))
_lay(complete, item)
target[key].append(complete)
else:
target[key] = value
def _completed(document: dict[str, Any]) -> dict[str, Any]:
"""An oci-compose written by hand says what its author decides: the image,
its ports, variables and volumes, and how it is installed. The rest is what
ProxMenux writes for any image, and is taken from the translation of that
same image with those ports, variables and volumes."""
contract = document['container_contract']
reference = contract['image']['reference']
title = (source_text((document.get('catalog_ui') or {}).get('title')) or contract.get('container_name')
or reference.rsplit('/', 1)[-1].split(':', 1)[0].split('@', 1)[0])
name = normalize_app_id(contract.get('service_name') or title)
service: dict[str, Any] = {'image': reference}
ports = [f"{port.get('published_example') or port['container_port']}:{port['container_port']}"
+ ('/udp' if port.get('protocol') == 'udp' else '') for port in contract.get('ports') or []]
if ports:
service['ports'] = ports
volumes = [f"./{Path(volume['container_path']).name or 'data'}:{volume['container_path']}"
for volume in contract.get('volumes') or []]
if volumes:
service['volumes'] = volumes
environment = {item['name']: str(item.get('example') or '') for item in contract.get('environment') or []}
if environment:
service['environment'] = environment
template = template_from_compose(_dump({'name': name, 'services': {name: service}}), title)
for item in template['container_contract'].get('environment', []):
# A value the author left empty is asked for or generated as the file says.
if not environment.get(item['name']):
item['example'] = ''
_lay(template, document)
ui = template['catalog_ui']
if 'category_label' not in (document.get('catalog_ui') or {}):
ui['category_label'] = _catalog().categories()['labels'].get(ui.get('category'), ui.get('category_label'))
return template
def template_from_oci_compose(text: str) -> dict[str, Any]:
"""The template of an oci-compose: one written by hand, a file of the
catalog or the record of an installed application. Of a record only the
template is taken, never the answers of that installation."""
if len(text.encode('utf-8', errors='replace')) > MAX_OCI_COMPOSE_BYTES:
raise ConversionError(translate('The file is too large to be an oci-compose'))
try:
document = json.loads(text)
except ValueError as error:
raise ConversionError(f"{translate('The oci-compose is not valid JSON:')} {error}") from error
if isinstance(document, dict) and isinstance(document.get('template'), dict):
document = document['template']
if not isinstance(document, dict) or document.get('kind') != OCI_COMPOSE_KIND:
raise ConversionError(translate('The file is not a ProxMenux oci-compose.'))
contract = document.get('container_contract')
image = contract.get('image') if isinstance(contract, dict) else None
reference = image.get('reference') if isinstance(image, dict) else None
if not isinstance(reference, str) or not reference.strip():
raise ConversionError(translate('The oci-compose is incomplete: it does not say which image to install.'))
if not IMAGE_REFERENCE.fullmatch(reference):
raise ConversionError(f"{translate('This is not a valid image reference:')} {reference}")
faulty = translate('The oci-compose has errors:')
def refuse(errors: list[str]) -> None:
more = [f' ... (+{len(errors) - 8})'] if len(errors) > 8 else []
raise ConversionError('\n'.join([faulty, *(f' {error}' for error in errors[:8]), *more]))
template = json.loads(json.dumps(document))
offered = None
try:
if _errors({'first_run': {'endpoints': [], 'credentials': []}, **template}):
# Written by hand: what it says has to be in its form before it is completed.
misshapen = [error for error in _catalog().shape_errors(template) if not error.endswith(': required')]
misshapen += [f'container_contract/{name}/{position}/{field}: required'
for name, field in CONTRACT_LISTS.items() if isinstance(contract.get(name), list)
for position, item in enumerate(contract[name])
if isinstance(item, dict) and field not in item]
if misshapen:
refuse(misshapen)
# The paths the author gives as a host directory stay offered as one.
offered = frozenset(volume['container_path'] for volume in contract.get('volumes') or []
if 'host-bind' in (volume.get('installation_choice') or [])
or volume.get('default') == 'host-bind')
template = _completed(template)
template.setdefault('first_run', {'endpoints': [], 'credentials': []})
template = _catalog().finish(template, offered)
except (KeyError, TypeError, AttributeError, IndexError, ValueError) as error:
raise ConversionError(f'{faulty} {type(error).__name__} {error}') from error
errors = _errors(template)
if errors:
refuse(errors)
return template
def describe_oci_compose(template: dict[str, Any]) -> str:
"""What an oci-compose adds to what a Compose file can say: the containers
it is made of and the files it writes in them."""
lines = []
services = [service.get('name') for service in (template.get('compose_stack') or {}).get('services') or []
if isinstance(service, dict) and service.get('name')]
if len(services) > 1:
lines += ['', translate('Containers of the application:'), f" {', '.join(services)}"]
profile = (template.get('proxmox') or {}).get('installer_profile') or {}
written = [item.get('container_path') for item in profile.get('generated_files') or [] if item.get('container_path')]
if written:
lines += ['', translate('Files it writes in the container:')] + [f' {path}' for path in written]
lines += ['', translate('This file decides what is mounted and with which privileges. '
'Install it only if it comes from a source you trust.')]
# It follows the description of the image, after an empty line.
return '\n' + '\n'.join(lines)
def _read_pasted(ui, title: str | None = None) -> str:
"""The definition is pasted in the terminal: dialog cannot take it."""
console.show_logo()
@@ -322,7 +468,8 @@ def _read_pasted(ui, title: str | None = None) -> str:
def read_definition(ui) -> tuple[str, str]:
"""The Compose file of the application and where it came from."""
"""The Compose file of the application and where it came from, or an
oci-compose, named by OCI_COMPOSE."""
from .cli import MENU_SIZE
source = ui.choose(translate('How is the image described?'), [
('paste', translate('Paste its Compose file in the terminal')),
@@ -330,9 +477,16 @@ def read_definition(ui) -> tuple[str, str]:
('url', translate('Download its Compose file from an address')),
('run', translate('Paste its docker run command in the terminal')),
('image', translate('Only the image reference, with no Compose file')),
('oci-compose', translate('Read an oci-compose from a file of this host')),
('oci-compose-url', translate('Download an oci-compose from an address')),
], 'paste', title=translate('Image that is not in the catalog'), size=MENU_SIZE)
if source is None:
raise UserCancelled(translate('Image selection was cancelled'))
if source == 'oci-compose':
return _read_file(ui.ask(translate('Path of the oci-compose file'), '/root/oci-compose.json'),
MAX_OCI_COMPOSE_BYTES), OCI_COMPOSE
if source == 'oci-compose-url':
return _read_url(ui.ask(translate('Address of the oci-compose file'), ''), MAX_OCI_COMPOSE_BYTES), OCI_COMPOSE
if source == 'paste':
return _read_pasted(ui), translate('pasted Compose file')
if source == 'file':
@@ -536,14 +690,19 @@ def _name(ui, template: dict[str, Any]) -> str:
def explore(ui) -> None:
"""Reads a Compose file, a docker run command or an image reference,
reports what ProxMenux would install from it and installs it."""
"""Reads a Compose file, a docker run command, an image reference or an
oci-compose, reports what ProxMenux would install from it and installs it."""
text, origin = read_definition(ui)
notes = [line.partition(':')[2].strip() for line in text.splitlines() if line.startswith('#note:')]
notes += ignored_settings(text)
template = template_from_compose(text)
written = origin == OCI_COMPOSE
if written:
notes = []
template = template_from_oci_compose(text)
else:
notes = [line.partition(':')[2].strip() for line in text.splitlines() if line.startswith('#note:')]
notes += ignored_settings(text)
template = template_from_compose(text)
title = translate('Image that is not in the catalog')
summary = describe(template)
summary = describe(template) + (describe_oci_compose(template) if written else '')
available, report = registry_report(template['container_contract']['image']['reference'])
summary += '\n\n' + report
if notes:
@@ -556,12 +715,14 @@ def explore(ui) -> None:
if not template.get('compatibility', {}).get('automatic_install_candidate'):
ui.message(f"{translate('This image cannot be installed as it is described:')}\n\n{summary}", title)
return
secrets = _ask_secrets(template)
# An oci-compose already says which of its values are secret and which it generates.
secrets = [] if written else _ask_secrets(template)
if secrets:
summary += ('\n\n' + translate('These values are asked during the installation:') + ' '
+ ', '.join(secrets))
if not ui.review(f"{translate('This is what ProxMenux understood from the')} {origin}:\n\n{summary}",
title, question=translate('Install this image?'), default=False):
heading = (translate('This is what the oci-compose installs:') if written
else f"{translate('This is what ProxMenux understood from the')} {origin}:")
if not ui.review(f'{heading}\n\n{summary}', title, question=translate('Install this image?'), default=False):
return
mode = ui.choose(translate('How is it installed?'),
[(DEFAULT_MODE, translate('Default: only what the application needs')),
@@ -571,4 +732,9 @@ def explore(ui) -> None:
return
# The record of the instance keeps the whole template, which is what an
# update, a recreation or a removal read later.
install_template(ui, template, _name(ui, template), mode)
if written:
# An oci-compose keeps the name it was written with.
name = normalize_app_id(source_text(template['catalog_ui'].get('title')) or str(template.get('id') or 'oci-app'))
else:
name = _name(ui, template)
install_template(ui, template, name, mode)
+268
View File
@@ -0,0 +1,268 @@
"""An image that is not in the catalog can also be installed from an
oci-compose: one written by hand with what its author decides, a file of the
catalog, or the record of an installed application, of which only the template
is read."""
import copy
import json
from pathlib import Path
import sys
import tempfile
import unittest
from unittest.mock import patch
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "src"))
from proxmenux_oci import custom
from proxmenux_oci.catalog import Catalog
from proxmenux_oci.converter import ConversionError
CATALOG = Catalog(ROOT)
EXAMPLES = ("jellyfin-official", "duplicati-official")
# Fields that describe where a template came from and are not read to install it.
ORIGIN = {"id", "compose_source_example", "source", "pull_policy"}
def catalog_file(app):
return (ROOT / "catalog/apps" / f"{app}.json").read_text(encoding="utf-8")
def example(app):
return (ROOT / "examples" / f"{app}.oci-compose.json").read_text(encoding="utf-8")
def installed(value):
"""A block without the fields that only say where it came from."""
if isinstance(value, dict):
return {key: installed(item) for key, item in value.items() if key not in ORIGIN}
if isinstance(value, list):
return [installed(item) for item in value]
return value
class FakeUI:
def __init__(self, choices, answers=()):
self.choices, self.answers = list(choices), list(answers)
self.offered, self.reviews, self.asked = [], [], []
def choose(self, prompt, options, default=None, **kwargs):
self.offered.append([key for key, _ in options])
return self.choices.pop(0)
def ask(self, prompt, default="", **kwargs):
self.asked.append(prompt)
return self.answers.pop(0)
def review(self, text, title, **kwargs):
self.reviews.append(text)
return True
def message(self, text, title=None):
self.reviews.append(text)
class CompleteFileTests(unittest.TestCase):
def test_a_file_of_the_catalog_is_read_as_it_is(self):
template = custom.template_from_oci_compose(catalog_file("mariadb"))
self.assertEqual(template["kind"], "proxmenux.oci-template")
self.assertEqual(template["container_contract"]["image"]["reference"], "lscr.io/linuxserver/mariadb:latest")
def test_of_an_installation_record_only_the_template_is_taken(self):
record = {"vmid": 114, "status": "installed", "template": json.loads(catalog_file("mariadb")),
"deployment": {"environment": [{"name": "MYSQL_ROOT_PASSWORD", "value": "the-secret-of-someone"}]}}
template = custom.template_from_oci_compose(json.dumps(record))
self.assertEqual(template["container_contract"]["image"]["reference"], "lscr.io/linuxserver/mariadb:latest")
self.assertNotIn("the-secret-of-someone", json.dumps(template))
self.assertNotIn("deployment", template)
def test_a_template_taken_from_an_installation_installs_the_same(self):
# The record of an installation holds the template as the catalog
# composed it; reading it again must not change it.
index = json.loads((ROOT / "catalog/index.json").read_text(encoding="utf-8"))["applications"]
checked = 0
for item in index:
if item.get("hidden") or not item.get("automatic_install_candidate"):
continue
composed = CATALOG.compose(item["id"])
again = custom.template_from_oci_compose(json.dumps(composed))
for template in (composed, again):
template.pop("status", None)
template.pop("community_tested", None)
self.assertEqual(again, composed, item["id"])
checked += 1
self.assertGreater(checked, 300)
def test_it_is_not_verified_because_it_says_so(self):
document = json.loads(catalog_file("mariadb"))
document["status"] = "laboratory-validated"
document["community_tested"] = {"by": "someone"}
template = custom.template_from_oci_compose(json.dumps(document))
self.assertNotEqual(template["status"], "laboratory-validated")
self.assertNotIn("community_tested", template)
class HandWrittenTests(unittest.TestCase):
"""An oci-compose written by hand says what its author decides; ProxMenux
completes it with what it writes for any image."""
def test_the_examples_install_what_the_catalog_installs(self):
for app in EXAMPLES:
with self.subTest(app=app):
written = json.loads(example(app))
self.assertNotIn("compose_stack", written)
self.assertNotIn("source", written)
template = custom.template_from_oci_compose(example(app))
catalog = CATALOG.compose(app)
for field in ("image", "ports", "environment", "restart", "service_name", "container_name"):
self.assertEqual(installed(template["container_contract"][field]),
installed(catalog["container_contract"][field]), field)
self.assertEqual(installed(template["container_contract"]["volumes"]),
installed(catalog["container_contract"]["volumes"]))
self.assertEqual(template["proxmox"]["defaults"], catalog["proxmox"]["defaults"])
for name in sorted(set(template["proxmox"]["installer_profile"]) | set(catalog["proxmox"]["installer_profile"])):
if name == "network":
# Says how Docker connected it; nothing reads it here.
continue
self.assertEqual(template["proxmox"]["installer_profile"].get(name),
catalog["proxmox"]["installer_profile"].get(name), name)
self.assertEqual(template["first_run"], catalog["first_run"])
self.assertEqual(template["catalog_ui"]["category_label"], catalog["catalog_ui"]["category_label"])
self.assertTrue(template["compatibility"]["automatic_install_candidate"])
self.assertEqual(CATALOG.schema_errors(template), [])
def test_the_least_an_oci_compose_says_is_its_image(self):
template = custom.template_from_oci_compose(json.dumps(
{"kind": "proxmenux.oci-template", "container_contract": {"image": {"reference": "traefik/whoami:latest"}}}))
self.assertEqual(template["container_contract"]["image"]["repository"], "traefik/whoami")
self.assertEqual(template["catalog_ui"]["title"], {"en_US": "whoami"})
self.assertEqual(template["container_contract"]["volumes"], [])
self.assertEqual(CATALOG.schema_errors(template), [])
def test_what_the_author_writes_is_kept(self):
document = json.loads(example("duplicati-official"))
document["container_contract"]["ports"] = [{"container_port": 5514, "protocol": "udp", "required": False}]
document["container_contract"]["volumes"][0].update(read_only=True, managed_volume={"default_size_gb": 40})
document["container_contract"]["volumes"].append(
{"container_path": "/exports", "installation_choice": ["managed-volume", "host-bind"]})
document["container_contract"]["volumes"].append({"container_path": "/state"})
document["catalog_ui"]["launch"] = {"scheme": "http", "port": None, "path": "/"}
document["first_run"] = {"endpoints": [], "credentials": []}
document["proxmox"]["defaults"] = {"memory_mb": 512}
template = custom.template_from_oci_compose(json.dumps(document))
port = template["container_contract"]["ports"][0]
self.assertEqual((port["container_port"], port["protocol"], port["required"]), (5514, "udp", False))
data, _, _, exports, state = template["container_contract"]["volumes"]
self.assertTrue(data["read_only"])
self.assertEqual(data["managed_volume"], {"backup": True, "default_size_gb": 40})
# A path its author gives as a host directory is offered as one; the
# others follow the rule of the catalog and stay in a container volume.
self.assertEqual(exports["installation_choice"], ["managed-volume", "host-bind"])
self.assertEqual(state["installation_choice"], ["managed-volume"])
self.assertEqual(data["installation_choice"], ["managed-volume"])
self.assertEqual(template["first_run"]["endpoints"], [])
self.assertIsNone(template["catalog_ui"]["launch"]["port"])
self.assertEqual(template["proxmox"]["defaults"]["memory_mb"], 512)
self.assertEqual(template["proxmox"]["defaults"]["cores"], 2)
def test_a_secret_left_empty_is_not_given_a_value(self):
template = custom.template_from_oci_compose(example("duplicati-official"))
self.assertEqual([item["example"] for item in template["container_contract"]["environment"]], ["", ""])
class FaultTests(unittest.TestCase):
def test_what_is_not_an_oci_compose_is_refused(self):
for text in ("services:\n app:\n image: x", "[1, 2]", json.dumps({"kind": "something-else"}),
json.dumps({"kind": "proxmenux.oci-template", "container_contract": {"image": {}}}),
json.dumps({"kind": "proxmenux.oci-template", "container_contract": "jellyfin"})):
with self.subTest(text=text[:30]):
with self.assertRaises(ConversionError):
custom.template_from_oci_compose(text)
def test_a_fault_says_where_it_is(self):
for edit, where in ((lambda d: d["container_contract"].update(ports="8200"), "ports"),
(lambda d: d["container_contract"]["volumes"].append({"path": "/x"}),
"container_contract/volumes/3/container_path: required"),
(lambda d: d["first_run"].update(endpoints=[{"port": 8200}]), "first_run/endpoints/0"),
(lambda d: d["catalog_ui"].update(launch={"port": "8200"}), "catalog_ui/launch"),
(lambda d: d["container_contract"]["image"].update(reference="not an image"), "not an image")):
with self.subTest(where=where):
document = json.loads(example("duplicati-official"))
edit(document)
with self.assertRaises(ConversionError) as raised:
custom.template_from_oci_compose(json.dumps(document))
self.assertIn(where, str(raised.exception))
def test_a_fault_is_reported_on_a_node_that_cannot_check_the_schema(self):
document = json.loads(example("duplicati-official"))
document["first_run"]["endpoints"] = [{"port": 8200}]
with patch.object(Catalog, "schema_errors", return_value=None):
custom.template_from_oci_compose(example("duplicati-official"))
with self.assertRaises(ConversionError) as raised:
custom.template_from_oci_compose(json.dumps(document))
self.assertIn("The oci-compose has errors:", str(raised.exception))
self.assertIn("first_run/endpoints/0/label: required", str(raised.exception))
def test_the_check_without_the_schema_library_accepts_every_file_of_the_catalog(self):
index = json.loads((ROOT / "catalog/index.json").read_text(encoding="utf-8"))["applications"]
for item in index:
if item.get("hidden") or not item.get("automatic_install_candidate"):
continue
self.assertEqual(CATALOG.shape_errors(CATALOG.compose(item["id"])), [], item["id"])
document = json.loads(catalog_file("mariadb"))
document["container_contract"]["ports"] = "3306"
document["container_contract"]["volumes"][0]["container_path"] = 7
self.assertEqual(CATALOG.shape_errors(document),
["container_contract/volumes/0/container_path: string", "container_contract/ports: array"])
class MenuTests(unittest.TestCase):
def test_the_review_says_what_an_oci_compose_adds(self):
text = custom.describe_oci_compose(CATALOG.compose("filebrowser-quantum"))
self.assertIn("/home/filebrowser/data/config.yaml", text)
self.assertIn("Install it only if it comes from a source you trust.", text)
self.assertTrue(text.startswith("\n\n"))
def test_the_menu_offers_an_oci_compose_from_a_file_or_an_address(self):
with tempfile.TemporaryDirectory() as folder:
path = Path(folder) / "oci-compose.json"
path.write_text(example("jellyfin-official"), encoding="utf-8")
ui = FakeUI(["oci-compose"], [str(path)])
text, origin = custom.read_definition(ui)
self.assertEqual(ui.offered[0][-2:], ["oci-compose", "oci-compose-url"])
self.assertEqual(origin, custom.OCI_COMPOSE)
self.assertEqual(json.loads(text)["catalog_ui"]["title"], {"en_US": "Jellyfin Official"})
def test_an_oci_compose_keeps_its_name_and_its_own_questions(self):
with tempfile.TemporaryDirectory() as folder:
path = Path(folder) / "oci-compose.json"
path.write_text(example("duplicati-official"), encoding="utf-8")
ui = FakeUI(["oci-compose", custom.DEFAULT_MODE], [str(path)])
with patch.object(custom, "registry_report", return_value=(True, "image found")), \
patch.object(custom, "install_template") as install, \
patch.object(custom, "_ask_secrets", side_effect=AssertionError("not for an oci-compose")), \
patch.object(custom, "_name", side_effect=AssertionError("not for an oci-compose")):
custom.explore(ui)
_, template, name, mode = install.call_args.args
self.assertEqual(name, "duplicati-official")
self.assertEqual(mode, custom.DEFAULT_MODE)
self.assertEqual(len(ui.asked), 1)
self.assertTrue(ui.reviews[0].startswith("This is what the oci-compose installs:"))
self.assertIn("Install it only if it comes from a source you trust.", ui.reviews[0])
def test_a_compose_file_is_read_as_before(self):
compose = "services:\n whoami:\n image: traefik/whoami:latest\n ports:\n - 8080:80\n"
with tempfile.TemporaryDirectory() as folder:
path = Path(folder) / "docker-compose.yml"
path.write_text(compose, encoding="utf-8")
ui = FakeUI(["file", None], [str(path), "whoami"])
with patch.object(custom, "registry_report", return_value=(True, "image found")), \
patch.object(custom, "install_template") as install:
custom.explore(ui)
install.assert_not_called()
self.assertNotIn("Install it only if it comes from a source you trust.", ui.reviews[0])
self.assertNotIn("oci-compose", ui.reviews[0])
if __name__ == "__main__":
unittest.main()
+15
View File
@@ -401,3 +401,18 @@ pmx_rebuild_dkms_after_kernel() {
fi
return 0
}
# Log2RAM syncs only the archived journal files, so the active file it found
# on disk when it was installed stays there. Restored at every boot, it makes
# journald continue an old sequence and delete its newest archives first.
# The sync removes that file from the disk copy. Returns 0 when it changes
# the given Log2RAM script.
pmx_log2ram_drop_stale_journal() {
local log2ram_bin="$1"
[[ -f "$log2ram_bin" ]] || return 1
grep -q -- '--exclude=journal/\*/\*' "$log2ram_bin" 2>/dev/null || return 1
grep -q -- '--delete-excluded' "$log2ram_bin" 2>/dev/null && return 1
[[ -e "${log2ram_bin}.proxmenux.bak" ]] || cp -a "$log2ram_bin" "${log2ram_bin}.proxmenux.bak"
sed '/optional_params+=("--exclude=journal\/\*\/\*")/a\ optional_params+=("--delete-excluded")' "$log2ram_bin" \
| pmx_write_file "$log2ram_bin"
}
+7 -1
View File
@@ -887,6 +887,11 @@ _update_existing_log2ram_auto() {
sed 's/rsync -aAXv /rsync -aXv --no-acls /g' "$log2ram_bin" | pmx_write_file "$log2ram_bin"
fi
if pmx_log2ram_drop_stale_journal "$log2ram_bin"; then
NECESSARY_REBOOT=1
msg_ok "$(translate "Log2RAM keeps the journal in order from the next reboot")"
fi
if dpkg-query -W -f='${Status}' proxmox-backup-server 2>/dev/null \
| grep -q 'install ok installed'; then
tmp_file="$(mktemp /etc/logrotate.d/.proxmox-backup-api.XXXXXX)" || return 1
@@ -995,7 +1000,7 @@ EOF
install_log2ram_auto() {
local FUNC_VERSION="1.6"
local FUNC_VERSION="1.7"
local existing_log2ram_bin=""
pmx_journal_context "install_log2ram_auto" "$FUNC_VERSION"
@@ -1113,6 +1118,7 @@ install_log2ram_auto() {
cp -a "$_l2r_bin" "${_l2r_bin}.proxmenux.bak"
pmx_edit_file "$_l2r_bin" 's/rsync -aAXv /rsync -aXv --no-acls /g'
fi
pmx_log2ram_drop_stale_journal "$_l2r_bin" || true
break
done
@@ -3026,6 +3026,11 @@ _update_existing_log2ram_custom() {
sed 's/rsync -aAXv /rsync -aXv --no-acls /g' "$log2ram_bin" | pmx_write_file "$log2ram_bin"
fi
if pmx_log2ram_drop_stale_journal "$log2ram_bin"; then
NECESSARY_REBOOT=1
msg_ok "$(translate "Log2RAM keeps the journal in order from the next reboot")"
fi
if dpkg-query -W -f='${Status}' proxmox-backup-server 2>/dev/null \
| grep -q 'install ok installed'; then
tmp_file="$(mktemp /etc/logrotate.d/.proxmox-backup-api.XXXXXX)" || return 1
@@ -3119,7 +3124,7 @@ EOF
configure_log2ram() {
local FUNC_VERSION="1.6"
local FUNC_VERSION="1.7"
local existing_log2ram_bin=""
pmx_journal_context "configure_log2ram" "$FUNC_VERSION"
# description: Install Log2RAM with user-chosen RAM size; prompts for size and SSD/M.2 awareness before applying.
@@ -3242,6 +3247,7 @@ configure_log2ram() {
cp -a "$_l2r_bin" "${_l2r_bin}.proxmenux.bak"
pmx_edit_file "$_l2r_bin" 's/rsync -aAXv /rsync -aXv --no-acls /g'
fi
pmx_log2ram_drop_stale_journal "$_l2r_bin" || true
break
done
@@ -5,6 +5,10 @@ import { DocPage } from "@/components/docs/DocBlocks"
const NAMESPACE = "docs.ociManager.customImage"
const URL = "https://proxmenux.com/docs/oci-manager/custom-image"
const LINKS = {
readmeLink: "https://github.com/MacRimi/ProxMenux/blob/main/oci/README.md#the-oci-compose-block-by-block",
}
export async function generateMetadata({ params }: { params: Promise<{ locale: string }> }): Promise<Metadata> {
const { locale } = await params
const t = await getTranslations({ locale, namespace: `${NAMESPACE}.meta` })
@@ -19,5 +23,5 @@ export async function generateMetadata({ params }: { params: Promise<{ locale: s
export default async function Page({ params }: { params: Promise<{ locale: string }> }) {
const { locale } = await params
setRequestLocale(locale)
return <DocPage locale={locale} namespace={NAMESPACE} minutes={6} />
return <DocPage locale={locale} namespace={NAMESPACE} minutes={7} links={LINKS} />
}
@@ -23,7 +23,7 @@
{
"id": "sources",
"title": "How the image is described",
"intro": "The first screen asks how the image is described. There are four ways to give a complete definition and a fifth that uses only the image name.",
"intro": "The first screen asks how the image is described. Four ways give a definition that is translated, a fifth uses only the image name, and two read an oci-compose that is already written.",
"blocks": [
{
"table": {
@@ -33,7 +33,9 @@
["Read its Compose file from a file of this host", "A path on the node, <code>/root/docker-compose.yml</code> by default. The file must exist and be at most 256 KiB."],
["Download its Compose file from an address", "An <code>http://</code> or <code>https://</code> address that serves the raw YAML. At most 256 KiB are downloaded."],
["Paste its docker run command in the terminal", "The command published by the project. Ports, environment, volumes, devices, capabilities, user, shared memory and the other supported options are translated."],
["Only the image reference, with no Compose file", "A reference such as <code>ghcr.io/user/application:latest</code>. The registry is queried and the OCI metadata of the image is kept."]
["Only the image reference, with no Compose file", "A reference such as <code>ghcr.io/user/application:latest</code>. The registry is queried and the OCI metadata of the image is kept."],
["Read an oci-compose from a file of this host", "A JSON file on the node, <code>/root/oci-compose.json</code> by default, of at most 2 MiB. It is not translated: it is checked and installed as it is written."],
["Download an oci-compose from an address", "An <code>http://</code> or <code>https://</code> address that serves the JSON file. At most 2 MiB are downloaded."]
]
}
},
@@ -41,7 +43,7 @@
"figure": {
"src": "/oci-manager/custom-source-menu.png",
"alt": "Menu that asks how an image that is not in the catalog is described",
"caption": "The five ways to describe an image that is not in the catalog."
"caption": "The ways to describe an image that is not in the catalog."
}
}
]
@@ -112,6 +114,43 @@
}
]
},
{
"id": "oci-compose",
"title": "An oci-compose written by hand",
"intro": "An oci-compose is the JSON file an application is installed from. The catalog keeps one for each of its applications, and the same format can be written by hand and installed with this option.",
"blocks": [
{
"p": "One written by hand states what its author decides: the image, its ports, variables and volumes, and what the installation asks. ProxMenux completes the rest when it reads the file. It can say what a Compose file cannot: the text of a question, a password generated during the installation, a GPU profile, the owner of each volume, the check of the first start or the credentials shown at the end."
},
{
"steps": {
"items": [
{
"title": "Check",
"body": "A file with errors is not installed. Each error names the field it is in, and a file that is not valid JSON reports the line and the column."
},
{
"title": "Review",
"body": "A valid file shows what it installs: image, web address, ports, persistent paths, the variables the installation asks for and the files it writes."
},
{
"title": "Install",
"body": "The installation is the one of any application of the catalog, with default or advanced settings."
}
]
}
},
{
"calloutWarning": {
"title": "A file that decides privileges",
"body": "An oci-compose decides what is mounted in the container and with which privileges. The review says it before installing: install only files that come from a source you trust."
}
},
{
"p": "The format is described block by block, with two complete examples, in the <readmeLink>README of the OCI engine</readmeLink>. The record of an installation can be read the same way: only the template it was installed from is used, and none of the answers of that installation."
}
]
},
{
"id": "limits",
"title": "Definitions that are not installed",
@@ -23,7 +23,7 @@
{
"id": "sources",
"title": "Cómo se describe la imagen",
"intro": "La primera pantalla pregunta cómo se describe la imagen. Hay cuatro formas de aportar una definición completa y una quinta que usa solo el nombre de la imagen.",
"intro": "La primera pantalla pregunta cómo se describe la imagen. Cuatro formas aportan una definición que se traduce, una quinta usa solo el nombre de la imagen y dos leen un oci-compose ya escrito.",
"blocks": [
{
"table": {
@@ -33,7 +33,9 @@
["Leer su archivo Compose desde un archivo de este host", "Una ruta del nodo, <code>/root/docker-compose.yml</code> por defecto. El archivo debe existir y ocupar como máximo 256 KiB."],
["Descargar su archivo Compose desde una dirección", "Una dirección <code>http://</code> o <code>https://</code> que sirva el YAML sin procesar. Se descargan como máximo 256 KiB."],
["Pegar su comando docker run en la terminal", "El comando que publica el proyecto. Se traducen puertos, entorno, volúmenes, dispositivos, capacidades, usuario, memoria compartida y el resto de opciones admitidas."],
["Solo la referencia de la imagen, sin archivo Compose", "Una referencia como <code>ghcr.io/usuario/aplicacion:latest</code>. Se consulta el registro y se conservan los metadatos OCI de la imagen."]
["Solo la referencia de la imagen, sin archivo Compose", "Una referencia como <code>ghcr.io/user/application:latest</code>. Se consulta el registro y se conservan los metadatos OCI de la imagen."],
["Leer un oci-compose desde un archivo de este host", "Un archivo JSON del nodo, <code>/root/oci-compose.json</code> por defecto, de 2 MiB como máximo. No se traduce: se comprueba y se instala tal como está escrito."],
["Descargar un oci-compose desde una dirección", "Una dirección <code>http://</code> o <code>https://</code> que sirva el archivo JSON. Se descargan como máximo 2 MiB."]
]
}
},
@@ -41,7 +43,7 @@
"figure": {
"src": "/oci-manager/custom-source-menu.png",
"alt": "Menú que pregunta cómo se describe una imagen que no está en el catálogo",
"caption": "Las cinco formas de describir una imagen que no está en el catálogo."
"caption": "Las formas de describir una imagen que no está en el catálogo."
}
}
]
@@ -112,6 +114,43 @@
}
]
},
{
"id": "oci-compose",
"title": "Un oci-compose escrito a mano",
"intro": "Un oci-compose es el archivo JSON desde el que se instala una aplicación. El catálogo guarda uno por cada aplicación, y el mismo formato puede escribirse a mano e instalarse con esta opción.",
"blocks": [
{
"p": "El que se escribe a mano indica lo que decide su autor: la imagen, sus puertos, variables y volúmenes, y lo que pregunta la instalación. ProxMenux completa el resto al leer el archivo. Puede decir lo que un archivo Compose no puede: el texto de una pregunta, una contraseña generada durante la instalación, un perfil de GPU, el propietario de cada volumen, la comprobación del primer arranque o las credenciales que se muestran al terminar."
},
{
"steps": {
"items": [
{
"title": "Comprobar",
"body": "Un archivo con errores no se instala. Cada error indica el campo en el que está, y un archivo que no es JSON válido indica la línea y la columna."
},
{
"title": "Revisar",
"body": "Un archivo válido muestra lo que instala: imagen, dirección web, puertos, rutas persistentes, las variables que pide la instalación y los archivos que escribe."
},
{
"title": "Instalar",
"body": "La instalación es la de cualquier aplicación del catálogo, con ajustes por defecto o avanzados."
}
]
}
},
{
"calloutWarning": {
"title": "Un archivo que decide privilegios",
"body": "Un oci-compose decide qué se monta en el contenedor y con qué privilegios. El resumen lo indica antes de instalar: instala solo archivos que vengan de una fuente de confianza."
}
},
{
"p": "El formato se describe bloque a bloque, con dos ejemplos completos, en el <readmeLink>README del motor OCI</readmeLink>. El registro de una instalación se lee de la misma forma: solo se usa la plantilla con la que se instaló, y ninguna de las respuestas de esa instalación."
}
]
},
{
"id": "limits",
"title": "Definiciones que no se instalan",
@@ -123,7 +162,7 @@
"Un Dockerfile sin imagen publicada: la imagen debe construirse y publicarse antes en un registro OCI.",
"Un archivo Compose con varios servicios: queda bloqueado porque describe más de una imagen. Esta opción instala un único contenedor.",
"<code>configs</code>, secretos externos o formatos de dispositivo que no pueden traducirse sin ambigüedad.",
"Sustituciones del shell como <code>$(comando)</code>: debe escribirse el valor resultante.",
"Sustituciones del shell como <code>$(command)</code>: debe escribirse el valor resultante.",
"Opciones de <code>docker run</code> no reconocidas y <code>--env-file</code>: se lee en su lugar un archivo Compose explícito."
]
}