Mirrors/ProxMenux
1
0
Fork 0
mirror of https://github.com/MacRimi/ProxMenux.git synced 2026-11-03 11:22:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #550 from jcastro/feature/oci-custom-compose-stacks

Browse Source 
feat(oci): import multi-service Compose files from Custom image
This commit is contained in:
MacRimi
2026-10-28 23:05:36 +01:00
committed by GitHub
parent 2d4b0cb746 2ab83c0b88
commit d27dc834db
71 changed files with 5677 additions and 176 deletions
Download Patch File Download Diff File Expand all files Collapse all files
oci/README.md
+77 -2
View File
@@ -209,7 +209,82 @@ The native path checks the saved configuration and installation identity, keeps
## From a Compose file or a docker run command to an oci-compose
The converter reads a Compose file with one service. A `docker run` command and an image reference alone are first written as a Compose file and follow the same path, so the three produce the same kind of oci-compose.
The converter reads Compose files with one or several services. Multi-service
files use the native stack compiler: each service runs in its own LXC, with
dependency ordering and a private network. The first service that publishes a
port is selected as the main service, unless `x-casaos.main` selects another one.
Every image is checked for registry access before installation; written tags,
digests and image entrypoints are preserved.
The file importer resolves an explicitly selected project `.env` and service
`env_file` files, then applies Compose interpolation and service overrides. It
does not silently import the host process environment. Missing required values
are requested before installation, and summaries hide secret values. An inline
Compose definition can also supply an environment file or enter values through
the assistant.
Published web services receive individual LAN addresses. The main web check uses
the same LAN URL shown to the user; private backend aliases stay on the internal
network. Internal services can use declared or inherited image health checks,
exposed TCP ports, or a clearly labelled process-running check. A running process
alone does not satisfy `service_healthy`.
For custom stacks, loopback public URLs in variables such as `PUBLIC_URL` or
`PENPOT_PUBLIC_URI` are mapped to the service publishing their original Compose
host port, using that service's LAN address and container port. Explicit external
URLs and internal loopback endpoints remain unchanged. Automatic public URL
mapping requires a static frontend address; with DHCP, provide the application's
explicit public URL instead. This lets every dependent service receive its final
public URL before its first start, including migration jobs.
Only names containing a `PUBLIC` or `EXTERNAL` segment and ending in `URL`,
`URI` or `ORIGIN` qualify for this automatic mapping. Other application-specific
variables, such as `NEXTAUTH_URL` or `ROOT_URL`, must contain the intended URL in
the supplied Compose or environment file. Changing a frontend IP later also
requires updating its application URLs.
Private members receive specific routes to their own stack's static frontend
addresses before the application starts. These routes stay inside the container's
network namespace and use the frontend's private address as the next hop. Native
update and recovery preserve their ownership records and rebuild the required
host configuration, including when a restore changes container IDs.
Repeated identical volume sources share one host directory, selected once;
separate configuration volumes stay independent. Host directories are not part
of a normal Proxmox container backup. Newly created private data volumes use the
imported image user unless an explicit volume owner or PUID/PGID overrides it.
SELinux `z`/`Z` labels are reported as not applied on a non-enforcing host; an
enforcing host is refused rather than silently losing their protection.
An explicit Compose `stop_signal` overrides the image's stop signal through
`lxc.signal.halt`. Without it, the image signal stays unchanged, including when
Compose replaces its entrypoint. A worker sharing a web image may therefore
need its own declared stop signal. Native updates abort safely if a service
does not stop; they do not silently force termination to replace its disks.
Some programs ignore termination signals when running as PID 1. Setting a signal
does not add an init process or guarantee graceful shutdown; an explicit
`SIGKILL` requests forced termination, not a graceful stop.
A dependency declared with `service_completed_successfully` runs as a one-shot
job. Its original command must finish with a verified exit code of zero before
the dependent service starts. Such jobs are excluded from onboot and watchdog
restarts. Update and restore invalidate the completion proof when the image,
runtime configuration or recovery identity changes. A migration update is
refused when database or unclassified backend data lives in host directories
that the transaction cannot safely restore.
The native exit-status monitor currently accepts LXC 7.0.0, the verified protocol
version. Other versions refuse one-shot execution rather than treating a stopped
container as proof of success.
Before each native stack backup, the engine writes the validated original
installation record into every stopped member and reads it back. A missing or
unverifiable copy aborts the backup, so database and worker archives do not
silently lose the metadata needed to register the restored stack.
The bridges are local to the installation node; they are not a cluster SDN.
Unsupported network and runtime semantics are reported before installation.
Support for several services does not imply complete Docker Compose compatibility.
A `docker run` command or image reference is converted to Compose first and
follows the same importer.
Starting from this definition:
@@ -251,7 +326,7 @@ each key becomes a field of the oci-compose:
| `stop_grace_period` | `proxmox.defaults.shutdown_timeout_seconds` | The time a shutdown waits |
| `labels`, `logging` | Kept in `container_contract.original_compose` only | Not applied: they are read by Docker tools |
What has no translation stops the installation and is named: `pid: host`, a Compose file with several services given to this option, `configs`, external secrets, a `build` section without a published image, and shell substitutions such as `$(command)`.
What has no translation stops the installation and is named: `pid: host`, `configs`, external secrets, a `build` section without a published image, and shell substitutions such as `$(command)`.
Besides those fields, the converter writes the blocks that describe the translation itself: where the definition came from, the keys it understood, the adaptations it applied and the state of each validation. They are listed in [Blocks ProxMenux writes](#blocks-proxmenux-writes).
oci/remote/allocate_private_network.py
+49 -8
View File
@@ -16,6 +16,47 @@ def command_output(*command: str) -> str:
return result.stdout
def _interface_files() -> list[Path]:
"""Include pending Proxmox network edits and down interface definitions."""
paths = [Path('/etc/network/interfaces'), Path('/etc/network/interfaces.new')]
paths.extend(Path('/etc/network/interfaces.d').glob('*'))
return [path for path in paths if path.is_file()]
def _configured_networks(text: str) -> list[ipaddress.IPv4Network]:
networks = []
addresses = []
netmask = None
inet = False
def collect():
for address in addresses:
value = address if '/' in address else address+'/'+(netmask or '32')
try:
interface = ipaddress.ip_interface(value)
if interface.version == 4:
networks.append(interface.network)
except ValueError:
continue
for line in text.splitlines():
words = line.partition('#')[0].split()
if not words:
continue
if words[0] == 'iface':
collect()
addresses = []
netmask = None
inet = len(words) >= 4 and words[2] == 'inet'
elif inet and len(words) >= 2:
if words[0] == 'address':
addresses.append(words[1])
elif words[0] == 'netmask':
netmask = words[1]
collect()
return networks
def existing_bridges() -> set[str]:
bridges: set[str] = set()
for line in command_output("ip", "-o", "link", "show").splitlines():
@@ -25,15 +66,13 @@ def existing_bridges() -> set[str]:
for path in Path("/etc/pve/lxc").glob("*.conf"):
text = path.read_text(encoding="utf-8", errors="ignore")
bridges.update(re.findall(r"(?:^|,)bridge=([^,\s]+)", text, re.MULTILINE))
interface_paths = [Path("/etc/network/interfaces")]
interface_paths.extend(Path("/etc/network/interfaces.d").glob("*"))
for path in interface_paths:
if not path.is_file():
continue
for path in _interface_files():
text = path.read_text(encoding="utf-8", errors="ignore")
bridges.update(
re.findall(r"^(?:auto|iface)\s+(vmbr\d+)\b", text, re.MULTILINE)
)
for line in text.splitlines():
words = line.partition('#')[0].split()
if words and words[0] in {'auto', 'iface'}:
names = words[1:] if words[0] == 'auto' else words[1:2]
bridges.update(name for name in names if re.fullmatch(r'vmbr\d+', name))
return bridges
@@ -54,6 +93,8 @@ def existing_networks() -> list[ipaddress.IPv4Network]:
networks.append(ipaddress.ip_interface(address).network)
except ValueError:
pass
for path in _interface_files():
networks.extend(_configured_networks(path.read_text(encoding='utf-8', errors='ignore')))
return networks
oci/remote/install_generic_stack.py
+264 -46
View File
@@ -26,6 +26,7 @@ import xml.etree.ElementTree as ET
from allocate_private_network import allocate_network, existing_bridges, existing_networks
import oci_instances
import oci_nested_mounts
import oci_stack_public_routes
from oci_ui import translate, msg_info, msg_ok, msg_warn, msg_error, msg_info2, stop_spinner, log
HERE = Path(__file__).resolve().parent
@@ -96,7 +97,7 @@ def report_error(text, tail=True):
def persist_instances(deployment, services, primary_id=None):
template = json.loads(Path(sys.argv[1]).read_text()) if primary_id is not None else {}
with oci_instances.locked(oci_instances.ROOT):
with oci_instances.locked(oci_instances.ROOT, timeout=60):
if primary_id is not None:
oci_instances.save_assembly(oci_instances.ROOT, primary_id, template, deployment, services)
oci_instances.resume_assembly(oci_instances.ROOT, primary_id)
@@ -244,6 +245,122 @@ def relay_child_output(stream):
return result
def assign_health_address(service):
"""Bind a generated dependency probe to its allocated private address."""
check = service['healthcheck']
if check['type'] == 'http':
endpoint = check['endpoint']
check['url'] = f"{endpoint['scheme']}://{service['ip']}:{endpoint['port']}{endpoint['path']}"
elif check['type'] == 'tcp':
check['host'] = service['ip']
def public_bindings(services):
"""Validate compiler-owned targets before any host/container mutation."""
names = {service['name']: service for service in services}
result = []
failure = translate('Invalid stack contract')
for service in services:
bindings = service.get('public_environment_bindings', {})
if not isinstance(bindings, dict):
raise RuntimeError(failure)
environment = service['deployment'].get('environment', [])
for key, binding in bindings.items():
if (not isinstance(key, str) or not re.fullmatch(r'[A-Za-z_][A-Za-z0-9_]*', key)
or not isinstance(binding, dict)
or set(binding) != {'target_service', 'container_port', 'value'}
or not isinstance(binding.get('target_service'), str)):
raise RuntimeError(failure)
target = names.get(binding['target_service'])
value = binding.get('value')
if (target is None or not (target.get('frontend') or target.get('main'))
or type(binding.get('container_port')) is not int
or not 0 < binding['container_port'] < 65536
or not isinstance(value, str) or value.count('@STACK_LAN_IP@') != 1
or any(char in value for char in '\r\n')
or sum(entry.get('name') == key for entry in environment) != 1):
raise RuntimeError(failure)
result.append((service, key, binding, target))
if any(target.get('frontend_ipv4') == 'dhcp' for _, _, _, target in result):
raise RuntimeError(translate('Automatic public URLs require static frontend addresses or explicit URLs'))
return result
def _set_public_value(service, key, value):
for entry in service['deployment']['environment']:
if entry['name'] == key:
entry['value'] = value
return
raise RuntimeError(translate('Invalid stack contract'))
def prepare_public_environment(services):
"""Known static destinations are final before any application's first start."""
for service, key, binding, target in public_bindings(services):
address = target.get('frontend_ipv4')
try:
address = ipaddress.ip_interface(address).ip
if address.version != 4:
raise ValueError()
except (ValueError, TypeError) as error:
raise RuntimeError(translate('Invalid stack contract')) from error
_set_public_value(service, key, binding['value'].replace('@STACK_LAN_IP@', str(address)))
def install_public_routes(services, primary_id, bridge):
"""Private members can reach only this stack's own static frontend IPs."""
frontends = [service for service in services if service.get('main') or service.get('frontend')]
targets = {}
for service in frontends:
address = service.get('frontend_ipv4')
if not address or address == 'dhcp':
continue
destination = str(ipaddress.IPv4Interface(address).ip)
if destination in targets:
raise RuntimeError(translate('Duplicate stack frontend address'))
targets[destination] = service
if not targets:
return
records = {service['vmid']: oci_instances.read(oci_instances.ROOT, service['vmid']) for service in services}
stack_id = records[primary_id]['installation_id']
for service in services:
if service.get('main') or service.get('frontend'):
continue
vmid = service['vmid']
identity = records[vmid]['installation_id']
spec = {'schema': 1, 'stack_id': stack_id, 'primary_vmid': primary_id,
'vmid': vmid, 'installation_id': identity, 'private_bridge': bridge,
'private_address': service['ip'] + '/24', 'routes': [
{'destination': destination, 'via': target['ip'], 'target_vmid': target['vmid'],
'target_installation_id': records[target['vmid']]['installation_id']}
for destination, target in sorted(targets.items())]}
plan = service['deployment']
plan['stack_public_routes'] = spec
oci_stack_public_routes.validate(plan, vmid, identity)
config = Path(f'/etc/pve/lxc/{vmid}.conf')
text = config.read_text()
# Never replace another include or accept a stale route contract.
oci_stack_public_routes.check(text, {}, vmid, require_host_assets=False)
line = oci_stack_public_routes.install(plan, vmid, identity)
config.write_text(text.rstrip('\n') + '\n' + line + '\n')
oci_stack_public_routes.check(config.read_text(), plan, vmid, identity)
def bind_completed_jobs(services):
"""All members and private aliases exist before a job can be authorized."""
from oci_compose_jobs import job_spec_from_record
for service in services:
check = service['healthcheck']
if check.get('type') != 'completed':
continue
vmid = int(service['vmid'])
run('pct', 'set', vmid, '--onboot', '0')
service['deployment']['onboot'] = False
record = oci_instances.read(oci_instances.ROOT, vmid)
check['job'] = job_spec_from_record(record, run('pct', 'config', vmid),
int(check.get('timeout_seconds', 300)))
def lan_access_urls(services, subnet, strict=True):
urls = []
for service in services:
@@ -271,15 +388,43 @@ def lan_access_urls(services, subnet, strict=True):
return urls
def _last_int_env(service, names):
"""Last matching PUID/PGID-style value from deployment.environment,
or 0 if absent/unparsable — mirrors install_oci.sh's
`[.environment[]? | select(.name == ...) | .value] | last // "0"`
plus its numeric-guard `[[ $X =~ ^[0-9]+$ ]] || X=0`."""
def wait_main_web(service, subnet):
"""Check the advertised frontend, preserving private IPs for backend aliases.
A service may legitimately refuse HTTP requests whose Host is its private
stack address. The endpoint shown to the user is the readiness contract.
"""
deadline = time.monotonic() + min(120, int(service['healthcheck']['timeout_seconds']))
url = None
while time.monotonic() < deadline:
try:
found = lan_access_urls([service], subnet)
if found:
url = found[0]['url']
break
except (RuntimeError, OSError, subprocess.SubprocessError):
pass
time.sleep(1)
if url is None:
raise RuntimeError(translate('No LAN address was obtained'))
service['healthcheck']['url'] = url
wait_web(service, url)
return url
def _last_int_env(service, names, default=0):
"""Use the last explicit environment override, falling back only if absent.
Invalid explicit values retain the installer's existing zero fallback.
"""
value = None
found = False
for entry in service['deployment'].get('environment', []):
if entry.get('name') in names:
found = True
value = entry.get('value')
if not found:
return default
try:
parsed = int(value)
except (TypeError, ValueError):
@@ -287,28 +432,78 @@ def _last_int_env(service, names):
return parsed if parsed >= 0 else 0
def resolve_volume_owner(service, owner_strategy):
"""(uid, gid) for a given volume_preparations owner_strategy,
mirroring install_oci.sh's HOST_ROOT_UID/HOST_ROOT_GID (mapped-root)
and HOST_BIND_UID/HOST_BIND_GID (mapped-application-user) — these
are deliberately DIFFERENT values, not the same computation:
def runtime_volume_owner(service, root=None):
"""Container IDs from the OCI import, with image USER as a fallback.
- mapped-root: 100000:100000 for an unprivileged container, 0:0
otherwise. Never depends on PUID/PGID or volume_owner (matches
install_oci.sh's HOST_ROOT_UID/HOST_ROOT_GID, which are pinned
constants that do not read CONTAINER_PUID/CONTAINER_PGID at all).
- mapped-application-user: the container's own PUID/PGID (last of
PUID/USER_ID/UID and PGID/GROUP_ID/GID in deployment.environment,
overridden by the template's volume_owner.uid/gid when present —
matches install_oci.sh's IMAGE_VOLUME_UID/GID override of
CONTAINER_PUID/CONTAINER_PGID before the offset is applied),
offset by 100000 for an unprivileged container (matches
HOST_BIND_UID/HOST_BIND_GID). volume_owner only ever overrides
this application-user value, never mapped-root's.
Proxmox has already resolved a named USER and Compose user override into
lxc.init.uid/gid. No ownership is changed here. The mounted image account
files are only needed on versions that omit one of those numeric fields.
"""
vmid = service.get('vmid')
config = {}
if type(vmid) is int and vmid >= 100:
from oci_installation_state import parse_config
config = parse_config(run('pct', 'config', vmid).encode())
resolved = {}
for name, key in (('uid', 'lxc.init.uid'), ('gid', 'lxc.init.gid')):
value = config.get(key)
if value is not None:
if not re.fullmatch(r'[0-9]+', value):
raise RuntimeError(translate('The imported OCI user or group is not numeric'))
resolved[name] = int(value)
if len(resolved) == 2:
return resolved['uid'], resolved['gid']
image = (service.get('observed') or {}).get('image') or {}
if not image and type(vmid) is int and vmid >= 100:
try:
record = oci_instances.read(oci_instances.ROOT, vmid)
image = (record.get('observed') or {}).get('image') or {}
except (OSError, ValueError, KeyError):
pass
declared = str((image.get('defaults') or {}).get('User') or '')
user, separator, group = declared.partition(':')
accounts, groups = {}, {}
if root is not None:
root = Path(root)
for filename, target in (('passwd', accounts), ('group', groups)):
path = root / 'etc' / filename
if path.resolve().is_relative_to(root.resolve()) and path.is_file() and not path.is_symlink():
for line in path.read_text(errors='replace').splitlines():
fields = line.split(':')
if len(fields) >= (4 if filename == 'passwd' else 3) and fields[2].isdigit():
if filename == 'passwd' and fields[3].isdigit():
target[fields[0]] = (int(fields[2]), int(fields[3]))
elif filename == 'group':
target[fields[0]] = int(fields[2])
if not user:
uid, gid = 0, 0
elif user.isdigit():
uid = int(user)
gid = next((value[1] for value in accounts.values() if value[0] == uid), 0)
elif user in accounts:
uid, gid = accounts[user]
elif 'uid' in resolved:
uid = resolved['uid']
gid = next((value[1] for value in accounts.values() if value[0] == uid), 0)
else:
raise RuntimeError(translate('The imported OCI user or group is not numeric'))
if separator and group:
if group.isdigit():
gid = int(group)
elif group in groups:
gid = groups[group]
elif 'gid' not in resolved:
raise RuntimeError(translate('The imported OCI user or group is not numeric'))
return resolved.get('uid', uid), resolved.get('gid', gid)
Any value that cannot be parsed as a non-negative integer is
treated as 0, matching install_oci.sh's `[[ ... =~ ^[0-9]+$ ]] ||
...=0` guard.
def resolve_volume_owner(service, owner_strategy, imported_owner=None):
"""Resolve fresh volume ownership without changing existing host directories.
Mapped root remains fixed. Application ownership starts with the actual
OCI import user, then honors explicit PUID/PGID aliases and the recipe's
declared volume_owner override. All values are container IDs until the
standard unprivileged offset is applied exactly once.
"""
unprivileged = bool(service['deployment']['security']['unprivileged'])
if owner_strategy == 'mapped-root':
@@ -316,8 +511,9 @@ def resolve_volume_owner(service, owner_strategy):
if owner_strategy != 'mapped-application-user':
raise RuntimeError(f"{translate('Unsupported owner strategy:')} {owner_strategy}")
puid = _last_int_env(service, ('PUID', 'USER_ID', 'UID'))
pgid = _last_int_env(service, ('PGID', 'GROUP_ID', 'GID'))
imported_uid, imported_gid = (runtime_volume_owner(service) if imported_owner is None else imported_owner)
puid = _last_int_env(service, ('PUID', 'USER_ID', 'UID'), imported_uid)
pgid = _last_int_env(service, ('PGID', 'GROUP_ID', 'GID'), imported_gid)
volume_owner = service.get('template', {}).get('proxmox', {}) \
.get('installer_profile', {}).get('volume_owner') or {}
@@ -338,7 +534,7 @@ def resolve_volume_owner(service, owner_strategy):
return (puid + offset, pgid + offset)
def apply_volume_preparation(service, mount, target):
def apply_volume_preparation(service, mount, target, imported_owner=None):
"""Declarative volume_preparations for one already-attached (but
still empty/freshly-mounted) mount, mirroring install_oci.sh's
apply_installer_profile() semantics (only_when_mount_type /
@@ -381,7 +577,7 @@ def apply_volume_preparation(service, mount, target):
if lost.is_dir() and not lost.is_symlink():
lost.rmdir()
owner_strategy = match.get('owner_strategy')
owner = resolve_volume_owner(service, owner_strategy)
owner = resolve_volume_owner(service, owner_strategy, imported_owner)
log(LOG, f"Volume prepared before the first start: {mount['container_path']}")
return True, owner
@@ -390,6 +586,7 @@ def attach_mounts(service, temporary):
"""Populate new managed volumes from the image, preserving its ownership."""
vmid = service['vmid']
root = Path(f'/var/lib/lxc/{vmid}/rootfs')
imported_owner = None
for index, mount in enumerate(service['deployment']['mounts']):
target = root / mount['container_path'].lstrip('/')
seed = temporary / f'seed-{vmid}-{index}'
@@ -398,6 +595,8 @@ def attach_mounts(service, temporary):
try:
if not target.resolve().is_relative_to(root.resolve()) or target.is_symlink():
raise RuntimeError(translate('Unsafe volume path'))
if imported_owner is None:
imported_owner = runtime_volume_owner(service, root)
if target.is_dir():
stat = target.stat()
owner = (stat.st_uid,stat.st_gid,stat.st_mode & 0o777)
@@ -421,6 +620,12 @@ def attach_mounts(service, temporary):
if not source.is_absolute() or ',' in str(source) or '\n' in str(source):
raise RuntimeError(translate('Invalid shared path'))
if not source.exists():
shared_owner = mount.get('host_owner')
if shared_owner is not None:
if (not isinstance(shared_owner, list) or len(shared_owner) != 2
or any(type(value) is not int or not 0 <= value <= 2**32-2 for value in shared_owner)):
raise RuntimeError(translate('Invalid shared path'))
owner = (*shared_owner, owner[2])
source.mkdir(parents=True)
os.chown(source,*owner[:2])
source.chmod(owner[2])
@@ -431,7 +636,7 @@ def attach_mounts(service, temporary):
if mount['type']=='managed-volume':
run('pct','mount',vmid)
try:
applied, declared_owner = apply_volume_preparation(service, mount, target)
applied, declared_owner = apply_volume_preparation(service, mount, target, imported_owner)
if not applied:
lost=target/'lost+found'
if lost.is_dir() and not lost.is_symlink():
@@ -466,6 +671,13 @@ def wait_web(primary, url):
try:
with opener.open(url,timeout=4) as response:
if response.status<400:
check = primary['healthcheck']
if check.get('type') == 'exec':
remaining = max(1, deadline-time.monotonic())
process = subprocess.run(['pct', 'exec', str(primary['vmid']), '--', *check['argv']],
capture_output=True, timeout=min(30, remaining), check=False)
if process.returncode != 0:
raise RuntimeError('Health check has not succeeded yet')
return
except Exception:
pass
@@ -738,6 +950,12 @@ def finish_independent_suite(services, subnet):
return result
def check_selinux_labels(template, enforcing_path=Path('/sys/fs/selinux/enforce')):
if (template.get('proxmox', {}).get('custom_compose_selinux_labels')
and enforcing_path.is_file() and enforcing_path.read_text().strip() == '1'):
raise RuntimeError(translate('SELinux volume labels cannot be ignored while SELinux is enforcing.'))
def main():
deployment = json.loads(Path(sys.argv[2]).read_text())
if deployment.get('deployment_kind') != 'generic-multi-lxc-stack':
@@ -750,10 +968,12 @@ def main():
return
if os.geteuid()!=0:
raise RuntimeError(translate('The installer must run as root on Proxmox VE'))
check_selinux_labels(json.loads(Path(sys.argv[1]).read_text()))
init_log(deployment.get('stack_name') or 'stack')
services = copy.deepcopy(deployment['services'])
independent = bool(deployment.get('suite_arr'))
primary = None if independent else next(s for s in services if s['main'])
prepare_public_environment(services)
created = []
bridge_created = False
lifecycle = None
@@ -779,7 +999,7 @@ def main():
s['vmid'] = base+s['offset']
s['ip'] = str(subnet.network_address+30+s['offset'])
for alias in s['aliases']:
if not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9_.-]*',alias):
if not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9_.-]{0,252}',alias):
raise RuntimeError(translate('Invalid service alias'))
aliases.append({'hostname':alias,'address':s['ip']})
if not independent:
@@ -805,7 +1025,7 @@ def main():
results = {}
for order,s in enumerate(services,1):
plan = s['deployment']
s['public_environment'] = {e['name']:e['value'] for e in plan['environment'] if '@STACK_LAN_IP@' in e['value']}
s['public_environment'] = {e['name']:e['value'] for e in plan['environment'] if '@STACK_LAN_IP@' in e['value'] and e['name'] not in s.get('public_environment_bindings', {})}
for e in plan['environment']:
e['value'] = e['value'].replace('@STACK_LAN_IP@',s['ip'])
plan['vmid'] = s['vmid']
@@ -831,17 +1051,19 @@ def main():
if s['main'] or s.get('frontend'):
address = access_address(s.get('frontend_ipv4') or 'dhcp', deployment['network'].get('frontend_gateway'))
run('pct','set',s['vmid'],'--net1',f"name=eth1,bridge={deployment['network']['frontend_bridge']},{address},host-managed=1,firewall=1,type=veth")
if s['healthcheck']['type']=='http':
endpoint = s['healthcheck']['endpoint']
s['healthcheck']['url'] = f"{endpoint['scheme']}://{s['ip']}:{endpoint['port']}{endpoint['path']}"
assign_health_address(s)
# Existing CT/network configurations now reserve every allocated ID and subnet.
# Never retain the allocation lock while startup/hooks wait for another registry operation.
fcntl.flock(lock,fcntl.LOCK_UN)
if independent:
fcntl.flock(lock,fcntl.LOCK_UN)
result = finish_independent_suite(services,subnet)
persist_instances(deployment, services)
refresh_notes(services)
result.update(completion_notes=list(deployment.get('completion_notes',[])), log=LOG)
print('PROXMENUX_RESULT='+base64.b64encode(json.dumps(result).encode()).decode(),flush=True)
return
install_public_routes(services, primary_id, bridge)
bind_completed_jobs(services)
hook_spec = Path(tmp)/'lifecycle.json'
write_json(hook_spec,{'schema':1,'stack':deployment['stack_name'],'dependencies':[
{'vmid':s['vmid'],'label':s['name'],'healthcheck':s['healthcheck']} for s in services if not s['main'] and not s.get('deferred_setup')]})
@@ -849,20 +1071,15 @@ def main():
msg_info(translate('Installing the stack startup hook...'))
run('bash',HERE/'stack_dependency_hook.sh','--install',primary_id,hook_spec,capture=False)
msg_ok(translate('Stack startup hook installed'))
fcntl.flock(lock,fcntl.LOCK_UN)
msg_info(translate('Starting the main container and its dependencies...') if len(services) > 1
else translate('Starting the container...'))
run('pct','start',primary_id,capture=False,timeout=600)
msg_ok(f"{translate('Container started')}: CT {primary_id} ({primary['name']})")
endpoint = primary['healthcheck']['endpoint']
url = f"{endpoint['scheme']}://{primary['ip']}:{endpoint['port']}{endpoint['path']}"
msg_info(translate('Waiting for the application to respond...'))
wait_web(primary,url)
addresses=run('lxc-info','-n',primary_id,'-iH').splitlines()
lan = next((a for a in addresses if re.fullmatch(r'\d+\.\d+\.\d+\.\d+',a) and ipaddress.ip_address(a) not in subnet),None)
if not lan:
raise RuntimeError(translate('No LAN address was obtained'))
public_url = f"{endpoint['scheme']}://{lan}:{endpoint['port']}{endpoint['path']}"
public_url = wait_main_web(primary, subnet)
url = public_url
lan = urllib.parse.urlsplit(public_url).hostname
msg_ok(f"{translate('Application responding:')} {public_url}")
if primary['public_environment']:
msg_info(translate('Applying the LAN address to the application URLs...'))
@@ -883,7 +1100,8 @@ def main():
refresh_notes(services)
# The credentials of the main service come from its own installation.
result.update(vmid=primary_id,ip=lan,stack_vmids={s['name']:s['vmid'] for s in services},
urls=[{'label':'Web UI','url':public_url}],
urls=(lan_access_urls(services, subnet) if sum(bool(s['main'] or s.get('frontend')) for s in services) > 1
else [{'label':'Web UI','url':public_url}]),
credentials=result.get('credentials') or [])
result.update(completion_notes=[note.replace('{main_vmid}',str(primary_id))
for note in deployment.get('completion_notes', [])], log=LOG)
oci/remote/install_oci.sh
+16 -1
View File
@@ -994,6 +994,11 @@ prepare_nvidia_driver_links() {
source "$SCRIPT_DIR/oci_nvidia_setup.sh"
oci_valid_extra_hostname() {
# Compose network aliases may contain underscores; each entry stays one hosts-file token.
[[ ${#1} -le 253 && $1 =~ ^[A-Za-z0-9][A-Za-z0-9_.-]*$ ]]
}
apply_extra_hosts() {
local count rootfs hosts_file temporary address hostname bridge_address failed=0
count=$(jq '.extra_hosts? // [] | length' "$DEPLOYMENT_FILE")
@@ -1021,7 +1026,7 @@ apply_extra_hosts() {
|| { oci_log "Could not resolve host-gateway on $BRIDGE"; failed=1; break; }
address=$bridge_address
fi
[[ $hostname =~ ^[A-Za-z0-9][A-Za-z0-9.-]*$ ]] \
oci_valid_extra_hostname "$hostname" \
|| { oci_log "Invalid extra hostname: $hostname"; failed=1; break; }
[[ $address =~ ^[0-9A-Fa-f:.]+$ ]] \
|| { oci_log "Invalid extra host address: $address"; failed=1; break; }
@@ -2102,6 +2107,16 @@ apply_rlimits
apply_host_monitor
apply_kept_settings
# Recreated stack members retain their owned pre-init routes. Append rather
# than replacing lxc.include: other managed runtime includes must survive.
if jq -e '.stack_public_routes != null' "$DEPLOYMENT_FILE" >/dev/null; then
ROUTE_INCLUDE=$(python3 "${SCRIPT_DIR}/oci_stack_public_routes.py" install "$VMID" \
--deployment "$DEPLOYMENT_FILE" --installation-id "$INSTANCE_ID") \
|| die "$(translate "Managed public route installation failed")"
[[ -n "$ROUTE_INCLUDE" ]] || die "$(translate "Managed public route include is missing")"
grep -Fxq "$ROUTE_INCLUDE" "$CONF" || printf '%s\n' "$ROUTE_INCLUDE" >>"$CONF"
fi
while IFS= read -r REPAIR_ENCODED; do
[[ -n $REPAIR_ENCODED ]] || continue
run_pre_start_repair "$(printf '%s' "$REPAIR_ENCODED" | base64 -d)"
oci/remote/oci_carried_record.py
+71 -1
View File
@@ -31,7 +31,7 @@ import sys
import uuid
import oci_instances as instances
from oci_installation_state import sha
from oci_installation_state import sha, parse_config
from oci_ui import translate
DIRECTORY = '.proxmenux'
@@ -40,6 +40,7 @@ KIND = 'proxmenux.oci-carried-record'
STAMP = 'carried.sha256'
CONFIG_CACHE = 'carried-config.json'
LOCAL_CONFIGS = Path('/etc/pve/lxc')
ROOTFS = Path('/var/lib/lxc')
LIMIT = 16 * 1024 * 1024
STACK_CONTRACT = '/etc/pve/priv/proxmenux-stack-{}.json'
NVIDIA_HOOK = re.compile(r'/usr/local/lib/proxmenux/oci/nvidia-mount-([a-f0-9]{64})\.sh')
@@ -400,6 +401,75 @@ def carry(root, vmid, mount_stopped=True, verify=False, adopted=False):
return outcome
def carry_backup(root, vmid, original_record, validated_config_text):
"""Require a verified original record inside a stopped CT before vzdump.
A coordinated transaction has already marked the host registry pending.
The explicit original is the validated plan record, never that pending
record or an untrusted copy from inside the guest. Unlike best-effort
carry(), any missing copy, configuration drift or I/O failure aborts backup.
"""
failure = translate('The carried record could not be verified before the backup')
if isinstance(validated_config_text, bytes):
validated_config_text = validated_config_text.decode('utf-8')
if (not isinstance(original_record, dict) or original_record.get('vmid') != vmid
or original_record.get('status') != 'installed'
or original_record.get('pending_transaction') or original_record.get('pending_stack_transaction')
or not isinstance(validated_config_text, str)):
raise ValueError(failure)
identity = original_record.get('installation_id')
if not identity or instances.identity(validated_config_text.encode()) != identity:
raise ValueError(failure)
def checked_stopped():
status = _run('pct', 'status', str(int(vmid)))
if status.returncode or status.stdout.strip() != 'status: stopped':
raise ValueError(failure)
def checked_configuration():
result = _run('pct', 'config', str(int(vmid)))
if result.returncode or instances.identity(result.stdout.encode()) != identity:
raise ValueError(failure)
def projection(text):
if re.search(r'^\[[^]\r\n]+\]\s*$', text, re.MULTILINE):
raise ValueError(failure)
values = parse_config(text.encode())
# The identity is checked independently; notes and maintenance
# locks do not change the image, mounts, network, user or devices.
values.pop('description', None)
lock = values.pop('lock', None)
if lock not in (None, 'backup', 'mounted'):
raise ValueError(failure)
return values
if projection(result.stdout) != projection(validated_config_text):
raise ValueError(failure)
return result.stdout
checked_stopped()
config = checked_configuration()
if _host_directory_missing(vmid) or _run('pct', 'mount', str(int(vmid))).returncode:
raise ValueError(failure)
try:
checked_stopped()
checked_configuration()
rootfs = ROOTFS / str(int(vmid)) / 'rootfs'
value = bundle(original_record, config)
value['saved_at'] = datetime.datetime.now(datetime.timezone.utc).isoformat()
value['generation'] = uuid.uuid4().hex
write_copy(rootfs, value, mapped_root(config))
copied = read_copy(rootfs)
if copied != value or digest(copied) != digest(value):
raise ValueError(failure)
checked_stopped()
checked_configuration()
finally:
if _run('pct', 'unmount', str(int(vmid))).returncode:
raise ValueError(failure)
instances.write(instances.location(root, vmid).parent / STAMP,
{'digest': digest(value), 'generation': value['generation']})
return {'status': 'carried', 'generation': value['generation'], 'digest': digest(value)}
def registered(root):
if not root.is_dir():
return []
oci/remote/oci_compose_jobs.py
+479
View File
@@ -0,0 +1,479 @@
#!/usr/bin/env python3
"""Verify native OCI one-shot completion using authenticated raw LXC events.
No image wrapper, shell injection or STOPPED-state inference is used. The
supported monitor ABI was verified against upstream LXC v7.0.0 monitor.h,
monitor.c and tools/lxc_monitor.c. The textual CLI loses signal information.
This standalone helper is copied beside the persistent dependency hook.
"""
from __future__ import annotations
import argparse
import contextlib
import datetime
import fcntl
import hashlib
import json
import os
from pathlib import Path
import re
import select
import socket
import stat
import struct
import subprocess
import tempfile
import time
import urllib.parse
import uuid
STATE_ROOT = Path('/var/lib/proxmenux/oci/jobs')
INSTANCE_ROOT = Path('/usr/local/share/proxmenux/oci/instances')
PCT = '/usr/sbin/pct'
MONITOR = '/usr/bin/lxc-monitor'
LXC_PATH = '/var/lib/lxc'
FRAME = struct.Struct('@i256si')
SUPPORTED_VERSIONS = frozenset({'7.0.0'})
DIGEST = re.compile(r'sha256:[0-9a-f]{64}\Z')
class JobError(RuntimeError):
pass
def _vmid(value):
if type(value) is not int or not 100 <= value <= 999999999:
raise JobError('Invalid OCI job VMID')
return value
def _root():
if os.geteuid() != 0:
raise JobError('OCI job execution requires root')
def _trusted_read(path, *, owner=0):
"""Open a root-owned regular file without following its final symlink."""
fd = os.open(path, os.O_RDONLY | os.O_CLOEXEC | os.O_NOFOLLOW)
try:
info = os.fstat(fd)
if not stat.S_ISREG(info.st_mode) or info.st_uid != owner or info.st_mode & 0o022:
raise JobError('Untrusted OCI job metadata file')
if info.st_size > 8 * 1024 * 1024:
raise JobError('OCI job metadata file is too large')
with os.fdopen(fd, 'r', encoding='utf-8') as source:
fd = -1
return source.read()
finally:
if fd >= 0:
os.close(fd)
def _private_directory(path):
path.mkdir(parents=True, mode=0o700, exist_ok=True)
info = path.lstat()
if (not stat.S_ISDIR(info.st_mode) or info.st_uid != 0 or info.st_mode & 0o077):
raise JobError('OCI job state directory is not private')
def config_fingerprint(text):
"""Presentation and resource changes do not re-run a completed migration."""
selected = []
for line in text.split('\n'):
if line.startswith('['):
break # Pending/snapshot sections are not the active definition.
key, separator, value = line.partition(': ')
if not separator:
continue
if (key in {'entrypoint', 'env', 'rootfs', 'unprivileged', 'features', 'ostype'}
or re.fullmatch(r'(?:mp|dev)[0-9]+', key)
or key.startswith(('lxc.init.', 'lxc.environment.', 'lxc.mount.',
'lxc.cgroup', 'lxc.apparmor', 'lxc.seccomp'))):
if key == 'rootfs' or re.fullmatch(r'mp[0-9]+', key):
value = ','.join(part for part in value.split(',') if not part.startswith('size='))
selected.append((key, value))
if not any(key == 'rootfs' for key, _ in selected):
raise JobError('OCI job has no root filesystem configuration')
selected.sort(key=lambda item: item[0])
return hashlib.sha256(json.dumps(selected, separators=(',', ':'), ensure_ascii=True).encode()).hexdigest()
def _config_text(vmid):
# Validate the actual PVE file's ownership; use the same API rendering
# whose exact hash the coordinated transaction authorizes.
_trusted_read(Path('/etc/pve/lxc') / f'{vmid}.conf')
result = subprocess.run([PCT, 'config', str(vmid)], capture_output=True, check=True, timeout=15)
return result.stdout.decode('utf-8')
def _record_digest(record, config_text, *, executing=False):
pending = record.get('pending_transaction')
if not pending:
return record.get('observed', {}).get('resolved_registry_digest')
vmid = _vmid(record.get('vmid'))
if record.get('status') != 'updating' or not record.get('pending_stack_transaction'):
raise JobError('OCI job has no authorized coordinated update context')
journal = Path(pending)
try:
relative = journal.relative_to(INSTANCE_ROOT / str(vmid) / 'transactions')
except ValueError as error:
raise JobError('OCI job transaction is outside its private instance state') from error
if (len(relative.parts) != 2 or relative.parts[1] != 'transaction.json'
or not re.fullmatch(r'[0-9a-f]{32}', relative.parts[0])):
raise JobError('Invalid OCI job transaction path')
child = json.loads(_trusted_read(journal))
coordinated = child.get('coordinated') or {}
if (child.get('schema_version') != 1 or child.get('id') != record.get('transaction_id')
or child.get('id') != relative.parts[0] or child.get('vmid') != vmid
or child.get('phase') not in {'candidate-installed', 'health-passed', 'committing'}
or child.get('record', {}).get('installation_id') != record.get('installation_id')
or coordinated.get('journal') != record.get('pending_stack_transaction')
or child.get('staged_config_sha256') != hashlib.sha256(config_text.encode()).hexdigest()):
raise JobError('OCI job staged image is not bound to this actual container configuration')
stack_journal = Path(record['pending_stack_transaction'])
try:
stack_path = stack_journal.relative_to(INSTANCE_ROOT)
except ValueError as error:
raise JobError('OCI job stack transaction is outside private instance state') from error
if (len(stack_path.parts) != 4 or not stack_path.parts[0].isdigit()
or stack_path.parts[1] != 'stack-transactions'
or not re.fullmatch(r'[0-9a-f]{32}', stack_path.parts[2])
or stack_path.parts[3] != 'transaction.json'):
raise JobError('Invalid OCI job stack transaction path')
stack = json.loads(_trusted_read(stack_journal))
phases = {'starting', 'checking', 'publishing', 'recovering'}
if not executing:
phases.add('replacing')
members = stack.get('plan', {}).get('members') or []
if (stack.get('schema_version') != 1 or stack.get('id') != coordinated.get('id')
or stack.get('phase') not in phases
or not any(member.get('vmid') == vmid
and member.get('installation_id') == record.get('installation_id')
for member in members)):
raise JobError('OCI job is not authorized by the active stack transaction')
return child.get('registry_digest')
def job_spec_from_record(record, config_text, timeout_seconds=300):
"""Bind a first installation, staged update or restored member for its hook."""
if isinstance(config_text, bytes):
config_text = config_text.decode('utf-8')
if not isinstance(config_text, str) or not isinstance(record, dict):
raise JobError('Invalid OCI job binding input')
if type(timeout_seconds) is not int or not 30 <= timeout_seconds <= 3600:
raise JobError('OCI job timeout must be between 30 and 3600 seconds')
_vmid(record.get('vmid'))
try:
installation_id = str(uuid.UUID(record['installation_id']))
except (KeyError, ValueError, TypeError, AttributeError) as error:
raise JobError('OCI job requires an installation identity') from error
digest = _record_digest(record, config_text)
if not isinstance(digest, str) or not DIGEST.fullmatch(digest):
raise JobError('OCI job requires a verified image digest')
return {'installation_id': installation_id, 'image_digest': digest,
'expected_config_fingerprint': config_fingerprint(config_text),
'timeout_seconds': timeout_seconds}
def current_identity(vmid, spec):
"""Bind the request to both a trusted instance record and the actual CT."""
_vmid(vmid)
if not isinstance(spec, dict):
raise JobError('Invalid OCI job specification')
try:
installation_id = str(uuid.UUID(spec['installation_id']))
except (KeyError, ValueError, TypeError, AttributeError) as error:
raise JobError('OCI job requires an installation identity') from error
digest = spec.get('image_digest')
if not isinstance(digest, str) or not DIGEST.fullmatch(digest):
raise JobError('OCI job requires a verified image digest')
record = json.loads(_trusted_read(INSTANCE_ROOT / str(vmid) / 'oci-compose.json'))
if record.get('vmid') != vmid or record.get('installation_id') != installation_id:
raise JobError('OCI job installation identity changed')
config = _config_text(vmid)
if _record_digest(record, config, executing=True) != digest:
raise JobError('OCI job image digest changed')
description = next((line.partition(': ')[2] for line in config.split('\n')
if line.startswith('description: ')), '')
markers = re.findall(r'proxmenux-instance=([0-9a-fA-F-]{36})', urllib.parse.unquote(description))
if installation_id not in markers:
raise JobError('The CT does not carry the OCI job installation identity')
fingerprint = config_fingerprint(config)
if spec.get('expected_config_fingerprint', fingerprint) != fingerprint:
raise JobError('OCI job runtime configuration changed')
recovery = record.get('recovered_at', '')
if not isinstance(recovery, str):
raise JobError('Invalid OCI job recovery generation')
return {'vmid': vmid, 'installation_id': installation_id, 'image_digest': digest,
'config_fingerprint': fingerprint, 'recovery_generation': recovery}
def _status(vmid):
result = subprocess.run([PCT, 'status', str(vmid)], capture_output=True, text=True, timeout=15)
value = result.stdout.strip()
if result.returncode or value not in {'status: stopped', 'status: running'}:
raise JobError('Cannot establish the OCI job container state')
return value.removeprefix('status: ')
def _socket_name(lxcpath=LXC_PATH):
value = 14695981039346656037
for byte in ('lxc/'+lxcpath+'/monitor-sock').encode():
value = ((value ^ byte) * 1099511628211) & ((1 << 64)-1)
return f'lxc/{value:016x}/{lxcpath}'
@contextlib.contextmanager
def _subscription():
version = subprocess.run([MONITOR, '--version'], capture_output=True, text=True,
check=True, timeout=10).stdout.strip()
if version not in SUPPORTED_VERSIONS or FRAME.size != 264:
raise JobError('Unsupported native LXC monitor ABI; refusing to guess job completion')
# This CLI only bootstraps monitord when no other container has started it.
# Its lossy textual exit status is NEVER consumed.
primer = subprocess.Popen([MONITOR, '-n', '^$'], stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
close_fds=True)
receiver = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
try:
receiver.settimeout(1)
deadline = time.monotonic()+5
name = _socket_name()
while True:
try:
receiver.connect(b'\0'+name.encode())
break
except (ConnectionRefusedError, FileNotFoundError):
if time.monotonic() >= deadline:
raise JobError('Native LXC monitor subscription could not be established')
time.sleep(.05)
peer = struct.unpack('3i', receiver.getsockopt(socket.SOL_SOCKET, socket.SO_PEERCRED,
struct.calcsize('3i')))
pid, uid, gid = peer
executable = str(Path(f'/proc/{pid}/exe').resolve())
if uid != 0 or gid != 0 or executable not in {
'/usr/libexec/lxc/lxc-monitord', '/usr/lib/lxc/lxc-monitord'}:
raise JobError('Native LXC monitor peer is not the trusted root daemon')
receiver.setblocking(False)
yield receiver, {'version': version, 'peer_uid': uid, 'peer_gid': gid,
'peer_executable': executable, 'protocol': 'lxc-msg-native264'}
finally:
receiver.close()
primer.terminate()
try:
primer.wait(timeout=3)
except subprocess.TimeoutExpired:
primer.kill()
primer.wait(timeout=3)
def _frames(buffer):
messages = []
while len(buffer) >= FRAME.size:
chunk, buffer = buffer[:FRAME.size], buffer[FRAME.size:]
kind, name, value = FRAME.unpack(chunk)
if kind not in {0, 1, 2}:
raise JobError('Unknown native LXC monitor event')
try:
name = name.split(b'\0', 1)[0].decode('ascii')
except UnicodeDecodeError as error:
raise JobError('Malformed native LXC monitor event') from error
messages.append((kind, name, value))
return messages, buffer
def _run_attempt(vmid, deadline):
with _subscription() as (receiver, proof):
# Drain traffic before our start; events from unrelated guests do not
# become evidence for this job. There is no historical event replay.
pending = b''
while select.select([receiver], [], [], 0)[0]:
data = receiver.recv(8192)
if not data:
raise JobError('Native LXC monitor closed before job start')
messages, pending = _frames(pending+data)
if any(name == str(vmid) for _, name, _ in messages):
raise JobError('OCI job changed state before its start attempt')
started = False
# Fast successful PID1 jobs can make pct start return 255 because
# Proxmox can no longer query the PID. Only raw exit evidence decides.
process = subprocess.Popen([PCT, 'start', str(vmid)], stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
close_fds=True)
try:
while time.monotonic() < deadline:
if not select.select([receiver], [], [], min(1, max(.01, deadline-time.monotonic())))[0]:
continue
data = receiver.recv(8192)
if not data:
raise JobError('Native LXC monitor closed without a job exit event')
messages, pending = _frames(pending+data)
for kind, name, value in messages:
if name != str(vmid):
continue
if kind == 0 and value in {1, 2}:
started = True
if kind != 2:
continue
if not started or not 0 <= value <= 65535:
raise JobError('No fresh, valid native OCI job exit proof')
if not os.WIFEXITED(value):
reason = f'signal {os.WTERMSIG(value)}' if os.WIFSIGNALED(value) else 'unknown wait status'
raise JobError(f'OCI job CT {vmid} did not complete normally ({reason})')
exitcode = os.WEXITSTATUS(value)
if exitcode != 0:
raise JobError(f'OCI job CT {vmid} exited with code {exitcode}')
while _status(vmid) != 'stopped':
if time.monotonic() >= deadline:
raise JobError('OCI job reported exit zero but did not stop')
time.sleep(.05)
process.wait(timeout=max(.1, min(5, deadline-time.monotonic())))
return {**proof, 'raw_waitstatus': value, 'exit_code': exitcode,
'fresh_start_observed': True, 'pct_start_returncode': process.returncode}
raise JobError('OCI job timed out without verified native exit zero')
finally:
if process.poll() is None:
process.terminate()
try:
process.wait(timeout=3)
except subprocess.TimeoutExpired:
process.kill()
process.wait(timeout=3)
def _cache_matches(cache, identity):
return (isinstance(cache, dict) and cache.get('schema') == 1
and cache.get('status') == 'completed' and cache.get('identity') == identity
and isinstance(cache.get('proof'), dict)
and type(cache['proof'].get('raw_waitstatus')) is int
and cache['proof']['raw_waitstatus'] == 0
and cache['proof'].get('fresh_start_observed') is True)
def _read_cache(path):
try:
return json.loads(_trusted_read(path))
except FileNotFoundError:
return None
def _sync_directory(path):
descriptor = os.open(path, os.O_RDONLY | os.O_DIRECTORY | os.O_CLOEXEC)
try:
os.fsync(descriptor)
finally:
os.close(descriptor)
def _write_cache(path, value):
descriptor, name = tempfile.mkstemp(prefix='.'+path.name+'.', dir=path.parent)
try:
with os.fdopen(descriptor, 'w', encoding='utf-8') as output:
json.dump(value, output, sort_keys=True)
output.write('\n')
output.flush()
os.fsync(output.fileno())
os.replace(name, path)
_sync_directory(path.parent)
finally:
with contextlib.suppress(FileNotFoundError):
os.unlink(name)
@contextlib.contextmanager
def _job_lock(vmid, deadline):
_private_directory(STATE_ROOT)
fd = os.open(STATE_ROOT/f'{vmid}.lock', os.O_CREAT|os.O_RDWR|os.O_NOFOLLOW|os.O_CLOEXEC, 0o600)
try:
info = os.fstat(fd)
if not stat.S_ISREG(info.st_mode) or info.st_uid != 0 or info.st_mode & 0o077:
raise JobError('Untrusted OCI job lock file')
while True:
try:
fcntl.flock(fd, fcntl.LOCK_EX|fcntl.LOCK_NB)
break
except BlockingIOError:
if time.monotonic() >= deadline:
raise JobError('Timed out waiting for another OCI job attempt')
time.sleep(.05)
yield
finally:
os.close(fd)
def ensure_completed(vmid, job_spec, force=False):
_root()
_vmid(vmid)
if type(force) is not bool:
raise JobError('Invalid OCI job force flag')
timeout = job_spec.get('timeout_seconds', 300) if isinstance(job_spec, dict) else None
if type(timeout) is not int or not 30 <= timeout <= 3600:
raise JobError('OCI job timeout must be between 30 and 3600 seconds')
deadline = time.monotonic()+timeout
with _job_lock(vmid, deadline):
identity = current_identity(vmid, job_spec)
if _status(vmid) != 'stopped':
raise JobError('OCI job is already running without this attempt\'s exit subscription')
path = STATE_ROOT/f'{vmid}.json'
cache = _read_cache(path)
if not force and _cache_matches(cache, identity):
return {'status': 'completed', 'cached': True, **identity, 'proof': cache['proof']}
# A forced failed retry must never leave an earlier success reusable.
if path.exists():
path.unlink()
_sync_directory(path.parent)
proof = _run_attempt(vmid, deadline)
if current_identity(vmid, job_spec) != identity:
raise JobError('OCI job identity or configuration changed during execution')
if _status(vmid) != 'stopped':
raise JobError('OCI job restarted before completion could be recorded')
value = {'schema': 1, 'status': 'completed', 'identity': identity, 'proof': proof,
'attempt_id': str(uuid.uuid4()),
'completed_at': datetime.datetime.now(datetime.timezone.utc).isoformat()}
_write_cache(path, value)
return {'status': 'completed', 'cached': False, **identity, 'proof': proof}
def forget_completed(vmid, installation_id):
"""Remove only this installation's completion, never a reused VMID's state."""
_vmid(vmid)
try:
installation_id = str(uuid.UUID(installation_id))
except (ValueError, TypeError, AttributeError):
return False # Older non-job records may use a legacy identity format.
if not STATE_ROOT.exists():
return False
_root()
with _job_lock(vmid, time.monotonic()+30):
path = STATE_ROOT/f'{vmid}.json'
cache = _read_cache(path)
if (not isinstance(cache, dict) or not isinstance(cache.get('identity'), dict)
or cache['identity'].get('installation_id') != installation_id):
return False
path.unlink()
_sync_directory(path.parent)
# The stable lock inode stays: unlinking it could split concurrent
# waiters across two locks when this numeric VMID is reused.
return True
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('operation', choices=['ensure'])
parser.add_argument('vmid', type=int)
parser.add_argument('--spec-json', required=True)
parser.add_argument('--force', action='store_true')
args = parser.parse_args()
try:
if len(args.spec_json) > 65536:
raise JobError('OCI job specification is too large')
result = ensure_completed(args.vmid, json.loads(args.spec_json), force=args.force)
except (JobError, OSError, ValueError, KeyError, subprocess.SubprocessError) as error:
print(str(error), file=__import__('sys').stderr)
return 1
print(json.dumps(result, sort_keys=True))
return 0
if __name__ == '__main__':
raise SystemExit(main())
oci/remote/oci_instance_transaction.py
+16 -5
View File
@@ -33,6 +33,7 @@ import oci_host_mounts as host_mounts
import oci_nested_mounts as nested_mounts
import oci_accelerators as gpu_devices
import oci_runtime_settings as runtime_settings
import oci_stack_public_routes as public_routes
import oci_image_cache as image_cache
import oci_work_backup as work_backup
import oci_ui
@@ -52,7 +53,7 @@ BASIC = {'arch', 'cmode', 'console', 'tty', 'cores', 'cpulimit', 'cpuunits', 'de
# Their output is data (and may hold saved secrets); it is never logged.
DATA_COMMANDS = {('pct', 'config'), ('pvesh', 'get'), ('pct', 'df')}
# Kept across a modification: the watchdog and where the backups are made.
HOST_PREFERENCES = ('watchdog', 'work_storage')
HOST_PREFERENCES = ('watchdog', 'work_storage', 'stack_public_routes')
LOG_DIR = Path(os.environ.get('OCI_LOG_DIR', '/var/log/proxmenux/oci'))
ANSI = re.compile(r'\x1b\[[0-9;?]*[A-Za-z]')
SPINNER_FRAME = re.compile('^ ?[' + ''.join(oci_ui.FRAMES) + ']')
@@ -476,7 +477,7 @@ def candidate_contract(record, operation, proposal=None):
# of what is being edited: it stays as the record has it now.
for key in HOST_PREFERENCES:
if key in record['deployment']:
result['deployment'][key] = record['deployment'][key]
result['deployment'][key] = copy.deepcopy(record['deployment'][key])
else:
result['deployment'].pop(key, None)
return with_default_healthcheck(result)
@@ -519,6 +520,11 @@ def external_changes(record, config, adopt=True):
return values
def without_public_routes(config, vmid):
"""The managed include is validated separately from sysctl includes."""
return b'\n'.join(line for line in config.splitlines() if not public_routes.own(line, vmid)) + b'\n'
def preflight(record, candidate, config, coordinated=None, in_place=False, baseline=True):
deployment = record['deployment']
desired = candidate['deployment']
@@ -555,7 +561,10 @@ def preflight(record, candidate, config, coordinated=None, in_place=False, basel
files = runtime_settings.file_binds(config)
if files:
runtime_keys.add('lxc.mount.entry')
runtime_settings.check(config, deployment, record['vmid'])
public_routes.check(config, deployment, record['vmid'], record['installation_id'])
if deployment.get('stack_public_routes'):
runtime_keys.add('lxc.include')
runtime_settings.check(without_public_routes(config, record['vmid']), deployment, record['vmid'])
coordinated_keys = {'hookscript', *KEPT_AS_IS} if coordinated else set(KEPT_AS_IS)
preserved_keys = {'nameserver', 'searchdomain', 'timezone', 'protection'} if in_place else set()
detached = unused_volumes(config)
@@ -567,7 +576,7 @@ def preflight(record, candidate, config, coordinated=None, in_place=False, basel
or any(not re.fullmatch(r'[A-Za-z0-9_.-]+:[^\s,]+', value) or value in used_volumes
for value in detached.values())):
raise ValueError(translate('The container disks do not match the saved record'))
if '[' in config.decode() or keys - BASIC - runtime_keys - coordinated_keys - preserved_keys - {k for k in keys if re.fullmatch(r'(mp|dev|unused)[0-9]+', k)}:
if re.search(r'^\[[^]\r\n]+\]\s*$', config.decode(), re.MULTILINE) or keys - BASIC - runtime_keys - coordinated_keys - preserved_keys - {k for k in keys if re.fullmatch(r'(mp|dev|unused)[0-9]+', k)}:
raise ValueError(translate('The container has advanced Proxmox settings outside the supported profile'))
unprivileged = deployment.get('security', {}).get('unprivileged', True) is not False
if (desired.get('security', {}).get('unprivileged', True) is not False) != unprivileged \
@@ -680,7 +689,8 @@ def freeze_host_sources(record, candidate, acknowledge_external_data, files=()):
def check_runtime_mounts(config, deployment):
runtime_settings.check(config, deployment, deployment['vmid'])
public_routes.check(config, deployment, deployment['vmid'])
runtime_settings.check(without_public_routes(config, deployment['vmid']), deployment, deployment['vmid'])
nested_mounts.check(config)
actual = mounts(config)
files = runtime_settings.file_binds(config)
@@ -1303,6 +1313,7 @@ def complete_recovery(root, journal, state):
raise ValueError(translate('The configuration changed after the backup was restored; the recovery is not confirmed'))
cleanup_restored_format_dirs(vmid, state['record']['deployment'], state['record']['installation_id'])
runtime_settings.restore(state['record']['deployment'], vmid)
public_routes.restore(state['record']['deployment'], vmid, state['record']['installation_id'])
check_runtime_mounts(config, state['record']['deployment'])
gpu_devices.check(config, state['record']['deployment'])
if show:
oci/remote/oci_instances.py
+18 -3
View File
@@ -15,6 +15,7 @@ import re
import subprocess
import sys
import tempfile
import time
import uuid
from oci_installation_state import command, image_from_archive, private_directory, sha
@@ -42,8 +43,22 @@ def location(root, vmid):
return path / 'oci-compose.json'
def _acquire_lock(fd, timeout):
"""Wait only for acquisition; a failure inside the operation is never replayed."""
deadline = time.monotonic() + max(0, float(timeout))
while True:
try:
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
return
except BlockingIOError:
remaining = deadline - time.monotonic()
if remaining <= 0:
raise
time.sleep(min(0.2, remaining))
@contextmanager
def locked(root):
def locked(root, timeout=0):
private_directory(root)
path = root / '.lock'
if path.is_symlink():
@@ -53,12 +68,12 @@ def locked(root):
fd = int(inherited)
if os.fstat(fd).st_ino != path.stat().st_ino or os.fstat(fd).st_dev != path.stat().st_dev:
raise ValueError(translate('Wrong inherited registry lock'))
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
_acquire_lock(fd, timeout)
yield
else:
with path.open('a') as lock:
os.chmod(path, 0o600)
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
_acquire_lock(lock, timeout)
yield
oci/remote/oci_remove.py
+16 -2
View File
@@ -19,6 +19,8 @@ import sys
import oci_image_cache as image_cache
import oci_work_backup as work_backup
import oci_instances as instances
import oci_compose_jobs as compose_jobs
import oci_stack_public_routes as public_routes
from oci_installation_state import parse_config
import oci_console
import oci_runtime_settings as runtime_settings
@@ -197,7 +199,7 @@ def _leftovers(vmid):
"""Whether anything of the container is still on the host."""
paths = [runtime_settings.include_path(vmid), runtime_settings.legacy_include_path(vmid),
runtime_settings.seccomp_path(vmid), SNIPPETS / f'proxmenux-rclone-{int(vmid)}-fuse-hook.sh', MONITOR_APPS / f'{int(vmid)}.json',
CLUSTER_RECORDS / f'{int(vmid)}.json',
CLUSTER_RECORDS / f'{int(vmid)}.json', public_routes.include_path(vmid), public_routes.contract_path(vmid),
*oci_console.LOG_DIR.glob(f'{int(vmid)}.console.log*')]
return any(path.exists() for path in paths)
@@ -213,7 +215,8 @@ def sweep_orphans(root):
(runtime_settings.seccomp_path(0).parent, r'([0-9]+)\.proxmenux-seccomp'),
(oci_console.LOG_DIR, r'([0-9]+)\.console\.log.*'),
(SNIPPETS, r'proxmenux-rclone-([0-9]+)-fuse-hook\.sh'),
(CLUSTER_RECORDS, r'([0-9]+)\.json')):
(CLUSTER_RECORDS, r'([0-9]+)\.json'),
(public_routes.DIRECTORY, r'([0-9]+)\.(?:conf|json)')):
if directory.is_dir():
found.update(int(m.group(1)) for m in (re.fullmatch(pattern, p.name) for p in directory.iterdir()) if m)
# Only the App tab registrations of OCI installs; the other ones belong to
@@ -230,10 +233,17 @@ def sweep_orphans(root):
if instances.guest_exists(vmid):
continue
record = instances.has_contract(root, vmid)
route_assets = public_routes.contract_path(vmid).exists() or public_routes.include_path(vmid).exists()
route_identity = instances.read(root, vmid)['installation_id'] if record and route_assets else None
if record and not instances.release_orphan(root, vmid):
continue
if not (record or _leftovers(vmid)):
continue
try:
public_routes.remove(vmid, route_identity)
except (ValueError, OSError):
# Changed or unowned assets remain for inspection, never guessed.
continue
oci_console.remove_log(vmid)
remove_host_state(vmid)
cleaned.append(vmid)
@@ -339,6 +349,7 @@ def remove(root, vmid):
msg_warn(f"{translate('The container no longer exists:')} CT {member}")
incomplete = True
oci_console.remove_log(member)
public_routes.remove(member, record['installation_id'])
remove_host_state(member)
elif instances.identity(config) != record['installation_id']:
msg_warn(f"{translate('The VMID belongs to another container now and is not touched:')} CT {member}")
@@ -348,6 +359,7 @@ def remove(root, vmid):
subprocess.run(['pct', 'stop', str(member), '--skiplock', '1'], check=False, capture_output=True)
run('pct', 'destroy', str(member), '--purge', '1', '--destroy-unreferenced-disks', '1')
oci_console.remove_log(member)
public_routes.remove(member, record['installation_id'])
remove_host_state(member)
msg_ok(f"{translate('Container removed:')} CT {member}")
if bridge and not bridge_in_use(bridge, set(members)):
@@ -365,6 +377,8 @@ def remove(root, vmid):
if lifecycle.exists() and not lifecycle.is_symlink():
lifecycle.unlink()
for member in members:
record = instances.read(root, member)
compose_jobs.forget_completed(member, record['installation_id'])
directory = instances.location(root, member).parent
work_backup.forget(root, member)
if directory.is_dir() and not directory.is_symlink():
oci/remote/oci_restore_recovery.py
+105 -2
View File
@@ -31,7 +31,10 @@ import tempfile
import oci_carried_record as carried
import oci_console
import oci_stack_public_routes as public_routes
import oci_instances as instances
import oci_compose_jobs as compose_jobs
import oci_stack_plan
import oci_instance_transaction as transaction
import oci_nested_mounts
import oci_nvidia_refresh
@@ -57,7 +60,7 @@ HOST_MONITOR_CONTENT = 'lxc.namespace.share.pid = 1\nlxc.namespace.share.net = 1
# them from, so they are not offered again.
UNRECOVERABLE = '.unrecoverable.json'
# The fields of a record that name a container of the application.
VMID_KEYS = ('vmid', 'primary_vmid', 'base_vmid')
VMID_KEYS = ('vmid', 'primary_vmid', 'base_vmid', 'target_vmid')
def run(*args):
@@ -285,7 +288,7 @@ def renumbered_lines(text, old, new):
f'lxc.include: {runtime_settings.legacy_include_path(old)}': f'lxc.include: {runtime_settings.include_path(new)}'}
hook = re.compile(rf'(hookscript: \S+:snippets/proxmenux-rclone-){int(old)}(-fuse-hook\.sh)')
lines = [hook.sub(rf'\g<1>{int(new)}\g<2>', swaps.get(line, line)) if line.startswith('hookscript: ')
else swaps.get(line, line) for line in text.split('\n')]
else swaps.get(line, public_routes.renumber_line(line, old, new)) for line in text.split('\n')]
return '\n'.join(lines)
@@ -439,6 +442,11 @@ def check_host_resources(plan):
vmid, label = entry['vmid'], f"CT {entry['vmid']}"
values = parse_config(entry['config'].encode())
deployment = entry['copy']['record']['deployment']
projected = renumbered_lines(entry['config'], entry.get('recorded', vmid), vmid)
try:
public_routes.check(projected, deployment, vmid, entry.get('installation_id'), require_host_assets=False)
except ValueError as error:
plan['blockers'].append(f"{label}: {error}")
directories = [value.split(',', 1)[0] for key, value in values.items()
if re.fullmatch(r'mp[0-9]+', key) and value.startswith('/')]
devices = [oci_stack_modify.device_path(value) for key, value in values.items()
@@ -480,6 +488,9 @@ def check_host_resources(plan):
if key != 'lxc.include':
continue
path = Path(value)
if any(public_routes.own(f'lxc.include: {path}', number) for number in {vmid, entry.get('recorded', vmid)}):
# Canonical assets are emitted from the validated deployment before start.
continue
if path in HOST_MONITOR:
if path.exists() and path.read_text() != HOST_MONITOR_CONTENT:
plan['blockers'].append(f"{label}: {translate('a different host monitor include already exists:')} {path}")
@@ -660,6 +671,19 @@ def valid_contract(contract, dependents):
return False
except ValueError:
return False
elif check.get('type') == 'tcp':
try:
if (not isinstance(check.get('host'), str)
or ipaddress.IPv4Address(check['host']) not in PRIVATE
or type(check.get('port')) is not int or not 0 < check['port'] < 65536):
return False
except ValueError:
return False
elif check.get('type') == 'completed':
try:
oci_stack_plan.validate_job_check(check)
except (ValueError, TypeError, KeyError):
return False
elif check.get('type') != 'running':
return False
return True
@@ -693,12 +717,27 @@ def check_records(root, plan):
plan['blockers'].append(f"CT {entry['vmid']}: {translate('this host keeps a record of another installation with an operation pending')}")
def check_public_route_members(root, plan):
"""Guest-carried routes can target only members validated in this recovery."""
restored = {entry['vmid']: entry['copy']['record'] for entry in plan['restored'] if entry.get('copy')}
try:
records = {member['vmid']: restored.get(member['vmid']) or instances.read(root, member['vmid'])
for member in plan['members']}
if not any(record.get('deployment', {}).get('stack_public_routes') for record in records.values()):
return
oci_stack_plan.validate_public_route_members(plan['primary'], records)
except (ValueError, KeyError, OSError) as error:
plan['blockers'].append(translate('Invalid stack contract') + ': ' + str(error))
def prepare(root):
"""Every restored application of this node, with what prevents its
recovery and what the user has to know."""
plans = applications(root, [examine(entry) for entry in pending(root)])
for plan in plans:
check_records(root, plan)
if not plan['blockers']:
check_public_route_members(root, plan)
if not plan['blockers']:
check_networks(plan)
check_host_resources(plan)
@@ -725,6 +764,7 @@ def restore_host_files(entry):
vmid = entry['vmid']
deployment = entry['copy']['record']['deployment']
renumber_config(entry)
public_routes.restore(deployment, vmid, entry.get('installation_id'))
for line in entry['config'].splitlines():
key, _, value = line.partition(': ')
if key == 'lxc.include':
@@ -903,6 +943,7 @@ def register(root, plan):
# leaves it; this also proves the stack can still be updated.
oci_stack_modify.register(root, vmid)
refresh_service_plans(root, [vmid for vmid, stack in written if stack])
refresh_completed_jobs(root, plan)
except BaseException:
for vmid, _ in written:
path = instances.location(root, vmid)
@@ -945,6 +986,65 @@ def refresh_service_plans(root, vmids):
instances.write(instances.location(root, vmid), record)
def refresh_completed_jobs(root, plan):
"""Rebind restored job proofs before a start, including a deferred start."""
primaries = set()
for entry in plan['restored']:
record = instances.read(root, entry['vmid'])
primary = record.get('stack_member', {}).get('primary_vmid')
if record.get('stack'):
primary = record['vmid']
if primary is not None:
primaries.add(primary)
for primary_id in primaries:
primary = instances.read(root, primary_id)
services = primary.get('stack', {}).get('deployment', {}).get('services', [])
jobs = [service for service in services if service.get('healthcheck', {}).get('type') == 'completed']
if not jobs:
continue
for service in jobs:
vmid = service['vmid']
record = instances.read(root, vmid)
if (record['deployment'].get('onboot') or record['deployment'].get('watchdog')
or record['deployment'].get('execution_mode') != 'one-shot'):
raise ValueError(translate('Invalid service check arguments'))
config = checked('pct', 'config', str(vmid))
# Any restored member may include database data from before the cached proof.
record['recovered_at'] = instances.now()
instances.write(instances.location(root, vmid), record)
timeout = service['healthcheck'].get('timeout_seconds', 300)
service['healthcheck'] = {'type': 'completed', 'timeout_seconds': timeout,
'job': compose_jobs.job_spec_from_record(record, config, timeout)}
dependencies = [{'vmid': service['vmid'], 'label': service['name'],
'healthcheck': service['healthcheck']} for service in services
if service['vmid'] != primary_id and not service.get('deferred_setup')]
contract = {'schema': 1, 'stack': primary['stack']['deployment'].get('stack_name', ''),
'dependencies': dependencies}
with tempfile.NamedTemporaryFile('w', suffix='.json') as file:
json.dump(contract, file)
file.flush()
checked('bash', str(ENGINE / 'stack_dependency_hook.sh'), '--install', str(primary_id), file.name)
for entry in plan['restored']:
if entry['vmid'] == primary_id:
entry['contract'] = contract
primary['stack']['deployment']['services'] = services
members = primary['stack'].get('members', [])
for index, member in enumerate(members):
if any(service['vmid'] == member.get('vmid') for service in jobs):
current = copy.deepcopy(instances.read(root, member['vmid']))
current.pop('stack', None)
members[index] = current
instances.write(instances.location(root, primary_id), primary)
def completed_job(root, vmid):
record = instances.read(root, vmid)
primary = record.get('stack_member', {}).get('primary_vmid', vmid)
main = record if primary == vmid else instances.read(root, primary)
return any(service.get('vmid') == vmid and service.get('healthcheck', {}).get('type') == 'completed'
for service in main.get('stack', {}).get('deployment', {}).get('services', []))
def start_order(entry):
order = re.search(r'^startup: .*?order=([0-9]+)', entry['config'], re.MULTILINE)
return int(order[1]) if order else 9999
@@ -990,6 +1090,9 @@ def recover(root, plan, start=False, host_firewall=()):
msg_warn(f"{translate('Could not add the confirmed host firewall rule')}: {text}")
if start and not plan['hold']:
for entry in sorted(plan['restored'], key=start_order):
if completed_job(root, entry['vmid']):
# The main hook executes jobs in graph order with authenticated exit proofs.
continue
if 'running' in run('pct', 'status', str(entry['vmid'])).stdout:
continue
msg_info(f"{translate('Starting the container...')} CT {entry['vmid']}")
oci/remote/oci_stack_native.py
+173 -7
View File
@@ -3,15 +3,20 @@
import argparse
import copy
import json
import ipaddress
import os
from pathlib import Path
import socket
import stat
import subprocess
import time
import tempfile
import uuid
import oci_image_cache as image_cache
import oci_carried_record as carried
import oci_stack_public_routes as public_routes
import oci_compose_jobs as compose_jobs
import oci_instances as instances
import oci_instance_transaction as member_tx
import oci_stack_plan
@@ -23,6 +28,17 @@ from oci_update_current import resolve_archive
from oci_ui import translate, msg_info, msg_ok, msg_warn, msg_error
def tcp_target(check):
"""Saved dependency probes only connect to numeric addresses and real TCP ports."""
host, port = check.get('host'), check.get('port')
if not isinstance(host, str) or type(port) is not int or not 0 < port < 65536:
raise ValueError(translate('Invalid service check arguments'))
try:
return str(ipaddress.IPv4Address(host)), port
except ValueError as error:
raise ValueError(translate('Invalid service check arguments')) from error
def validate_database_transition(previous, candidate):
def major(image):
values = image.get('defaults', {}).get('Env') or []
@@ -152,6 +168,28 @@ def immich_plan(primary, records, inventory, lifecycle, operation):
return plan
def write_hook_contract(path, value):
"""Atomically replace a pmxcfs private file without changing its directory mode.
pmxcfs assigns permissions from the path and rejects chmod on /etc/pve/priv.
The registry writer manages ordinary private directories, so it is not used here.
"""
for target, directory in ((path.parent, True), (path, False)):
info = target.lstat()
correct_type = stat.S_ISDIR(info.st_mode) if directory else stat.S_ISREG(info.st_mode)
if not correct_type or info.st_uid != os.geteuid() or info.st_mode & 0o077:
raise ValueError(translate('Unsafe dependency hook contract'))
fd, temporary = tempfile.mkstemp(dir=path.parent, prefix='.proxmenux-job-')
try:
with os.fdopen(fd, 'w') as output:
json.dump(value, output, indent=2)
output.write('\n')
os.replace(temporary, path)
finally:
if os.path.exists(temporary):
os.unlink(temporary)
class NativeAdapter:
def __init__(self, root, journal, plan, acknowledge_external_data=False):
self.root, self.journal, self.plan = root, Path(journal), plan
@@ -190,6 +228,7 @@ class NativeAdapter:
if row['type'] != 'lxc' or row.get('node') != socket.gethostname().split('.')[0]:
raise ValueError(translate('A member VMID is in use by another guest or is on another node'))
config = instances.command('pct', 'config', str(vmid))
public_routes.check(config, record['deployment'], vmid, record['installation_id'])
if instances.identity(config) != record['installation_id']:
raise ValueError(translate('The identity of a member was replaced'))
if (not self.journal.exists() or not self.state().get('replacement_intent')) and not instances.same_config_except_notes(record, config, ignore_tags=True):
@@ -206,8 +245,29 @@ class NativeAdapter:
if pending and pending != str(self.journal):
raise ValueError(translate('Another stack operation is pending'))
def _migration_backups(self):
if not any(self.is_job(vmid) for vmid in self.services):
return
database_kinds = {'postgres', 'mariadb', 'linuxserver/mariadb', 'mongo',
'getmeili/meilisearch', 'redis', 'valkey/valkey'}
markers = ('PG_MAJOR=', 'MYSQL_MAJOR=', 'MYSQL_VERSION=', 'MARIADB_MAJOR=',
'MARIADB_VERSION=', 'MONGO_VERSION=', 'MONGODB_VERSION=')
for vmid, record in self.records.items():
service = self.services[vmid]
if self.is_job(vmid):
continue
environment = record.get('observed', {}).get('image', {}).get('defaults', {}).get('Env') or []
database = service.get('kind') in database_kinds or any(
isinstance(value, str) and value.startswith(markers) for value in environment)
unknown_backend = vmid != self.plan['primary_vmid'] and not service.get('frontend')
external = any(mount.get('type') == 'host-bind' and not mount.get('read_only')
for mount in record['deployment'].get('mounts', []))
if external and (database or unknown_backend):
raise ValueError(translate('Migration jobs cannot safely restore database or unclassified backend data from host directories'))
def preflight(self):
self.validate(self.plan)
self._migration_backups()
ha = json.loads(instances.command('pvesh', 'get', '/cluster/ha/resources', '--output-format', 'json'))
if not isinstance(ha, list) or any(r.get('sid') == 'ct:%s' % vmid for r in ha for vmid in self.records):
raise ValueError(translate('High availability resources are not supported for stacks'))
@@ -220,7 +280,7 @@ class NativeAdapter:
member_tx.preflight(record, record, config, coordinated=True)
member_tx.freeze_host_sources(record, record, self.acknowledge)
check = self.services[vmid].get('healthcheck', {})
if check.get('type') not in ('exec', 'http', 'running'):
if check.get('type') not in ('exec', 'http', 'running', 'tcp', 'completed'):
raise ValueError(translate('A member has no reproducible service check'))
if check['type'] == 'exec' and not check.get('argv'):
raise ValueError(translate('Empty exec service check'))
@@ -231,6 +291,16 @@ class NativeAdapter:
raise ValueError(translate('HTTP service check without a saved URL'))
if check['type'] == 'http' and not check['url'].startswith(('http://', 'https://')):
raise ValueError(translate('Invalid service check URL'))
if check['type'] == 'tcp':
tcp_target(check)
if check['type'] == 'completed':
oci_stack_plan.validate_job_check(check, record)
expected = compose_jobs.job_spec_from_record(record, config.decode(), check['timeout_seconds'])
if (check['job'] != expected or record['deployment'].get('onboot')
or record['deployment'].get('watchdog')
or record['deployment'].get('execution_mode') != 'one-shot'
or parse_config(config).get('onboot', '0') != '0' or self.is_running(vmid)):
raise ValueError(translate('Invalid service check arguments'))
if not 0 < int(check.get('timeout_seconds', 120)) <= 3600:
raise ValueError(translate('Invalid service check timeout'))
primary_id = self.plan['primary_vmid']
@@ -340,7 +410,7 @@ class NativeAdapter:
if Path('/etc/pve/lxc/%s.conf' % vmid).exists():
member_tx.stop(vmid)
def backup(self, vmid, identity):
def backup(self, vmid, transaction_id):
self.validate(self.plan)
base = self.work_base()
if base != self.journal.parent and not (self.journal.parent / work_backup.MARK).exists():
@@ -350,6 +420,8 @@ class NativeAdapter:
f"{self.records[self.plan['primary_vmid']]['deployment'].get(work_backup.KEY)}")
directory = base / ('backup-%s' % vmid)
private_directory(directory)
config = member_tx.owned(vmid, self.records[vmid]['installation_id']).decode()
carried.carry_backup(self.root, vmid, self.records[vmid], config)
archive = member_tx.verified_backup(vmid, directory, 'zstd',
translate('The backup of a member could not be identified'))
return {'archive': str(archive), 'sha256': member_tx.filehash(archive)}
@@ -382,12 +454,83 @@ class NativeAdapter:
progress=f"{translate('Recreating') if operation == 'recreate' else translate('Updating')} "
f"{self.describe(vmid)}:")
def is_job(self, vmid):
return self.services[vmid].get('healthcheck', {}).get('type') == 'completed'
def _job_contract(self, rollback=False):
"""Rebind jobs to staged/restored images and journal the hook transition."""
if not any(self.is_job(vmid) for vmid in self.services):
return
primary = self.plan['primary_vmid']
path = Path('/etc/pve/priv/proxmenux-stack-%s.json' % primary)
info = path.lstat()
if not stat.S_ISREG(info.st_mode) or info.st_uid != 0 or info.st_mode & 0o077:
raise ValueError(translate('Unsafe dependency hook contract'))
current = json.loads(path.read_text())
operation = self.state()['id']
marker = self.journal.with_name('completed-job-contract.json')
if marker.is_symlink():
raise ValueError(translate('Unsafe dependency hook contract'))
state = {'schema': 1, 'operation_id': operation}
if marker.exists():
info = marker.lstat()
if not stat.S_ISREG(info.st_mode) or info.st_uid != os.geteuid() or info.st_mode & 0o077:
raise ValueError(translate('Unsafe dependency hook contract'))
state = json.loads(marker.read_text())
if state.get('schema') != 1 or state.get('operation_id') != operation:
raise ValueError(translate('The dependency hook and the stack recipe differ'))
original = next(member for member in self.plan['members'] if member['vmid'] == primary)
original_services = original['stack']['deployment']['services']
expected = state.get('job_hook_contract')
intent = state.get('job_hook_contract_intent')
if expected is None:
expected = {'schema': 1, 'stack': original['stack']['deployment'].get('stack_name', '')}
expected['dependencies'] = [{'vmid': s['vmid'], 'label': s['name'], 'healthcheck': s['healthcheck']}
for s in original_services if s['vmid'] != primary and not s.get('deferred_setup')]
accepted = [expected]
if intent is not None:
accepted.extend([intent['before'], intent['after']])
if current not in accepted:
raise ValueError(translate('The dependency hook and the stack recipe differ'))
if rollback:
self.services = {s['vmid']: copy.deepcopy(s) for s in original_services}
for vmid, service in self.services.items():
if not self.is_job(vmid):
continue
record = instances.read(self.root, vmid)
config = member_tx.owned(vmid, record['installation_id']).decode()
timeout = service['healthcheck'].get('timeout_seconds', 300)
service['healthcheck'] = {'type': 'completed', 'timeout_seconds': timeout,
'job': compose_jobs.job_spec_from_record(record, config, timeout)}
rebuilt = copy.deepcopy(current)
rebuilt['dependencies'] = [{'vmid': s['vmid'], 'label': s['name'], 'healthcheck': s['healthcheck']}
for s in self.services.values() if s['vmid'] != primary and not s.get('deferred_setup')]
if current != rebuilt or intent is not None:
state['job_hook_contract_intent'] = {'before': current, 'after': rebuilt}
instances.write(marker, state)
write_hook_contract(path, rebuilt)
state['job_hook_contract'] = rebuilt
state.pop('job_hook_contract_intent', None)
instances.write(marker, state)
def _ensure_job(self, vmid, force=False):
self._job_contract()
return compose_jobs.ensure_completed(vmid, self.services[vmid]['healthcheck']['job'], force=force)
def start(self, vmid):
self.validate(self.plan)
if self.is_job(vmid):
self._ensure_job(vmid)
return
if vmid == self.plan['primary_vmid']:
self._job_contract()
if not self.is_running(vmid):
member_tx.run('pct', 'start', str(vmid))
def healthcheck(self, vmid):
if self.is_job(vmid):
self._ensure_job(vmid)
return
check = self.services[vmid]['healthcheck']
timeout = int(check.get('timeout_seconds', 120))
if not 0 < timeout <= 3600:
@@ -399,8 +542,15 @@ class NativeAdapter:
try:
if check['type'] == 'exec':
member_tx.run('pct', 'exec', str(vmid), '--', *check['argv'])
# Main/custom web services retain both contracts: an exec
# check never replaces the advertised HTTP readiness probe.
if check.get('url') and (self.services[vmid].get('frontend') or check.get('endpoint')):
member_tx.run('curl', '-fsS', '--noproxy', '*', '--max-time', '5', check['url'])
elif check['type'] == 'http':
member_tx.run('curl', '-fsS', '--noproxy', '*', '--max-time', '5', check['url'])
elif check['type'] == 'tcp':
with socket.create_connection(tcp_target(check), timeout=5):
pass
member_tx.gpu_devices.validate_runtime(vmid, self.records[vmid]['deployment'])
deployment = self.records[vmid]['deployment']
if deployment.get('replay_profile') == {'adapter': 'install_immich_stack.sh', 'role': 'machine-learning'}:
@@ -419,7 +569,7 @@ class NativeAdapter:
'assert (any(d.startswith("GPU") for d in ort.capi._pybind_state.get_available_openvino_device_ids()) '
'if p=="openvino" else ctypes.CDLL("libcuda.so.1").cuInit(0)==0)', acceleration)
return
except RuntimeError:
except (RuntimeError, OSError):
time.sleep(2)
raise ValueError(f"{translate('A member did not pass its service check:')} {self.describe(vmid)}")
@@ -441,12 +591,22 @@ class NativeAdapter:
instances.write(journal, child)
def restore_running_state(self, running, order):
recovering = self.state().get('phase') == 'recovering'
has_jobs = any(self.is_job(vmid) for vmid in order)
for vmid in order:
if self.is_job(vmid):
if recovering and running.get(str(self.plan['primary_vmid'])):
self._ensure_job(vmid, force=True)
continue
if running[str(vmid)]:
self.start(vmid)
for vmid in order:
if running[str(vmid)]:
self.healthcheck(vmid)
if has_jobs:
# A rollback migration may require the restored database to be ready.
self.healthcheck(vmid)
if not has_jobs:
for vmid in order:
if running[str(vmid)]:
self.healthcheck(vmid)
# Starting the primary can start dependencies through its native hook.
for vmid in reversed(order):
if not running[str(vmid)]:
@@ -465,7 +625,7 @@ class NativeAdapter:
# Keep child journals available even if publication is interrupted.
links = self.journal.parent / ('member-%s.json' % vmid)
instances.write(links, {'journal': str(journal)})
member_tx.run('pct', 'set', str(vmid), '--onboot', '1' if original['deployment']['onboot'] else '0')
member_tx.run('pct', 'set', str(vmid), '--onboot', '1' if original['deployment']['onboot'] and not self.is_job(vmid) else '0')
after = member_tx.owned(vmid, original['installation_id'])
if ([line for line in before.splitlines() if not line.startswith(b'onboot: ')]
!= [line for line in after.splitlines() if not line.startswith(b'onboot: ')]):
@@ -476,6 +636,7 @@ class NativeAdapter:
primary_id = self.plan['primary_vmid']
primary = instances.read(self.root, primary_id)
primary['stack']['members'] = []
primary['stack']['deployment']['services'] = copy.deepcopy(list(self.services.values()))
for vmid in self.plan['start_order']:
record = instances.read(self.root, vmid)
record.pop('stack', None)
@@ -555,8 +716,12 @@ class NativeAdapter:
if any(self.plan.get(flag) for flag in ('nextcloud_replay', 'paperless_replay', 'tandoor_replay', 'immich_replay')):
record['deployment']['native_config'] = record['observed']['config']
record['deployment']['member_replay_projection'] = replay.normalize(record)
if self.is_job(vmid) and self.state().get('replacement_intent'):
# Restored database contents invalidate an earlier completion proof too.
record['recovered_at'] = instances.now()
record['pending_stack_transaction'] = str(self.journal)
instances.write(instances.location(self.root, vmid), record)
self._job_contract(rollback=True)
primary_id = plan['primary_vmid']
primary = instances.read(self.root, primary_id)
snapshots = []
@@ -566,6 +731,7 @@ class NativeAdapter:
snapshot.pop('pending_stack_transaction', None)
snapshots.append(snapshot)
primary['stack']['members'] = snapshots
primary['stack']['deployment']['services'] = copy.deepcopy(list(self.services.values()))
instances.write(instances.location(self.root, primary_id), primary)
def finalize(self, state):
oci/remote/oci_stack_plan.py
+58
View File
@@ -1,9 +1,59 @@
"""Read-only validation of a coordinated stack operation, before host changes."""
import copy
import oci_stack_public_routes as public_routes
import re
import uuid
from oci_ui import translate
def validate_job_check(check, record=None):
"""Validate a completion proof request without treating STOPPED as success."""
failure = translate('Invalid service check arguments')
if not isinstance(check, dict) or check.get('type') != 'completed':
raise ValueError(failure)
job = check.get('job')
if not isinstance(job, dict):
raise ValueError(failure)
try:
identity = str(uuid.UUID(job['installation_id']))
except (KeyError, ValueError, TypeError, AttributeError) as error:
raise ValueError(failure) from error
timeout = job.get('timeout_seconds')
if (type(timeout) is not int or not 30 <= timeout <= 3600
or type(check.get('timeout_seconds')) is not int or check['timeout_seconds'] != timeout
or not isinstance(job.get('image_digest'), str)
or not re.fullmatch(r'sha256:[0-9a-f]{64}', job['image_digest'])
or not isinstance(job.get('expected_config_fingerprint'), str)
or not re.fullmatch(r'[0-9a-f]{64}', job['expected_config_fingerprint'])):
raise ValueError(failure)
if record is not None and (record.get('installation_id') != identity
or record.get('observed', {}).get('resolved_registry_digest') != job['image_digest']):
raise ValueError(failure)
return job
def validate_public_route_members(primary_id, records):
"""A route may name only the frontends of this exact recorded stack."""
for vmid, record in records.items():
spec = public_routes.validate(record.get('deployment', {}), vmid, record['installation_id'])
if spec is None:
continue
primary = records.get(primary_id)
if (primary is None or spec['primary_vmid'] != primary_id
or spec['stack_id'] != primary['installation_id']):
raise ValueError(translate('Invalid stack contract'))
members = {member['vmid']: member['installation_id']
for member in primary.get('stack', {}).get('members', [])}
if members.get(vmid) != record['installation_id']:
raise ValueError(translate('Invalid stack contract'))
for route in spec['routes']:
target = records.get(route['target_vmid'])
if (target is None or target['installation_id'] != route['target_installation_id']
or members.get(route['target_vmid']) != route['target_installation_id']):
raise ValueError(translate('Invalid stack contract'))
def build(primary, records, inventory, operation):
"""Inventory maps VMIDs to installation UUIDs, including unrelated guests.
@@ -15,6 +65,7 @@ def build(primary, records, inventory, operation):
stack = primary.get('stack')
if not stack or not stack.get('members'):
raise ValueError(translate('This is not a coordinated stack'))
validate_public_route_members(primary['vmid'], records)
snapshots = stack['members']
ids = [member['vmid'] for member in snapshots]
if any(type(vmid) is not int or vmid < 100 for vmid in ids) or len(set(ids)) != len(ids):
@@ -48,6 +99,13 @@ def build(primary, records, inventory, operation):
missing.append(vmid)
if current.get('deployment', {}).get('rootfs_adaptation_replay_required'):
blockers.append({'vmid': vmid, 'reason': 'dedicated-adapter-replay-required'})
service = next(service for service in services if service['vmid'] == vmid)
if service.get('healthcheck', {}).get('type') == 'completed':
validate_job_check(service['healthcheck'], current)
if (vmid == primary['vmid'] or current.get('deployment', {}).get('onboot')
or current.get('deployment', {}).get('watchdog')
or current.get('deployment', {}).get('execution_mode') != 'one-shot'):
raise ValueError(translate('Invalid service check arguments'))
members.append(copy.deepcopy(current))
if missing and operation == 'update':
raise ValueError(translate('Stack members are missing; recreate them after verifying their volumes'))
oci/remote/oci_stack_public_routes.py
+351
View File
@@ -0,0 +1,351 @@
#!/usr/bin/env python3
"""Owned /32 routes to a stack's own frontends, installed before OCI init."""
import argparse
import copy
import fcntl
import ipaddress
import json
import os
from pathlib import Path
import re
import stat
import subprocess
import tempfile
import uuid
DIRECTORY = Path('/etc/pve/priv/proxmenux-stack-public-routes')
HELPER = Path('/usr/local/share/proxmenux/oci/engine/remote/oci_stack_public_routes.py')
CONFIGS = Path('/etc/pve/lxc')
ROOT_UID = 0
PRIVATE = ipaddress.ip_network('10.77.0.0/16')
IP = '/usr/sbin/ip'
NSENTER = '/usr/bin/nsenter'
NS_GET_NSTYPE = 0xb703
CLONE_NEWNET = 0x40000000
class RouteError(ValueError):
pass
def _vmid(value):
if type(value) is not int or not 100 <= value < 1000000000:
raise RouteError('Invalid public route VMID')
return value
def _identity(value):
try:
result = str(uuid.UUID(value))
except (ValueError, TypeError, AttributeError) as error:
raise RouteError('Invalid public route identity') from error
if result != value:
raise RouteError('Invalid public route identity')
return result
def include_path(vmid):
return DIRECTORY / f'{_vmid(vmid)}.conf'
def contract_path(vmid):
return DIRECTORY / f'{_vmid(vmid)}.json'
def _address(value):
if not isinstance(value, str):
raise RouteError('Invalid public route address')
try:
address = ipaddress.IPv4Address(value)
except (ValueError, TypeError) as error:
raise RouteError('Invalid public route address') from error
if address.is_loopback or address.is_multicast or address.is_unspecified or address.is_link_local:
raise RouteError('Invalid public route address')
return address
def validate(deployment, vmid, identity=None):
spec = deployment.get('stack_public_routes')
if spec is None:
return None
required = {'schema', 'stack_id', 'primary_vmid', 'vmid', 'installation_id',
'private_bridge', 'private_address', 'routes'}
if not isinstance(spec, dict) or set(spec) != required or type(spec['schema']) is not int or spec['schema'] != 1:
raise RouteError('Invalid public route contract')
if _vmid(spec['vmid']) != _vmid(vmid):
raise RouteError('Public route owner differs')
_vmid(spec['primary_vmid'])
_identity(spec['stack_id'])
if _identity(spec['installation_id']) != (identity or spec['installation_id']):
raise RouteError('Public route owner identity differs')
if not isinstance(spec['private_bridge'], str) or not re.fullmatch(r'vmbr[0-9]+', spec['private_bridge']):
raise RouteError('Invalid public route bridge')
if not isinstance(spec['private_address'], str):
raise RouteError('Invalid public route private address')
try:
source = ipaddress.IPv4Interface(spec['private_address'])
except (ValueError, TypeError) as error:
raise RouteError('Invalid public route private address') from error
if source.network.prefixlen != 24 or source.ip not in PRIVATE or source.ip in (source.network.network_address, source.network.broadcast_address):
raise RouteError('Invalid public route private address')
routes = spec['routes']
if not isinstance(routes, list) or not routes or len(routes) > 64:
raise RouteError('Invalid public route list')
destinations = set()
for route in routes:
if not isinstance(route, dict) or set(route) != {'destination', 'via', 'target_vmid', 'target_installation_id'}:
raise RouteError('Invalid public route entry')
destination, via = _address(route['destination']), _address(route['via'])
if (destination in source.network or destination in destinations or via not in source.network
or via in (source.ip, source.network.network_address, source.network.broadcast_address)
or _vmid(route['target_vmid']) == vmid):
raise RouteError('Invalid public route destination or next hop')
_identity(route['target_installation_id'])
destinations.add(destination)
return copy.deepcopy(spec)
def include_content():
return f'lxc.hook.start-host = /usr/bin/python3 -I {HELPER} start\n'
def own(line, vmid):
if isinstance(line, bytes):
line = line.decode()
return line.rstrip('\n') == f'lxc.include: {include_path(vmid)}'
def renumber_line(line, old, new):
binary = isinstance(line, bytes)
text = line.decode() if binary else line
if own(text, old):
text = f'lxc.include: {include_path(new)}' + ('\n' if text.endswith('\n') else '')
return text.encode() if binary else text
def _read(path):
info = path.lstat()
if not stat.S_ISREG(info.st_mode) or info.st_uid != ROOT_UID or info.st_mode & 0o022:
raise RouteError('Unsafe managed public route file')
if info.st_size > 262144:
raise RouteError('Managed public route file is too large')
return path.read_bytes()
def _managed_directory(path, create=False):
if create:
path.mkdir(parents=True, exist_ok=True)
info = path.lstat()
if not stat.S_ISDIR(info.st_mode) or info.st_uid != ROOT_UID or info.st_mode & 0o022:
raise RouteError('Unsafe managed public route directory')
def _write(path, data, mode=0o600):
_managed_directory(path.parent, create=True)
if path.is_symlink():
raise RouteError('Managed public route path is a symbolic link')
fd, temporary = tempfile.mkstemp(prefix='.public-routes-', dir=path.parent)
try:
with os.fdopen(fd, 'wb') as output:
output.write(data)
output.flush()
os.fsync(output.fileno())
# pmxcfs supplies the permissions of files below its private directory.
if Path('/etc/pve') not in path.parents:
os.chmod(temporary, mode)
os.replace(temporary, path)
finally:
if os.path.exists(temporary):
os.unlink(temporary)
def install(deployment, vmid, identity=None):
spec = validate(deployment, vmid, identity)
if spec is None:
return
# The installed engine is the authority; upgrades replace it normally.
_managed_directory(HELPER.parent)
_read(HELPER)
_write(contract_path(vmid), (json.dumps(spec, sort_keys=True) + '\n').encode())
_write(include_path(vmid), include_content().encode())
return f'lxc.include: {include_path(vmid)}'
restore = install
publish = install
def check(config, deployment, vmid, identity=None, require_host_assets=True):
spec = validate(deployment, vmid, identity)
text = config.decode() if isinstance(config, bytes) else config
includes = [line for line in text.splitlines() if line.startswith('lxc.include: ')
and str(DIRECTORY) + '/' in line]
expected = [f'lxc.include: {include_path(vmid)}'] if spec else []
if includes != expected:
raise RouteError('Managed public route include differs from the contract')
if spec and require_host_assets:
_managed_directory(DIRECTORY)
_managed_directory(HELPER.parent)
if _read(include_path(vmid)) != include_content().encode():
raise RouteError('Managed public route include was changed')
if json.loads(_read(contract_path(vmid))) != spec:
raise RouteError('Managed public route contract was changed')
_read(HELPER)
return spec
def remove(vmid, identity=None):
path = contract_path(vmid)
if not path.exists() and not path.is_symlink():
if include_path(vmid).exists() or include_path(vmid).is_symlink():
raise RouteError('Public route include has no ownership contract')
return False
_managed_directory(DIRECTORY)
spec = json.loads(_read(path))
validate({'stack_public_routes': spec}, vmid, identity)
include = include_path(vmid)
if include.exists() or include.is_symlink():
if _read(include) != include_content().encode():
raise RouteError('Managed public route include was changed')
include.unlink()
path.unlink()
return True
def _config(vmid):
text = _read(CONFIGS / f'{_vmid(vmid)}.conf').decode()
matches = re.findall(r'proxmenux-instance=([0-9a-f-]{36})(?![0-9a-f-])', text)
if len(set(matches)) != 1:
raise RouteError('Public route container identity is unknown')
values = {}
for line in text.splitlines():
if line.startswith('#') or ': ' not in line:
continue
key, value = line.split(': ', 1)
if key in ('net0', 'net1'):
if key in values:
raise RouteError('Ambiguous public route network')
parts = [part.split('=', 1) for part in value.split(',') if '=' in part]
if len({key for key, _ in parts}) != len(parts):
raise RouteError('Ambiguous public route network')
values[key] = dict(parts)
return matches[0], values, text
def live_contract(spec):
"""Validate actual PVE ownership and both ends of every generated route."""
vmid = spec['vmid']
identity, networks, text = _config(vmid)
check(text, {'stack_public_routes': spec}, vmid, identity)
source = networks.get('net0', {})
if source.get('bridge') != spec['private_bridge'] or source.get('ip') != spec['private_address'] or source.get('name') != 'eth0':
raise RouteError('Public route private interface differs')
primary_identity, primary_networks, _ = _config(spec['primary_vmid'])
if primary_identity != spec['stack_id'] or primary_networks.get('net0', {}).get('bridge') != spec['private_bridge']:
raise RouteError('Public route stack ownership differs')
for route in spec['routes']:
target_identity, target_networks, _ = _config(route['target_vmid'])
private, frontend = target_networks.get('net0', {}), target_networks.get('net1', {})
try:
private_ip = ipaddress.IPv4Interface(private.get('ip'))
public_ip = ipaddress.IPv4Interface(frontend.get('ip'))
except (ValueError, TypeError) as error:
raise RouteError('Public route target is not a static frontend') from error
if (target_identity != route['target_installation_id'] or private.get('bridge') != spec['private_bridge']
or private_ip.network != ipaddress.IPv4Interface(spec['private_address']).network
or str(private_ip.ip) != route['via'] or str(public_ip.ip) != route['destination']):
raise RouteError('Public route target ownership or addresses differ')
mac = source.get('hwaddr', '').lower()
if not re.fullmatch(r'(?:[0-9a-f]{2}:){5}[0-9a-f]{2}', mac):
raise RouteError('Public route private hardware address is unknown')
return mac
def namespace_fd(path):
if not isinstance(path, str) or not re.fullmatch(r'/proc/[1-9][0-9]*/(?:fd/[0-9]+|ns/net)', path):
raise RouteError('LXC did not provide a network namespace')
fd = os.open(path, os.O_RDONLY)
try:
if fcntl.ioctl(fd, NS_GET_NSTYPE) != CLONE_NEWNET:
raise RouteError('LXC namespace is not a network namespace')
info, host = os.fstat(fd), os.stat('/proc/self/ns/net')
if (info.st_dev, info.st_ino) == (host.st_dev, host.st_ino):
raise RouteError('Refusing to configure the host network namespace')
return fd
except BaseException:
os.close(fd)
raise
def _namespace_command(fd, *args):
process = subprocess.run([NSENTER, f'--net=/proc/self/fd/{fd}', '--', IP, *args],
pass_fds=(fd,), capture_output=True, check=True, timeout=15)
return process.stdout
def apply(spec, fd, mac):
source = ipaddress.IPv4Interface(spec['private_address'])
interfaces = json.loads(_namespace_command(fd, '-j', 'address', 'show', 'dev', 'eth0'))
if (len(interfaces) != 1 or interfaces[0].get('address', '').lower() != mac
or not any(item.get('family') == 'inet' and item.get('local') == str(source.ip)
and item.get('prefixlen') == source.network.prefixlen
for item in interfaces[0].get('addr_info', []))):
raise RouteError('Network namespace does not belong to the public route owner')
for route in spec['routes']:
_namespace_command(fd, '-4', 'route', 'replace', route['destination'] + '/32',
'via', route['via'], 'dev', 'eth0', 'src', str(source.ip))
def start():
if os.geteuid() != 0:
raise RouteError('Public routes require root')
name = os.environ.get('LXC_NAME', '')
if not re.fullmatch(r'[1-9][0-9]*', name):
raise RouteError('LXC did not provide its container identity')
vmid = _vmid(int(name))
spec = json.loads(_read(contract_path(vmid)))
validate({'stack_public_routes': spec}, vmid)
mac = live_contract(spec)
pid = os.environ.get('LXC_PID', '')
if not re.fullmatch(r'[1-9][0-9]*', pid):
raise RouteError('LXC did not provide its init process identity')
fd = namespace_fd(f'/proc/{pid}/ns/net')
try:
apply(spec, fd, mac)
finally:
os.close(fd)
def main():
parser = argparse.ArgumentParser(description=__doc__)
commands = parser.add_subparsers(dest='operation', required=True)
start_parser = commands.add_parser('start')
start_parser.add_argument('hook_arguments', nargs='*')
install_parser = commands.add_parser('install')
install_parser.add_argument('vmid', type=int)
install_parser.add_argument('--deployment', required=True)
install_parser.add_argument('--installation-id', required=True)
args = parser.parse_args()
if os.geteuid() != 0:
raise RouteError('Public routes require root')
if args.operation == 'start':
# Proxmox's network scripts use the default LXC version-0 hook ABI.
expected = [os.environ.get('LXC_NAME', ''), 'lxc', 'start-host']
if args.hook_arguments != expected:
raise RouteError('Unexpected public route hook invocation')
start()
else:
line = install(json.loads(Path(args.deployment).read_text()), args.vmid, args.installation_id)
if line:
print(line)
if __name__ == '__main__':
try:
main()
except (ValueError, OSError, subprocess.SubprocessError) as error:
detail = str(error)
if isinstance(error, subprocess.CalledProcessError) and error.stderr:
output = error.stderr.decode(errors='replace') if isinstance(error.stderr, bytes) else error.stderr
detail += ': ' + output.strip()[:1000]
raise SystemExit('Managed public routes failed: ' + detail)
oci/remote/oci_watchdog.py
+4 -2
View File
@@ -124,7 +124,8 @@ def watched(root):
try:
vmid = int(path.parent.name)
record = instances.read(root, vmid)
if record.get('status') != 'installed' or record.get('deployment', {}).get('watchdog') is not True:
if (record.get('status') != 'installed' or record.get('deployment', {}).get('watchdog') is not True
or record.get('deployment', {}).get('execution_mode') == 'one-shot'):
continue
if record.get('pending_stack_transaction') or record.get('pending_transaction'):
continue
@@ -326,7 +327,8 @@ def set_watchdog(root, vmid, enabled):
vmids = application(root, vmid)
for member in vmids:
record = instances.read(root, member)
record.setdefault('deployment', {})['watchdog'] = bool(enabled)
deployment = record.setdefault('deployment', {})
deployment['watchdog'] = bool(enabled) and deployment.get('execution_mode') != 'one-shot'
instances.write(instances.location(root, member), record)
oci_carried_record.carry(root, member, mount_stopped=False)
if enabled:
oci/remote/stack_dependency_hook.sh
+62 -2
View File
@@ -4,6 +4,7 @@ set -Eeuo pipefail
PATH=/usr/sbin:/usr/bin:/sbin:/bin
HOOK_NAME=proxmenux-stack-dependencies.sh
COMPOSE_JOB_HELPER=/var/lib/vz/snippets/proxmenux-compose-jobs.py
# Longest wait for a dependency that is being backed up, and for any other lock.
BACKUP_LOCK_WAIT=${PROXMENUX_STACK_BACKUP_LOCK_WAIT:-14400}
OTHER_LOCK_WAIT=${PROXMENUX_STACK_OTHER_LOCK_WAIT:-300}
@@ -12,6 +13,8 @@ RESTART_GRACE=${PROXMENUX_STACK_RESTART_GRACE:-20}
POLL=${PROXMENUX_STACK_POLL:-5}
# Earlier published versions of this script, replaced on update.
PREVIOUS_RELEASES=(
4bc477bcc35e229d1c964c9372838f0973cbe6f1e3e41f0fb486e9b724247310
243b0dca81b1b155ca169a4d0e11cdb8ca55974c65ede6a4a9be8f4d536a141e
c28b69acd7cc8e66d3aec3e8b2ec86bda0e4f91193ec78549f1965f2c859c0d1
02c50ef17f2411ef22f4d382c51b34a7a0974a446d4a155ecb63fb1e805a7c59
)
@@ -58,11 +61,40 @@ install_hook() {
(all(.dependencies[];
(.vmid | type == "number") and
(.label | type == "string") and
(.healthcheck.type | IN("exec", "http", "running")) and
(.healthcheck.timeout_seconds | type == "number")
(.healthcheck.type | IN("exec", "http", "running", "tcp", "completed")) and
(.healthcheck.timeout_seconds | type == "number") and
(if .healthcheck.type == "tcp" then
(.healthcheck.host | type == "string") and
(.healthcheck.host | test("^([0-9]{1,3}\\.){3}[0-9]{1,3}$")) and
(.healthcheck.host | split(".") | all(.[]; tonumber < 256)) and
(.healthcheck.port | type == "number") and
(.healthcheck.port > 0 and .healthcheck.port < 65536 and .healthcheck.port == (.healthcheck.port | floor))
elif .healthcheck.type == "completed" then
(.healthcheck.job | type == "object") and
(.healthcheck.job.installation_id | type == "string") and
(.healthcheck.job.installation_id | test("^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")) and
(.healthcheck.job.image_digest | type == "string") and
(.healthcheck.job.image_digest | test("^sha256:[0-9a-f]{64}$")) and
(.healthcheck.job.expected_config_fingerprint | type == "string") and
(.healthcheck.job.expected_config_fingerprint | test("^[0-9a-f]{64}$")) and
(.healthcheck.job.timeout_seconds == .healthcheck.timeout_seconds) and
(.healthcheck.timeout_seconds >= 30 and .healthcheck.timeout_seconds <= 3600 and .healthcheck.timeout_seconds == (.healthcheck.timeout_seconds | floor))
else true end)
))
' "$source_config" >/dev/null || die "Invalid dependency contract"
if jq -e 'any(.dependencies[]; .healthcheck.type == "completed")' "$source_config" >/dev/null; then
local helper_source helper_temporary
helper_source="$(dirname -- "$0")/oci_compose_jobs.py"
[[ -f $helper_source && ! -L $helper_source ]] || die "Native OCI job helper is missing"
install -d -m 0755 "$(dirname -- "$COMPOSE_JOB_HELPER")"
helper_temporary=$(mktemp "${COMPOSE_JOB_HELPER}.XXXXXX")
if ! install -m 0755 "$helper_source" "$helper_temporary" || ! mv -f -- "$helper_temporary" "$COMPOSE_JOB_HELPER"; then
rm -f -- "$helper_temporary"
die "Could not install the persistent native OCI job helper"
fi
fi
storage=$(find_snippet_storage)
hook_volume="${storage}:snippets/${HOOK_NAME}"
hook_path=$(pvesm path "$hook_volume")
@@ -144,6 +176,20 @@ dependency_is_healthy() {
url=$(jq -r '.url' <<<"$healthcheck")
curl -fsS --max-time 3 "$url" >/dev/null 2>&1
;;
tcp)
local host port
host=$(jq -r '.host' <<<"$healthcheck")
port=$(jq -r '.port' <<<"$healthcheck")
python3 - "$host" "$port" <<'PYTCP' >/dev/null 2>&1
import ipaddress, socket, sys
host = str(ipaddress.IPv4Address(sys.argv[1]))
port = int(sys.argv[2])
if not 0 < port < 65536:
raise ValueError('Invalid TCP port')
with socket.create_connection((host, port), timeout=3):
pass
PYTCP
;;
*) return 1 ;;
esac
}
@@ -167,6 +213,20 @@ start_dependencies() {
pct config "$id" >/dev/null 2>&1 \
|| die "Dependency $label (CT $id) does not exist"
if [[ $(jq -r '.type' <<<"$healthcheck") == completed ]]; then
[[ -f $COMPOSE_JOB_HELPER && ! -L $COMPOSE_JOB_HELPER ]] || die "Native OCI job helper is missing"
local helper_owner helper_mode
helper_owner=$(stat -c '%u' "$COMPOSE_JOB_HELPER")
helper_mode=$(stat -c '%a' "$COMPOSE_JOB_HELPER")
[[ $helper_owner == 0 && $helper_mode =~ ^[0-7]{3,4}$ ]] && (( (8#$helper_mode & 8#022) == 0 )) \
|| die "Native OCI job helper is not owned safely by root"
printf 'Waiting for completed job %s (CT %s)...\n' "$label" "$id"
python3 -I "$COMPOSE_JOB_HELPER" ensure "$id" --spec-json "$(jq -c '.job' <<<"$healthcheck")" >/dev/null 9>&- \
|| die "Job $label (CT $id) did not complete successfully"
printf 'Dependency %s (CT %s): ready.\n' "$label" "$id"
continue
fi
is_running "$id" || wait_for_release "$id" "$label"
if ! is_running "$id"; then
printf 'Starting dependency %s (CT %s)...\n' "$label" "$id"
oci/src/proxmenux_oci/casaos.py
+1 -1
View File
@@ -906,7 +906,7 @@ def convert_casaos_compose(
"environment": "import-image-env-then-apply-compose-overrides",
"user": "import-from-oci-image",
"working_dir": "import-from-oci-image",
"stop_signal": "import-from-oci-image",
"stop_signal": "apply-compose-override" if service.get("stop_signal") is not None else "import-from-oci-image",
},
**(
{"installer_profile": _compose_installer_profile(service, markers["devices"], markers["security_opt"])}
oci/src/proxmenux_oci/cli.py
+14 -2
View File
@@ -265,9 +265,18 @@ def _deployment_summary_text(template: dict[str, Any], deployment: dict[str, Any
child = service["deployment"]
lines.append(f" {service['name']}: {child['resources']['cores']} CPU, "
f"{child['resources']['memory_mb']} MB RAM")
check = service.get('healthcheck', {}).get('type')
if check:
label = {'running': translate('Process running only (no health check declared)'),
'completed': translate('Must finish successfully (exit code 0)'),
'exec': translate('Command'), 'http': 'HTTP', 'tcp': 'TCP'}.get(check, check)
lines.append(f" {translate('Startup check')}: {label}")
for mount in child["mounts"]:
target = (f"{mount['size_gb']} GB {on} {mount['source']}"
if mount["type"] == "managed-volume" else mount["source"])
if mount["type"] == "managed-volume" else
f"{translate('Host directory (not included in Proxmox backups)')}: {mount['source']}")
if mount['type'] == 'managed-volume' and mount.get('backup'):
target += f" ({translate('in backups')})"
lines.append(f" {mount['container_path']} → {target}")
for key, label in (("media", N_("Library")), ("application", N_("Application data")), ("transfer", N_("Consume/export"))):
storage = plan.get(key)
@@ -457,7 +466,10 @@ def install_template(ui, template: dict[str, Any], identifier: str, mode: str) -
_print_installation_summary(result, removed)
if watchdog:
from .management import set_watchdog
if set_watchdog(PROJECT_ROOT, sorted(vmids), True):
job_names = {service['name'] for service in deployment.get('services', [])
if service.get('deployment', {}).get('execution_mode') == 'one-shot'}
job_vmids = {int(vmid) for name, vmid in (result.get('stack_vmids') or {}).items() if name in job_names}
if set_watchdog(PROJECT_ROOT, sorted(vmids-job_vmids), True):
console.msg_ok(translate("Watchdog enabled: the application is restarted when it crashes."))
else:
console.msg_warn(translate("The watchdog could not be enabled. Turn it on from Manage installed OCI applications."))
oci/src/proxmenux_oci/compose_input.py
+258
View File
@@ -0,0 +1,258 @@
"""Resolve explicitly supplied Compose inputs without inheriting the host environment.
No command substitutions, shell evaluation or image-specific rewrites are performed.
"""
from __future__ import annotations
import json
from pathlib import Path
import re
import yaml
from .converter import ConversionError, _split_short_mount
from .i18n import translate
NAME = re.compile(r'[A-Za-z_][A-Za-z0-9_]*')
SECRET = re.compile(r'pass|secret|token|key|credential|url', re.I)
LIMIT = 256 * 1024
class Interpolator:
def __init__(self, values=None, missing=None):
self.values = dict(values or {})
self.missing = missing
def expand(self, value, depth=0):
if depth > 20:
raise ConversionError(translate('Unresolved variable:') + ' nesting')
result, i = [], 0
while i < len(value):
if value[i] != '$':
result.append(value[i]); i += 1; continue
if value[i:i+2] == '$$':
result.append('$'); i += 2; continue
if value[i:i+2] == '${':
end, nested = i+2, 1
while end < len(value) and nested:
if value[end:end+2] == '${':
nested += 1; end += 2; continue
if value[end] == '}':
nested -= 1
end += 1
if nested:
raise ConversionError(translate('Unresolved variable:') + ' ${...}')
expression = value[i+2:end-1]
match = re.fullmatch(r'([A-Za-z_][A-Za-z0-9_]*)(?:(:?[-+?])(.*))?', expression, re.S)
if not match:
raise ConversionError(translate('Unresolved variable:') + ' ${...}')
name, op, argument = match.groups()
i = end
else:
match = NAME.match(value, i+1)
if not match:
result.append('$'); i += 1; continue
name, op, argument = match[0], None, None
i = match.end()
present = name in self.values
nonempty = present and self.values[name] != ''
use = nonempty if op and op.startswith(':') else present
if op and op.endswith('-'):
replacement = self.values[name] if use else self.expand(argument, depth+1)
elif op and op.endswith('+'):
replacement = self.expand(argument, depth+1) if use else ''
elif op and op.endswith('?'):
if not use:
raise ConversionError(translate('Unresolved variable:') + ' ' + name)
replacement = self.values[name]
elif present:
replacement = self.values[name]
elif self.missing is not None:
replacement = self.missing(name)
self.values[name] = replacement
else:
raise ConversionError(translate('Unresolved variable:') + ' ' + name)
result.append(str(replacement))
return ''.join(result)
def walk(self, value):
if isinstance(value, str):
return self.expand(value)
if isinstance(value, dict):
return {key: self.walk(item) for key, item in value.items()}
if isinstance(value, list):
return [self.walk(item) for item in value]
return value
def dotenv(text, interpolator):
"""The common KEY=value dotenv grammar; quoted values are never executed."""
result = {}
local = Interpolator(interpolator.values, interpolator.missing)
for line in text.splitlines():
raw = line.strip()
if not raw or raw.startswith('#'):
continue
if raw.startswith('export '):
raw = raw[7:].lstrip()
key, separator, value = raw.partition('=')
key = key.strip()
if not NAME.fullmatch(key):
raise ConversionError(translate('Invalid environment variable name:') + ' ' + key)
if not separator:
result[key] = None
continue
value = value.strip()
if value.startswith("'"):
end, quoted = 1, []
while end < len(value):
if value[end:end+2] == "\\'":
quoted.append("'"); end += 2; continue
if value[end] == "'":
break
quoted.append(value[end]); end += 1
if end >= len(value) or (value[end+1:].strip() and not value[end+1:].lstrip().startswith('#')):
raise ConversionError(translate('Invalid environment variable name:') + ' ' + key)
value = ''.join(quoted)
else:
if value.startswith('"'):
quoted = value
try:
value, end = json.JSONDecoder().raw_decode(value)
if not isinstance(value, str):
raise ValueError()
except (ValueError, TypeError):
raise ConversionError(translate('Invalid environment variable name:') + ' ' + key) from None
suffix = quoted[end:].strip()
if suffix and not suffix.startswith('#'):
raise ConversionError(translate('Invalid environment variable name:') + ' ' + key)
else:
value = re.split(r'\s+#', value, maxsplit=1)[0].rstrip()
value = local.expand(value)
result[key] = value
local.values[key] = value
return result
def resolve_document(document, project=None, env_files=None, missing=None):
"""Pure preparation of a full document; preserve every service and image."""
resolver = Interpolator(project, missing)
result = resolver.walk(document)
for service in result.get('services', {}).values():
if not isinstance(service, dict):
raise ConversionError(translate('The Compose file does not contain a Compose document'))
merged = {}
files = service.pop('env_file', [])
if isinstance(files, str):
files = [files]
if not isinstance(files, list):
raise ConversionError(translate('Unresolved variable:') + ' env_file')
for item in files:
if isinstance(item, dict):
if set(item) - {'path', 'required'}:
raise ConversionError(translate('Unresolved variable:') + ' env_file')
filename, required = item.get('path'), item.get('required', True)
else:
filename, required = item, True
if not isinstance(filename, str) or not filename:
raise ConversionError(translate('Unresolved variable:') + ' env_file')
if filename not in (env_files or {}):
if not required:
continue
raise ConversionError(translate('Unresolved variable:') + ' ' + filename)
merged.update(dotenv(env_files[filename], resolver))
environment = service.get('environment', {})
if isinstance(environment, list):
if any(not isinstance(item, str) for item in environment):
raise ConversionError('environment must be a list or a mapping')
pairs = [item.partition('=') for item in environment]
environment = {key: value if separator else resolver.values.get(key)
for key, separator, value in pairs}
if not isinstance(environment, dict):
raise ConversionError('environment must be a list or a mapping')
for key, value in environment.items():
merged[key] = resolver.values.get(str(key)) if value is None else value
service['environment'] = {key: value for key, value in merged.items() if value is not None}
return result
def prepare(ui, document, directory=None, *, metadata=None):
"""Read companion files explicitly, then ask once for each missing value."""
directory = Path(directory) if directory is not None else None
cache = {}
entered = {}
def read(path):
try:
data = Path(path).expanduser().read_bytes()
except OSError as error:
raise ConversionError(translate('The file could not be read:') + ' ' + str(path)) from error
if len(data) > LIMIT:
raise ConversionError(translate('The Compose file is too large'))
try:
return data.decode('utf-8-sig')
except UnicodeError:
raise ConversionError(translate('The Compose file is not valid YAML:') + ' UTF-8') from None
def missing(name):
if name not in entered:
prompt = translate('Value for') + ' ' + name
entered[name] = (ui.password(prompt, required=True) if SECRET.search(name)
else ui.ask(prompt, '', required=True))
return entered[name]
resolver = Interpolator(missing=missing)
default = directory / '.env' if directory else None
if default is not None and default.is_file():
cache['.env'] = read(default)
resolver.values.update({k: v for k, v in dotenv(cache['.env'], resolver).items() if v is not None})
elif re.search(r'\$(?:\{|[A-Za-z_])', yaml.safe_dump(document)):
path = ui.ask(translate('Project environment file (.env), or leave empty to enter values'), '', required=False)
if path:
cache['.env'] = read(path)
resolver.values.update({k: v for k, v in dotenv(cache['.env'], resolver).items() if v is not None})
for service in document.get('services', {}).values():
files = service.get('env_file', []) if isinstance(service, dict) else []
if isinstance(files, str):
files = [files]
if not isinstance(files, list):
raise ConversionError(translate('Unresolved variable:') + ' env_file')
for item in files:
raw = item.get('path', '') if isinstance(item, dict) else item
if not isinstance(raw, str) or not raw:
raise ConversionError(translate('Unresolved variable:') + ' env_file')
filename = resolver.expand(raw)
if filename in cache:
continue
candidate = directory / filename if directory and not Path(filename).is_absolute() else None
if candidate is not None and candidate.resolve().is_relative_to(directory.resolve()) and candidate.is_file():
cache[filename] = read(candidate)
continue
required = item.get('required', True) if isinstance(item, dict) else True
if type(required) is not bool:
raise ConversionError(translate('Unresolved variable:') + ' env_file.required')
if not required:
continue
path = ui.ask(translate('Path of environment file') + ': ' + filename,
str(candidate or filename), required=required)
if path:
cache[filename] = read(path)
result = resolve_document(document, resolver.values, cache, missing)
for service in result.get('services', {}).values():
mounts = []
for mount in service.get('volumes') or []:
if isinstance(mount, str):
source, target, mode = _split_short_mount(mount)
flags = (mode or '').split(',')
if any(flag in {'z', 'Z'} for flag in flags):
if any(flag not in {'', 'ro', 'rw', 'z', 'Z'} for flag in flags) or {'ro','rw'} <= set(flags) or {'z','Z'} <= set(flags):
raise ConversionError(translate('Unsupported volume options:') + ' ' + str(mode))
mount = {'target':target, 'read_only':'ro' in flags}
if source is not None:
mount['source'] = source
if metadata is not None:
metadata['selinux_labels'] = True
mounts.append(mount)
if 'volumes' in service:
service['volumes'] = mounts
return result
oci/src/proxmenux_oci/converter.py
+21 -2
View File
@@ -40,6 +40,7 @@ SUPPORTED_SERVICE_KEYS = {
"shm_size",
"stdin_open",
"stop_grace_period",
"stop_signal",
"sysctls",
"tty",
"ulimits",
@@ -59,6 +60,18 @@ LINUX_CAPABILITIES = {
"SYS_TTY_CONFIG", "SYSLOG", "WAKE_ALARM",
}
# Linux signal names accepted by LXC. Keep independent of the review host OS.
_STOP_SIGNALS = set("HUP INT QUIT ILL TRAP ABRT IOT BUS FPE KILL USR1 SEGV USR2 PIPE ALRM TERM STKFLT CHLD CLD CONT STOP TSTP TTIN TTOU URG XCPU XFSZ VTALRM PROF WINCH IO POLL PWR SYS UNUSED RTMIN RTMAX".split())
def _stop_signal(value: Any) -> str | None:
if not isinstance(value, str):
return None
name = value.strip().upper()
if name.startswith("SIG"):
name = name[3:]
return "SIG" + name if name in _STOP_SIGNALS else None
SENSITIVE_NAME = re.compile(
r"(?:PASS|PASSWORD|TOKEN|SECRET|API_?KEY|PRIVATE_KEY|CREDENTIAL|(?:^|_)KEY(?:$|_))",
re.I,
@@ -463,7 +476,7 @@ def _extra_hosts(value: Any) -> list[dict[str, str]] | None:
for raw_host, raw_address in entries:
host = str(raw_host).strip()
address = str(raw_address).strip()
if not re.fullmatch(r"[A-Za-z0-9](?:[A-Za-z0-9.-]{0,251}[A-Za-z0-9])?", host):
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,252}", host):
return None
if address != "host-gateway" and not re.fullmatch(r"[0-9A-Fa-f:.]+", address):
return None
@@ -889,6 +902,8 @@ def _compose_option_blockers(
blockers.append("mac-address-format")
_, device_blockers = _device_installer_profile(service, optional_devices)
blockers.extend(device_blockers)
if "stop_signal" in service and _stop_signal(service["stop_signal"]) is None:
blockers.append("stop-signal-format")
runtime = service.get("runtime")
if runtime is not None and str(runtime).casefold() != "nvidia":
blockers.append(f"runtime-value:{runtime}")
@@ -949,6 +964,10 @@ def _compose_installer_profile(
profile.setdefault("runtime", {})["command"] = service["command"]
if "entrypoint" in service and service["entrypoint"] is not None:
profile.setdefault("runtime", {})["compose_entrypoint"] = service["entrypoint"]
if service.get("stop_signal") is not None:
halt_signal = _stop_signal(service["stop_signal"])
if halt_signal is not None:
profile.setdefault("runtime", {})["halt_signal"] = halt_signal
if service.get("working_dir") is not None:
profile.setdefault("runtime", {})["working_directory"] = str(service["working_dir"])
if service.get("user") is not None:
@@ -1580,7 +1599,7 @@ def convert_readme(
"environment": "import-image-env-then-apply-compose-overrides",
"user": "import-from-oci-image",
"working_dir": "import-from-oci-image",
"stop_signal": "import-from-oci-image",
"stop_signal": "apply-compose-override" if service.get("stop_signal") is not None else "import-from-oci-image",
},
**(
{"installer_profile": _compose_installer_profile(service, markers["devices"], markers["security_opt"])}
oci/src/proxmenux_oci/custom.py
+202 -39
View File
@@ -2,6 +2,7 @@
is translated into the same template the catalog applications use."""
from __future__ import annotations
import copy
import json
import os
import re
@@ -20,7 +21,7 @@ from .catalog import TIME_SETTINGS
from .cli import install_template
from .i18n import source_text
from .installer import ADVANCED_MODE, DEFAULT_MODE
from .converter import ConversionError
from .converter import ConversionError, _image_contract
from .i18n import N_, translate
from .registry_auth import RegistryError, refuses_access, validate_image
from .ui import UserCancelled
@@ -33,6 +34,18 @@ def _valid_image_reference(reference: str) -> bool:
return False
def _registry_lookup(reference: str) -> str:
"""Use the pull engine's repository/digest form only for registry transport.
Skopeo rejects Docker's valid tag@digest spelling. The written reference
remains the recipe identity; its tag is removed only from the query.
"""
if not _valid_image_reference(reference):
raise ConversionError(f"{translate('This is not a valid image reference:')} {reference}")
image = _image_contract(reference)
return image['repository'] + '@' + image['digest'] if image.get('digest') else reference
MAX_COMPOSE_BYTES = 256 * 1024
# An oci-compose: the file ProxMenux installs an application from. The catalog
# keeps one for each of its applications and every installation records its own.
@@ -53,8 +66,8 @@ class _Dumper(yaml.SafeDumper):
return super().increase_indent(flow, False)
def _dump(document: dict[str, Any]) -> str:
return yaml.dump(document, Dumper=_Dumper, default_flow_style=False, sort_keys=True)
def _dump(document: dict[str, Any], *, sort_keys: bool = True) -> str:
return yaml.dump(document, Dumper=_Dumper, default_flow_style=False, sort_keys=sort_keys)
def _services(compose: dict[str, Any]) -> dict[str, Any]:
@@ -107,22 +120,14 @@ def _check_mounts(service: dict[str, Any]) -> None:
def _keep_reference(template: dict[str, Any], reference: str) -> None:
"""The catalog normalizes every image to its latest tag; an image given by
hand keeps the tag or the digest that was written."""
repository, _, digest = reference.partition('@')
tag = ''
if ':' in repository.rsplit('/', 1)[-1]:
repository, _, tag = repository.rpartition(':')
image = template['container_contract']['image']
image['reference'] = reference if (tag or digest) else f'{repository}:latest'
image['repository'] = repository
image['tag'] = tag or ('' if digest else 'latest')
image['digest'] = f'@{digest}' if digest else None
if tag or digest:
image = _image_contract(reference)
if image.get('digest') or ':' in reference.split('@', 1)[0].rsplit('/', 1)[-1]:
image['pull_policy'] = 'resolve-written-reference-to-architecture-digest-at-install'
template['container_contract']['image'] = image
def template_from_compose(text: str, title: str | None = None) -> dict[str, Any]:
"""The template of a Compose file with a single service. The metadata the
importer expects is taken from the service itself."""
def template_from_compose(text: str, title: str | None = None, *, interpolated: bool = False, image_entrypoints: dict[str, list[str]] | None = None, input_metadata: dict[str, Any] | None = None) -> dict[str, Any]:
"""Translate a Compose application through the native single or stack installer."""
try:
compose = yaml.safe_load(text)
except yaml.YAMLError as error:
@@ -131,9 +136,7 @@ def template_from_compose(text: str, title: str | None = None) -> dict[str, Any]
raise ConversionError(translate('The Compose file does not contain a Compose document'))
services = _services(compose)
if len(services) > 1:
raise ConversionError(
f"{translate('The Compose file describes several services:')} {', '.join(services)}. "
f"{translate('Only one service at a time can be installed this way.')}")
return _template_from_stack(compose, title, interpolated=interpolated, image_entrypoints=image_entrypoints, input_metadata=input_metadata)
name = next(iter(services))
service = services[name] or {}
if not service.get('image'):
@@ -181,12 +184,132 @@ def template_from_compose(text: str, title: str | None = None) -> dict[str, Any]
return template
def _template_from_stack(compose: dict[str, Any], title: str | None, *, interpolated: bool = False, image_entrypoints: dict[str, list[str]] | None = None, input_metadata: dict[str, Any] | None = None) -> dict[str, Any]:
"""Reuse the catalog stack compiler, keeping explicitly selected image versions."""
from .stack import apply_stack_support, normalized_mounts
services = _services(compose)
singles = {}
for name, service in services.items():
if not isinstance(service, dict):
raise ConversionError(f"{translate('The service declares no image:')} {name}")
singles[name] = template_from_compose(_dump({'services': {name: service}}), title)
metadata = compose.get('x-casaos', {})
if metadata is None:
metadata = {}
if not isinstance(metadata, dict):
raise ConversionError('x-casaos.main no identifica un servicio valido')
main = metadata.get('main') or next(
(name for name, service in services.items() if _published_port(service)), next(iter(services)))
if not isinstance(main, str) or main not in services:
raise ConversionError('x-casaos.main no identifica un servicio valido')
document = copy.deepcopy(compose)
document['x-casaos'] = {**metadata, 'main': main,
'title': {'en_US': title or main}}
port = _published_port(services[main])
if port:
document['x-casaos'].update(port_map=port, scheme='http', index='/')
identifier = normalize_app_id(str(compose.get('name') or title or main))
template = convert_casaos_compose(_dump(document), 'local', '',
f'local/{identifier}/docker-compose.yml', '')
template['proxmox']['custom_compose_stack'] = True
template['proxmox']['custom_compose_interpolated'] = interpolated
template['proxmox']['stack_image_entrypoints'] = copy.deepcopy(image_entrypoints or {})
if (input_metadata or {}).get('selinux_labels'):
template['proxmox']['custom_compose_selinux_labels'] = True
_keep_reference(template, str(services[main]['image']))
# Catalog imports use latest. A pasted Compose must keep every written tag/digest.
for service in template['compose_stack']['services']:
reference = singles[service['name']]['container_contract']['image']['reference']
service['image'] = reference
service['compose']['image'] = reference
if 'environment' in services[service['name']]:
service['compose']['environment'] = copy.deepcopy(services[service['name']]['environment'])
service['compose']['volumes'] = [
mount for mount in normalized_mounts(service['compose'].get('volumes', []))
if mount['target'] not in TIME_SETTINGS]
template['container_contract']['original_compose'] = _dump({
**template['compose_stack']['top_level'],
'services': {service['name']: service['compose'] for service in template['compose_stack']['services']}})
for service in template['container_contract']['related_services']:
service['image'] = singles[service['name']]['container_contract']['image']['reference']
preparations = list({entry['container_path']: entry for single in singles.values()
for entry in single['proxmox'].get('installer_profile', {}).get('volume_preparations', [])}.values())
template['proxmox'].setdefault('installer_profile', {})['volume_preparations'] = preparations
apply_stack_support(template)
# Keep compiler diagnostics visible at the custom-image boundary.
if template['proxmox'].get('generic_stack_review'):
template['compatibility']['automatic_install_candidate'] = False
template['compatibility']['untranslated_blockers'] = template['proxmox']['generic_stack_review']
return template
def enrich_image_defaults(ui, document: dict[str, Any], *, image_entrypoints: dict[str, list[str]] | None = None) -> dict[str, Any]:
"""Read implicit ports/health checks from OCI metadata, never an app-name table."""
from .stack import kind
document = copy.deepcopy(document)
inspected = {}
services = _services(document)
for name, service in services.items():
if not isinstance(service, dict) or not service.get('image'):
continue # The converter supplies the normal validation diagnostic.
health = service.get('healthcheck')
inherit_health = isinstance(health, dict) and not health.get('disable') and 'test' not in health
application = kind(str(service['image'])) == 'application'
internal = application and not service.get('ports') and not service.get('expose')
command = service.get('command')
arguments = shlex.split(command) if isinstance(command, str) else command or []
option_command = bool(arguments and isinstance(arguments[0], str) and arguments[0].startswith('-'))
if not internal and not inherit_health and not (application and health is None) and not option_command:
continue
reference = str(service['image'])
if not _valid_image_reference(reference):
raise ConversionError(f"{translate('This is not a valid image reference:')} {reference}")
if reference not in inspected:
command = ['skopeo', 'inspect', '--config', f'docker://{_registry_lookup(reference)}']
result = subprocess.run(command, capture_output=True, text=True, check=False, timeout=120)
if result.returncode and _registry_refused(reference):
from .registry_accounts import offer_sign_in
if offer_sign_in(ui, reference, translate('Image that is not in the catalog')):
result = subprocess.run(command, capture_output=True, text=True, check=False, timeout=120)
if result.returncode:
raise ConversionError(f"{translate('The image could not be read from its registry:')} {reference}")
try:
inspected[reference] = (json.loads(result.stdout).get('config') or {})
except (ValueError, AttributeError):
raise ConversionError(f"{translate('The image could not be read from its registry:')} {reference}") from None
defaults = inspected[reference]
if image_entrypoints is not None:
entrypoint = defaults.get('Entrypoint') or []
if isinstance(entrypoint, list) and all(isinstance(arg, str) for arg in entrypoint):
image_entrypoints[reference] = entrypoint
if internal and service.get('entrypoint') is None and service.get('command') is None:
ports = [port for port in (defaults.get('ExposedPorts') or {})
if str(port).endswith('/tcp') or '/' not in str(port)]
if ports:
service['expose'] = sorted(ports)
# Compose healthcheck overrides inherit the image's test unless disabled.
if (health is None or inherit_health) and (defaults.get('Healthcheck') or {}).get('Test'):
source = defaults['Healthcheck']
effective = {'test': source['Test']}
for old, new in [('Interval', 'interval'), ('Timeout', 'timeout'), ('StartPeriod', 'start_period')]:
if source.get(old):
effective[new] = str(source[old])+'ns'
if source.get('Retries'):
effective['retries'] = source['Retries']
effective.update(health or {})
service['healthcheck'] = effective
elif inherit_health:
service.pop('healthcheck', None)
return document
def compose_from_image(reference: str, title: str | None = None) -> str:
"""A Compose file written from what the image declares: its ports, its
persistent paths and its variables."""
if not _valid_image_reference(reference):
raise ConversionError(f"{translate('This is not a valid image reference:')} {reference}")
result = subprocess.run(['skopeo', 'inspect', '--config', f'docker://{reference}'],
result = subprocess.run(['skopeo', 'inspect', '--config', f'docker://{_registry_lookup(reference)}'],
capture_output=True, text=True, check=False, timeout=120)
if result.returncode != 0:
raise ConversionError(f"{translate('The image could not be read from its registry:')} "
@@ -492,6 +615,7 @@ def read_definition(ui) -> tuple[str, str]:
"""The Compose file of the application and where it came from, or an
oci-compose, named by OCI_COMPOSE."""
from .cli import MENU_SIZE
ui._compose_source_dir = None
source = ui.choose(translate('How is the image described?'), [
('paste', translate('Paste its Compose file in the terminal')),
('file', translate('Read its Compose file from a file of this host')),
@@ -515,8 +639,9 @@ def read_definition(ui) -> tuple[str, str]:
if source == 'paste':
return _read_pasted(ui), translate('pasted Compose file')
if source == 'file':
return _read_file(ui.ask(translate('Path of the Compose file'), '/root/docker-compose.yml')), \
translate('Compose file of this host')
path = ui.ask(translate('Path of the Compose file'), '/root/docker-compose.yml')
ui._compose_source_dir = Path(path).expanduser().resolve().parent
return _read_file(path), translate('Compose file of this host')
if source == 'url':
return _read_url(ui.ask(translate('Address of the Compose file'), '')), translate('downloaded Compose file')
if source == 'run':
@@ -589,7 +714,7 @@ def ignored_settings(text: str) -> list[str]:
notes.append(translate('The container takes its time zone from Proxmox, so the time files '
'of the host are not attached to it.'))
break
if compose.get('networks') or compose.get('volumes'):
if len(compose.get('services') or {}) == 1 and (compose.get('networks') or compose.get('volumes')):
notes.append(translate('The container gets its own address and its own volumes, so the networks '
'and volumes declared in the file are not used.'))
return notes
@@ -598,7 +723,7 @@ def ignored_settings(text: str) -> list[str]:
def registry_report(reference: str) -> tuple[bool, str]:
"""Whether the image really exists in its registry, and for which
architectures. A name written by hand is easy to get wrong."""
result = subprocess.run(['skopeo', 'inspect', '--raw', f'docker://{reference}'],
result = subprocess.run(['skopeo', 'inspect', '--raw', f'docker://{_registry_lookup(reference)}'],
capture_output=True, text=True, check=False, timeout=120)
if result.returncode != 0:
return False, f"{translate('Could not inspect the image in its registry:')} {reference}"
@@ -609,7 +734,7 @@ def registry_report(reference: str) -> tuple[bool, str]:
manifests = document.get('manifests')
if not manifests:
# The image publishes a single manifest: its architecture is in the image itself.
detail = subprocess.run(['skopeo', 'inspect', f'docker://{reference}'],
detail = subprocess.run(['skopeo', 'inspect', f'docker://{_registry_lookup(reference)}'],
capture_output=True, text=True, check=False, timeout=120)
try:
architecture = json.loads(detail.stdout).get('Architecture') if detail.returncode == 0 else None
@@ -627,9 +752,9 @@ def registry_report(reference: str) -> tuple[bool, str]:
def _registry_refused(reference: str) -> bool:
"""Whether the registry answered that the image needs an account."""
try:
result = subprocess.run(['skopeo', 'inspect', '--raw', f'docker://{reference}'],
result = subprocess.run(['skopeo', 'inspect', '--raw', f'docker://{_registry_lookup(reference)}'],
capture_output=True, text=True, check=False, timeout=120)
except (OSError, subprocess.TimeoutExpired):
except (OSError, subprocess.TimeoutExpired, ConversionError):
return False
return result.returncode != 0 and refuses_access(result.stderr)
@@ -642,6 +767,13 @@ def describe(template: dict[str, Any]) -> str:
profile = proxmox.get('installer_profile', {}) or {}
security = proxmox.get('security_profile', {}) or {}
lines = [f"{translate('Image') + ':':<14} {contract['image']['reference']}"]
if proxmox.get('custom_compose_selinux_labels'):
lines += [translate('SELinux volume labels are not applied on this Proxmox host.')]
services = (template.get('compose_stack') or {}).get('services') or []
if len(services) > 1:
lines += ['', translate('Containers of the application:')]
lines += [f" {service['name']}: {service['image']}" for service in services]
endpoints = template.get('first_run', {}).get('endpoints') or []
if endpoints:
# The address is read from the published port; the image does not say
@@ -677,9 +809,11 @@ def describe(template: dict[str, Any]) -> str:
for volume in volumes]
environment = contract.get('environment') or []
if environment:
lines += ['', translate('Variables the installation asks for:')]
# A stack takes its values from the Compose file; none is asked here.
lines += ['', translate('Variables the installation asks for:') if len(services) <= 1 else
f"{translate('Variables')} ({template['compose_stack']['main_service']}):"]
lines += [f" {item['name']}"
+ (f" = {item['example']}" if item.get('example') and not item['sensitive'] else '')
+ (f" = {item['example']}" if item.get('example') and not item['sensitive'] and len(services) <= 1 else '')
+ ('' if item.get('required', True) else f" ({translate('optional')})")
for item in environment]
devices = profile.get('device_requests') or []
@@ -720,13 +854,17 @@ def _ask_secrets(template: dict[str, Any]) -> list[str]:
def _name(ui, template: dict[str, Any]) -> str:
default = normalize_app_id(source_text(template['catalog_ui'].get('title'))
stack = template.get('compose_stack') if len((template.get('compose_stack') or {}).get('services') or []) > 1 else None
default = normalize_app_id((stack or {}).get('project_name') or source_text(template['catalog_ui'].get('title'))
or template['container_contract'].get('container_name') or 'oci-app')
name = normalize_app_id(ui.ask(translate('Name for this application'), default))
if not name:
raise UserCancelled(translate('No name was given'))
template['catalog_ui']['title'] = {'en_US': name}
template['container_contract']['container_name'] = name
if stack:
# The containers of the stack are named after it.
stack['project_name'] = name
return name
@@ -741,16 +879,34 @@ def _explore(ui) -> None:
else:
notes = [line.partition(':')[2].strip() for line in text.splitlines() if line.startswith('#note:')]
notes += ignored_settings(text)
template = template_from_compose(text)
try:
document = yaml.safe_load(text)
except yaml.YAMLError as error:
raise ConversionError(f"{translate('The Compose file is not valid YAML:')} {error}") from error
multiple = isinstance(document, dict) and len(document.get('services') or {}) > 1
if multiple:
from .compose_input import prepare
input_metadata = {}
document = prepare(ui, document, getattr(ui, '_compose_source_dir', None), metadata=input_metadata)
text = _dump(document, sort_keys=False)
template = template_from_compose(text, interpolated=True, input_metadata=input_metadata) if multiple else template_from_compose(text)
title = translate('Image that is not in the catalog')
summary = describe(template) + (describe_oci_compose(template) if written else '')
reference = template['container_contract']['image']['reference']
available, report = registry_report(reference)
if not available and _registry_refused(reference):
from .registry_accounts import offer_sign_in
if offer_sign_in(ui, reference, title):
available, report = registry_report(reference)
summary += '\n\n' + report
stack_services = (template.get('compose_stack') or {}).get('services') or []
references = list(dict.fromkeys(service['image'] for service in stack_services)) if len(stack_services) > 1 else [template['container_contract']['image']['reference']]
available = True
reports = []
for reference in references:
accessible, report = registry_report(reference)
if not accessible and _registry_refused(reference):
from .registry_accounts import offer_sign_in
if offer_sign_in(ui, reference, title):
accessible, report = registry_report(reference)
available = available and accessible
if accessible and len(references) > 1:
report = f'{reference}: {report}'
reports.append(report)
summary += '\n\n' + report
if notes:
summary += '\n\n' + '\n'.join(notes)
if not available:
@@ -758,11 +914,18 @@ def _explore(ui) -> None:
'try again later. For a private image, sign in from Registry accounts in the OCI menu.')
ui.message(f'{summary}\n\n{advice}', title)
return
if not written and multiple:
image_entrypoints = {}
document = enrich_image_defaults(ui, document, image_entrypoints=image_entrypoints)
template = template_from_compose(_dump(document, sort_keys=False), interpolated=True, image_entrypoints=image_entrypoints, input_metadata=input_metadata)
summary = describe(template) + ''.join('\n\n'+report for report in reports)
if notes:
summary += '\n\n'+'\n'.join(notes)
if not template.get('compatibility', {}).get('automatic_install_candidate'):
ui.message(f"{translate('This image cannot be installed as it is described:')}\n\n{summary}", title)
return
# An oci-compose already says which of its values are secret and which it generates.
secrets = [] if written else _ask_secrets(template)
secrets = [] if written or len(stack_services) > 1 else _ask_secrets(template)
if secrets:
summary += ('\n\n' + translate('These values are asked during the installation:') + ' '
+ ', '.join(secrets))
oci/src/proxmenux_oci/installer.py
+2 -2
View File
@@ -1598,7 +1598,7 @@ def _run_remote_install(
shutil.copy2(project_root / 'remote' / 'oci_installation_state.py', temporary / 'oci_installation_state.py')
shutil.copy2(project_root / 'remote' / 'oci_instances.py', temporary / 'oci_instances.py')
shutil.copy2(project_root / 'remote' / 'oci_image_mountpoints.py', temporary / 'oci_image_mountpoints.py')
for helper in ('oci_ui.sh', 'oci_ui.py', 'oci_description.py', 'oci_console.py', 'oci_native_stack.py', 'oci_native_stack.sh', 'oci_instance_transaction.py', 'oci_managed_volumes.py', 'oci_host_mounts.py', 'oci_nested_mounts.py', 'oci_runtime_settings.py', 'oci_gpu_devices.py', 'oci_accelerators.py', 'oci_nvidia_runtime.py', 'oci_nvidia_refresh.py', 'oci_nvidia_dynamic.py', 'oci_update_current.py', 'oci_resource_modify.py', 'oci_native_modify.py', 'oci_stack_replay.py', 'oci_stack_plan.py', 'oci_stack_transaction.py', 'oci_stack_native.py', 'oci_image_cache.py', 'nvidia_lxc_mount_lab.sh', 'oci_nvidia_setup.sh', 'oci_immich_ml.sh', 'oci_rocm_check.py'):
for helper in ('oci_carried_record.py', 'oci_compose_jobs.py', 'oci_stack_public_routes.py', 'oci_ui.sh', 'oci_ui.py', 'oci_description.py', 'oci_console.py', 'oci_native_stack.py', 'oci_native_stack.sh', 'oci_instance_transaction.py', 'oci_managed_volumes.py', 'oci_host_mounts.py', 'oci_nested_mounts.py', 'oci_runtime_settings.py', 'oci_gpu_devices.py', 'oci_accelerators.py', 'oci_nvidia_runtime.py', 'oci_nvidia_refresh.py', 'oci_nvidia_dynamic.py', 'oci_update_current.py', 'oci_resource_modify.py', 'oci_native_modify.py', 'oci_stack_replay.py', 'oci_stack_plan.py', 'oci_stack_transaction.py', 'oci_stack_native.py', 'oci_image_cache.py', 'nvidia_lxc_mount_lab.sh', 'oci_nvidia_setup.sh', 'oci_immich_ml.sh', 'oci_rocm_check.py'):
shutil.copy2(project_root / 'remote' / helper, temporary / helper)
if deployment_kind == 'generic-multi-lxc-stack':
shutil.copy2(project_root / 'remote' / 'install_generic_stack.py', temporary / 'install_generic_stack.py')
@@ -1635,7 +1635,7 @@ def _run_remote_install(
archive.add(project_root / 'remote' / 'haos_healthcheck.py', arcname='haos_healthcheck.py')
archive.add(project_root / 'remote' / 'oci_installation_state.py', arcname='oci_installation_state.py')
archive.add(project_root / 'remote' / 'oci_instances.py', arcname='oci_instances.py')
for helper in ('oci_ui.sh', 'oci_ui.py', 'oci_description.py', 'oci_console.py', 'oci_native_stack.py', 'oci_native_stack.sh', 'oci_instance_transaction.py', 'oci_managed_volumes.py', 'oci_image_mountpoints.py', 'oci_host_mounts.py', 'oci_nested_mounts.py', 'oci_runtime_settings.py', 'oci_gpu_devices.py', 'oci_accelerators.py', 'oci_nvidia_runtime.py', 'oci_nvidia_refresh.py', 'oci_nvidia_dynamic.py', 'oci_update_current.py', 'oci_resource_modify.py', 'oci_native_modify.py', 'oci_stack_replay.py', 'oci_stack_plan.py', 'oci_stack_transaction.py', 'oci_stack_native.py', 'oci_image_cache.py', 'nvidia_lxc_mount_lab.sh', 'oci_nvidia_setup.sh', 'oci_immich_ml.sh', 'oci_rocm_check.py'):
for helper in ('oci_carried_record.py', 'oci_compose_jobs.py', 'oci_stack_public_routes.py', 'oci_ui.sh', 'oci_ui.py', 'oci_description.py', 'oci_console.py', 'oci_native_stack.py', 'oci_native_stack.sh', 'oci_instance_transaction.py', 'oci_managed_volumes.py', 'oci_image_mountpoints.py', 'oci_host_mounts.py', 'oci_nested_mounts.py', 'oci_runtime_settings.py', 'oci_gpu_devices.py', 'oci_accelerators.py', 'oci_nvidia_runtime.py', 'oci_nvidia_refresh.py', 'oci_nvidia_dynamic.py', 'oci_update_current.py', 'oci_resource_modify.py', 'oci_native_modify.py', 'oci_stack_replay.py', 'oci_stack_plan.py', 'oci_stack_transaction.py', 'oci_stack_native.py', 'oci_image_cache.py', 'nvidia_lxc_mount_lab.sh', 'oci_nvidia_setup.sh', 'oci_immich_ml.sh', 'oci_rocm_check.py'):
archive.add(project_root / 'remote' / helper, arcname=helper)
if deployment_kind == 'generic-multi-lxc-stack':
archive.add(project_root / 'remote' / 'install_generic_stack.py', arcname='install_generic_stack.py')
oci/src/proxmenux_oci/stack.py
+337 -29
View File
@@ -5,13 +5,16 @@ import copy
import base64
import re
import secrets
import hashlib
import math
import shlex
from urllib.parse import urlsplit, urlunsplit, quote
import yaml
from .casaos import convert_casaos_compose, canonical_image_repository
from .converter import ConversionError
from .converter import _split_short_mount
from .converter import _image_contract, _split_short_mount, _port_contract
from .i18n import translate
@@ -51,6 +54,12 @@ def environment(value):
def kind(image):
repo = canonical_image_repository(image)
for prefix in ('docker.io/', 'index.docker.io/', 'registry-1.docker.io/'):
if repo.startswith(prefix):
repo = repo[len(prefix):]
break
if repo.startswith('library/'):
repo = repo[len('library/'):]
return repo if repo in {'postgres', 'redis', 'valkey/valkey', 'mariadb', 'linuxserver/mariadb', 'mongo', 'getmeili/meilisearch'} else 'application'
@@ -77,6 +86,47 @@ def authenticated_redis(service):
and bool(environment(c.get('environment')).get('REDISCLI_AUTH')))
def custom_dependency_command(service, image_entrypoint=None):
"""Known server commands may add flags; another executable is not equivalent."""
if service['compose'].get('entrypoint'):
return False
command = service['compose'].get('command')
if not command:
return True
try:
argv = shlex.split(command) if isinstance(command, str) else command
except ValueError:
return False
if (not isinstance(argv, list) or not argv or any(not isinstance(arg, str) for arg in argv)):
return False
allowed = {'postgres': {'postgres'}, 'redis': {'redis-server'},
'valkey/valkey': {'valkey-server'}, 'mariadb': {'mariadbd', 'mysqld'},
'linuxserver/mariadb': {'mariadbd', 'mysqld'}, 'mongo': {'mongod'},
'getmeili/meilisearch': {'meilisearch'}}
# Password-protected Redis still needs its own validated auth check.
if kind(service['image']) in {'redis', 'valkey/valkey'} and any(
arg.split('=', 1)[0] in {'--requirepass', '--masterauth', '--aclfile'} for arg in argv):
return authenticated_redis(service)
if argv[0].startswith('-'):
# Docker passes options through the actual image entrypoint. Do not
# guess or prepend a database executable on behalf of the image.
return (isinstance(image_entrypoint, list) and bool(image_entrypoint)
and all(isinstance(arg, str) and arg for arg in image_entrypoint))
return argv[0] in allowed.get(kind(service['image']), set())
def completed_services(template):
"""Jobs are declared by Compose's successful-completion dependency condition."""
jobs = set()
for service in template.get('compose_stack', {}).get('services', []):
dependencies = service['compose'].get('depends_on') or {}
if isinstance(dependencies, dict):
for name, requirement in dependencies.items():
if isinstance(requirement, dict) and requirement.get('condition') == 'service_completed_successfully':
jobs.add(name)
return jobs
def ordered_services(template):
stack = template['compose_stack']
services = {s['name']: copy.deepcopy(s) for s in stack['services']}
@@ -104,7 +154,10 @@ def ordered_services(template):
for dep in raw:
if isinstance(raw, dict):
condition = raw[dep].get('condition', 'service_started')
if condition not in {'service_started', 'service_healthy'}:
allowed = {'service_started', 'service_healthy'}
if template.get('proxmox', {}).get('custom_compose_stack'):
allowed.add('service_completed_successfully')
if condition not in allowed:
raise StackError(f"{name}: {translate('unsupported dependency condition')} {condition}")
if raw[dep].get('required', True) is False:
raise StackError(f"{name}: {translate('optional dependencies are not yet supported')}")
@@ -143,7 +196,10 @@ def service_template(parent, service, main=False):
parent['catalog_ui'].get('category_label'), service['name'],
)
# Preserve selected image and resolved stack values; the importer normalizes secrets.
single['container_contract']['image']['reference'] = service['image']
image = _image_contract(service['image'])
if image.get('digest') or ':' in service['image'].split('@', 1)[0].rsplit('/', 1)[-1]:
image['pull_policy'] = 'resolve-written-reference-to-architecture-digest-at-install'
single['container_contract']['image'] = image
single['container_contract']['environment'] = [
{'name': k, 'example': v, 'required': True, 'sensitive': True, 'prompt_user': False}
for k, v in c['environment'].items()
@@ -166,36 +222,211 @@ def service_template(parent, service, main=False):
return single
def service_endpoint(service, include_expose=False):
"""Return a TCP port; only a published port implies a web frontend here."""
ports = list(service['compose'].get('ports') or [])
if include_expose:
ports += list(service['compose'].get('expose') or [])
for port in ports:
if isinstance(port, dict):
protocol = port.get('protocol', 'tcp')
target = port.get('target')
else:
raw, _, protocol = str(port).partition('/')
protocol = protocol or 'tcp'
target = raw.rsplit(':', 1)[-1]
if protocol == 'tcp' and str(target).isdigit() and 0 < int(target) < 65536:
return {'scheme': 'http', 'port': int(target), 'path': '/'}
return None
def public_environment_bindings(services, main_name, environments=None):
"""Bind loopback URL origins to the service publishing their Compose host port."""
frontends = [service for service in services if service_endpoint(service)]
result = {}
for service in services:
bindings = {}
values = (environments or {}).get(service['name'], environment(service['compose'].get('environment')))
for key, value in values.items():
words = key.upper().split('_')
if not ({'PUBLIC', 'EXTERNAL'} & set(words[:-1]) and words[-1] in {'URL', 'URI', 'ORIGIN'}):
continue
if not re.match(r'^https?://', value, re.I):
continue
url = urlsplit(value)
if url.scheme not in {'http', 'https'} or url.hostname not in {'localhost', '127.0.0.1', '0.0.0.0', '::1'}:
continue
try:
origin_port = url.port or (443 if url.scheme == 'https' else 80)
except ValueError as error:
raise StackError(translate('Invalid service check URL')) from error
matches = set()
for target in frontends:
for port in _port_contract(target['compose'].get('ports'), set()):
if (port['protocol'] == 'tcp' and port.get('published_example') == origin_port
and not port.get('container_port_end') and not port.get('published_example_end')):
matches.add((target['name'], port['container_port']))
if len(matches) > 1:
raise StackError(f"{service['name']}: {key}: {translate('Ambiguous local public URL; use an explicit URL')}")
if not matches:
raise StackError(f"{service['name']}: {key}: {translate('Local public URL has no matching published port; use an explicit URL')}")
target_name, target_port = next(iter(matches))
credentials = url.netloc.rpartition('@')[0]+'@' if '@' in url.netloc else ''
rendered = urlunsplit((url.scheme, credentials+'@STACK_LAN_IP@:'+str(target_port),
url.path, url.query, url.fragment))
bindings[key] = {'target_service': target_name, 'container_port': target_port, 'value': rendered}
if bindings:
result[service['name']] = bindings
return result
def _check_duration(value, default):
if value is None:
return default
text = str(value).strip().lower()
parts = list(re.finditer(r'(\d+(?:\.\d+)?)(ns|us|ms|h|m|s)', text))
if not parts or ''.join(item.group() for item in parts) != text:
raise StackError(translate('Invalid service check timeout'))
factors = {'ns': 1e-9, 'us': 1e-6, 'ms': 1e-3, 's': 1, 'm': 60, 'h': 3600}
duration = sum(float(item.group(1)) * factors[item.group(2)] for item in parts)
if not math.isfinite(duration) or duration < 0 or duration > 3600:
raise StackError(translate('Invalid service check timeout'))
return duration
def compose_healthcheck(service):
"""An explicit worker readiness command executed inside its own LXC."""
raw = service['compose'].get('healthcheck')
if raw is None:
return None
if not isinstance(raw, dict):
raise StackError(translate('Invalid service check arguments'))
if raw.get('disable') is True:
return None
test = raw.get('test')
if test == ['NONE']:
return None
if isinstance(test, str):
test = ['CMD-SHELL', test]
if (not isinstance(test, list) or len(test) < 2
or any(not isinstance(arg, str) or any(ch in arg for ch in '\0\n\r') for arg in test)):
raise StackError(translate('Invalid service check arguments'))
if test[0] == 'CMD':
argv = test[1:]
elif test[0] == 'CMD-SHELL' and len(test) == 2 and test[1].strip():
# pct exec does not promise the image PID1 environment. Reject checks
# that would silently evaluate an unset runtime variable.
reference = re.search(r'\$(?:[A-Za-z_][A-Za-z0-9_]*|\{[^}]+\})', test[1])
if reference:
raise StackError(f"healthcheck: {translate('needs stack review')}: {reference.group(0)}")
argv = ['sh', '-c', test[1]]
else:
raise StackError(translate('Invalid service check arguments'))
if not argv or not argv[0].strip():
raise StackError(translate('Empty exec service check'))
interval = _check_duration(raw.get('interval'), 30)
request = _check_duration(raw.get('timeout'), 5)
start_period = _check_duration(raw.get('start_period'), 0)
retries = raw.get('retries', 3)
if (type(retries) is not int or retries < 1 or retries > 1000
or interval <= 0 or request <= 0):
raise StackError(translate('Invalid service check timeout'))
total = max(30, math.ceil(start_period + (interval + request) * retries))
if total > 3600:
raise StackError(translate('Invalid service check timeout'))
return {'type': 'exec', 'argv': argv, 'timeout_seconds': total}
def readiness_mode(service):
"""A declared healthy check is distinct from port liveness or running only."""
if compose_healthcheck(service) or kind(service['image']) != 'application':
return 'exec'
if service_endpoint(service):
return 'http'
if service_endpoint(service, include_expose=True):
return 'tcp'
return 'running'
def _entrypoint_for(template, service):
entries = template.get('proxmox', {}).get('stack_image_entrypoints', {})
reference = service['image']
return entries.get(reference) or (entries.get(reference[:-7]) if reference.endswith(':latest') else None)
def shared_sources(services):
"""Source identities mounted more than once, before storage is chosen."""
users = {}
for service in services:
for mount in normalized_mounts(service['compose'].get('volumes', [])):
source = mount.get('source')
if source:
users.setdefault(source, []).append(service['name'])
return {source for source, names in users.items() if len(names) > 1}
def assess(template):
"""Reject untranslated semantics before any host resources are allocated."""
errors = []
custom = template.get('proxmox', {}).get('custom_compose_stack', False)
try:
services = ordered_services(template)
literal = template.get('proxmox', {}).get('custom_compose_interpolated', False)
top = template['compose_stack'].get('top_level', {})
if any(k not in {'name', 'networks', 'volumes', 'version'} for k in top):
if any(k not in {'name', 'networks', 'volumes', 'version'}
and not (custom and str(k).startswith('x-')) for k in top):
raise StackError(translate('Top-level configs, secrets and other global options are not yet supported'))
networks = top.get('networks', {})
if len(networks) > 1 or any(v and any(not ((k == 'driver' and x == 'bridge') or (k == 'name' and isinstance(x,str))) for k,x in v.items()) for v in networks.values()):
raise StackError(translate('Multiple networks or external networks are not yet supported'))
if any(v for v in top.get('volumes', {}).values()):
raise StackError(translate('Top-level volume options are not yet supported'))
healthy_required = set()
# Short-form dependencies mean service_started; collect long-form
# healthy requirements even when other services use the short form.
for service in services:
dependencies = service['compose'].get('depends_on') or {}
if isinstance(dependencies, dict):
healthy_required.update(name for name, condition in dependencies.items()
if isinstance(condition, dict) and condition.get('condition') == 'service_healthy')
jobs = completed_services(template) if custom else set()
if custom:
public_environment_bindings(services, template['compose_stack']['main_service'])
if template['compose_stack']['main_service'] in jobs or jobs & healthy_required:
raise StackError(translate('unsupported dependency condition') + ': service_completed_successfully')
for service in services:
if service['name'] in jobs and service['compose'].get('restart') not in (None, 'no'):
raise StackError(translate('unsupported dependency condition') + ': restart')
seen_sources = set()
for s in services:
c = s['compose']
for key in ('profiles','build','configs','secrets','env_file','pid','privileged','devices','runtime','cap_add','security_opt','network_mode','extra_hosts'):
if c.get(key):
errors.append(f"{s['name']}: {key} {translate('needs stack review')}")
try:
explicit_check = compose_healthcheck(s) if custom else None
except StackError as error:
errors.append(f"{s['name']}: {error}")
explicit_check = None
if (custom and s['name'] == template['compose_stack']['main_service']
and explicit_check and service_endpoint(s) is None):
errors.append(f"{s['name']}: {translate('Main endpoint not yet defined')}")
if s['name'] != template['compose_stack']['main_service'] and kind(s['image']) == 'application':
errors.append(f"{s['name']}: {translate('health and persistence profile not yet defined')}")
if kind(s['image']) != 'application' and (c.get('command') or c.get('entrypoint')) and not authenticated_redis(s):
if not custom:
errors.append(f"{s['name']}: {translate('health and persistence profile not yet defined')}")
elif s['name'] in healthy_required and explicit_check is None:
errors.append(f"{s['name']}: service_healthy {translate('needs stack review')}: "
f"{translate('health and persistence profile not yet defined')}")
if (kind(s['image']) != 'application' and (c.get('command') or c.get('entrypoint'))
and not authenticated_redis(s) and not (custom and custom_dependency_command(s, _entrypoint_for(template, s)))):
errors.append(f"{s['name']}: {translate('custom dependency commands are not yet supported')}")
for m in normalized_mounts(c.get('volumes', [])):
source = m.get('source')
if source is not None and source in seen_sources:
if source is not None and source in seen_sources and not custom:
errors.append(translate('Volumes shared between services are not yet supported'))
if source is not None:
seen_sources.add(source)
if set(m) - {'type','source','target','read_only'} or m.get('read_only'):
if set(m) - {'type','source','target','read_only'} or (m.get('read_only') and not custom):
errors.append(translate('Volume options are not yet supported'))
if m['target'].startswith(('/etc/', '/var/run/', '/run/', '/dev/', '/proc/', '/sys/')):
errors.append(translate('System path mounts are not yet supported'))
@@ -203,27 +434,38 @@ def assess(template):
for k,v in env.items():
for a,b in VARIABLE.findall(v):
variable = a or b
if variable not in {'TZ','PUID','PGID'} and not variable.startswith('GENERATED_'):
if not literal and variable not in {'TZ','PUID','PGID'} and not variable.startswith('GENERATED_'):
errors.append(f'{s["name"]}: {translate("unsupported variable")} {variable}')
if variable.startswith('GENERATED_') and re.search(r'API_KEY|CLIENT_SECRET|CREDENTIALS_ENABLED', variable):
errors.append(f'{s["name"]}: {translate("unsupported external credential or boolean")} {variable}')
single = service_template(template,s)
errors.extend(f'{s["name"]}: {b}' for b in single['compatibility']['untranslated_blockers'])
if custom:
for source in shared_sources(services):
owners = set()
for service in services:
if any(m.get('source') == source for m in normalized_mounts(service['compose'].get('volumes', []))):
env = environment(service['compose'].get('environment'))
owners.add((env.get('PUID', '1000'), env.get('PGID', '1000')))
if len(owners) > 1:
errors.append(f"{source}: {translate('Volumes shared between services are not yet supported')}")
if not template['first_run'].get('endpoints'):
errors.append(translate('Main endpoint not yet defined'))
resolve_environments(services, 'UTC', template.get('proxmox', {}).get('stack_generators', {}))
resolve_environments(services, 'UTC', template.get('proxmox', {}).get('stack_generators', {}), literal=literal)
except (ValueError, KeyError, TypeError, ConversionError) as e:
errors.append(str(e))
return sorted(set(errors))
def resolve_environments(services, timezone, generators=None):
def resolve_environments(services, timezone, generators=None, literal=False):
tokens = {'TZ': timezone, 'PUID': '1000', 'PGID': '1000'}
resolved = {}
for s in services:
env = environment(s['compose'].get('environment'))
def replace(match):
name = match.group(1) or match.group(2)
if literal:
return match.group(0)
if name.startswith('GENERATED_'):
if name not in tokens:
generator = (generators or {}).get(name)
@@ -237,7 +479,10 @@ def resolve_environments(services, timezone, generators=None):
raise StackError(f"{translate('Unresolved variable:')} {name}")
return tokens[name]
resolved[s['name']] = {k: VARIABLE.sub(replace, v) for k,v in env.items()}
# Bind URL credentials using the destination service, never a global text replacement.
if literal:
return resolved
# Catalog-generated credentials are bound only for legacy catalog definitions.
# A fully interpolated custom Compose keeps its explicit connection strings.
databases = {}
for s in services:
if kind(s['image']) == 'postgres':
@@ -300,10 +545,15 @@ def build_stack(template, ui, mode='advanced'):
raise StackError('; '.join(problems))
services = copy.deepcopy(ordered_services(template))
defaults = template['proxmox']['defaults']
custom = template.get('proxmox', {}).get('custom_compose_stack', False)
jobs = completed_services(template) if custom else set()
shared = shared_sources(services) if custom else set()
shared_paths = {}
# A default installation takes the name, the VMID and the settings from the
# recipe; it still asks the storage, the address and how the stack starts.
quiet = DefaultsUI() if mode == DEFAULT_MODE else ui
name = _hostname_default(quiet.ask(translate('Stack name'), template['compose_stack']['project_name']))
# An imported Compose file was already named before its installation.
name = _hostname_default((DefaultsUI() if custom else quiet).ask(translate('Stack name'), template['compose_stack']['project_name']))
vmid = quiet.ask(translate('Base VMID (empty = next free block)'), '', required=False)
from . import host
from . import network as access
@@ -315,25 +565,55 @@ def build_stack(template, ui, mode='advanced'):
root = ask_storage(ui, translate('Storage for rootfs'), 'rootdir', defaults['rootfs_storage'])
cache = ask_storage(ui, translate('Storage for the OCI image cache'), 'vztmpl', defaults['template_storage'], mode)
generators = template.get('proxmox', {}).get('stack_generators', {})
literal = template.get('proxmox', {}).get('custom_compose_interpolated', False)
# The data paths of every member are settled before the network and the settings.
volumes = root
draft_envs = resolve_environments(copy.deepcopy(services), host.timezone(), generators)
draft_envs = resolve_environments(copy.deepcopy(services), host.timezone(), generators, literal=literal)
drafts = []
selected_resources = {}
for s in services:
draft = copy.deepcopy(s)
draft['compose']['environment'] = draft_envs[s['name']]
_, plan = _service_plan(template, draft)
if mode != DEFAULT_MODE and s['name'] == template['compose_stack']['main_service']:
resources = {'cores': int(ui.ask(translate('CPU cores'), str(plan['resources']['cores']))),
'memory_mb': int(ui.ask(translate('Memory in MB'), str(plan['resources']['memory_mb'])))}
if mode != DEFAULT_MODE and (custom or s['name'] == template['compose_stack']['main_service']):
prefix = s['name']+': ' if custom and s['name'] != template['compose_stack']['main_service'] else ''
resources = {'cores': int(ui.ask(prefix+translate('CPU cores'), str(plan['resources']['cores']))),
'memory_mb': int(ui.ask(prefix+translate('Memory in MB'), str(plan['resources']['memory_mb'])))}
if resources['cores'] < 1 or resources['memory_mb'] < 256:
raise StackError(translate('Invalid resources'))
selected_resources[s['name']] = resources
# The paths of the application itself hold the user's data, so where
# they go is asked in a default installation too; the databases and
# caches beside it keep the values of the recipe.
asked = mode != DEFAULT_MODE or s['name'] == template['compose_stack']['main_service']
source_mounts = {m['target']: m for m in normalized_mounts(s['compose'].get('volumes', []))}
for m in plan['mounts']:
label = f"{s['name']}: {m['container_path']}"
source_mount = source_mounts.get(m['container_path'], {})
source = source_mount.get('source')
if source in shared:
if source not in shared_paths:
slug = re.sub(r'[^A-Za-z0-9_.-]+', '-', source).strip('.-') or 'data'
identity = hashlib.sha256(source.encode()).hexdigest()[:8]
default_path = source if source.startswith('/') else '/mnt/oci-shared/'+name+'/shared/'+slug+'-'+identity
shared_paths[source] = ui.ask(f"{translate('Host path for')} {source}", default_path)
chosen = shared_paths[source]
if not chosen.startswith('/') or any(c in chosen for c in ',\n\r') or '..' in chosen.split('/'):
raise StackError(translate('Invalid volume target'))
if chosen.rstrip('/') in {'', '/etc', '/var', '/usr', '/root', '/dev', '/proc', '/sys', '/run', '/boot', '/bin', '/sbin', '/lib', '/lib64'}:
raise StackError(translate('Invalid volume target'))
env = draft_envs[s['name']]
m.update(type='host-bind', source=shared_paths[source], backup=False, size_gb=None,
read_only=bool(source_mount.get('read_only')))
# Without an explicit PUID/PGID, use the imported image user
# when creating the directory, just like private volumes.
if 'PUID' in env or 'PGID' in env:
uid, gid = env.get('PUID', '1000'), env.get('PGID', '1000')
if not str(uid).isdigit() or not str(gid).isdigit() or int(uid) > 65535 or int(gid) > 65535:
raise StackError(translate('Invalid resources'))
offset = 100000 if plan.get('security', {}).get('unprivileged', True) else 0
m['host_owner'] = [int(uid)+offset, int(gid)+offset]
continue
mount_mode = (ui.choose(f"{translate('Where to store')} {label}",
[('managed-volume', translate('Container volume (included in backups)')),
('host-bind', translate('Host directory (not included in Proxmox backups)'))],
@@ -358,9 +638,11 @@ def build_stack(template, ui, mode='advanced'):
from .custom_mounts import ask_stack_custom_mounts
ask_stack_custom_mounts(ui, drafts, volumes)
bridge = ask_bridge(ui, translate('Access bridge'), defaults['bridge'], mode)
addresses, gateway = access.ask_addresses(ui, bridge, [''])
frontend_names = [s['name'] for s in services if s['name'] == template['compose_stack']['main_service'] or (custom and service_endpoint(s))]
address_labels = frontend_names if custom else ['']
addresses, gateway = access.ask_addresses(ui, bridge, address_labels)
timezone = quiet.ask(translate('Timezone'), host.timezone())
envs = resolve_environments(services, timezone, generators)
envs = resolve_environments(services, timezone, generators, literal=literal)
for group in template.get('proxmox', {}).get('stack_optional_environment', []):
service_name = group['service']
if service_name not in envs:
@@ -375,13 +657,18 @@ def build_stack(template, ui, mode='advanced'):
if not value or any(c in value for c in '\r\n'):
raise StackError(translate('External credential is empty or spans multiple lines'))
envs[service_name][field['name']] = value
public_bindings = public_environment_bindings(services, template['compose_stack']['main_service'], envs) if custom else {}
if any(addresses[binding['target_service']] == 'dhcp' for bindings in public_bindings.values() for binding in bindings.values()):
raise StackError(translate('Automatic public URLs require static frontend addresses or explicit URLs'))
plans = []
for index, s in enumerate(services):
main = s['name'] == template['compose_stack']['main_service']
env = envs[s['name']]
# Public application URLs cannot retain localhost in a remote deployment.
for key,value in list(env.items()):
if value.startswith(('http://localhost', 'http://127.0.0.1', 'http://0.0.0.0')):
if custom and key in public_bindings.get(s['name'], {}):
env[key] = public_bindings[s['name']][key]['value']
elif not custom and value.startswith(('http://localhost', 'http://127.0.0.1', 'http://0.0.0.0')):
url = urlsplit(value)
endpoint = template['first_run']['endpoints'][0]
env[key] = urlunsplit((url.scheme, '@STACK_LAN_IP@:'+str(endpoint['port']), url.path, url.query, url.fragment))
@@ -389,16 +676,27 @@ def build_stack(template, ui, mode='advanced'):
single, plan = _service_plan(template, s)
k = kind(s['image'])
# Values are already resolved; do not reinterpret user credentials as Compose variables.
plan['environment'] = [{'name':key,'value':value,'sensitive':True} for key,value in env.items() if value != '']
plan['environment'] = [{'name':key,'value':value,'sensitive':True} for key,value in env.items() if value != '' or literal]
plan.update(hostname=_hostname_default(name+'-'+s['name']), template_storage=cache,
start_after_create=False)
plan['rootfs']['storage'] = root
if 'memory_default_mb' not in single['proxmox'].get('installer_profile', {}).get('resources', {}):
plan['resources']['memory_mb'] = max(1024 if k in {'postgres','mariadb','linuxserver/mariadb','mongo','getmeili/meilisearch'} else 512, plan['resources']['memory_mb'])
plan['mounts'] = drafts[index]['deployment']['mounts']
if main and mode != DEFAULT_MODE:
plan['resources'].update(resources)
if k == 'postgres':
if s['name'] in selected_resources:
plan['resources'].update(selected_resources[s['name']])
explicit_check = compose_healthcheck(s) if custom else None
if s['name'] in jobs:
health = {'type': 'completed', 'timeout_seconds': 300}
plan['execution_mode'] = 'one-shot'
elif explicit_check:
health = explicit_check
if main:
endpoint = service_endpoint(s)
if endpoint is None:
raise StackError(f"{s['name']}: {translate('Main endpoint not yet defined')}")
health['endpoint'] = endpoint
elif k == 'postgres':
health = {'type':'exec','timeout_seconds':180,'argv':['pg_isready','-h','127.0.0.1','-U',env.get('POSTGRES_USER','postgres'),'-d',env.get('POSTGRES_DB',env.get('POSTGRES_USER','postgres'))]}
elif k == 'mariadb':
health = {'type':'exec','timeout_seconds':240,'argv':['healthcheck.sh','--connect','--innodb_initialized']}
@@ -418,19 +716,29 @@ def build_stack(template, ui, mode='advanced'):
check = "REDISCLI_AUTH=${REDISCLI_AUTH:-$(tr '\\000' '\\n' < /proc/1/environ | sed -n 's/^REDISCLI_AUTH=//p')}; export REDISCLI_AUTH; " + check
health = {'type':'exec','timeout_seconds':90,'argv':['sh','-c',check]}
else:
endpoint = template['first_run']['endpoints'][0]
health = {'type':'http','timeout_seconds':360,'endpoint':endpoint}
endpoint = service_endpoint(s, include_expose=not main) if custom else template['first_run']['endpoints'][0]
if endpoint is None:
if main or not custom:
raise StackError(f"{s['name']}: {translate('Main endpoint not yet defined')}")
health = {'type': 'running', 'timeout_seconds': 30}
elif custom and not main and service_endpoint(s) is None:
health = {'type': 'tcp', 'timeout_seconds': 180, 'port': endpoint['port']}
else:
health = {'type':'http','timeout_seconds':360,'endpoint':endpoint}
plans.append({'name':s['name'],'main':main,'kind':k,'offset':0 if main else len(plans)+1,
'aliases':list(dict.fromkeys([s['name'],s['compose'].get('container_name',s['name'])])),
'template':single,'deployment':plan,'healthcheck':health})
if main:
plans[-1]['frontend_ipv4'] = addresses['']
if public_bindings.get(s['name']):
plans[-1]['public_environment_bindings'] = public_bindings[s['name']]
if s['name'] in frontend_names and s['name'] not in jobs:
plans[-1]['frontend'] = True
plans[-1]['frontend_ipv4'] = addresses[s['name']] if custom else addresses['']
if mode != DEFAULT_MODE:
from .extra_devices import ask_stack_extra_devices
ask_stack_extra_devices(ui, plans)
onboot = ui.confirm(translate('Start the stack with Proxmox'), False)
for plan in plans:
plan['deployment']['onboot'] = onboot
plan['deployment']['onboot'] = onboot and plan['name'] not in jobs
return {'deployment_kind':'generic-multi-lxc-stack','stack_name':name,'base_vmid':int(vmid) if vmid else None,
'completion_notes':template.get('proxmox',{}).get('stack_completion_notes',[]),
'rootfs_storage':root,'template_storage':cache,'onboot':onboot,'start_after_create':True,
oci/tests/test_backup_carried_record.py
+87
View File
@@ -0,0 +1,87 @@
"""A backup must contain its validated original metadata, despite a pending registry."""
import copy
import json
import os
from pathlib import Path
import sys
import tempfile
from types import SimpleNamespace
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'remote'))
import oci_carried_record as carried
VMID = 4801
IDENTITY = '488ed3cb-1145-477a-b90f-c46777d69fb2'
CONFIG = ('description: <!-- proxmenux-instance='+IDENTITY+' -->\n'
'rootfs: local-zfs:subvol-4801-disk-0,size=8G\n'
'unprivileged: 1\nlxc.init.uid: 999\n'
'net0: name=eth0,bridge=vmbr10,ip=10.77.0.31/24\n')
ORIGINAL = {'vmid': VMID, 'installation_id': IDENTITY, 'status': 'installed',
'deployment': {}, 'observed': {'config': CONFIG}}
class BackupCarriedRecordTests(unittest.TestCase):
def setUp(self):
temporary = tempfile.TemporaryDirectory(); self.addCleanup(temporary.cleanup)
self.root = Path(temporary.name)/'registry'; self.rootfs_base = Path(temporary.name)/'lxc'
self.rootfs = self.rootfs_base/str(VMID)/'rootfs'; self.rootfs.mkdir(parents=True)
self.config = CONFIG; self.status = 'stopped'; self.unmount_code = 0; self.commands = []
def run(*args):
self.commands.append(args)
value = self.config if args[1]=='config' else 'status: '+self.status if args[1]=='status' else ''
code = self.unmount_code if args[1]=='unmount' else 0
return SimpleNamespace(returncode=code, stdout=value)
for name, value in [('_run',run),('_host_directory_missing',lambda _:False),
('ROOTFS',self.rootfs_base),('STACK_CONTRACT',str(Path(temporary.name)/'stack-{}.json')),
('mapped_root',lambda _: (os.getuid(),os.getgid()))]:
context=patch.object(carried,name,value);context.start();self.addCleanup(context.stop)
def carry(self, original=None):
return carried.carry_backup(self.root,VMID,original or ORIGINAL,CONFIG)
def test_pending_host_registry_does_not_replace_the_validated_original(self):
pending=copy.deepcopy(ORIGINAL);pending.update(status='updating',pending_stack_transaction='/transaction')
carried.instances.write(carried.instances.location(self.root,VMID),pending)
result=self.carry()
copy_in_backup=carried.read_copy(self.rootfs)
self.assertEqual(copy_in_backup['record'],ORIGINAL)
self.assertNotIn('pending_stack_transaction',copy_in_backup['record'])
self.assertEqual(result['digest'],carried.digest(copy_in_backup))
self.assertEqual(self.commands.count(('pct','mount',str(VMID))),1)
self.assertEqual(self.commands.count(('pct','unmount',str(VMID))),1)
self.assertEqual(json.loads(carried.instances.location(self.root,VMID).read_text()),pending)
def test_notes_and_backup_lock_changes_do_not_hide_identity(self):
self.config=CONFIG.replace('description: ','description: user notes ')+'lock: backup\n'
self.carry()
self.assertEqual(carried.read_copy(self.rootfs)['record'],ORIGINAL)
def test_runtime_network_mount_user_and_identity_drift_are_refused(self):
for before,after in [('10.77.0.31','10.77.0.99'),('disk-0','disk-1'),('uid: 999','uid: 1000'),(IDENTITY,'11111111-1111-4111-8111-111111111111')]:
with self.subTest(after=after):
self.config=CONFIG.replace(before,after);self.commands=[]
with self.assertRaises(ValueError):self.carry()
self.assertNotIn(('pct','mount',str(VMID)),self.commands)
def test_running_guest_and_pending_original_are_refused(self):
self.status='running'
with self.assertRaises(ValueError):self.carry()
self.status='stopped'
for field,value in [('status','updating'),('pending_transaction','/transaction'),('pending_stack_transaction','/stack')]:
record=dict(ORIGINAL,**{field:value})
with self.subTest(field=field),self.assertRaises(ValueError):self.carry(record)
def test_write_failure_is_not_silenced_and_always_unmounts(self):
with patch.object(carried,'write_copy',side_effect=OSError('disk full')):
with self.assertRaisesRegex(OSError,'disk full'):self.carry()
self.assertIn(('pct','unmount',str(VMID)),self.commands)
def test_roundtrip_mismatch_and_unmount_failure_abort_before_success_stamp(self):
with patch.object(carried,'read_copy',return_value=None):
with self.assertRaises(ValueError):self.carry()
self.assertFalse((carried.instances.location(self.root,VMID).parent/carried.STAMP).exists())
self.unmount_code=1
with self.assertRaises(ValueError):self.carry()
if __name__=='__main__':unittest.main()
oci/tests/test_carried_config_cache.py
+3
View File
@@ -19,6 +19,9 @@ class CarriedConfigCacheTests(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.addCleanup(self.tmp.cleanup)
contract = patch.object(carried, 'STACK_CONTRACT', str(Path(self.tmp.name) / 'stack-{}.json'))
contract.start()
self.addCleanup(contract.stop)
self.root = Path(self.tmp.name) / 'instances'
self.source = Path(self.tmp.name) / 'lxc'
self.source.mkdir()
oci/tests/test_cluster_paths.py
+3
View File
@@ -1,5 +1,6 @@
"""A migrated container finds what it needs on any node of a cluster."""
import os
from pathlib import Path
import sys
import tempfile
@@ -50,6 +51,7 @@ class SysctlIncludeTests(unittest.TestCase):
def config(self, include):
return f"arch: amd64\nlxc.include: {include}\n".encode()
@unittest.skipUnless(os.geteuid() == 0, "the sysctl include must belong to root")
def test_cluster_path_and_the_path_of_earlier_installs_are_accepted(self):
runtime_settings.check(self.config(self.new), DEPLOYMENT, 101)
runtime_settings.check(self.config(self.old), DEPLOYMENT, 101)
@@ -58,6 +60,7 @@ class SysctlIncludeTests(unittest.TestCase):
with self.assertRaises(ValueError):
runtime_settings.check(self.config(self.root / "elsewhere"), DEPLOYMENT, 101)
@unittest.skipUnless(os.geteuid() == 0, "the sysctl include must belong to root")
def test_recovery_accepts_both_paths_only(self):
state = {"vmid": 101, "record": {"deployment": DEPLOYMENT}}
runtime_settings.check_recovery(self.config(self.new), state)
oci/tests/test_completed_job_lifecycle.py
+280
View File
@@ -0,0 +1,280 @@
"""Completion proofs survive native staging, rollback and renumbered restore."""
import copy
import json
from pathlib import Path
import stat
import sys
import tempfile
from types import SimpleNamespace
import unittest
from unittest.mock import patch
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / 'remote'))
import oci_compose_jobs as jobs
import oci_instances as instances
import oci_stack_native as native
import oci_stack_plan as plans
import oci_restore_recovery as recovery
MAIN = '11111111-1111-4111-8111-111111111111'
JOB = '22222222-2222-4222-8222-222222222222'
DB = '33333333-3333-4333-8333-333333333333'
OLD = 'sha256:' + 'a' * 64
NEW = 'sha256:' + 'b' * 64
CONFIG = 'rootfs: local-zfs:subvol-4901-disk-0,size=8G\nunprivileged: 1\nentrypoint: migrate\n'
SPEC = {'installation_id': JOB, 'image_digest': OLD,
'expected_config_fingerprint': jobs.config_fingerprint(CONFIG), 'timeout_seconds': 300}
CHECK = {'type': 'completed', 'timeout_seconds': 300, 'job': SPEC}
def fixture():
services = [{'vmid': 4902, 'name': 'database', 'kind': 'postgres', 'healthcheck': {'type': 'running', 'timeout_seconds': 120}},
{'vmid': 4901, 'name': 'migration', 'healthcheck': copy.deepcopy(CHECK)},
{'vmid': 4900, 'name': 'web', 'frontend': True, 'healthcheck': {'type': 'running', 'timeout_seconds': 120}}]
records = []
for vmid, identity in ((4900, MAIN), (4901, JOB), (4902, DB)):
records.append({'schema_version': 1, 'vmid': vmid, 'installation_id': identity, 'status': 'installed',
'deployment': {'onboot': False, 'mounts': []},
'observed': {'resolved_registry_digest': OLD, 'image': {'defaults': {}}},
'stack_member': {'stack_id': MAIN, 'primary_vmid': 4900}})
records[1]['deployment']['execution_mode'] = 'one-shot'
records[0]['stack'] = {'id': MAIN, 'deployment': {'stack_name': 'test', 'services': services},
'members': copy.deepcopy(records)}
plan = {'operation': 'update', 'primary_vmid': 4900, 'members': records,
'start_order': [4902, 4901, 4900], 'stop_order': [4900, 4901, 4902]}
return plan
class NativeCompletedJobs(unittest.TestCase):
def adapter(self):
return native.NativeAdapter(Path('/unused'), Path('/unused/journal'), fixture())
def test_stopped_job_is_not_accepted_without_an_authenticated_success(self):
adapter = self.adapter()
with patch.object(adapter, '_job_contract'), \
patch.object(jobs, 'ensure_completed', side_effect=jobs.JobError('exit 7')), \
patch.object(adapter, 'is_running', return_value=False), \
patch.object(native.member_tx, 'run') as run:
with self.assertRaisesRegex(jobs.JobError, 'exit 7'):
adapter.healthcheck(4901)
run.assert_not_called()
def test_job_start_calls_completion_runner_not_pct_start(self):
adapter = self.adapter()
with patch.object(adapter, 'validate'), patch.object(adapter, '_job_contract'), \
patch.object(jobs, 'ensure_completed', return_value={'status': 'completed'}) as complete, \
patch.object(native.member_tx, 'run') as run:
adapter.start(4901)
complete.assert_called_once_with(4901, SPEC, force=False)
run.assert_not_called()
def test_database_host_bind_cannot_be_acknowledged_as_a_migration_backup(self):
adapter = self.adapter()
adapter.acknowledge = True
adapter.records[4902]['deployment']['mounts'] = [{'type': 'host-bind', 'source': '/database'}]
with self.assertRaisesRegex(ValueError, 'Migration jobs'):
adapter._migration_backups()
def test_unknown_internal_backend_host_bind_is_not_assumed_safe(self):
adapter = self.adapter()
adapter.services[4902]['kind'] = 'application'
adapter.records[4902]['deployment']['mounts'] = [{'type': 'host-bind', 'source': '/unknown'}]
with self.assertRaisesRegex(ValueError, 'Migration jobs'):
adapter._migration_backups()
def test_image_database_fingerprint_detects_an_unknown_tagged_repository(self):
adapter = self.adapter()
adapter.services[4902].update(kind='application', frontend=True)
adapter.records[4902]['observed']['image']['defaults']['Env'] = ['PG_MAJOR=16']
adapter.records[4902]['deployment']['mounts'] = [{'type': 'host-bind', 'source': '/database'}]
with self.assertRaisesRegex(ValueError, 'Migration jobs'):
adapter._migration_backups()
def test_primary_app_shared_files_and_readonly_backend_are_not_database_write_risks(self):
adapter = self.adapter()
adapter.records[4900]['deployment']['mounts'] = [{'type': 'host-bind', 'source': '/app'}]
adapter.records[4902]['deployment']['mounts'] = [{'type': 'host-bind', 'source': '/db', 'read_only': True}]
adapter._migration_backups()
def test_rollback_checks_database_before_rerunning_old_job_then_starts_main(self):
adapter = self.adapter()
events = []
with patch.object(adapter, 'state', return_value={'phase': 'recovering', 'replacement_intent': True}), \
patch.object(adapter, 'start', side_effect=lambda vmid: events.append(('start', vmid))), \
patch.object(adapter, 'healthcheck', side_effect=lambda vmid: events.append(('healthy', vmid))), \
patch.object(adapter, '_ensure_job', side_effect=lambda vmid, force=False: events.append(('job', vmid, force))), \
patch.object(adapter, 'stop', side_effect=lambda vmid: events.append(('stop', vmid))):
adapter.restore_running_state({'4900': True, '4901': False, '4902': True}, [4902, 4901, 4900])
self.assertLess(events.index(('healthy', 4902)), events.index(('job', 4901, True)))
self.assertLess(events.index(('job', 4901, True)), events.index(('start', 4900)))
self.assertNotIn(('start', 4901), events)
def job_contract(self, tamper=False, rollback=False):
adapter = self.adapter()
original = {'schema': 1, 'stack': 'test', 'dependencies': [
{'vmid': s['vmid'], 'label': s['name'], 'healthcheck': copy.deepcopy(s['healthcheck'])}
for s in adapter.services.values() if s['vmid'] != 4900]}
contract = copy.deepcopy(original)
if tamper:
contract['dependencies'][0]['label'] = 'externally changed'
state = {'id': 'operation', 'phase': 'recovering' if rollback else 'starting'}
writes = []
path = SimpleNamespace(lstat=lambda: SimpleNamespace(st_mode=stat.S_IFREG | 0o600, st_uid=0),
read_text=lambda: json.dumps(contract))
staged = copy.deepcopy(adapter.records[4901])
staged['observed']['resolved_registry_digest'] = OLD if rollback else NEW
updated = {**SPEC, 'image_digest': OLD if rollback else NEW,
'expected_config_fingerprint': 'c' * 64}
def write(target, value):
writes.append(copy.deepcopy(value))
if target is path:
contract.clear(); contract.update(copy.deepcopy(value))
else:
state.clear(); state.update(copy.deepcopy(value))
with patch.object(native, 'Path', return_value=path), \
patch.object(adapter, 'state', side_effect=lambda: copy.deepcopy(state)), \
patch.object(instances, 'write', side_effect=write), \
patch.object(native, 'write_hook_contract', side_effect=write), \
patch.object(instances, 'read', return_value=staged), \
patch.object(native.member_tx, 'owned', return_value=CONFIG.encode()), \
patch.object(jobs, 'job_spec_from_record', return_value=updated) as bind:
if tamper:
with self.assertRaisesRegex(ValueError, 'hook and the stack recipe differ'):
adapter._job_contract()
bind.assert_not_called()
self.assertEqual(writes, [])
return
adapter._job_contract(rollback=rollback)
self.assertEqual(bind.call_args.args[0]['observed']['resolved_registry_digest'], updated['image_digest'])
self.assertEqual(adapter.services[4901]['healthcheck']['job'], updated)
self.assertEqual(contract['dependencies'][1]['healthcheck']['job'], updated)
self.assertIn('job_hook_contract_intent', writes[0])
self.assertNotIn('job_hook_contract_intent', state)
def test_new_image_binds_new_job_digest_before_main_hook(self):
self.job_contract()
def test_rollback_rebinds_old_image_with_actual_restored_config(self):
self.job_contract(rollback=True)
def test_concurrent_hook_edit_is_not_overwritten(self):
self.job_contract(tamper=True)
def test_job_hook_intent_survives_orchestrator_rewriting_its_main_journal(self):
with tempfile.TemporaryDirectory() as directory:
adapter = self.adapter()
adapter.journal = Path(directory) / 'transaction.json'
adapter.journal.write_text(json.dumps({'id': 'operation', 'phase': 'starting'}))
contract = {'schema': 1, 'stack': 'test', 'dependencies': [
{'vmid': s['vmid'], 'label': s['name'], 'healthcheck': copy.deepcopy(s['healthcheck'])}
for s in adapter.services.values() if s['vmid'] != 4900]}
hook = SimpleNamespace(lstat=lambda: SimpleNamespace(st_mode=stat.S_IFREG | 0o600, st_uid=0),
read_text=lambda: json.dumps(contract))
staged = copy.deepcopy(adapter.records[4901])
staged['observed']['resolved_registry_digest'] = NEW
updated = {**SPEC, 'image_digest': NEW, 'expected_config_fingerprint': 'c' * 64}
actual_write = instances.write
def write(path, value):
if path is hook:
contract.clear(); contract.update(copy.deepcopy(value))
else:
actual_write(path, value)
with patch.object(native, 'Path', return_value=hook), \
patch.object(instances, 'write', side_effect=write), \
patch.object(native, 'write_hook_contract', side_effect=write), \
patch.object(instances, 'read', return_value=staged), \
patch.object(native.member_tx, 'owned', return_value=CONFIG.encode()), \
patch.object(jobs, 'job_spec_from_record', return_value=updated):
adapter._job_contract()
# The orchestration engine checkpoints its own original state object.
adapter.journal.write_text(json.dumps({'id': 'operation', 'phase': 'checking'}))
adapter._job_contract()
marker = json.loads((Path(directory) / 'completed-job-contract.json').read_text())
self.assertEqual(marker['operation_id'], 'operation')
self.assertEqual(marker['job_hook_contract']['dependencies'][1]['healthcheck']['job'], updated)
self.assertNotIn('job_hook_contract_intent', marker)
class JobPlanValidation(unittest.TestCase):
def test_plan_preserves_database_job_main_order(self):
fixture_plan = fixture()
primary = fixture_plan['members'][0]
records = {r['vmid']: r for r in fixture_plan['members']}
plan = plans.build(primary, records, {r['vmid']: r['installation_id'] for r in records.values()}, 'update')
self.assertEqual(plan['start_order'], [4902, 4901, 4900])
def test_job_cannot_be_onboot_or_bound_to_another_image(self):
for mutation in ('onboot', 'watchdog', 'mode', 'digest'):
with self.subTest(mutation=mutation):
value = fixture()
records = {r['vmid']: r for r in value['members']}
if mutation in ('onboot', 'watchdog'):
records[4901]['deployment'][mutation] = True
elif mutation == 'mode':
records[4901]['deployment'].pop('execution_mode')
else:
value['members'][0]['stack']['deployment']['services'][1]['healthcheck']['job']['image_digest'] = NEW
with self.assertRaises(ValueError):
plans.build(value['members'][0], records, {r['vmid']: r['installation_id'] for r in records.values()}, 'update')
class RestoreCompletedJobs(unittest.TestCase):
def test_only_bound_completed_contracts_are_accepted(self):
contract = {'schema': 1, 'dependencies': [{'vmid': 4901, 'label': 'migration', 'healthcheck': copy.deepcopy(CHECK)}]}
self.assertTrue(recovery.valid_contract(contract, {4901}))
for field, value in (('image_digest', 'latest'), ('installation_id', 'arbitrary'),
('expected_config_fingerprint', ''), ('timeout_seconds', True)):
broken = copy.deepcopy(contract)
broken['dependencies'][0]['healthcheck']['job'][field] = value
self.assertFalse(recovery.valid_contract(broken, {4901}), field)
def test_restore_never_starts_a_job_as_an_ordinary_daemon(self):
entries = [{'vmid': vmid, 'config': 'startup: order=' + str(order), 'contract': None}
for order, vmid in enumerate((4902, 4901, 4900), 1)]
plan = {'bridges': {}, 'renumbered': {}, 'restored': entries, 'hold': False, 'notes': []}
calls = []
def run(*args):
calls.append(args)
return SimpleNamespace(stdout='status: stopped', stderr='', returncode=0)
with patch.object(recovery, 'restore_host_files'), patch.object(recovery, 'clean_format_directories'), \
patch.object(recovery, 'register', return_value=[4902, 4901, 4900]), \
patch.object(recovery.carried, 'carry'), \
patch.object(recovery, 'completed_job', side_effect=lambda root, vmid: vmid == 4901), \
patch.object(recovery, 'run', side_effect=run):
recovery.recover(Path('/unused'), plan, start=True)
self.assertEqual([call for call in calls if call[:2] == ('pct', 'start')],
[('pct', 'start', '4902'), ('pct', 'start', '4900')])
def test_renumbered_restore_rebinds_current_config_and_invalidates_deferred_cache(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
plan = fixture()
for record in plan['members']:
instances.write(instances.location(root, record['vmid']), record)
entry = {'vmid': 4900, 'contract': {'schema': 1}}
restore_plan = {'restored': [entry]}
calls = []
def checked(*args):
if args[:2] == ('pct', 'config'):
return CONFIG.replace('subvol-4901', 'subvol-NEW-4901')
calls.append((args, json.loads(Path(args[-1]).read_text())))
return ''
with patch.object(recovery, 'checked', side_effect=checked):
recovery.refresh_completed_jobs(root, restore_plan)
restored = instances.read(root, 4901)
main = instances.read(root, 4900)
check = main['stack']['deployment']['services'][1]['healthcheck']
self.assertTrue(restored['recovered_at'])
self.assertEqual(check['job']['installation_id'], JOB)
self.assertEqual(check['job']['image_digest'], OLD)
self.assertNotEqual(check['job']['expected_config_fingerprint'], SPEC['expected_config_fingerprint'])
self.assertEqual(calls[0][0][-2], '4900')
self.assertEqual([d['vmid'] for d in calls[0][1]['dependencies']], [4902, 4901])
self.assertTrue(recovery.completed_job(root, 4901))
self.assertFalse(recovery.completed_job(root, 4902))
if __name__ == '__main__':
unittest.main()
oci/tests/test_compose_host_aliases.py
+41
View File
@@ -0,0 +1,41 @@
"""Compose aliases remain single, safe /etc/hosts tokens, including underscores."""
from pathlib import Path
import re
import subprocess
import sys
import unittest
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / 'src'))
from proxmenux_oci.converter import _extra_hosts
class ComposeHostAliasTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
source = (ROOT / 'remote/install_oci.sh').read_text()
cls.shell = re.search(r'^oci_valid_extra_hostname\(\) \{\n.*?^\}', source, re.M | re.S)[0]
def shell_accepts(self, value):
return subprocess.run(['bash', '-c', self.shell + '\noci_valid_extra_hostname "$1"', 'alias-test', value],
capture_output=True).returncode == 0
def test_compose_container_aliases_are_accepted_by_import_and_installer(self):
for value in ('affine_postgres', 'affine_migration_job', 'db-1', 'db.internal', 'a', 'x' * 253):
with self.subTest(value=value):
self.assertTrue(self.shell_accepts(value))
self.assertEqual(_extra_hosts({value: '10.77.8.2'}),
[{'hostname': value, 'address': '10.77.8.2'}])
def test_multiline_whitespace_and_shell_syntax_are_refused(self):
for value in ('', 'db name', 'db\tname', 'db\n127.0.0.1 injected', '#comment',
'-db', '_db', '$(id)', 'db;id', 'db/../host', 'x' * 254):
with self.subTest(value=value):
self.assertFalse(self.shell_accepts(value))
self.assertIsNone(_extra_hosts({value: '10.77.8.2'}))
def test_apply_extra_hosts_uses_validated_token_before_writing(self):
source = (ROOT / 'remote/install_oci.sh').read_text()
apply = source.split('apply_extra_hosts() {', 1)[1].split('\n}', 1)[0]
self.assertLess(apply.index('oci_valid_extra_hostname "$hostname"'),
apply.index("printf '%s %s\\n'"))
oci/tests/test_compose_input.py
+132
View File
@@ -0,0 +1,132 @@
"""Compose inputs are explicit, deterministic, and never taken from the host."""
import os
from pathlib import Path
import sys
import tempfile
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'src'))
from proxmenux_oci.compose_input import Interpolator, dotenv, prepare, resolve_document
from proxmenux_oci.converter import ConversionError
from proxmenux_oci.custom import template_from_compose, _dump
class ComposeInputs(unittest.TestCase):
def test_default_required_alternative_nested_and_escaped_dollar(self):
expand = Interpolator({'EMPTY': '', 'SET': 'value'}).expand
for source, expected in [
('${MISSING:-fallback}', 'fallback'), ('${EMPTY:-fallback}', 'fallback'),
('${EMPTY-fallback}', ''), ('${SET:+yes}', 'yes'), ('${MISSING+yes}', ''),
('${EMPTY+yes}', 'yes'), ('${MISSING:-${SET}}', 'value'),
('$$HOME $SET ${SET}', '$HOME value value'), ('$5', '$5')]:
with self.subTest(source=source):
self.assertEqual(expand(source), expected)
for source in ['${MISSING?needed}', '${EMPTY:?needed}', '${NO}', '${BAD', '${X:=bad}']:
with self.subTest(source=source), self.assertRaises(ConversionError):
expand(source)
def test_never_reads_host_environment(self):
with patch.dict(os.environ, {'COMPOSE_SECRET': 'must-not-inherit'}):
with self.assertRaises(ConversionError):
Interpolator().expand('${COMPOSE_SECRET}')
def test_dotenv_quotes_comments_interpolation_and_no_shell_execution(self):
parsed = dotenv('''A=hello
B="${A} world" # note
C='literal$NO'
D=unquoted # comment
E=$(touch /tmp/should-not-exist)
F=
G
''', Interpolator())
self.assertEqual(parsed['B'], 'hello world')
self.assertEqual(parsed['C'], 'literal$NO')
self.assertEqual(parsed['D'], 'unquoted')
self.assertEqual(parsed['E'], '$(touch /tmp/should-not-exist)')
self.assertEqual(parsed['F'], '')
self.assertIsNone(parsed['G'])
for text in ['NOT A KEY=value', 'A="x" junk', "A='unterminated"]:
with self.subTest(text=text), self.assertRaises(ConversionError):
dotenv(text, Interpolator())
def test_file_merge_order_and_environment_override(self):
doc = {'services': {'app': {'image': 'example:${VERSION:-stable}',
'env_file': ['first.env', {'path':'optional.env', 'required':False}, 'second.env'],
'environment': {'A':'explicit', 'B':None}}}}
result = resolve_document(doc, {'B':'inherited-explicitly'},
{'first.env':'A=one\nB=first\nC=first', 'second.env':'C=second'})
self.assertEqual(result['services']['app']['image'], 'example:stable')
self.assertEqual(result['services']['app']['environment'],
{'A':'explicit', 'B':'inherited-explicitly', 'C':'second'})
self.assertNotIn('env_file', result['services']['app'])
self.assertIn('env_file', doc['services']['app'])
def test_unset_bare_variables_are_removed_and_optional_file_is_not_prompted(self):
class UI:
def ask(self, *args, **kwargs): raise AssertionError('Optional input must not be required')
document={'services': {'app': {'image':'example:1', 'env_file':[{'path':'missing.env','required':False}],
'environment':['UNSET', 'EMPTY=']}}}
with tempfile.TemporaryDirectory() as directory:
result=prepare(UI(),document,directory)
self.assertEqual(result['services']['app']['environment'],{'EMPTY':''})
def test_single_quoted_escape_and_dollar_value_are_literal(self):
self.assertEqual(dotenv("PASSWORD='it\\'s$literal'", Interpolator())['PASSWORD'],"it's$literal")
def test_selinux_mount_labels_are_normalized_with_explicit_host_guard_metadata(self):
class UI:
def ask(self,*args,**kwargs): raise AssertionError('No input required')
doc={'services':{'app':{'image':'example:1','volumes':['shared:/data:ro,z']},
'worker':{'image':'example:1','volumes':['shared:/data:Z']}}}
metadata={}
result=prepare(UI(),doc,metadata=metadata)
self.assertTrue(metadata['selinux_labels'])
self.assertEqual(result['services']['app']['volumes'],[{'source':'shared','target':'/data','read_only':True}])
self.assertEqual(result['services']['worker']['volumes'][0]['read_only'],False)
template=template_from_compose(_dump({'services':{'app':{'image':'adminer:4','ports':['8080:8080']},
'db':{'image':'postgres:16','environment':{'POSTGRES_PASSWORD':'test-only'}}}}),
interpolated=True,input_metadata=metadata)
self.assertTrue(template['proxmox']['custom_compose_selinux_labels'])
def test_literal_secret_is_not_expanded_again_by_stack_compiler(self):
document = {'services': {'db': {'image':'postgres:16',
'environment':{'POSTGRES_PASSWORD':'${PASSWORD}'}},
'app':{'image':'adminer:4','ports':['8080:8080'],
'environment':{'DATABASE_URL':'postgres://user:${PASSWORD}@db/test'}}}}
resolved = resolve_document(document, {'PASSWORD':'literal$PASSWORD${ALSO_LITERAL}'})
template = template_from_compose(_dump(resolved), interpolated=True)
self.assertTrue(template['compatibility']['automatic_install_candidate'], template['compatibility'])
db = next(s for s in template['compose_stack']['services'] if s['name']=='db')
self.assertEqual(db['compose']['environment']['POSTGRES_PASSWORD'], 'literal$PASSWORD${ALSO_LITERAL}')
def test_companion_dotenv_and_env_file_read_from_explicit_source_directory(self):
class UI:
def ask(self, *args, **kwargs):
raise AssertionError('No extra input needed for supplied companion files')
password = ask
with tempfile.TemporaryDirectory() as directory:
Path(directory, '.env').write_text('VERSION=16\nPASSWORD=test-only\n')
Path(directory, 'db.env').write_text('POSTGRES_PASSWORD=${PASSWORD}\n')
doc = {'services':{'db':{'image':'postgres:${VERSION}', 'env_file':'db.env'},
'app':{'image':'adminer:4','ports':['8080:8080']}}}
result = prepare(UI(), doc, directory)
self.assertEqual(result['services']['db']['image'], 'postgres:16')
self.assertEqual(result['services']['db']['environment']['POSTGRES_PASSWORD'], 'test-only')
def test_missing_values_are_asked_once_and_secrets_use_password_input(self):
class UI:
def __init__(self): self.prompts=[]
def ask(self, prompt, default='', **kwargs):
return ''
def password(self, prompt, **kwargs):
self.prompts.append(prompt); return 'synthetic-secret'
ui=UI()
doc={'services':{'one':{'image':'example:1','environment':{'PASSWORD':'${PASSWORD}'}},
'two':{'image':'example:2','environment':{'PASSWORD':'${PASSWORD}'}}}}
result=prepare(ui,doc)
self.assertEqual(len(ui.prompts),1)
self.assertEqual(result['services']['one']['environment'],result['services']['two']['environment'])
if __name__ == '__main__': unittest.main()
oci/tests/test_compose_job_integration.py
+96
View File
@@ -0,0 +1,96 @@
"""One-shot dependencies use the persistent verified helper, never bare pct start."""
import io
import json
import os
from pathlib import Path
import shutil
import subprocess
import sys
import tarfile
import tempfile
import unittest
from unittest.mock import patch
ROOT=Path(__file__).resolve().parents[1]
sys.path.insert(0,str(ROOT/'remote'))
sys.path.insert(0,str(ROOT/'src'))
import install_generic_stack as stack
import oci_compose_jobs as jobs
from proxmenux_oci import installer
IID='55bb1839-f087-4b03-a9b8-75bec7537c9c'
DIGEST='sha256:'+'a'*64
CONFIG=f'description: proxmenux-instance={IID}\nrootfs: local-zfs:subvol-4791-disk-0,size=8G\nentrypoint: sh -c true\n'
RECORD={'vmid':4791,'installation_id':IID,'observed':{'resolved_registry_digest':DIGEST}}
class JobInstallIntegrationTests(unittest.TestCase):
def test_installer_binds_completed_job_after_actual_runtime_configuration(self):
service={'vmid':4791,'deployment':{'onboot':True},'healthcheck':{'type':'completed','timeout_seconds':300}}
def run(*args):
if args==('pct','config',4791):return CONFIG
self.assertEqual(args,('pct','set',4791,'--onboot','0'));return ''
with patch.object(stack,'run',side_effect=run),patch.object(stack.oci_instances,'read',return_value=RECORD):
stack.bind_completed_jobs([service])
self.assertFalse(service['deployment']['onboot'])
self.assertEqual(service['healthcheck']['job'],jobs.job_spec_from_record(RECORD,CONFIG))
def test_local_transport_really_copies_standalone_job_helper(self):
with tempfile.TemporaryDirectory() as directory:
template=Path(directory)/'template.json';template.write_text('{}')
def inspect(command,*args):
staged=Path(command[1]).parent/'oci_compose_jobs.py'
self.assertEqual(staged.read_bytes(),(ROOT/'remote/oci_compose_jobs.py').read_bytes())
self.assertEqual((staged.parent/'oci_carried_record.py').read_bytes(),(ROOT/'remote/oci_carried_record.py').read_bytes())
self.assertEqual((staged.parent/'oci_stack_public_routes.py').read_bytes(),(ROOT/'remote/oci_stack_public_routes.py').read_bytes())
return None
with patch.object(installer,'_stream_process',side_effect=inspect):
installer._run_remote_install(ROOT,template,{'deployment_kind':'generic-multi-lxc-stack'},'local',True)
def test_ssh_transport_really_archives_standalone_job_helper(self):
with tempfile.TemporaryDirectory() as directory:
template=Path(directory)/'template.json';template.write_text('{}')
def inspect(command,data):
self.assertEqual(command[0],'ssh')
with tarfile.open(fileobj=io.BytesIO(data),mode='r:gz') as archive:
self.assertEqual(archive.extractfile('oci_compose_jobs.py').read(),(ROOT/'remote/oci_compose_jobs.py').read_bytes())
self.assertEqual(archive.extractfile('oci_carried_record.py').read(),(ROOT/'remote/oci_carried_record.py').read_bytes())
self.assertEqual(archive.extractfile('oci_stack_public_routes.py').read(),(ROOT/'remote/oci_stack_public_routes.py').read_bytes())
return None
with patch.object(installer,'_stream_process',side_effect=inspect):
installer._run_remote_install(ROOT,template,{'deployment_kind':'generic-multi-lxc-stack'},'test-node',True)
@unittest.skipUnless(shutil.which('jq'),'jq required')
def test_hook_contract_rejects_incomplete_or_unsafe_job_spec(self):
text=(ROOT/'remote/stack_dependency_hook.sh').read_text()
start=text.index(" jq -e '")+len(" jq -e '");end=text.index("\n ' \"$source_config\"",start)
expression=text[start:end]
spec=jobs.job_spec_from_record(RECORD,CONFIG)
contract={'schema':1,'dependencies':[{'vmid':4791,'label':'migration','healthcheck':{'type':'completed','timeout_seconds':300,'job':spec}}]}
good=subprocess.run(['jq','-e',expression],input=json.dumps(contract),text=True,capture_output=True)
self.assertEqual(good.returncode,0,good.stderr)
contract['dependencies'][0]['healthcheck']['job']['image_digest']='latest'
bad=subprocess.run(['jq','-e',expression],input=json.dumps(contract),text=True,capture_output=True)
self.assertNotEqual(bad.returncode,0)
@unittest.skipUnless(shutil.which('jq') and shutil.which('flock'),'jq/flock required')
def test_hook_calls_completed_helper_without_starting_job_directly(self):
with tempfile.TemporaryDirectory() as directory:
root=Path(directory);priv=root/'priv';priv.mkdir();locks=root/'locks';locks.mkdir();binary=root/'bin';binary.mkdir()
helper=root/'jobs.py';helper.write_text("import json,os,sys\nfrom pathlib import Path\nPath(os.environ['JOB_CALLS']).write_text(json.dumps(sys.argv[1:]))\n")
pct=binary/'pct';pct.write_text('#!/bin/sh\n[ "$1" = config ] && exit 0\necho "UNEXPECTED_PCT_START" >&2\nexit 99\n');pct.chmod(0o755)
stat=binary/'stat';stat.write_text('#!/bin/sh\n[ "$2" = %u ] && { echo 0; exit 0; }\n[ "$2" = %a ] && { echo 755; exit 0; }\nexit 1\n');stat.chmod(0o755)
spec=jobs.job_spec_from_record(RECORD,CONFIG)
(priv/'proxmenux-stack-4790.json').write_text(json.dumps({'schema':1,'dependencies':[{'vmid':4791,'label':'migration','healthcheck':{'type':'completed','timeout_seconds':300,'job':spec}}]}))
text=(ROOT/'remote/stack_dependency_hook.sh').read_text()
functions=text[text.index('die() {'):text.index('\nif [[ ${1:-} == "--install"')]
functions=functions.replace('/etc/pve/priv',str(priv)).replace('/run/lock',str(locks))
command='COMPOSE_JOB_HELPER='+str(helper)+'\n'+functions+'\nstart_dependencies 4790\n'
calls=root/'calls.json';env={**os.environ,'PATH':str(binary)+':'+os.environ['PATH'],'JOB_CALLS':str(calls)}
result=subprocess.run(['bash','-c',command],env=env,text=True,capture_output=True)
self.assertEqual(result.returncode,0,result.stderr)
self.assertEqual(json.loads(calls.read_text())[:2],['ensure','4791'])
self.assertNotIn('UNEXPECTED_PCT_START',result.stderr)
if __name__=='__main__':unittest.main()
oci/tests/test_compose_jobs.py
+219
View File
@@ -0,0 +1,219 @@
"""A completed OCI job needs real exit-zero proof, not stopped/CLI output."""
import contextlib
import copy
import json
import os
from pathlib import Path
import socket
import sys
import tempfile
import time
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1]/'remote'))
import oci_compose_jobs as jobs
IID='55bb1839-f087-4b03-a9b8-75bec7537c9c'
DIGEST='sha256:'+'a'*64
IDENTITY={'vmid':4790,'installation_id':IID,'image_digest':DIGEST,'config_fingerprint':'f'*64}
PROOF={'raw_waitstatus':0,'exit_code':0,'fresh_start_observed':True,'protocol':'lxc-msg-native264'}
CONFIG=f'''description: OCI %3C!-- proxmenux-instance={IID} --%3E
rootfs: local-zfs:subvol-4790-disk-0,size=8G
entrypoint: sh -c 'exit 0'
env: TOKEN=literal-secret-value
unprivileged: 1
lxc.init.uid: 1000
lxc.init.gid: 1000
mp0: /mnt/test/shared,mp=/data,backup=0,size=8G
cores: 1
memory: 256
tags: temporary
'''
def pack(kind,name,value):return jobs.FRAME.pack(kind,str(name).encode(),value)
class FakeProcess:
def __init__(self,rc=255):self.returncode=rc
def poll(self):return self.returncode
def wait(self,timeout=None):return self.returncode
def terminate(self):self.returncode=-15
def kill(self):self.returncode=-9
class NativeExitTests(unittest.TestCase):
def attempt(self, events):
receiver,sender=socket.socketpair()
with receiver,sender:
receiver.setblocking(False)
@contextlib.contextmanager
def subscription():yield receiver,{'protocol':'test-native-frames'}
def start(*args,**kwargs):
sender.sendall(b''.join(pack(*event) for event in events))
return FakeProcess()
with patch.object(jobs,'_subscription',subscription), \
patch.object(jobs.subprocess,'Popen',side_effect=start), \
patch.object(jobs,'_status',return_value='stopped'):
return jobs._run_attempt(4790,time.monotonic()+.2)
def test_exit_zero_is_proven_even_when_pct_start_loses_the_fast_pid(self):
proof=self.attempt([(0,4790,1),(0,4790,2),(2,4790,0)])
self.assertEqual(proof['raw_waitstatus'],0)
self.assertEqual(proof['pct_start_returncode'],255)
def test_nonzero_and_sigkill_never_become_completed(self):
for raw in (7<<8,9):
with self.subTest(raw=raw),self.assertRaises(jobs.JobError):
self.attempt([(0,4790,1),(0,4790,2),(2,4790,raw)])
def test_stopped_or_missing_exit_is_not_success(self):
with self.assertRaisesRegex(jobs.JobError,'timed out'):
self.attempt([(0,4790,1),(0,4790,2),(0,4790,0)])
def test_exit_without_a_fresh_start_event_is_rejected(self):
with self.assertRaisesRegex(jobs.JobError,'fresh'):
self.attempt([(2,4790,0)])
def test_unrelated_guest_exit_zero_does_not_complete_this_job(self):
with self.assertRaisesRegex(jobs.JobError,'timed out'):
self.attempt([(0,4790,1),(2,123,0),(0,4790,0)])
def test_partial_frames_are_buffered_and_unknown_abi_is_rejected(self):
data=pack(2,4790,1792)
self.assertEqual(jobs._frames(data[:8]),([],data[:8]))
self.assertEqual(jobs._frames(data),([(2,'4790',1792)],b''))
with self.assertRaises(jobs.JobError):jobs._frames(pack(100,4790,0))
class IdentityAndTrustTests(unittest.TestCase):
def test_command_environment_user_and_mount_changes_invalidate_fingerprint(self):
first=jobs.config_fingerprint(CONFIG)
for before,after in [('exit 0','exit 7'),('literal-secret-value','changed'),
('lxc.init.uid: 1000','lxc.init.uid: 1001'),('/mnt/test/shared','/mnt/test/other')]:
with self.subTest(before=before):
self.assertNotEqual(jobs.config_fingerprint(CONFIG.replace(before,after)),first)
altered=CONFIG.replace('cores: 1','cores: 4').replace('memory: 256','memory: 4096').replace('tags: temporary','tags: renamed').replace('size=8G','size=16G')
self.assertEqual(jobs.config_fingerprint(altered),first)
self.assertNotIn('literal-secret-value',first)
def test_instance_uuid_digest_and_container_marker_are_bound(self):
record={'vmid':4790,'installation_id':IID,'observed':{'resolved_registry_digest':DIGEST}}
spec={'installation_id':IID,'image_digest':DIGEST}
def read(path):return json.dumps(record) if str(path).endswith('.json') else CONFIG
with patch.object(jobs,'_trusted_read',side_effect=read), patch.object(jobs,'_config_text',return_value=CONFIG):
result=jobs.current_identity(4790,spec)
self.assertEqual(result['installation_id'],IID)
with self.assertRaises(jobs.JobError):jobs.current_identity(4790,{**spec,'image_digest':'sha256:'+'b'*64})
with self.assertRaises(jobs.JobError):jobs.current_identity(4790,{**spec,'installation_id':str(__import__('uuid').uuid4())})
with self.assertRaises(jobs.JobError):jobs.current_identity(4790,{**spec,'expected_config_fingerprint':'b'*64})
with patch.object(jobs,'_trusted_read',return_value=json.dumps(record)), patch.object(jobs,'_config_text',return_value=CONFIG.replace(IID,'another-instance')):
with self.assertRaises(jobs.JobError):jobs.current_identity(4790,spec)
def test_staged_digest_requires_matching_child_and_stack_transactions(self):
txid='a'*32
stackid='b'*32
journal=jobs.INSTANCE_ROOT/'4790'/'transactions'/txid/'transaction.json'
parent=jobs.INSTANCE_ROOT/'4792'/'stack-transactions'/stackid/'transaction.json'
new_digest='sha256:'+'b'*64
record={'vmid':4790,'installation_id':IID,'status':'updating',
'observed':{'resolved_registry_digest':DIGEST},'transaction_id':txid,
'pending_transaction':str(journal),'pending_stack_transaction':str(parent)}
child={'schema_version':1,'id':txid,'vmid':4790,'phase':'candidate-installed',
'record':{'installation_id':IID},'registry_digest':new_digest,
'staged_config_sha256':__import__('hashlib').sha256(CONFIG.encode()).hexdigest(),
'coordinated':{'journal':str(parent),'id':'active-stack-operation'}}
stack={'schema_version':1,'id':'active-stack-operation','phase':'starting',
'plan':{'members':[{'vmid':4790,'installation_id':IID}]}}
def read(path):
return json.dumps(child if path==journal else stack if path==parent else record)
with patch.object(jobs,'_trusted_read',side_effect=read),patch.object(jobs,'_config_text',return_value=CONFIG):
spec=jobs.job_spec_from_record(record,CONFIG)
self.assertEqual(spec['image_digest'],new_digest)
self.assertEqual(jobs.current_identity(4790,spec)['image_digest'],new_digest)
child['staged_config_sha256']='0'*64
with self.assertRaises(jobs.JobError):jobs.current_identity(4790,spec)
child['staged_config_sha256']=__import__('hashlib').sha256(CONFIG.encode()).hexdigest()
stack['id']='another-operation'
with self.assertRaises(jobs.JobError):jobs.current_identity(4790,spec)
stack['id']='active-stack-operation';stack['phase']='replacing'
self.assertEqual(jobs.job_spec_from_record(record,CONFIG)['image_digest'],new_digest)
with self.assertRaises(jobs.JobError):jobs.current_identity(4790,spec)
def test_restore_generation_changes_identity_even_with_identical_config(self):
record={'vmid':4790,'installation_id':IID,'observed':{'resolved_registry_digest':DIGEST}}
spec={'installation_id':IID,'image_digest':DIGEST}
with patch.object(jobs,'_trusted_read',side_effect=lambda path:json.dumps(record)),patch.object(jobs,'_config_text',return_value=CONFIG):
before=jobs.current_identity(4790,spec)
record['recovered_at']='2026-10-28T12:00:00Z'
after=jobs.current_identity(4790,spec)
self.assertEqual(before['config_fingerprint'],after['config_fingerprint'])
self.assertNotEqual(before,after)
def test_metadata_final_symlinks_and_world_writes_are_rejected(self):
with tempfile.TemporaryDirectory() as directory:
source=Path(directory)/'record';source.write_text('{}');source.chmod(0o600)
link=Path(directory)/'link';link.symlink_to(source)
self.assertEqual(jobs._trusted_read(source,owner=os.getuid()),'{}')
with self.assertRaises(OSError):jobs._trusted_read(link,owner=os.getuid())
source.chmod(0o666)
with self.assertRaises(jobs.JobError):jobs._trusted_read(source,owner=os.getuid())
def test_socket_name_is_the_source_verified_lxc_hash(self):
self.assertEqual(jobs._socket_name(),'lxc/ad055575fe28ddd5//var/lib/lxc')
class CompletionCacheTests(unittest.TestCase):
def setUp(self):
directory=tempfile.TemporaryDirectory();self.addCleanup(directory.cleanup)
self.root=Path(directory.name)
for name,replacement in [('STATE_ROOT',self.root),('_root',lambda:None),
('_job_lock',lambda *args:contextlib.nullcontext()),
('_status',lambda *args:'stopped')]:
item=patch.object(jobs,name,replacement);item.start();self.addCleanup(item.stop)
item=patch.object(jobs,'current_identity',return_value=IDENTITY);item.start();self.addCleanup(item.stop)
item=patch.object(jobs,'_read_cache',side_effect=lambda path:json.loads(path.read_text()) if path.exists() else None);item.start();self.addCleanup(item.stop)
self.spec={'installation_id':IID,'image_digest':DIGEST,'timeout_seconds':30}
def test_only_matching_proven_cache_skips_job_start(self):
with patch.object(jobs,'_run_attempt',return_value=PROOF) as attempt:
self.assertFalse(jobs.ensure_completed(4790,self.spec)['cached'])
self.assertTrue(jobs.ensure_completed(4790,self.spec)['cached'])
attempt.assert_called_once()
cache=json.loads((self.root/'4790.json').read_text())
self.assertEqual(cache['identity'],IDENTITY)
self.assertEqual(cache['proof']['raw_waitstatus'],0)
def test_changed_image_identity_or_config_never_reuses_old_completion(self):
cache={'schema':1,'status':'completed','identity':{**IDENTITY,'image_digest':'sha256:'+'b'*64},'proof':PROOF}
(self.root/'4790.json').write_text(json.dumps(cache))
with patch.object(jobs,'_run_attempt',return_value=PROOF) as attempt:
self.assertFalse(jobs.ensure_completed(4790,self.spec)['cached'])
attempt.assert_called_once()
def test_force_failure_invalidates_earlier_success(self):
with patch.object(jobs,'_run_attempt',return_value=PROOF):jobs.ensure_completed(4790,self.spec)
with patch.object(jobs,'_run_attempt',side_effect=jobs.JobError('signal 9')):
with self.assertRaises(jobs.JobError):jobs.ensure_completed(4790,self.spec,force=True)
self.assertFalse((self.root/'4790.json').exists())
def test_remove_forgets_only_matching_installation_uuid(self):
with patch.object(jobs,'_run_attempt',return_value=PROOF):jobs.ensure_completed(4790,self.spec)
other='45e85d70-4e28-416f-8ae8-325ae9153272'
self.assertFalse(jobs.forget_completed(4790,other))
self.assertTrue((self.root/'4790.json').exists())
self.assertTrue(jobs.forget_completed(4790,IID))
self.assertFalse((self.root/'4790.json').exists())
self.assertFalse(jobs.forget_completed(4790,IID))
def test_running_job_and_bad_timeout_are_refused(self):
with patch.object(jobs,'_status',return_value='running'),patch.object(jobs,'_run_attempt') as attempt:
with self.assertRaisesRegex(jobs.JobError,'already running'):jobs.ensure_completed(4790,self.spec)
attempt.assert_not_called()
for timeout in (True,0,29,3601,'300'):
with self.subTest(timeout=timeout),self.assertRaises(jobs.JobError):
jobs.ensure_completed(4790,{**self.spec,'timeout_seconds':timeout})
if __name__=='__main__':unittest.main()
oci/tests/test_compose_stop_signal.py
+57
View File
@@ -0,0 +1,57 @@
"""Explicit Compose stop signals override OCI image metadata without guessing."""
from pathlib import Path
import sys
import unittest
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'src'))
from proxmenux_oci.custom import template_from_compose
from proxmenux_oci.stack import assess, service_template
from proxmenux_oci.converter import _stop_signal
SOURCE = '''services:
web:
image: nginx:alpine
ports: ['8080:80']
depends_on: [worker]
worker:
image: example/worker:latest
entrypoint: /worker.sh
'''
class ComposeStopSignalTests(unittest.TestCase):
def test_explicit_worker_signal_reaches_native_runtime(self):
template = template_from_compose(SOURCE + ' stop_signal: SIGTERM\n')
self.assertEqual(assess(template), [])
worker = next(s for s in template['compose_stack']['services'] if s['name'] == 'worker')
native = service_template(template, worker)
self.assertEqual(native['proxmox']['installer_profile']['runtime']['halt_signal'], 'SIGTERM')
self.assertEqual(native['proxmox']['installer_profile']['runtime']['compose_entrypoint'], '/worker.sh')
def test_main_signal_is_applied_too(self):
document = SOURCE.replace(" ports: ['8080:80']", " ports: ['8080:80']\n stop_signal: SIGQUIT")
template = template_from_compose(document)
web = next(s for s in template['compose_stack']['services'] if s['name'] == 'web')
native = service_template(template, web)
self.assertEqual(native['proxmox']['installer_profile']['runtime']['halt_signal'], 'SIGQUIT')
self.assertEqual(native['proxmox']['image_metadata_policy']['stop_signal'], 'apply-compose-override')
def test_absent_signal_does_not_guess_from_entrypoint_override(self):
template = template_from_compose(SOURCE)
worker = next(s for s in template['compose_stack']['services'] if s['name'] == 'worker')
self.assertNotIn('halt_signal', service_template(template, worker)['proxmox']['installer_profile']['runtime'])
def test_invalid_signal_is_a_review_blocker(self):
for value in ('SIGTYPO', 'SIGTERM;touch /tmp/x', '', 99, None):
with self.subTest(value=value):
document = SOURCE + ' stop_signal: ' + (repr(value) if isinstance(value, str) else str(value).lower()) + '\n'
template = template_from_compose(document)
self.assertTrue(any('stop-signal-format' in e for e in assess(template)))
def test_named_linux_signals_normalize_independent_of_host_os(self):
self.assertEqual(_stop_signal('term'), 'SIGTERM')
self.assertEqual(_stop_signal('SIGWINCH'), 'SIGWINCH')
self.assertEqual(_stop_signal('SIGPWR'), 'SIGPWR')
self.assertIsNone(_stop_signal('SIGWINCH\nSIGTERM'))
if __name__ == '__main__':
unittest.main()
oci/tests/test_custom_compose_job_plan.py
+71
View File
@@ -0,0 +1,71 @@
"""Successful-completion conditions represent jobs, not daemon liveness."""
import copy
from pathlib import Path
import sys
import unittest
from unittest.mock import patch
sys.path.insert(0,str(Path(__file__).resolve().parents[1]/'src'))
from proxmenux_oci.custom import template_from_compose
from proxmenux_oci.stack import assess,build_stack,DefaultsUI,ordered_services
from proxmenux_oci.installer import DEFAULT_MODE
COMPOSE='''services:
app:
image: example.org/application:3
ports: ['8080:8080']
depends_on:
migration: {condition: service_completed_successfully}
volumes: [storage:/data]
migration:
image: example.org/application:3
command: [sh, -c, run-migrations]
depends_on: [db]
volumes: [storage:/data]
db:
image: postgres:16
environment: {POSTGRES_PASSWORD: synthetic-only}
volumes:
storage:
'''
class UI(DefaultsUI):
def confirm(self,text,default=False):return True
class JobPlanning(unittest.TestCase):
def test_job_order_completion_check_and_no_autostart(self):
template=template_from_compose(COMPOSE,interpolated=True)
self.assertEqual(assess(template),[])
self.assertEqual([s['name'] for s in ordered_services(template)],['db','migration','app'])
with patch('proxmenux_oci.installer.ask_storage',return_value='local-zfs'), \
patch('proxmenux_oci.installer.ask_bridge',return_value='vmbr0'), \
patch('proxmenux_oci.host.timezone',return_value='UTC'), \
patch('proxmenux_oci.network.ask_addresses',return_value=({'app':'dhcp'},None)):
plan=build_stack(template,UI(),DEFAULT_MODE)
services={s['name']:s for s in plan['services']}
self.assertEqual(services['migration']['healthcheck']['type'],'completed')
self.assertEqual(services['migration']['deployment']['execution_mode'],'one-shot')
self.assertFalse(services['migration']['deployment']['onboot'])
self.assertTrue(services['app']['deployment']['onboot'])
self.assertNotIn('frontend',services['migration'])
shared=[m['source'] for name in ['app','migration'] for m in services[name]['deployment']['mounts'] if m['container_path']=='/data']
self.assertEqual(len(set(shared)),1)
def test_a_job_cannot_be_treated_as_a_continuously_healthy_daemon(self):
template=template_from_compose(COMPOSE,interpolated=True)
template['compose_stack']['services'][0]['compose'].setdefault('depends_on',{})
main=next(s for s in template['compose_stack']['services'] if s['name']=='app')
main['compose']['depends_on']['migration']['condition']='service_healthy'
# Without a health check, a migration-like worker cannot satisfy healthy.
self.assertTrue(assess(template))
def test_auto_restart_of_successful_jobs_is_not_silently_enabled(self):
template=template_from_compose(COMPOSE,interpolated=True)
job=next(s for s in template['compose_stack']['services'] if s['name']=='migration')
job['compose']['restart']='always'
self.assertTrue(any('restart' in error for error in assess(template)))
if __name__=='__main__':unittest.main()
oci/tests/test_custom_compose_stacks.py
+153
View File
@@ -0,0 +1,153 @@
"""Pasted Compose files enter the same checked stack flow as catalog recipes."""
import sys
import unittest
from pathlib import Path
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'src'))
from proxmenux_oci import custom
from proxmenux_oci.converter import ConversionError
from proxmenux_oci.stack import ordered_services, resolve_environments, service_template
COMPOSE = '''services:
db:
image: postgres:16
environment:
POSTGRES_PASSWORD: supplied-password
volumes: [dbdata:/var/lib/postgresql/data]
app:
image: adminer:4
ports: ['8080:8080']
depends_on: [db]
volumes:
dbdata:
'''
class CustomComposeStacks(unittest.TestCase):
def test_stack_uses_existing_installer_and_preserves_versions_and_environment(self):
template = custom.template_from_compose(COMPOSE)
self.assertTrue(template['compatibility']['automatic_install_candidate'])
self.assertEqual(template['proxmox']['installer_profile']['stack_driver'], 'generic-multi-lxc-stack')
self.assertEqual(template['compose_stack']['main_service'], 'app')
services = ordered_services(template)
self.assertEqual([s['name'] for s in services], ['db', 'app'])
self.assertEqual([s['image'] for s in services], ['postgres:16', 'adminer:4'])
self.assertEqual(resolve_environments(services, 'UTC')['db']['POSTGRES_PASSWORD'], 'supplied-password')
self.assertEqual(service_template(template, services[0])['container_contract']['image']['reference'], 'postgres:16')
self.assertNotIn('supplied-password', custom.describe(template))
def test_stack_summary_hides_credentials_embedded_in_a_url(self):
document = COMPOSE.replace(
" ports:",
" environment: {DATABASE_URL: 'postgres://user:embedded-secret@db/app'}\n ports:")
template = custom.template_from_compose(document)
summary = custom.describe(template)
self.assertIn('DATABASE_URL', summary)
self.assertNotIn('embedded-secret', summary)
self.assertNotIn('postgres://user:', summary)
app = next(service for service in ordered_services(template) if service['name'] == 'app')
self.assertIn('embedded-secret', app['compose']['environment']['DATABASE_URL'])
def test_invalid_stack_metadata_has_a_conversion_error(self):
for metadata in ('x-casaos: not-a-mapping', 'x-casaos: [app]',
'x-casaos: {main: missing-service}'):
with self.subTest(metadata=metadata), self.assertRaises(ConversionError):
custom.template_from_compose(COMPOSE + metadata + '\n')
def test_short_read_only_time_mount_is_removed_from_stack(self):
text = COMPOSE.replace(" ports:", " volumes: [/etc/localtime:/etc/localtime:ro]\n ports:")
template = custom.template_from_compose(text)
app = next(service for service in ordered_services(template) if service['name'] == 'app')
self.assertEqual(app['compose']['volumes'], [])
self.assertFalse(any('System path' in error for error in template['compatibility']['untranslated_blockers']))
self.assertTrue(template['compatibility']['automatic_install_candidate'])
def test_written_digest_and_tag_survive_in_full_image_contracts(self):
digest = 'sha256:' + 'a' * 64
reference = 'ghcr.io/example/adminer:4@' + digest
template = custom.template_from_compose(COMPOSE.replace('adminer:4', reference))
main_image = template['container_contract']['image']
for image in (main_image, service_template(template, next(
service for service in ordered_services(template) if service['name'] == 'app'),
main=True)['container_contract']['image']):
self.assertEqual(image['reference'], reference)
self.assertEqual(image['registry'], 'ghcr.io')
self.assertEqual(image['repository'], 'ghcr.io/example/adminer')
self.assertEqual(image['tag'], '4')
self.assertEqual(image['digest'], digest)
self.assertEqual(image['pull_policy'],
'resolve-written-reference-to-architecture-digest-at-install')
dependency = service_template(template, ordered_services(template)[0])['container_contract']['image']
self.assertEqual(dependency['reference'], 'postgres:16')
self.assertEqual(dependency['repository'], 'postgres')
self.assertEqual(dependency['tag'], '16')
self.assertIsNone(dependency['digest'])
def test_reject_unsafe_dependency_mount_or_build(self):
for value in (' volumes: [/var/run/docker.sock:/var/run/docker.sock]',
' build: .'):
with self.subTest(value=value), self.assertRaises(ConversionError):
custom.template_from_compose(COMPOSE.replace(' volumes: [dbdata:/var/lib/postgresql/data]', value))
def test_shared_storage_is_supported_by_custom_stack(self):
template = custom.template_from_compose(COMPOSE.replace(" ports:", " volumes: [dbdata:/shared]\n ports:"))
self.assertTrue(template['compatibility']['automatic_install_candidate'])
self.assertEqual(template['compatibility']['untranslated_blockers'], [])
def test_every_image_checked_before_install(self):
class UI:
def message(self, text, title):
self.text = text
ui = UI()
with patch.object(custom, 'read_definition', return_value=(COMPOSE, 'Compose')), \
patch.object(custom, 'registry_report', side_effect=[(True, 'db ok'), (False, 'app missing')]) as report, \
patch.object(custom, '_registry_refused', return_value=False), \
patch.object(custom, 'install_template') as install:
custom.explore(ui)
self.assertEqual([c.args[0] for c in report.call_args_list], ['postgres:16', 'adminer:4'])
install.assert_not_called()
self.assertIn('app missing', ui.text)
def test_each_registry_line_names_its_image(self):
class UI:
def message(self, text, title):
self.text = text
ui = UI()
with patch.object(custom, 'read_definition', return_value=(COMPOSE, 'Compose')), \
patch.object(custom, 'registry_report', side_effect=[(True, 'in registry'), (False, 'app missing')]), \
patch.object(custom, '_registry_refused', return_value=False), \
patch.object(custom, 'install_template'):
custom.explore(ui)
self.assertIn('postgres:16: in registry', ui.text)
self.assertNotIn('adminer:4: app missing', ui.text)
@patch('proxmenux_oci.i18n.language', return_value='en')
def test_stack_summary_does_not_announce_questions_about_its_variables(self, _):
template = custom.template_from_compose(COMPOSE.replace(
" ports:", " environment:\n ADMINER_DEFAULT_SERVER: db\n ports:"))
summary = custom.describe(template)
self.assertNotIn('Variables the installation asks for', summary)
self.assertIn('Variables (app):\n ADMINER_DEFAULT_SERVER', summary)
single = custom.template_from_compose('services: {app: {image: nginx:1.27, environment: {MODE: x}}}')
self.assertIn('Variables the installation asks for', custom.describe(single))
def test_the_name_given_names_the_stack(self):
template = custom.template_from_compose('name: adminer-postgres\n' + COMPOSE)
class UI:
def ask(self, text, default=None, required=True):
self.default = default
return 'tools'
ui = UI()
self.assertEqual(custom._name(ui, template), 'tools')
self.assertEqual(ui.default, 'adminer-postgres')
self.assertEqual(template['compose_stack']['project_name'], 'tools')
def test_single_image_still_uses_single_container_flow(self):
template = custom.template_from_compose('services: {app: {image: nginx:1.27}}')
self.assertNotEqual(template['proxmox'].get('installer_profile', {}).get('stack_driver'), 'generic-multi-lxc-stack')
self.assertEqual(template['container_contract']['image']['reference'], 'nginx:1.27')
if __name__ == '__main__':
unittest.main()
oci/tests/test_custom_image_defaults.py
+86
View File
@@ -0,0 +1,86 @@
"""Internal service metadata is inferred from OCI, not app-specific recipes."""
import json
from pathlib import Path
import subprocess
import sys
import unittest
from types import SimpleNamespace
from unittest.mock import patch
ROOT=Path(__file__).resolve().parents[1]
sys.path.insert(0,str(ROOT/'src'))
sys.path.insert(0,str(ROOT/'remote'))
from proxmenux_oci.custom import enrich_image_defaults
import install_generic_stack as generic
class ImageDefaults(unittest.TestCase):
def test_internal_expose_and_health_from_oci_are_preserved_without_publishing_ports(self):
doc={'services':{'helper':{'image':'example.org/team/helper:3','healthcheck':{'disable':False,'retries':5}}}}
config={'config':{'ExposedPorts':{'7000/tcp':{},'7001/udp':{}},'Healthcheck':{
'Test':['CMD','/usr/bin/check-ready'],'Interval':3000000000,'Timeout':1000000000}}}
process=SimpleNamespace(returncode=0,stdout=json.dumps(config))
with patch('proxmenux_oci.custom.subprocess.run',return_value=process):
result=enrich_image_defaults(None,doc)['services']['helper']
self.assertEqual(result['expose'],['7000/tcp'])
self.assertNotIn('ports',result)
self.assertEqual(result['healthcheck']['test'],['CMD','/usr/bin/check-ready'])
self.assertEqual(result['healthcheck']['retries'],5)
self.assertEqual(result['healthcheck']['interval'],'3000000000ns')
self.assertNotIn('expose',doc['services']['helper'])
def test_null_image_health_does_not_crash_and_no_port_is_invented(self):
doc={'services':{'helper':{'image':'example.org/team/helper:3','healthcheck':{'disable':False}}}}
with patch('proxmenux_oci.custom.subprocess.run',return_value=SimpleNamespace(returncode=0,stdout='{"config":{"Healthcheck":null}}')):
result=enrich_image_defaults(None,doc)['services']['helper']
self.assertNotIn('healthcheck',result)
self.assertNotIn('expose',result)
def test_explicit_disabled_healthcheck_is_not_reenabled(self):
doc={'services':{'web':{'image':'example.org/web:1','ports':['8080:8080'],'healthcheck':{'disable':True}}}}
with patch('proxmenux_oci.custom.subprocess.run') as inspect:
self.assertEqual(enrich_image_defaults(None,doc),doc)
inspect.assert_not_called()
def test_worker_entrypoint_or_command_does_not_inherit_a_false_web_listener(self):
for override in [{'entrypoint':'/cron.sh'}, {'command':['worker','run']}, {'command':[]}]:
with self.subTest(override=override):
doc={'services':{'worker':{'image':'example.org/web-and-worker:1',**override}}}
config={'config':{'ExposedPorts':{'80/tcp':{}},'Healthcheck':None}}
with patch('proxmenux_oci.custom.subprocess.run',return_value=SimpleNamespace(returncode=0,stdout=json.dumps(config))):
result=enrich_image_defaults(None,doc)['services']['worker']
self.assertNotIn('expose',result)
def test_option_command_uses_actual_image_entrypoint_metadata(self):
doc={'services':{'db':{'image':'mariadb:11','command':['--transaction-isolation=READ-COMMITTED']}}}
config={'config':{'Entrypoint':['docker-entrypoint.sh'],'Cmd':['mariadbd']}}
entrypoints={}
with patch('proxmenux_oci.custom.subprocess.run',return_value=SimpleNamespace(returncode=0,stdout=json.dumps(config))):
result=enrich_image_defaults(None,doc,image_entrypoints=entrypoints)
self.assertEqual(entrypoints,{'mariadb:11':['docker-entrypoint.sh']})
self.assertEqual(result,doc)
def test_enforcing_selinux_stops_before_install_when_labels_need_translation(self):
import tempfile
with tempfile.TemporaryDirectory() as directory:
path=Path(directory)/'enforce'; path.write_text('1')
with self.assertRaisesRegex(RuntimeError,'SELinux'):
generic.check_selinux_labels({'proxmox':{'custom_compose_selinux_labels':True}},path)
path.write_text('0')
generic.check_selinux_labels({'proxmox':{'custom_compose_selinux_labels':True}},path)
path.write_text('1')
generic.check_selinux_labels({'proxmox':{}},path)
def test_main_http_success_does_not_override_failed_declared_exec_health(self):
service={'vmid':4800,'name':'app','healthcheck':{'type':'exec','argv':['check-ready'],'timeout_seconds':30}}
opener=SimpleNamespace(open=lambda *args,**kwargs: __import__('contextlib').nullcontext(SimpleNamespace(status=200)))
with patch.object(generic,'run',return_value='status: running'), \
patch.object(generic.urllib.request,'build_opener',return_value=opener), \
patch.object(generic.subprocess,'run',side_effect=[SimpleNamespace(returncode=1),SimpleNamespace(returncode=0)]) as execute, \
patch.object(generic.time,'sleep'):
generic.wait_web(service,'http://192.0.2.1:8080/')
self.assertEqual(execute.call_count,2)
self.assertEqual(execute.call_args.args[0],['pct','exec','4800','--','check-ready'])
if __name__=='__main__':unittest.main()
oci/tests/test_custom_multiapp_stack.py
+153
View File
@@ -0,0 +1,153 @@
"""Custom stacks keep each app's endpoint and actual shared storage identity."""
from pathlib import Path
import sys
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'src'))
from proxmenux_oci.casaos import convert_casaos_compose
from proxmenux_oci.stack import apply_stack_support, assess, build_stack, DefaultsUI, service_template
from proxmenux_oci.installer import DEFAULT_MODE, ADVANCED_MODE
COMPOSE = '''name: arr-test
services:
sonarr:
image: lscr.io/linuxserver/sonarr:latest
ports: ['8989:8989']
environment: {PUID: '1000', PGID: '1000'}
volumes: [sonarr-config:/config, shared-data:/data]
radarr:
image: lscr.io/linuxserver/radarr:latest
ports: ['7878:7878']
environment: {PUID: '1000', PGID: '1000'}
volumes: [radarr-config:/config, shared-data:/data]
volumes:
sonarr-config:
radarr-config:
shared-data:
x-casaos:
main: sonarr
port_map: '8989'
scheme: http
index: /
title: {en_US: Arr Test}
'''
def template(custom=True, text=COMPOSE):
result = convert_casaos_compose(text, 'local', '', 'local/arr-test/docker-compose.yml', '')
result['proxmox']['custom_compose_stack'] = custom
apply_stack_support(result)
return result
class UI(DefaultsUI):
def __init__(self):
self.asks = []
def ask(self, text, default=None, **kwargs):
self.asks.append(text)
return default or ''
class CustomMultiAppStackTests(unittest.TestCase):
def test_custom_opt_in_supports_two_web_apps_but_catalog_remains_guarded(self):
self.assertEqual(assess(template()), [])
self.assertTrue(assess(template(custom=False)))
def test_custom_background_app_without_port_is_only_checked_running(self):
result = template(text=COMPOSE.replace(" ports: ['7878:7878']\n", ''))
self.assertEqual(assess(result), [])
def test_service_image_contract_preserves_tag_and_digest(self):
parent = template()
service = parent['compose_stack']['services'][0]
for reference, repository, tag, digest in (
('ghcr.io/example/app:2.4', 'ghcr.io/example/app', '2.4', None),
('ghcr.io/example/app:2.4@sha256:'+'a'*64, 'ghcr.io/example/app', '2.4', 'sha256:'+'a'*64),
):
with self.subTest(reference=reference):
service['image'] = reference
service['compose']['image'] = reference
image = service_template(parent, service)['container_contract']['image']
self.assertEqual((image['reference'], image['repository'], image['tag'], image['digest']),
(reference, repository, tag, digest))
self.assertEqual(image['registry'], 'ghcr.io')
def test_compose_extension_metadata_is_accepted_only_for_custom(self):
text = 'x-common-environment: &common {PUID: \'1000\'}\n' + COMPOSE
self.assertEqual(assess(template(text=text)), [])
self.assertTrue(assess(template(custom=False, text=text)))
def test_shared_owners_must_match(self):
result = template(text=COMPOSE.replace("environment: {PUID: '1000', PGID: '1000'}", "environment: {PUID: '1001', PGID: '1000'}", 1))
self.assertTrue(any('shared' in e for e in assess(result)))
def test_advanced_addresses_are_kept_per_service(self):
ui = UI()
supplied = {'sonarr': '192.0.2.21/24', 'radarr': '192.0.2.22/24'}
with patch('proxmenux_oci.installer.ask_storage', return_value='local-zfs'), \
patch('proxmenux_oci.installer.ask_bridge', return_value='vmbr0'), \
patch('proxmenux_oci.host.timezone', return_value='UTC'), \
patch('proxmenux_oci.custom_mounts.ask_stack_custom_mounts'), \
patch('proxmenux_oci.extra_devices.ask_stack_extra_devices'), \
patch('proxmenux_oci.network.ask_addresses', return_value=(supplied, '192.0.2.1')):
plan = build_stack(template(), ui, ADVANCED_MODE)
self.assertEqual({s['name']:s['frontend_ipv4'] for s in plan['services']}, supplied)
self.assertEqual(plan['network']['frontend_gateway'], '192.0.2.1')
def test_advanced_custom_resources_are_selected_per_service(self):
class ResourceUI(UI):
def ask(self,text,default=None,**kwargs):
if text=='radarr: Memory in MB': return '768'
if text=='radarr: CPU cores': return '3'
if text=='Memory in MB': return '2048'
if text=='CPU cores': return '2'
return super().ask(text,default,**kwargs)
with patch('proxmenux_oci.installer.ask_storage',return_value='local-zfs'), \
patch('proxmenux_oci.installer.ask_bridge',return_value='vmbr0'), \
patch('proxmenux_oci.host.timezone',return_value='UTC'), \
patch('proxmenux_oci.custom_mounts.ask_stack_custom_mounts'), \
patch('proxmenux_oci.extra_devices.ask_stack_extra_devices'), \
patch('proxmenux_oci.network.ask_addresses',return_value=({'sonarr':'dhcp','radarr':'dhcp'},None)):
plan=build_stack(template(),ResourceUI(),ADVANCED_MODE)
resources={service['name']:service['deployment']['resources'] for service in plan['services']}
self.assertEqual(resources['radarr']['memory_mb'],768)
self.assertEqual(resources['radarr']['cores'],3)
self.assertEqual(resources['sonarr']['memory_mb'],2048)
self.assertEqual(resources['sonarr']['cores'],2)
def test_build_assigns_distinct_endpoints_and_shared_bind_source(self):
ui = UI()
with patch('proxmenux_oci.installer.ask_storage', return_value='local-zfs'), \
patch('proxmenux_oci.installer.ask_bridge', return_value='vmbr0'), \
patch('proxmenux_oci.host.timezone', return_value='UTC'), \
patch('proxmenux_oci.network.ask_addresses', side_effect=lambda ui, bridge, labels: ({name:'dhcp' for name in labels}, None)):
plan = build_stack(template(), ui, DEFAULT_MODE)
members = {s['name']:s for s in plan['services']}
self.assertEqual(members['sonarr']['healthcheck']['endpoint']['port'], 8989)
self.assertEqual(members['radarr']['healthcheck']['endpoint']['port'], 7878)
for member in members.values():
self.assertTrue(member['frontend'])
self.assertEqual(member['frontend_ipv4'], 'dhcp')
binds = [next(m for m in s['deployment']['mounts'] if m['container_path']=='/data') for s in members.values()]
self.assertEqual(binds[0]['source'], binds[1]['source'])
self.assertEqual(binds[0]['host_owner'], [101000, 101000])
self.assertTrue(all(m['type']=='host-bind' and m['backup'] is False for m in binds))
self.assertEqual(len([text for text in ui.asks if text.endswith('shared-data')]), 1)
configs = [next(m for m in s['deployment']['mounts'] if m['container_path']=='/config') for s in members.values()]
self.assertTrue(all(m['type']=='managed-volume' for m in configs))
def test_shared_directory_without_puid_uses_imported_image_owner(self):
text = COMPOSE.replace(" environment: {PUID: '1000', PGID: '1000'}\n", '')
with patch('proxmenux_oci.installer.ask_storage', return_value='local-zfs'), \
patch('proxmenux_oci.installer.ask_bridge', return_value='vmbr0'), \
patch('proxmenux_oci.host.timezone', return_value='UTC'), \
patch('proxmenux_oci.network.ask_addresses', side_effect=lambda ui, bridge, labels: ({name:'dhcp' for name in labels}, None)):
plan = build_stack(template(text=text), UI(), DEFAULT_MODE)
for service in plan['services']:
shared = next(m for m in service['deployment']['mounts'] if m['container_path'] == '/data')
self.assertNotIn('host_owner', shared)
if __name__ == '__main__':
unittest.main()
oci/tests/test_custom_registry_transport.py
+79
View File
@@ -0,0 +1,79 @@
"""Registry queries accept pinned tag@digest images without rewriting recipes."""
from pathlib import Path
import json
import sys
from types import SimpleNamespace
import unittest
from unittest.mock import patch
import yaml
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / 'src'))
from proxmenux_oci import custom
from proxmenux_oci.converter import ConversionError
DIGEST = 'sha256:' + 'a' * 64
REFERENCE = 'registry.test:5443/team/app:release@' + DIGEST
LOOKUP = 'registry.test:5443/team/app@' + DIGEST
class CustomRegistryTransportTests(unittest.TestCase):
def test_existing_image_parser_keeps_ports_and_pinned_digest(self):
pairs = [(REFERENCE, LOOKUP), ('[::1]:5443/team/app:release@' + DIGEST,
'[::1]:5443/team/app@' + DIGEST),
('alpine:3', 'alpine:3'), ('registry.test:5443/team/app', 'registry.test:5443/team/app'),
('ghcr.io/team/app@' + DIGEST, 'ghcr.io/team/app@' + DIGEST)]
for written, expected in pairs:
with self.subTest(written=written):
self.assertEqual(custom._registry_lookup(written), expected)
def test_registry_report_normalizes_raw_and_platform_detail_queries(self):
responses = [SimpleNamespace(returncode=0, stdout='{"schemaVersion":2}'),
SimpleNamespace(returncode=0, stdout='{"Architecture":"amd64"}')]
with patch.object(custom.subprocess, 'run', side_effect=responses) as run:
available, report = custom.registry_report(REFERENCE)
self.assertTrue(available)
self.assertIn('amd64', report)
self.assertEqual([call.args[0][-1] for call in run.call_args_list], ['docker://' + LOOKUP] * 2)
def test_failed_registry_report_keeps_the_written_reference(self):
with patch.object(custom.subprocess, 'run', return_value=SimpleNamespace(returncode=1, stdout='')):
available, report = custom.registry_report(REFERENCE)
self.assertFalse(available)
self.assertIn(REFERENCE, report)
def test_image_metadata_cache_and_entrypoint_identity_keep_written_reference(self):
document = {'services': {'worker': {'image': REFERENCE}, 'worker2': {'image': REFERENCE}}}
defaults = {'config': {'Entrypoint': ['image-entrypoint'], 'ExposedPorts': {'8080/tcp': {}},
'Healthcheck': {'Test': ['CMD', 'healthcheck']}}}
entrypoints = {}
with patch.object(custom.subprocess, 'run', return_value=SimpleNamespace(
returncode=0, stdout=json.dumps(defaults))) as run:
enriched = custom.enrich_image_defaults(None, document, image_entrypoints=entrypoints)
run.assert_called_once()
self.assertEqual(run.call_args.args[0][-1], 'docker://' + LOOKUP)
self.assertEqual(entrypoints, {REFERENCE: ['image-entrypoint']})
self.assertTrue(all(service['image'] == REFERENCE for service in enriched['services'].values()))
self.assertEqual(enriched['services']['worker']['healthcheck']['test'], ['CMD', 'healthcheck'])
def test_image_to_compose_queries_digest_but_preserves_written_image(self):
with patch.object(custom.subprocess, 'run', return_value=SimpleNamespace(
returncode=0, stdout='{"config":{}}')) as run:
document = yaml.safe_load(custom.compose_from_image(REFERENCE))
self.assertEqual(run.call_args.args[0][-1], 'docker://' + LOOKUP)
self.assertEqual(next(iter(document['services'].values()))['image'], REFERENCE)
def test_private_registry_refusal_uses_the_same_transport(self):
with patch.object(custom.subprocess, 'run', return_value=SimpleNamespace(
returncode=1, stderr='unauthorized: authentication required')) as run:
self.assertTrue(custom._registry_refused(REFERENCE))
self.assertEqual(run.call_args.args[0][-1], 'docker://' + LOOKUP)
def test_invalid_references_are_rejected_before_any_registry_query(self):
for reference in ('ghcr.io/app@sha256:short', 'user:token@ghcr.io/app', 'ghcr.io/app;id'):
with self.subTest(reference=reference), patch.object(custom.subprocess, 'run') as run:
with self.assertRaises(ConversionError):
custom._registry_lookup(reference)
with self.assertRaises(ConversionError):
custom.registry_report(reference)
self.assertFalse(custom._registry_refused(reference))
run.assert_not_called()
oci/tests/test_custom_worker_stack.py
+206
View File
@@ -0,0 +1,206 @@
"""Custom internal helpers need explicit readiness and never gain a LAN by expose."""
from pathlib import Path
import sys
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'src'))
from proxmenux_oci.custom import template_from_compose
from proxmenux_oci.stack import (assess, build_stack, compose_healthcheck,
DefaultsUI, kind, resolve_environments, StackError)
from proxmenux_oci.installer import DEFAULT_MODE
COMPOSE = '''name: worker-example
services:
web:
image: nginx:1.27
ports: ['8080:80']
depends_on: [helper]
helper:
image: example/helper:1
expose: ['3000']
'''
def build(template):
with patch('proxmenux_oci.installer.ask_storage', return_value='local-zfs'), \
patch('proxmenux_oci.installer.ask_bridge', return_value='vmbr0'), \
patch('proxmenux_oci.host.timezone', return_value='UTC'), \
patch('proxmenux_oci.network.ask_addresses', side_effect=lambda ui, bridge, labels: ({name: 'dhcp' for name in labels}, None)):
return build_stack(template, DefaultsUI(), DEFAULT_MODE)
class WorkerStackTests(unittest.TestCase):
def test_exposed_helper_gets_internal_tcp_check_without_frontend(self):
template = template_from_compose(COMPOSE)
self.assertEqual(assess(template), [])
services = {service['name']: service for service in build(template)['services']}
self.assertEqual(services['helper']['healthcheck']['port'], 3000)
self.assertEqual(services['helper']['healthcheck']['type'], 'tcp')
self.assertNotIn('frontend', services['helper'])
self.assertTrue(services['web']['frontend'])
self.assertEqual(services['web']['healthcheck']['endpoint']['port'], 80)
def test_service_healthy_does_not_invent_an_http_or_tcp_health_contract(self):
document = COMPOSE.replace('depends_on: [helper]', 'depends_on: {helper: {condition: service_healthy}}')
template = template_from_compose(document)
self.assertFalse(template['compatibility']['automatic_install_candidate'])
self.assertTrue(any('health and persistence' in error for error in assess(template)))
checked = document + " healthcheck: {test: ['CMD', 'test', '-f', '/ready']}\n"
self.assertEqual(assess(template_from_compose(checked)), [])
def test_worker_exec_check_is_preserved_without_exposing_a_port(self):
document = COMPOSE.replace(" expose: ['3000']", " healthcheck:\n test: ['CMD', 'test', '-f', '/ready']\n interval: 10s\n timeout: 2s\n start_period: 20s\n retries: 3")
template = template_from_compose(document)
self.assertEqual(assess(template), [])
services = {service['name']: service for service in build(template)['services']}
check = services['helper']['healthcheck']
self.assertEqual(check['type'], 'exec')
self.assertEqual(check['argv'], ['test', '-f', '/ready'])
self.assertEqual(check['timeout_seconds'], 56)
self.assertNotIn('frontend', services['helper'])
def test_explicit_check_takes_precedence_over_inferred_internal_http(self):
document = COMPOSE + " healthcheck: {test: ['CMD-SHELL', 'test -f /ready && test -f /worker-ready']}\n"
template = template_from_compose(document)
self.assertEqual(assess(template), [])
helper = next(service for service in build(template)['services'] if service['name'] == 'helper')
self.assertEqual(helper['healthcheck']['type'], 'exec')
self.assertEqual(helper['healthcheck']['argv'], ['sh', '-c', 'test -f /ready && test -f /worker-ready'])
def test_main_exec_check_retains_the_published_web_endpoint(self):
document = COMPOSE.replace(" ports: ['8080:80']", " ports: ['8080:80']\n healthcheck: {test: ['CMD', 'test', '-f', '/ready']}")
template = template_from_compose(document)
self.assertEqual(assess(template), [])
web = next(service for service in build(template)['services'] if service['name'] == 'web')
self.assertEqual(web['healthcheck']['type'], 'exec')
self.assertEqual(web['healthcheck']['argv'], ['test', '-f', '/ready'])
self.assertEqual(web['healthcheck']['endpoint'], {'scheme': 'http', 'port': 80, 'path': '/'})
self.assertTrue(web['frontend'])
def test_main_dependency_is_still_rejected_before_allocation(self):
document = COMPOSE.replace(' depends_on: [helper]\n', '').replace(" expose: ['3000']", " expose: ['3000']\n depends_on: [web]")
template = template_from_compose(document)
self.assertFalse(template['compatibility']['automatic_install_candidate'])
self.assertTrue(any('main service' in error for error in assess(template)))
def test_disabled_check_without_endpoint_is_explicitly_running_only(self):
document = COMPOSE.replace(" expose: ['3000']", ' healthcheck: {disable: true}')
template = template_from_compose(document)
self.assertEqual(assess(template), [])
helper = next(service for service in build(template)['services'] if service['name'] == 'helper')
self.assertEqual(helper['healthcheck']['type'], 'running')
def test_invalid_healthcheck_or_runtime_environment_requirement_needs_review(self):
for check in ({'test': ['CMD-SHELL', 'test -n "$TOKEN"']},
{'test': ['CMD', '']}, {'test': ['CMD', 'a\nb']},
{'test': ['CMD', 'true'], 'interval': '0s'},
{'test': ['CMD', 'true'], 'retries': -1},
{'test': ['CMD', 'true'], 'start_period': '2h'},
{'test': ['CMD', 'true'], 'timeout': 'nonsense'}):
with self.subTest(check=check), self.assertRaises(StackError):
compose_healthcheck({'compose': {'healthcheck': check}})
def test_dockerhub_aliases_keep_the_existing_database_health_profiles(self):
for image, expected in [('docker.io/library/postgres:16', 'postgres'),
('docker.io/library/redis:7', 'redis'),
('docker.io/valkey/valkey:8', 'valkey/valkey'),
('index.docker.io/library/mariadb:11', 'mariadb')]:
with self.subTest(image=image):
self.assertEqual(kind(image), expected)
def test_custom_server_flags_preserve_native_health_check(self):
document = COMPOSE.replace('example/helper:1', 'docker.io/library/redis:7').replace(
" expose: ['3000']", " command: ['redis-server', '--appendonly', 'yes', '--maxmemory-policy', 'noeviction']")
template = template_from_compose(document)
self.assertEqual(assess(template), [])
helper = next(service for service in build(template)['services'] if service['name'] == 'helper')
self.assertEqual(helper['kind'], 'redis')
self.assertEqual(helper['healthcheck']['type'], 'exec')
self.assertIn('redis-cli', str(helper['healthcheck']['argv']))
def test_a_different_binary_or_unhandled_auth_does_not_use_default_database_readiness(self):
for command in ("['sh', '-c', 'redis-server']", "['redis-server', '--requirepass', 'secret']"):
document = COMPOSE.replace('example/helper:1', 'redis:7').replace(
" expose: ['3000']", ' command: ' + command)
template = template_from_compose(document)
self.assertTrue(any('custom dependency commands' in error for error in assess(template)))
def test_plain_worker_uses_running_only_when_dependency_requires_started(self):
template = template_from_compose(COMPOSE.replace(" expose: ['3000']", " command: ['worker', '--watch']"))
self.assertEqual(assess(template), [])
helper = next(service for service in build(template)['services'] if service['name'] == 'helper')
self.assertEqual(helper['healthcheck'], {'type': 'running', 'timeout_seconds': 30})
self.assertNotIn('frontend', helper)
self.assertEqual(helper['template']['proxmox']['installer_profile']['runtime']['command'], ['worker', '--watch'])
def test_running_only_worker_cannot_satisfy_service_healthy(self):
document = COMPOSE.replace('depends_on: [helper]', 'depends_on: {helper: {condition: service_healthy}}').replace(" expose: ['3000']", " command: ['worker', '--watch']")
template = template_from_compose(document)
self.assertFalse(template['compatibility']['automatic_install_candidate'])
self.assertTrue(any('helper: service_healthy' in error for error in assess(template)))
def test_exposed_tcp_port_alone_is_not_a_declared_service_healthy_check(self):
template = template_from_compose(COMPOSE.replace('depends_on: [helper]', 'depends_on: {helper: {condition: service_healthy}}'))
self.assertTrue(any('helper: service_healthy' in error for error in assess(template)))
def test_explicit_worker_check_can_satisfy_service_healthy(self):
document = COMPOSE.replace('depends_on: [helper]', 'depends_on: {helper: {condition: service_healthy}}').replace(" expose: ['3000']", " healthcheck: {test: ['CMD', 'test', '-f', '/ready']}")
template = template_from_compose(document)
self.assertEqual(assess(template), [])
helper = next(service for service in build(template)['services'] if service['name'] == 'helper')
self.assertEqual(helper['healthcheck']['type'], 'exec')
def test_option_only_database_command_requires_verified_entrypoint_and_keeps_argv(self):
command = ['--transaction-isolation=READ-COMMITTED', '--binlog-format=ROW']
document = COMPOSE.replace('example/helper:1', 'mariadb:11').replace(" expose: ['3000']", " command: ['--transaction-isolation=READ-COMMITTED', '--binlog-format=ROW']")
template = template_from_compose(document)
self.assertTrue(any('custom dependency commands' in error for error in assess(template)))
template = template_from_compose(document, image_entrypoints={'mariadb:11': ['docker-entrypoint.sh']})
self.assertTrue(template['compatibility']['automatic_install_candidate'])
self.assertEqual(assess(template), [])
helper = next(service for service in build(template)['services'] if service['name'] == 'helper')
self.assertEqual(helper['template']['proxmox']['installer_profile']['runtime']['command'], command)
self.assertEqual(helper['healthcheck']['argv'], ['healthcheck.sh', '--connect', '--innodb_initialized'])
def test_literal_environment_preserves_all_explicit_dollar_values(self):
services = [{'name': 'helper', 'image': 'example/helper:1',
'compose': {'environment': {'PASSWORD': 'cash$VALUE${OTHER}', 'TOKEN': '${GENERATED_LOCAL_TOKEN}'}}}]
result = resolve_environments(services, 'UTC', literal=True)['helper']
self.assertEqual(result['PASSWORD'], 'cash$VALUE${OTHER}')
self.assertEqual(result['TOKEN'], '${GENERATED_LOCAL_TOKEN}')
with self.assertRaises(StackError):
resolve_environments(services, 'UTC')
def test_literal_explicit_empty_environment_clears_inherited_image_value(self):
document = COMPOSE.replace(" expose: ['3000']", " expose: ['3000']\n environment: {CLEAR_IMAGE_DEFAULT: ''}")
template = template_from_compose(document, interpolated=True)
helper = next(service for service in build(template)['services'] if service['name'] == 'helper')
self.assertEqual({item['name']: item['value'] for item in helper['deployment']['environment']},
{'CLEAR_IMAGE_DEFAULT': ''})
template['proxmox']['custom_compose_interpolated'] = False
helper = next(service for service in build(template)['services'] if service['name'] == 'helper')
self.assertEqual(helper['deployment']['environment'], [])
def test_literal_database_url_is_not_rebound_or_forced_to_have_a_password(self):
services = [
{'name': 'db', 'image': 'postgres:16', 'compose': {'environment': {'POSTGRES_USER': 'bootstrap', 'POSTGRES_HOST_AUTH_METHOD': 'trust'}}},
{'name': 'app', 'image': 'example/app:1', 'compose': {'environment': {'DATABASE_URL': 'postgresql://explicit-user@db/app'}}},
]
result = resolve_environments(services, 'UTC', literal=True)
self.assertEqual(result['app']['DATABASE_URL'], 'postgresql://explicit-user@db/app')
services[0]['compose']['environment']['POSTGRES_PASSWORD'] = 'bootstrap-password'
services[1]['compose']['environment']['DATABASE_URL'] = 'postgresql://other-user:explicit-password@db/app'
result = resolve_environments(services, 'UTC', literal=True)
self.assertEqual(result['app']['DATABASE_URL'], 'postgresql://other-user:explicit-password@db/app')
def test_fully_interpolated_custom_stack_does_not_reparse_literal_environment(self):
template = template_from_compose(COMPOSE.replace(" expose: ['3000']", " expose: ['3000']\n environment: {PASSWORD: 'cash$VALUE'}"))
template['proxmox']['custom_compose_interpolated'] = True
self.assertEqual(assess(template), [])
helper = next(service for service in build(template)['services'] if service['name'] == 'helper')
self.assertEqual(helper['deployment']['environment'][0]['value'], 'cash$VALUE')
if __name__ == '__main__':
unittest.main()
oci/tests/test_instance_lock_wait.py
+101
View File
@@ -0,0 +1,101 @@
"""Real cross-process contention at the late stack publication boundary."""
import contextlib
from pathlib import Path
import subprocess
import sys
import tempfile
import threading
import time
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'remote'))
import oci_instances as instances
HOLDER = '''import fcntl,sys,time
with open(sys.argv[1], 'a') as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
print('locked', flush=True)
sys.stdin.readline()
'''
@contextlib.contextmanager
def competing_owner(root):
process = subprocess.Popen([sys.executable, '-u', '-c', HOLDER, str(root / '.lock')],
stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)
try:
if process.stdout.readline().strip() != 'locked':
raise RuntimeError('Lock fixture did not acquire the lock')
yield process
finally:
if process.poll() is None:
process.terminate()
process.wait(timeout=5)
process.stdout.close()
process.stdin.close()
class InstanceLockWaitTests(unittest.TestCase):
def test_waiting_publication_succeeds_after_another_process_releases(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
with competing_owner(root) as process:
def release():
process.stdin.write('release\n')
process.stdin.flush()
timer = threading.Timer(0.35, release)
start = time.monotonic()
timer.start()
try:
with instances.locked(root, timeout=2):
elapsed = time.monotonic() - start
self.assertGreaterEqual(elapsed, 0.2)
self.assertLess(elapsed, 2)
finally:
timer.cancel()
timer.join()
def test_default_remains_immediate_when_another_operation_owns_the_registry(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
with competing_owner(root):
start = time.monotonic()
with self.assertRaises(BlockingIOError), instances.locked(root):
self.fail('Contended default lock was acquired')
self.assertLess(time.monotonic() - start, 0.5)
def test_wait_has_a_finite_timeout(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
with competing_owner(root):
start = time.monotonic()
with self.assertRaises(BlockingIOError), instances.locked(root, timeout=0.1):
self.fail('Contended timed lock was acquired')
elapsed = time.monotonic() - start
self.assertGreaterEqual(elapsed, 0.08)
self.assertLess(elapsed, 0.6)
def test_error_inside_operation_is_propagated_without_repeating_it(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
entered = []
with self.assertRaises(BlockingIOError):
with instances.locked(root, timeout=2):
entered.append(True)
raise BlockingIOError('unrelated operation failure')
self.assertEqual(entered, [True])
with instances.locked(root):
pass
def test_permission_failure_is_not_retried(self):
with tempfile.TemporaryDirectory() as directory:
with patch.object(instances.fcntl, 'flock', side_effect=PermissionError('denied')), \
patch.object(instances.time, 'sleep') as sleep:
with self.assertRaises(PermissionError), instances.locked(Path(directory), timeout=2):
pass
sleep.assert_not_called()
if __name__ == '__main__':
unittest.main()
oci/tests/test_job_hook_pmxcfs.py
+54
View File
@@ -0,0 +1,54 @@
"""Job hook contracts can be published on pmxcfs without chmod of /etc/pve/priv."""
from pathlib import Path
import json
import sys
import tempfile
import unittest
from unittest.mock import patch
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / 'remote'))
import oci_stack_native as native
class JobHookPmxcfsTests(unittest.TestCase):
def test_atomic_publish_does_not_change_parent_permissions(self):
with tempfile.TemporaryDirectory() as folder:
parent = Path(folder)
path = parent / 'proxmenux-stack-4900.json'
path.write_text('{"old":true}')
path.chmod(0o600)
mode = parent.stat().st_mode
with patch.object(native, 'private_directory', side_effect=AssertionError('No directory chmod')), \
patch.object(native.os, 'chmod', side_effect=PermissionError('pmxcfs')):
native.write_hook_contract(path, {'schema': 1, 'dependencies': []})
self.assertEqual(json.loads(path.read_text()), {'schema': 1, 'dependencies': []})
self.assertEqual(parent.stat().st_mode, mode)
self.assertEqual(list(parent.iterdir()), [path])
self.assertEqual(path.stat().st_mode & 0o777, 0o600)
def test_failed_atomic_publish_preserves_existing_contract(self):
with tempfile.TemporaryDirectory() as folder:
path = Path(folder) / 'proxmenux-stack-4900.json'
path.write_text('{"old":true}')
path.chmod(0o600)
with patch.object(native.os, 'replace', side_effect=OSError('pmxcfs write failed')):
with self.assertRaisesRegex(OSError, 'write failed'):
native.write_hook_contract(path, {'schema': 1})
self.assertEqual(json.loads(path.read_text()), {'old': True})
self.assertEqual(list(Path(folder).iterdir()), [path])
def test_untrusted_parent_or_file_is_refused_without_writing(self):
for parent_mode, file_mode in ((0o755, 0o600), (0o700, 0o644)):
with self.subTest(parent_mode=parent_mode, file_mode=file_mode), tempfile.TemporaryDirectory() as folder:
parent = Path(folder); parent.chmod(parent_mode)
path = parent / 'contract.json'; path.write_text('{"old":true}'); path.chmod(file_mode)
with self.assertRaisesRegex(ValueError, 'Unsafe dependency hook contract'):
native.write_hook_contract(path, {'schema': 1})
self.assertEqual(json.loads(path.read_text()), {'old': True})
def test_symlink_contract_is_refused(self):
with tempfile.TemporaryDirectory() as folder:
real = Path(folder) / 'real.json'; real.write_text('{"old":true}'); real.chmod(0o600)
link = Path(folder) / 'contract.json'; link.symlink_to(real)
with self.assertRaisesRegex(ValueError, 'Unsafe dependency hook contract'):
native.write_hook_contract(link, {'schema': 1})
self.assertEqual(json.loads(real.read_text()), {'old': True})
oci/tests/test_main_frontend_readiness.py
+44
View File
@@ -0,0 +1,44 @@
"""Main readiness probes the same address the installation advertises."""
import copy
import ipaddress
from pathlib import Path
import sys
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'remote'))
import install_generic_stack as installer
SERVICE = {'vmid':4860, 'main':True, 'name':'app', 'ip':'10.77.6.30',
'healthcheck':{'type':'http','timeout_seconds':360,
'endpoint':{'scheme':'http','port':80,'path':'/'}}}
class MainFrontendReadiness(unittest.TestCase):
def test_probe_and_persistent_health_url_use_advertised_lan_not_backend_ip(self):
service=copy.deepcopy(SERVICE)
with patch.object(installer,'run',return_value='10.77.6.30\n172.30.249.70\n'), \
patch.object(installer,'wait_web') as web:
url=installer.wait_main_web(service,ipaddress.ip_network('10.77.6.0/24'))
self.assertEqual(url,'http://172.30.249.70:80/')
self.assertEqual(service['healthcheck']['url'],url)
self.assertEqual(service['ip'],'10.77.6.30')
web.assert_called_once_with(service,url)
def test_waits_for_dhcp_frontend_without_probing_private_host(self):
service=copy.deepcopy(SERVICE)
with patch.object(installer,'lan_access_urls',side_effect=[RuntimeError('not yet'),[{'url':'http://192.0.2.7:80/'}]]), \
patch.object(installer.time,'sleep') as sleep, patch.object(installer,'wait_web') as web:
url=installer.wait_main_web(service,ipaddress.ip_network('10.77.6.0/24'))
self.assertEqual(url,'http://192.0.2.7:80/')
sleep.assert_called_once_with(1)
web.assert_called_once_with(service,url)
def test_no_frontend_address_never_falls_back_to_private_probe(self):
service=copy.deepcopy(SERVICE)
with patch.object(installer.time,'monotonic',side_effect=[0,1,121]), \
patch.object(installer.time,'sleep'), \
patch.object(installer,'lan_access_urls',side_effect=RuntimeError('not yet')), \
patch.object(installer,'wait_web') as web, self.assertRaisesRegex(RuntimeError,'No LAN address'):
installer.wait_main_web(service,ipaddress.ip_network('10.77.6.0/24'))
web.assert_not_called()
if __name__=='__main__':unittest.main()
oci/tests/test_modify_review.py
+17
View File
@@ -27,6 +27,23 @@ def installation():
return result,config
class SnapshotSectionGuardTests(unittest.TestCase):
def test_ipv6_listen_address_in_oci_entrypoint_is_not_a_snapshot(self):
saved, config = installation()
config += b"entrypoint: entrypoint.sh php -S '[::]:8080' -t /var/www/html\n"
saved['observed'].update(config=config.decode(), config_sha256=transaction.sha(config))
with patch.object(transaction, 'external_changes', return_value={}):
transaction.preflight(saved, copy.deepcopy(saved), config)
def test_actual_proxmox_snapshot_section_is_still_refused(self):
saved, config = installation()
config += b"\n[before-upgrade]\nmemory: 512\n"
saved['observed'].update(config=config.decode(), config_sha256=transaction.sha(config))
with patch.object(transaction, 'external_changes', return_value={}), \
self.assertRaisesRegex(ValueError, 'advanced Proxmox settings'):
transaction.preflight(saved, copy.deepcopy(saved), config)
class FallbackTests(unittest.TestCase):
def test_all_installer_adaptations_fall_back_before_any_native_write(self):
saved,config=installation()
oci/tests/test_native_exec_web_readiness.py
+78
View File
@@ -0,0 +1,78 @@
"""A passing exec check cannot commit an update whose published web service fails."""
import http.server
from pathlib import Path
import subprocess
import sys
import threading
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'remote'))
import oci_stack_native as native
class Reply(http.server.BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(self.server.reply_status)
self.end_headers()
def log_message(self, *args):
pass
class NativeExecWebReadinessTests(unittest.TestCase):
def adapter(self, url=None, frontend=True):
check = {'type': 'exec', 'argv': ['true'], 'timeout_seconds': 1}
if url:
check.update(url=url, endpoint={'port': 80})
member = {'vmid': 4800, 'deployment': {}, 'stack': {'deployment': {'services': [
{'vmid': 4800, 'name': 'service', 'frontend': frontend, 'healthcheck': check}]}}}
return native.NativeAdapter(Path('/registry'), Path('/journal'),
{'members': [member], 'primary_vmid': 4800})
def check_web(self, status):
server = http.server.HTTPServer(('127.0.0.1', 0), Reply)
server.reply_status = status
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
adapter = self.adapter(f'http://127.0.0.1:{server.server_port}/')
probes = []
def execute(*args):
probes.append(args[0])
if args[0] == 'pct':
return b'' # The custom CMD succeeded; web readiness is independent.
result = subprocess.run(args, capture_output=True, timeout=5)
if result.returncode:
raise RuntimeError('HTTP endpoint is not ready')
return result.stdout
try:
with patch.object(adapter, 'is_running', return_value=True), \
patch.object(native.member_tx.gpu_devices, 'validate_runtime'), \
patch.object(native.member_tx, 'run', side_effect=execute), \
patch.object(native.time, 'monotonic', side_effect=[0, 0, 2]), \
patch.object(native.time, 'sleep'):
if status >= 400:
with self.assertRaisesRegex(ValueError, 'service check'):
adapter.healthcheck(4800)
else:
adapter.healthcheck(4800)
self.assertEqual(probes, ['pct', 'curl'])
finally:
server.shutdown(); server.server_close(); thread.join(timeout=2)
def test_http_failure_rejects_even_when_exec_succeeds(self):
self.check_web(503)
def test_web_and_exec_must_both_pass(self):
self.check_web(200)
def test_database_exec_without_frontend_keeps_its_existing_check(self):
adapter = self.adapter(frontend=False)
with patch.object(adapter, 'is_running', return_value=True), \
patch.object(native.member_tx.gpu_devices, 'validate_runtime'), \
patch.object(native.member_tx, 'run', return_value=b'') as run:
adapter.healthcheck(4800)
run.assert_called_once_with('pct', 'exec', '4800', '--', 'true')
if __name__ == '__main__':
unittest.main()
oci/tests/test_private_network_configured.py
+74
View File
@@ -0,0 +1,74 @@
"""Down and pending bridge configs must reserve their names and IPv4 subnets."""
import importlib.util
import ipaddress
from pathlib import Path
import sys
import tempfile
import unittest
from unittest.mock import patch
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / 'remote'))
spec = importlib.util.spec_from_file_location('configured_private_network', ROOT / 'remote/allocate_private_network.py')
allocator = importlib.util.module_from_spec(spec)
spec.loader.exec_module(allocator)
def deployment():
return {'network': {'private_allocation':'automatic', 'private_subnet':'10.77.0.0/24',
'private_bridge':'vmbr10', 'private_host_address':'10.77.0.1/24'}}
class ConfiguredPrivateNetworkTests(unittest.TestCase):
def test_interface_files_include_pending_main_and_include_fragments(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
(root/'interfaces').write_text('auto vmbr0\n')
(root/'interfaces.new').write_text('auto vmbr10\n')
(root/'interfaces.d').mkdir()
(root/'interfaces.d/down').write_text('iface vmbr11 inet static\n')
(root/'interfaces.d/nested').mkdir()
def fake_path(value):
return root / Path(value).relative_to('/etc/network')
with patch.object(allocator, 'Path', side_effect=fake_path):
files = allocator._interface_files()
self.assertEqual(set(files), {root/'interfaces',root/'interfaces.new',root/'interfaces.d/down'})
def test_down_and_pending_subnets_are_occupied_even_with_different_bridge_names(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
down = root/'interfaces'
down.write_text('iface vmbr999 inet static\n address 10.77.0.1\n netmask 255.255.255.0\n')
pending = root/'interfaces.new'
pending.write_text('auto vmbr11\niface vmbr11 inet static\n address 10.77.1.1/24\n')
with patch.object(allocator, '_interface_files', return_value=[down,pending]), \
patch.object(allocator, 'command_output', return_value=''), \
patch.object(allocator.Path, 'glob', return_value=[]):
bridges = allocator.existing_bridges()
occupied = allocator.existing_networks()
self.assertIn('vmbr11', bridges)
self.assertIn(ipaddress.ip_network('10.77.0.0/24'), occupied)
self.assertIn(ipaddress.ip_network('10.77.1.0/24'), occupied)
self.assertEqual(allocator.allocate_network(deployment(),bridges,occupied),
('vmbr12',ipaddress.ip_network('10.77.2.0/24')))
def test_pending_bridge_name_reserves_name_without_an_address(self):
with tempfile.TemporaryDirectory() as directory:
pending = Path(directory)/'interfaces.new'
pending.write_text('auto vmbr0 vmbr10\niface vmbr10 inet manual\n')
with patch.object(allocator, '_interface_files', return_value=[pending]), \
patch.object(allocator, 'command_output', return_value=''), \
patch.object(allocator.Path, 'glob', return_value=[]):
bridges = allocator.existing_bridges()
self.assertEqual(allocator.allocate_network(deployment(),bridges,[]),
('vmbr11',ipaddress.ip_network('10.77.1.0/24')))
def test_ipv6_and_invalid_addresses_do_not_reserve_ipv4_pool(self):
text = ('iface vmbr12 inet6 static\n address fd00::1/64\n'
'iface vmbr20 inet static\n address invalid\n'
'iface vmbr21 inet static\n address 10.77.8.1/24 # comment\n')
self.assertEqual(allocator._configured_networks(text), [ipaddress.ip_network('10.77.8.0/24')])
if __name__ == '__main__':
unittest.main()
oci/tests/test_removal_cleanup.py
+2
View File
@@ -45,6 +45,8 @@ class RemovalCleanupTests(unittest.TestCase):
patch.object(oci_remove, "MONITOR_APPS", self.apps),
patch.object(oci_remove, "CLUSTER_RECORDS", self.records),
patch.object(oci_remove, "HOST_MONITOR_INCLUDES", self.host_monitor),
patch.object(oci_remove.public_routes, "DIRECTORY", self.root / "public-routes"),
patch.object(runtime_settings, "seccomp_path", lambda vmid: self.cluster / f"{vmid}.proxmenux-seccomp"),
patch.object(runtime_settings, "include_path", lambda vmid: self.cluster / f"{vmid}.sysctls"),
patch.object(runtime_settings, "legacy_include_path", lambda vmid: self.legacy / f"{vmid}.proxmenux-sysctls"),
patch.object(oci_remove.subprocess, "run"),
oci/tests/test_stack_attach_mounts_volume_preparations.py
+57
View File
@@ -149,6 +149,63 @@ class ResolveVolumeOwnerTests(unittest.TestCase):
igs.resolve_volume_owner(service, 'some-future-strategy')
class ImportedRuntimeOwnerTests(unittest.TestCase):
def test_numeric_imported_user_controls_fresh_app_volume_without_puid(self):
service = make_service()
service['vmid'] = 4820
with patch.object(igs, 'run', return_value='lxc.init.uid: 1000\nlxc.init.gid: 1000\n'):
owner = igs.runtime_volume_owner(service)
self.assertEqual(owner, (1000, 1000))
self.assertEqual(igs.resolve_volume_owner(service, 'mapped-application-user', owner), (101000, 101000))
def test_explicit_zero_puid_and_volume_owner_override_the_imported_user(self):
service = make_service(environment=[{'name': 'PUID', 'value': '0'}, {'name': 'PGID', 'value': '2500'}])
self.assertEqual(igs.resolve_volume_owner(service, 'mapped-application-user', (1000, 1000)), (100000, 102500))
service['template']['proxmox']['installer_profile']['volume_owner'] = {'uid': 1234, 'gid': 2345}
self.assertEqual(igs.resolve_volume_owner(service, 'mapped-application-user', (1000, 1000)), (101234, 102345))
with patch.object(igs, 'runtime_volume_owner', side_effect=AssertionError('mapped root must not inspect USER')):
self.assertEqual(igs.resolve_volume_owner(service, 'mapped-root'), (100000, 100000))
def test_imported_root_is_not_replaced_by_a_guessed_app_user(self):
service = make_service()
service['vmid'] = 4820
service['observed'] = {'image': {'defaults': {'User': 'node'}}}
with patch.object(igs, 'run', return_value='lxc.init.uid: 0\nlxc.init.gid: 0\n'):
self.assertEqual(igs.runtime_volume_owner(service), (0, 0))
def test_named_image_user_and_group_resolve_from_mounted_rootfs(self):
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
(root / 'etc').mkdir()
(root / 'etc/passwd').write_text('root:x:0:0:root:/root:/bin/sh\nnode:x:1000:1000:Node:/home/node:/bin/sh\n')
(root / 'etc/group').write_text('root:x:0:\nnode:x:1000:\nshared:x:2500:\n')
service = make_service()
service['observed'] = {'image': {'defaults': {'User': 'node:shared'}}}
self.assertEqual(igs.runtime_volume_owner(service, root), (1000, 2500))
service['observed']['image']['defaults']['User'] = '1000'
self.assertEqual(igs.runtime_volume_owner(service, root), (1000, 1000))
def test_actual_compose_user_override_wins_over_image_user(self):
service = make_service(unprivileged=False)
service['vmid'] = 4820
service['observed'] = {'image': {'defaults': {'User': 'node'}}}
with patch.object(igs, 'run', return_value='lxc.init.uid: 2000\nlxc.init.gid: 3000\n'):
owner = igs.runtime_volume_owner(service)
self.assertEqual(igs.resolve_volume_owner(service, 'mapped-application-user', owner), (2000, 3000))
def test_named_user_cannot_be_resolved_through_a_host_file_symlink(self):
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp) / 'rootfs'
(root / 'etc').mkdir(parents=True)
outside = Path(tmp) / 'host-passwd'
outside.write_text('node:x:1000:1000:Node:/home/node:/bin/sh\n')
(root / 'etc/passwd').symlink_to(outside)
service = make_service()
service['observed'] = {'image': {'defaults': {'User': 'node'}}}
with self.assertRaises(RuntimeError):
igs.runtime_volume_owner(service, root)
class ApplyVolumePreparationTests(unittest.TestCase):
def _target(self, tmp, with_lost_found=True):
target = Path(tmp) / 'target'
oci/tests/test_stack_backup_carried_metadata.py
+44
View File
@@ -0,0 +1,44 @@
"""A coordinated native backup must contain its verified, original restore recipe."""
from pathlib import Path
import sys
import tempfile
import unittest
from unittest.mock import patch
ROOT=Path(__file__).resolve().parents[1]
sys.path.insert(0,str(ROOT/'remote'))
import oci_stack_native as native
class StackBackupCarriedMetadataTests(unittest.TestCase):
def adapter(self, folder):
adapter=object.__new__(native.NativeAdapter)
adapter.root=Path('/unused/registry')
adapter.journal=Path(folder)/'transaction.json'
adapter.plan={'primary_vmid':4901}
adapter.records={4901:{'vmid':4901,'installation_id':'original','status':'installed'}}
return adapter
def test_transaction_id_does_not_replace_container_identity_before_archive_creation(self):
with tempfile.TemporaryDirectory() as folder:
adapter=self.adapter(folder);events=[]
archive=Path(folder)/'archive.tar.zst'
with patch.object(adapter,'validate'),patch.object(adapter,'work_base',return_value=Path(folder)), \
patch.object(native.member_tx,'owned',return_value=b'validated original config') as owned, \
patch.object(native.carried,'carry_backup',side_effect=lambda *args:events.append(('carry',args)),create=True), \
patch.object(native.member_tx,'verified_backup',side_effect=lambda *args:events.append(('backup',args)) or archive), \
patch.object(native.member_tx,'filehash',return_value='verified-sha'):
result=adapter.backup(4901,'different-transaction-id')
owned.assert_called_once_with(4901,'original')
self.assertEqual([event[0] for event in events],['carry','backup'])
self.assertEqual(events[0][1],(adapter.root,4901,adapter.records[4901],'validated original config'))
self.assertEqual(result,{'archive':str(archive),'sha256':'verified-sha'})
def test_missing_restore_metadata_prevents_archive_creation(self):
with tempfile.TemporaryDirectory() as folder:
adapter=self.adapter(folder)
with patch.object(adapter,'validate'),patch.object(adapter,'work_base',return_value=Path(folder)), \
patch.object(native.member_tx,'owned',return_value=b'validated original config'), \
patch.object(native.carried,'carry_backup',side_effect=ValueError('metadata not verified'),create=True), \
patch.object(native.member_tx,'verified_backup') as backup:
with self.assertRaisesRegex(ValueError,'metadata not verified'):
adapter.backup(4901,'different-transaction-id')
backup.assert_not_called()
oci/tests/test_stack_public_environment.py
+133
View File
@@ -0,0 +1,133 @@
"""Only public loopback origins bind by exact Compose port and static target."""
import copy
from pathlib import Path
import sys
import unittest
from unittest.mock import patch
ROOT=Path(__file__).resolve().parents[1]
sys.path.insert(0,str(ROOT/'src'));sys.path.insert(0,str(ROOT/'remote'))
from proxmenux_oci.custom import template_from_compose
from proxmenux_oci import stack
from proxmenux_oci.stack import assess,public_environment_bindings,StackError,build_stack,DefaultsUI
from proxmenux_oci.installer import DEFAULT_MODE
import install_generic_stack as runtime
SOURCE='''services:
web:
image: nginx:alpine
ports: ['9001:8080']
environment: {PUBLIC_URI: 'http://localhost:9001/app?x=1#view'}
depends_on: [backend, mail]
backend:
image: example/backend:latest
environment: {PUBLIC_URI: 'http://localhost:9001/app?x=1#view', EXPLICIT_URL: 'https://example.com:9443/keep'}
mail:
image: axllent/mailpit:v1
ports: ['8025:8025']
environment: {PUBLIC_URI: 'http://localhost:8025/mail'}
'''
def fixture(dynamic=False,jobs=False):
result=[]
for name,vmid,main,address in [('backend',4801,False,None),('mail',4802,False,'172.30.249.11/24'),('web',4800,True,'dhcp' if dynamic else '172.30.249.10/24')]:
target='mail' if name=='mail' else 'web';port=8025 if name=='mail' else 8080
value='http://@STACK_LAN_IP@:'+str(port)+'/app?x=1#view'
result.append({'name':name,'vmid':vmid,'main':main,'frontend':name!='backend','frontend_ipv4':address,
'healthcheck':{'type':'running'},'deployment':{'environment':[{'name':'PUBLIC_URI','value':value}]},
'public_environment_bindings':{'PUBLIC_URI':{'target_service':target,'container_port':port,'value':value}}})
if jobs:result.append({'name':'migration','vmid':4803,'main':False,'healthcheck':{'type':'completed'},'deployment':{'environment':[]}})
return result
def build(template,dynamic=False):
with patch('proxmenux_oci.installer.ask_storage',return_value='local-zfs'), \
patch('proxmenux_oci.installer.ask_bridge',return_value='vmbr0'), \
patch('proxmenux_oci.host.timezone',return_value='UTC'), \
patch('proxmenux_oci.network.ask_addresses',side_effect=lambda ui,bridge,labels: ({name:('dhcp' if dynamic else '172.30.249.11/24' if name=='mail' else '172.30.249.10/24') for name in labels},None)):
return build_stack(template,DefaultsUI(),DEFAULT_MODE)
class PublicEnvironmentTests(unittest.TestCase):
def source(self,text=SOURCE):
template=template_from_compose(text)
return template,template['compose_stack']['services']
def test_origin_port_maps_shared_url_to_web_and_mail_to_itself(self):
template,services=self.source();bindings=public_environment_bindings(services,'web')
self.assertEqual(assess(template),[])
self.assertEqual(bindings['backend']['PUBLIC_URI']['target_service'],'web')
self.assertEqual(bindings['backend']['PUBLIC_URI']['container_port'],8080)
self.assertEqual(bindings['backend']['PUBLIC_URI']['value'],'http://@STACK_LAN_IP@:8080/app?x=1#view')
self.assertEqual(bindings['mail']['PUBLIC_URI']['target_service'],'mail')
self.assertNotIn('EXPLICIT_URL',bindings['backend'])
def test_compiler_carries_targets_and_resolves_static_before_first_start(self):
template,_=self.source();plan=build(template)
services={service['name']:service for service in plan['services']}
self.assertEqual(services['backend']['public_environment_bindings']['PUBLIC_URI']['target_service'],'web')
with patch.object(runtime,'run') as run:runtime.prepare_public_environment(plan['services'])
values={item['name']:item['value'] for item in services['backend']['deployment']['environment']}
self.assertEqual(values['PUBLIC_URI'],'http://172.30.249.10:8080/app?x=1#view')
self.assertEqual(values['EXPLICIT_URL'],'https://example.com:9443/keep')
run.assert_not_called()
def test_final_environment_override_is_not_replaced_by_original_compose_binding(self):
template,_=self.source();original=stack.resolve_environments
def final_override(*args,**kwargs):
values=original(*args,**kwargs)
values['backend']['PUBLIC_URI']='https://operator.example.com/custom'
return values
with patch.object(stack,'resolve_environments',side_effect=final_override):plan=build(template)
backend=next(s for s in plan['services'] if s['name']=='backend')
self.assertNotIn('PUBLIC_URI',backend.get('public_environment_bindings',{}))
runtime.prepare_public_environment(plan['services'])
self.assertEqual(next(e['value'] for e in backend['deployment']['environment'] if e['name']=='PUBLIC_URI'),'https://operator.example.com/custom')
def test_internal_health_metrics_and_admin_urls_are_never_automatic_public_fields(self):
extra="OTEL_EXPORTER_OTLP_ENDPOINT: 'http://localhost:4318', HEALTHCHECK_URL: 'http://127.0.0.1:9090', LOCAL_ADMIN_API: 'http://localhost:9999', TEXT: 'not:a[url', "
template,services=self.source(SOURCE.replace('EXPLICIT_URL:',extra+'EXPLICIT_URL:'))
self.assertEqual(assess(template),[])
bindings=public_environment_bindings(services,'web')['backend']
for key in ['OTEL_EXPORTER_OTLP_ENDPOINT','HEALTHCHECK_URL','LOCAL_ADMIN_API','TEXT']:self.assertNotIn(key,bindings)
plan=build(template);backend=next(s for s in plan['services'] if s['name']=='backend');runtime.prepare_public_environment(plan['services'])
values={e['name']:e['value'] for e in backend['deployment']['environment']}
self.assertEqual(values['OTEL_EXPORTER_OTLP_ENDPOINT'],'http://localhost:4318')
self.assertEqual(values['HEALTHCHECK_URL'],'http://127.0.0.1:9090')
self.assertEqual(values['LOCAL_ADMIN_API'],'http://localhost:9999')
def test_ambiguous_host_port_is_blocked(self):
template,services=self.source(SOURCE.replace("['8025:8025']","['9001:8025']"))
with self.assertRaisesRegex(StackError,'Ambiguous'):public_environment_bindings(services,'web')
self.assertTrue(any('Ambiguous' in error for error in assess(template)))
def test_unmatched_public_origin_does_not_guess_even_single_frontend(self):
_,services=self.source(SOURCE.replace('localhost:9001','localhost:9999'))
with self.assertRaisesRegex(StackError,'no matching'):public_environment_bindings(services,'web')
only_web=[service for service in services if service['name']!='mail']
with self.assertRaisesRegex(StackError,'no matching'):public_environment_bindings(only_web,'web')
def test_missing_origin_port_does_not_fall_back_to_the_main(self):
_,services=self.source(SOURCE.replace('localhost:9001','localhost'))
with self.assertRaisesRegex(StackError,'no matching'):public_environment_bindings(services,'web')
def test_static_bindings_with_migration_do_not_run_any_commands(self):
with patch.object(runtime,'run') as run:runtime.prepare_public_environment(fixture(jobs=True))
run.assert_not_called()
def test_every_automatic_dhcp_binding_is_refused_before_any_command(self):
for jobs in (False,True):
with self.subTest(jobs=jobs),patch.object(runtime,'run') as run,self.assertRaisesRegex(RuntimeError,'Automatic public URLs'):
runtime.prepare_public_environment(fixture(dynamic=True,jobs=jobs))
run.assert_not_called()
template,_=self.source()
with self.assertRaisesRegex(StackError,'Automatic public URLs'):build(template,dynamic=True)
def test_explicit_public_urls_allow_dhcp_and_are_unchanged(self):
template,_=self.source(SOURCE.replace('localhost','operator.example.com'))
plan=build(template,dynamic=True)
with patch.object(runtime,'run') as run:runtime.prepare_public_environment(plan['services'])
self.assertTrue(all(not s.get('public_environment_bindings') for s in plan['services']))
backend=next(s for s in plan['services'] if s['name']=='backend')
self.assertEqual(next(e['value'] for e in backend['deployment']['environment'] if e['name']=='PUBLIC_URI'),'http://operator.example.com:9001/app?x=1#view')
run.assert_not_called()
if __name__=='__main__':unittest.main()
oci/tests/test_stack_public_route_install.py
+68
View File
@@ -0,0 +1,68 @@
"""Persist route contracts before startup without losing other LXC includes."""
import copy
from pathlib import Path
import sys
import tempfile
import unittest
from unittest.mock import patch
import uuid
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / 'remote'))
import install_generic_stack as runtime
import oci_stack_public_routes as routes
class PublicRouteInstallTests(unittest.TestCase):
def test_private_members_route_to_all_static_frontends_with_real_identities(self):
services = [
{'name':'web','vmid':4810,'main':True,'ip':'10.77.0.30','frontend_ipv4':'10.0.1.246/24','deployment':{}},
{'name':'mail','vmid':4811,'main':False,'frontend':True,'ip':'10.77.0.31','frontend_ipv4':'10.0.1.245/24','deployment':{}},
{'name':'worker','vmid':4812,'main':False,'ip':'10.77.0.32','deployment':{}},
]
identities = {service['vmid']: {'installation_id':str(uuid.uuid4())} for service in services}
with tempfile.TemporaryDirectory() as directory:
config = Path(directory) / '4812.conf'
config.write_text('hostname: worker\nlxc.include: /existing/sysctl.conf\n')
def path(value):
return config if str(value) == '/etc/pve/lxc/4812.conf' else Path(value)
def install(deployment, vmid, identity):
routes.validate(deployment,vmid,identity)
return f'lxc.include: {routes.include_path(vmid)}'
with patch.object(runtime,'Path',side_effect=path), patch.object(runtime.oci_instances,'read',side_effect=lambda root,vmid:identities[vmid]), patch.object(routes,'install',side_effect=install), patch.object(routes,'check',wraps=lambda text,plan,vmid,identity=None,**kw:routes.validate(plan,vmid,identity)):
runtime.install_public_routes(services,4810,'vmbr10')
self.assertIn('lxc.include: /existing/sysctl.conf\n',config.read_text())
spec=services[2]['deployment']['stack_public_routes']
self.assertEqual(spec['installation_id'],identities[4812]['installation_id'])
self.assertEqual(spec['stack_id'],identities[4810]['installation_id'])
self.assertEqual(spec['primary_vmid'],4810)
self.assertEqual(spec['routes'],[
{'destination':'10.0.1.245','via':'10.77.0.31','target_vmid':4811,'target_installation_id':identities[4811]['installation_id']},
{'destination':'10.0.1.246','via':'10.77.0.30','target_vmid':4810,'target_installation_id':identities[4810]['installation_id']}])
self.assertNotIn('stack_public_routes',services[0]['deployment'])
self.assertNotIn('stack_public_routes',services[1]['deployment'])
def test_dhcp_without_static_targets_does_not_invent_routes(self):
services=[{'main':True,'frontend_ipv4':'dhcp','deployment':{}}]
with patch.object(runtime.oci_instances,'read') as read:
runtime.install_public_routes(services,4810,'vmbr10')
read.assert_not_called()
def test_duplicate_static_ip_rejected_before_read_or_write(self):
services=[{'main':True,'frontend_ipv4':'10.0.1.246/24'}, {'frontend':True,'frontend_ipv4':'10.0.1.246/24'}]
with patch.object(runtime.oci_instances,'read') as read:
with self.assertRaisesRegex(RuntimeError,'Duplicate'):runtime.install_public_routes(services,4810,'vmbr10')
read.assert_not_called()
def test_recreate_route_include_is_last_after_other_managed_includes(self):
source=(ROOT/'remote/install_oci.sh').read_text()
flow=source[source.index('prepare_managed_format_directories "/var/lib/lxc/'):]
self.assertLess(flow.index('apply_host_monitor\n'),flow.index('ROUTE_INCLUDE=$(python3'))
self.assertLess(flow.index('apply_kept_settings\n'),flow.index('ROUTE_INCLUDE=$(python3'))
def test_routes_installed_before_jobs_and_main_start(self):
source=(ROOT/'remote/install_generic_stack.py').read_text()
main=source[source.index('def main('):]
self.assertLess(main.index('install_public_routes(services, primary_id, bridge)'),main.index('bind_completed_jobs(services)'))
self.assertLess(main.index('install_public_routes(services, primary_id, bridge)'),main.index("run('pct','start',primary_id"))
if __name__ == '__main__':unittest.main()
oci/tests/test_stack_public_routes.py
+199
View File
@@ -0,0 +1,199 @@
"""Managed routes cannot configure an unowned guest or the host namespace."""
import copy
from contextlib import ExitStack
import json
import os
from pathlib import Path
import stat
import subprocess
import sys
import tempfile
from types import SimpleNamespace
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'remote'))
import oci_stack_public_routes as routes
OWNER = '11111111-1111-4111-8111-111111111111'
PRIMARY = '22222222-2222-4222-8222-222222222222'
MAC = '02:11:22:33:44:55'
def fixture():
return {'stack_public_routes': {'schema': 1, 'stack_id': PRIMARY, 'primary_vmid': 4900,
'vmid': 4901, 'installation_id': OWNER, 'private_bridge': 'vmbr15',
'private_address': '10.77.5.31/24', 'routes': [
{'destination': '172.30.249.90', 'via': '10.77.5.30',
'target_vmid': 4900, 'target_installation_id': PRIMARY}]}}
class PublicRouteTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
root = Path(self.temp.name)
self.stack = ExitStack()
self.addCleanup(self.stack.close)
for key, value in {'DIRECTORY': root/'owned', 'HELPER': root/'helper.py',
'CONFIGS': root/'configs', 'ROOT_UID': os.getuid()}.items():
self.stack.enter_context(patch.object(routes, key, value))
routes.CONFIGS.mkdir()
routes.HELPER.write_text(Path(routes.__file__).read_text())
routes.HELPER.chmod(0o644)
self.deployment = fixture()
self.spec = self.deployment['stack_public_routes']
def config(self, vmid, identity, private, public=None, mac=MAC):
text = f'# proxmenux-instance={identity}\nnet0: name=eth0,bridge=vmbr15,ip={private},hwaddr={mac}\n'
if public:
text += f'net1: name=eth1,bridge=vmbr909,ip={public}\n'
if vmid == 4901:
text += f'lxc.include: {routes.include_path(vmid)}\n'
path = routes.CONFIGS/f'{vmid}.conf'
path.write_text(text)
path.chmod(0o600)
return text
def install(self):
return routes.install(self.deployment, 4901, OWNER)
def test_install_check_canonical_include_and_root_owned_contract(self):
line = self.install()
self.assertEqual(line, f'lxc.include: {routes.include_path(4901)}')
self.assertNotIn('lxc.hook.version', routes.include_path(4901).read_text())
self.assertIn(' -I ', routes.include_content())
self.assertEqual(routes.check(line, self.deployment, 4901, OWNER), self.spec)
self.assertEqual(routes.contract_path(4901).stat().st_mode & 0o777, 0o600)
self.assertEqual(routes.HELPER.stat().st_mode & 0o777, 0o644)
def test_restore_can_validate_without_assets_but_not_a_foreign_include(self):
line = f'lxc.include: {routes.include_path(4901)}\n'
self.assertEqual(routes.check(line, self.deployment, 4901, OWNER, require_host_assets=False), self.spec)
with self.assertRaises(routes.RouteError):
routes.check(line.replace('4901.conf','4902.conf'), self.deployment, 4901, OWNER, require_host_assets=False)
self.assertEqual(routes.renumber_line(line.encode(),4901,5901), f'lxc.include: {routes.include_path(5901)}\n'.encode())
def test_reused_vmid_cannot_remove_another_installation_contract(self):
self.install()
with self.assertRaises(routes.RouteError):routes.remove(4901, PRIMARY)
self.assertTrue(routes.contract_path(4901).exists())
self.assertTrue(routes.remove(4901, OWNER))
self.assertFalse(routes.include_path(4901).exists())
def test_bad_identity_and_unsafe_routes_fail_before_commands(self):
changes = [lambda s:s.update(vmid=4902), lambda s:s.update(installation_id=PRIMARY),
lambda s:s['routes'][0].update(destination='0.0.0.0/0'),
lambda s:s['routes'][0].update(destination='127.0.0.1'),
lambda s:s['routes'][0].update(via='172.30.249.1'),
lambda s:s['routes'][0].update(destination=True),
lambda s:s['routes'].append(copy.deepcopy(s['routes'][0]))]
with patch.object(routes.subprocess,'run') as run:
for change in changes:
value=copy.deepcopy(self.deployment);change(value['stack_public_routes'])
with self.subTest(value=value),self.assertRaises(routes.RouteError):routes.validate(value,4901,OWNER)
run.assert_not_called()
def test_symlink_and_unsafe_helper_fail_closed(self):
routes.DIRECTORY.mkdir()
target=Path(self.temp.name)/'other';target.write_text('do not touch')
routes.include_path(4901).symlink_to(target)
with self.assertRaises(routes.RouteError):self.install()
self.assertEqual(target.read_text(),'do not touch')
routes.include_path(4901).unlink()
routes.HELPER.chmod(0o666)
with self.assertRaises(routes.RouteError):self.install()
def test_symlink_directory_and_foreign_file_owner_are_rejected(self):
outside=Path(self.temp.name)/'outside';outside.mkdir()
routes.DIRECTORY.symlink_to(outside, target_is_directory=True)
with self.assertRaises(routes.RouteError):self.install()
unsafe=SimpleNamespace(lstat=lambda:SimpleNamespace(st_mode=stat.S_IFREG|0o600,st_uid=routes.ROOT_UID+1,st_size=10))
with self.assertRaises(routes.RouteError):routes._read(unsafe)
def test_live_identity_and_frontend_addresses_are_verified(self):
self.install()
self.config(4901,OWNER,'10.77.5.31/24')
self.config(4900,PRIMARY,'10.77.5.30/24','172.30.249.90/24')
self.assertEqual(routes.live_contract(self.spec),MAC)
self.config(4900,OWNER,'10.77.5.30/24','172.30.249.90/24')
with self.assertRaises(routes.RouteError):routes.live_contract(self.spec)
self.config(4900,PRIMARY,'10.77.5.30/24','172.30.249.91/24')
with self.assertRaises(routes.RouteError):routes.live_contract(self.spec)
def test_tampered_or_world_writable_contract_is_rejected(self):
self.install();routes.contract_path(4901).chmod(0o666)
with self.assertRaises(routes.RouteError):routes.check(f'lxc.include: {routes.include_path(4901)}',self.deployment,4901,OWNER)
def test_namespace_is_a_net_namespace_and_never_the_host(self):
with patch.object(routes.os,'open',return_value=17),patch.object(routes.os,'close') as close, \
patch.object(routes.fcntl,'ioctl',return_value=routes.CLONE_NEWNET), \
patch.object(routes.os,'fstat',return_value=SimpleNamespace(st_dev=1,st_ino=9)), \
patch.object(routes.os,'stat',return_value=SimpleNamespace(st_dev=1,st_ino=9)):
with self.assertRaisesRegex(routes.RouteError,'host network'):routes.namespace_fd('/proc/123/fd/4')
close.assert_called_once_with(17)
with patch.object(routes.os,'open',return_value=17),patch.object(routes.os,'close'), \
patch.object(routes.fcntl,'ioctl',return_value=0):
with self.assertRaisesRegex(routes.RouteError,'not a network'):routes.namespace_fd('/proc/123/fd/4')
with patch.object(routes.os,'open') as opened:
for value in (None,'/proc/self/ns/net','/etc/passwd','/proc/123/ns/mnt'):
with self.assertRaises(routes.RouteError):routes.namespace_fd(value)
opened.assert_not_called()
def test_routes_are_only_32_in_verified_namespace_with_no_shell(self):
interface=[{'address':MAC,'addr_info':[{'family':'inet','local':'10.77.5.31','prefixlen':24}]}]
with patch.object(routes,'_namespace_command',side_effect=[json.dumps(interface).encode(),b'']) as command:
routes.apply(self.spec,17,MAC)
self.assertEqual(command.call_args_list[0].args,(17,'-j','address','show','dev','eth0'))
self.assertEqual(command.call_args.args,(17,'-4','route','replace','172.30.249.90/32','via','10.77.5.30','dev','eth0','src','10.77.5.31'))
with patch.object(routes.subprocess,'run',return_value=SimpleNamespace(stdout=b'ok')) as run:
routes._namespace_command(17,'-4','route','show')
self.assertEqual(run.call_args.args[0][:3],[routes.NSENTER,'--net=/proc/self/fd/17','--'])
self.assertEqual(run.call_args.kwargs['pass_fds'],(17,));self.assertNotIn('shell',run.call_args.kwargs)
def test_engine_helper_can_change_normally_without_stale_snapshot_adoption(self):
routes.HELPER.write_text('# trusted engine update, same stable path\n')
self.install()
self.assertEqual(routes.HELPER.read_text(),'# trusted engine update, same stable path\n')
def test_namespace_with_wrong_owner_address_cannot_receive_routes(self):
for value in ([], [{'address':'02:aa:bb:cc:dd:ee','addr_info':[{'family':'inet','local':'10.77.5.31','prefixlen':24}]}]):
with patch.object(routes,'_namespace_command',return_value=json.dumps(value).encode()) as command:
with self.assertRaises(routes.RouteError):routes.apply(self.spec,17,MAC)
self.assertEqual(command.call_count,1)
def test_ip_failure_propagates_so_startup_cannot_report_success(self):
interface=[{'address':MAC,'addr_info':[{'family':'inet','local':'10.77.5.31','prefixlen':24}]}]
with patch.object(routes,'_namespace_command',side_effect=[json.dumps(interface).encode(),subprocess.CalledProcessError(2,['ip'])]):
with self.assertRaises(subprocess.CalledProcessError):routes.apply(self.spec,17,MAC)
def test_start_uses_only_the_runtime_pid_namespace(self):
self.install()
with patch.dict(os.environ,{'LXC_NAME':'4901','LXC_PID':'12345'},clear=True), \
patch.object(routes.os,'geteuid',return_value=0), \
patch.object(routes,'live_contract',return_value=MAC), \
patch.object(routes,'namespace_fd',return_value=17) as namespace, \
patch.object(routes,'apply') as apply, patch.object(routes.os,'close') as close:
routes.start()
namespace.assert_called_once_with('/proc/12345/ns/net')
apply.assert_called_once_with(self.spec,17,MAC)
close.assert_called_once_with(17)
def test_default_hook_arguments_must_match_owner_and_phase(self):
with patch.dict(os.environ,{'LXC_NAME':'4901'},clear=True), \
patch.object(routes.os,'geteuid',return_value=0),patch.object(routes,'start') as start:
with patch.object(sys,'argv',['helper','start','4901','lxc','start-host']):routes.main()
start.assert_called_once()
start.reset_mock()
with patch.object(sys,'argv',['helper','start','4902','lxc','start-host']), self.assertRaises(routes.RouteError):routes.main()
with patch.object(sys,'argv',['helper','start','4901','lxc','post-stop']), self.assertRaises(routes.RouteError):routes.main()
start.assert_not_called()
def test_missing_lxc_pid_is_not_replaced_by_a_host_fallback(self):
self.install()
with patch.dict(os.environ,{'LXC_NAME':'4901'},clear=True),patch.object(routes.os,'geteuid',return_value=0),patch.object(routes,'live_contract',return_value=MAC),patch.object(routes,'apply') as apply:
with self.assertRaises(routes.RouteError):routes.start()
apply.assert_not_called()
if __name__=='__main__':unittest.main()
oci/tests/test_stack_public_routes_lifecycle.py
+138
View File
@@ -0,0 +1,138 @@
"""Managed frontend /32 routes remain bounded and portable across lifecycle APIs."""
import copy
import contextlib
import json
import os
from pathlib import Path
import sys
import tempfile
import unittest
from unittest.mock import patch
ROOT=Path(__file__).resolve().parents[1]
sys.path.insert(0,str(ROOT/'remote'))
import oci_instance_transaction as tx
import oci_restore_recovery as recovery
import oci_stack_plan as plans
import oci_stack_public_routes as routes
import oci_remove as removal
MAIN='11111111-1111-4111-8111-111111111111'
WORKER='22222222-2222-4222-8222-222222222222'
FOREIGN='33333333-3333-4333-8333-333333333333'
def deployment():
return {'vmid':4901,'mounts':[],'stack_public_routes':{'schema':1,'stack_id':MAIN,
'primary_vmid':4900,'vmid':4901,'installation_id':WORKER,'private_bridge':'vmbr15',
'private_address':'10.77.5.31/24','routes':[{'destination':'192.0.2.10','via':'10.77.5.30',
'target_vmid':4900,'target_installation_id':MAIN}]}}
def records():
return {4900:{'vmid':4900,'installation_id':MAIN,'deployment':{},'stack':{'members':[
{'vmid':4900,'installation_id':MAIN},{'vmid':4901,'installation_id':WORKER}]}},
4901:{'vmid':4901,'installation_id':WORKER,'deployment':deployment()}}
class ManagedRouteLifecycleTests(unittest.TestCase):
@contextlib.contextmanager
def assets(self,folder):
with patch.multiple(routes,DIRECTORY=Path(folder)/'private',HELPER=Path(folder)/'canonical.py',ROOT_UID=os.geteuid()):
# The installed engine is trusted; guest-carried code is never used.
routes.HELPER.write_bytes(Path(routes.__file__).read_bytes())
routes.HELPER.chmod(0o755)
yield
def test_managed_include_coexists_with_sysctl_validation_and_rejects_unknown_include(self):
with tempfile.TemporaryDirectory() as folder,self.assets(folder):
plan=deployment();routes.install(plan,4901,WORKER)
config=(f'description: proxmenux-instance={WORKER}\nrootfs: local-zfs:subvol-4901-disk-0,size=8G\n'
f'lxc.include: {routes.include_path(4901)}\n').encode()
tx.check_runtime_mounts(config,plan)
with self.assertRaisesRegex(ValueError,'sysctl include'):
tx.check_runtime_mounts(config+b'lxc.include: /unknown/profile.conf\n',plan)
with self.assertRaisesRegex(ValueError,'differs from the contract'):
tx.check_runtime_mounts(config,{'vmid':4901,'mounts':[]})
def test_plan_rejects_foreign_target_even_if_supplied_in_record_inventory(self):
good=records();plans.validate_public_route_members(4900,good)
bad=copy.deepcopy(good);bad[4999]={'vmid':4999,'installation_id':FOREIGN,'deployment':{}}
route=bad[4901]['deployment']['stack_public_routes']['routes'][0]
route.update(target_vmid=4999,target_installation_id=FOREIGN)
with self.assertRaisesRegex(ValueError,'Invalid stack contract'):
plans.validate_public_route_members(4900,bad)
def test_plan_rejects_changed_target_and_stack_identities(self):
for field in ('target_installation_id','stack_id'):
bad=records();spec=bad[4901]['deployment']['stack_public_routes']
if field=='stack_id':spec[field]=FOREIGN
else:spec['routes'][0][field]=FOREIGN
with self.subTest(field=field),self.assertRaisesRegex(ValueError,'Invalid stack contract'):
plans.validate_public_route_members(4900,bad)
def test_restore_renumbers_only_managed_paths_and_all_route_member_ids(self):
with tempfile.TemporaryDirectory() as folder,self.assets(folder):
mapped=recovery.renumber(records(),{4900:4800,4901:4801})
# Mapping dictionary keys are not VMID-valued fields: select records by their new vmid.
mapped={value['vmid']:value for value in mapped.values()}
plans.validate_public_route_members(4800,mapped)
spec=mapped[4801]['deployment']['stack_public_routes']
self.assertEqual((spec['vmid'],spec['primary_vmid'],spec['routes'][0]['target_vmid']),(4801,4800,4800))
self.assertEqual(spec['installation_id'],WORKER)
original=f'lxc.include: {routes.include_path(4901)}\nlxc.include: /unrelated/4901.conf\n'
changed=recovery.renumbered_lines(original,4901,4801)
self.assertIn(f'lxc.include: {routes.include_path(4801)}',changed)
self.assertIn('lxc.include: /unrelated/4901.conf',changed)
routes.check(changed.splitlines()[0]+'\n',mapped[4801]['deployment'],4801,WORKER,require_host_assets=False)
def test_restore_republishes_only_canonical_host_code_before_start(self):
with tempfile.TemporaryDirectory() as folder,self.assets(folder):
plan=recovery.renumber(deployment(),{4900:4800,4901:4801})
entry={'vmid':4801,'installation_id':WORKER,'config':f'lxc.include: {routes.include_path(4801)}\n',
'copy':{'record':{'deployment':plan},'untrusted_script':'do not execute'}}
with patch.object(recovery,'renumber_config'):
recovery.restore_host_files(entry)
self.assertEqual(routes.HELPER.read_bytes(),Path(routes.__file__).read_bytes())
self.assertEqual(json.loads(routes.contract_path(4801).read_text())['routes'][0]['target_vmid'],4800)
routes.check(entry['config'],plan,4801,WORKER)
def test_restore_plan_refuses_foreign_route_before_host_assets_are_created(self):
bad=records();bad[4901]['deployment']['stack_public_routes']['routes'][0]['target_vmid']=4999
plan={'primary':4900,'members':[{'vmid':v} for v in bad],'restored':[
{'vmid':v,'copy':{'record':record}} for v,record in bad.items()],'blockers':[]}
recovery.check_public_route_members(Path('/unused'),plan)
self.assertTrue(any('Invalid stack contract' in value for value in plan['blockers']))
def test_recreate_keeps_managed_routes_outside_editable_desired_state(self):
old={'vmid':4901,'installation_id':WORKER,'status':'installed',
'template':{'id':'fixture','proxmox':{'installer_profile':{}},
'first_run':{'endpoints':[{'scheme':'http','port':80,'path':'/'}]}},
'deployment':deployment()}
candidate=copy.deepcopy(old);candidate['deployment'].pop('stack_public_routes')
candidate['deployment']['resources']={'cores':2,'memory_mb':1024}
result=tx.candidate_contract(old,'recreate',{'operation':'recreate','candidate':candidate})
self.assertEqual(result['deployment']['stack_public_routes'],old['deployment']['stack_public_routes'])
result['deployment']['stack_public_routes']['routes'][0]['destination']='192.0.2.11'
self.assertEqual(old['deployment']['stack_public_routes']['routes'][0]['destination'],'192.0.2.10')
def test_orphan_sweep_cleans_only_absent_owned_routes_and_keeps_pending_or_reused_ids(self):
for case in ('absent','pending','reused','changed'):
with self.subTest(case=case),tempfile.TemporaryDirectory() as folder,self.assets(folder):
root=Path(folder)/'records';root.mkdir()
routes.install(deployment(),4901,WORKER)
if case=='changed':routes.include_path(4901).write_text('foreign hook')
with patch.multiple(removal,CLUSTER_NODES=Path(folder)/'nodes',
SNIPPETS=Path(folder)/'snippets',MONITOR_APPS=Path(folder)/'apps',
CLUSTER_RECORDS=Path(folder)/'cluster-records'), \
patch.object(removal.runtime_settings,'include_path',lambda vmid:Path(folder)/f'{vmid}.sysctls'), \
patch.object(removal.runtime_settings,'legacy_include_path',lambda vmid:Path(folder)/f'{vmid}.legacy'), \
patch.object(removal.runtime_settings,'seccomp_path',lambda vmid:Path(folder)/f'{vmid}.seccomp'), \
patch.object(removal.oci_console,'LOG_DIR',Path(folder)/'logs'), \
patch.object(removal.instances,'guest_exists',return_value=case=='reused'), \
patch.object(removal.instances,'has_contract',return_value=case=='pending'), \
patch.object(removal.instances,'read',return_value={'installation_id':WORKER}), \
patch.object(removal.instances,'release_orphan',return_value=False), \
patch.object(removal,'remove_host_state'), \
patch.object(removal.oci_console,'remove_log'), \
patch.object(removal,'release_shared_host_files'):
cleaned=removal.sweep_orphans(root)
self.assertEqual(cleaned,[4901] if case=='absent' else [])
self.assertEqual(routes.contract_path(4901).exists(),case!='absent')
self.assertEqual(routes.include_path(4901).exists(),case!='absent')
oci/tests/test_stack_public_url_lifecycle.py
+130
View File
@@ -0,0 +1,130 @@
"""Public origins remain fixed desired state across native stack lifecycle APIs."""
import copy
import json
from pathlib import Path
import sys
import tempfile
import unittest
import uuid
from unittest.mock import patch
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / 'src'))
sys.path.insert(0, str(ROOT / 'remote'))
from proxmenux_oci.custom import template_from_compose
from proxmenux_oci.stack import build_stack, DefaultsUI
from proxmenux_oci.installer import DEFAULT_MODE
import install_generic_stack as runtime
import oci_instances as instances
import oci_instance_transaction as transactions
import oci_stack_plan as plans
import oci_stack_native as native
import oci_restore_recovery as recovery
SOURCE = '''services:
web:
image: nginx:alpine
ports: ['9001:8080']
environment: {PUBLIC_ORIGIN: 'http://localhost:9001/entry'}
depends_on: [backend, secondary]
backend:
image: example/worker:latest
environment: {PUBLIC_ORIGIN: 'http://localhost:9001/entry'}
secondary:
image: nginx:alpine
ports: ['8025:80']
environment: {PUBLIC_ORIGIN: 'http://localhost:8025/mail'}
'''
DIGEST = 'sha256:' + 'a' * 64
class StaticPublicURLLifecycle(unittest.TestCase):
def publish(self, root):
template = template_from_compose(SOURCE)
with patch('proxmenux_oci.installer.ask_storage', return_value='local-zfs'), \
patch('proxmenux_oci.installer.ask_bridge', return_value='vmbr909'), \
patch('proxmenux_oci.host.timezone', return_value='UTC'), \
patch('proxmenux_oci.network.ask_addresses', side_effect=lambda ui, bridge, labels:
({name: ('172.30.249.91/24' if name == 'secondary' else '172.30.249.90/24')
for name in labels}, None)):
deployment = build_stack(template, DefaultsUI(), DEFAULT_MODE)
services = deployment['services']
original = copy.deepcopy(services)
runtime.prepare_public_environment(services)
configs, observed = {}, {}
for index, service in enumerate(services):
vmid = 4900 + service['offset']
service['vmid'] = vmid
service['deployment']['vmid'] = vmid
service['ip'] = f'10.77.5.{30 + service["offset"]}'
runtime.assign_health_address(service)
identity = str(uuid.uuid4())
config = f'description: proxmenux-instance={identity}\nrootfs: local-zfs:subvol-{vmid}-disk-0,size=8G\n'
configs[vmid] = config.encode()
observed[vmid] = {'config': config, 'archive_path': '/fixture/image.tar',
'resolved_registry_digest': DIGEST,
'image': {'architecture': 'amd64', 'os': 'linux', 'defaults': {}}}
# The installer must publish final runtime plans, not stale input snapshots.
instances.write(instances.location(root, vmid), {
'schema_version': 1, 'vmid': vmid, 'installation_id': identity,
'status': 'assembling', 'template': copy.deepcopy(service['template']),
'deployment': copy.deepcopy(original[index]['deployment']),
'observed': copy.deepcopy(observed[vmid])})
contract = {'schema': 1, 'stack': 'public-url-fixture', 'dependencies': [
{'vmid': s['vmid'], 'label': s['name'], 'healthcheck': copy.deepcopy(s['healthcheck'])}
for s in services if not s['main']]}
with patch.object(instances, 'command', side_effect=lambda *args: configs[int(args[-1])]), \
patch.object(instances, 'observe', side_effect=lambda vmid, *args: copy.deepcopy(observed[vmid])):
instances.publish_stack(root, 4900, template, deployment, services)
return contract, {s['name']: s['vmid'] for s in services}
def assert_origins(self, records, names):
for name, vmid in names.items():
values = {item['name']: item['value'] for item in records[vmid]['deployment']['environment']}
self.assertEqual(values['PUBLIC_ORIGIN'], 'http://172.30.249.91:80/mail'
if name == 'secondary' else 'http://172.30.249.90:8080/entry')
self.assertNotIn('@STACK_LAN_IP@', json.dumps(records[vmid]['deployment']))
self.assertNotIn('localhost', values['PUBLIC_ORIGIN'])
def test_published_static_origins_survive_native_update_contract_and_private_probes(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
contract, names = self.publish(root)
records = {vmid: instances.read(root, vmid) for vmid in names.values()}
self.assert_origins(records, names)
plan = plans.build(records[4900], records,
{v: r['installation_id'] for v, r in records.items()}, 'update')
adapter = native.NativeAdapter(root, root / 'journal.json', plan)
candidates = {r['vmid']: transactions.candidate_contract(r, 'update') for r in plan['members']}
self.assert_origins(candidates, names)
expected = [{'vmid': s['vmid'], 'label': s['name'], 'healthcheck': s['healthcheck']}
for s in adapter.services.values() if s['vmid'] != 4900]
self.assertEqual(expected, contract['dependencies'])
secondary = next(s for s in adapter.services.values() if s['name'] == 'secondary')
self.assertTrue(secondary['healthcheck']['url'].startswith('http://10.77.5.'))
self.assertTrue(recovery.valid_contract(contract, set(names.values()) - {4900}))
def test_restore_refresh_adopts_storage_without_reinferring_url_or_lan_healthchecks(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
contract, names = self.publish(root)
for vmid in names.values():
record = instances.read(root, vmid)
record['deployment']['rootfs']['storage'] = 'restored-zfs'
record['recovered_at'] = instances.now()
instances.write(instances.location(root, vmid), record)
recovery.refresh_service_plans(root, list(names.values()))
records = {v: instances.read(root, v) for v in names.values()}
self.assert_origins(records, names)
primary = records[4900]
for service in primary['stack']['deployment']['services']:
self.assertEqual(service['deployment'], records[service['vmid']]['deployment'])
self.assertEqual(service['deployment']['rootfs']['storage'], 'restored-zfs')
expected = [{'vmid': s['vmid'], 'label': s['name'], 'healthcheck': s['healthcheck']}
for s in primary['stack']['deployment']['services'] if s['vmid'] != 4900]
self.assertEqual(expected, contract['dependencies'])
self.assertTrue(recovery.valid_contract(contract, set(names.values()) - {4900}))
if __name__ == '__main__':
unittest.main()
oci/tests/test_stack_tcp_readiness.py
+64
View File
@@ -0,0 +1,64 @@
"""Internal TCP readiness survives initial install, native lifecycle and restore."""
import copy
from pathlib import Path
import socket
import subprocess
import sys
import unittest
from unittest.mock import patch
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT/'remote'))
import install_generic_stack as installer
import oci_stack_native as native
import oci_restore_recovery as recovery
class TCPReadinessTests(unittest.TestCase):
def test_initial_runtime_probe_uses_allocated_private_address(self):
service={'ip':'10.77.20.31','healthcheck':{'type':'tcp','port':9998,'timeout_seconds':120}}
installer.assign_health_address(service)
self.assertEqual(service['healthcheck']['host'],'10.77.20.31')
self.assertNotIn('url',service['healthcheck'])
self.assertNotIn('frontend',service)
def test_native_health_probes_real_socket_without_http(self):
with socket.socket() as server:
server.bind(('127.0.0.1',0));server.listen()
port=server.getsockname()[1]
member={'vmid':4800,'deployment':{},'stack':{'deployment':{'services':[
{'vmid':4800,'name':'helper','healthcheck':{'type':'tcp','host':'127.0.0.1','port':port,'timeout_seconds':5}}]}}}
adapter=native.NativeAdapter(Path('/registry'),Path('/journal'),{'members':[member],'primary_vmid':4800})
with patch.object(adapter,'is_running',return_value=True), \
patch.object(native.member_tx.gpu_devices,'validate_runtime'), \
patch.object(native.member_tx,'run') as run:
adapter.healthcheck(4800)
run.assert_not_called()
def test_native_tcp_target_rejects_names_and_invalid_ports(self):
for host,port in [('example.com',3000),('127.0.0.1',True),('127.0.0.1',0),('127.0.0.1',65536),('127.0.0.1','3000')]:
with self.subTest(host=host,port=port),self.assertRaises(ValueError):
native.tcp_target({'host':host,'port':port})
def test_restore_retains_only_private_tcp_probes_with_valid_ports(self):
contract={'schema':1,'dependencies':[{'vmid':4801,'label':'helper',
'healthcheck':{'type':'tcp','host':'10.77.20.31','port':9998,'timeout_seconds':120}}]}
self.assertTrue(recovery.valid_contract(contract,{4801}))
for changes in ({'host':'192.168.1.1'},{'host':'localhost'},{'port':False},{'port':65536}):
invalid=copy.deepcopy(contract);invalid['dependencies'][0]['healthcheck'].update(changes)
self.assertFalse(recovery.valid_contract(invalid,{4801}))
def test_hook_tcp_probe_checks_connection_not_http_status(self):
text=(ROOT/'remote/stack_dependency_hook.sh').read_text()
functions=text[text.index('dependency_is_healthy() {'):text.index('\nstart_dependencies()')]
with socket.socket() as server:
server.bind(('127.0.0.1',0));server.listen()
port=server.getsockname()[1]
command=functions+'\ndependency_is_healthy 4801 \'{"type":"tcp","host":"127.0.0.1","port":'+str(port)+'}\''
result=subprocess.run(['bash','-c',command],text=True,capture_output=True)
self.assertEqual(result.returncode,0,result.stderr)
failed=subprocess.run(['bash','-c',command],text=True,capture_output=True)
self.assertNotEqual(failed.returncode,0)
if __name__=='__main__':unittest.main()
oci/tests/test_watchdog.py
+13
View File
@@ -351,6 +351,19 @@ class Switch(unittest.TestCase):
return {int(path.parent.name): json.loads(path.read_text())["deployment"].get("watchdog")
for path in root.glob("*/oci-compose.json")}
def test_completed_jobs_never_receive_or_use_watchdog(self):
with tempfile.TemporaryDirectory() as folder:
root=self.registry(folder)
path=root/'102'/'oci-compose.json'
record=json.loads(path.read_text())
record['deployment'].update(execution_mode='one-shot',watchdog=True)
path.write_text(json.dumps(record))
self.assertNotIn(102,watchdog.watched(root))
with patch.object(watchdog,'ensure_service'),patch('oci_carried_record.carry'):
watchdog.set_watchdog(root,101,True)
self.assertEqual(self.flags(root)[101],True)
self.assertEqual(self.flags(root)[102],False)
def test_every_container_of_the_application_takes_the_choice_and_keeps_the_rest_of_its_record(self):
from unittest.mock import patch
import oci_carried_record