ProxMenux/web/messages/en/docs/monitor/architecture.json

{
  "meta": {
    "title": "ProxMenux Monitor Architecture — AppImage, Flask, SQLite, WebSocket | ProxMenux",
    "description": "How ProxMenux Monitor is built: AppImage layout, Flask blueprints, background workers, data sources (psutil, pvesh, smartctl, journalctl), SQLite persistence, WebSocket terminal, AI providers, notification channels, reverse proxy and optional Fail2Ban integration.",
    "ogTitle": "ProxMenux Monitor Architecture",
    "ogDescription": "Inside ProxMenux Monitor — AppImage layout, Flask blueprints, background workers, SQLite, WebSocket, AI providers, notification channels.",
    "twitterTitle": "ProxMenux Monitor Architecture",
    "twitterDescription": "AppImage, Flask, SQLite, WebSocket, AI providers and notification channels — inside the Monitor."
  },
  "header": {
    "title": "Architecture",
    "description": "How ProxMenux Monitor is packaged, what runs inside the AppImage, and how requests flow from the browser through the Flask backend to the host's tooling and SQLite store.",
    "section": "ProxMenux Monitor"
  },
  "intro": {
    "title": "One process, many responsibilities",
    "body": "A single Python process listens on TCP 8008. It serves the static Next.js build, exposes the REST API, handles the WebSocket terminal, runs the periodic Health Monitor, and dispatches notifications. There is no separate web server, no message broker, no external database."
  },
  "requestFlow": {
    "heading": "Request flow",
    "intro": "A typical dashboard data request follows this path; browser-only preferences can be handled locally:",
    "diagramCaption": "Protected routes validate JWT when authentication is enabled. Core routes and blueprints return cached or newly collected host data. The optional Fail2Ban hook checks its cached jail ban list. Forwarded client-IP headers are trusted only with PROXMENUX_TRUST_PROXY=1; otherwise the peer address is used. Persistent state includes SQLite databases and JSON files, including auth.json.",
    "diagramArrowLabel": "HTTP / WS",
    "nodes": {
      "clientLabel": "Client",
      "clientDetail": "Browser or PWA\n+ optional\nNginx / Caddy /\nTraefik proxy",
      "flaskLabel": "Flask :8008",
      "flaskDetail": "Blueprints\nJWT middleware\nFail2Ban hook\n(if installed)",
      "hostLabel": "Host tools",
      "hostDetail": "psutil\npvesh\nsmartctl\njournalctl",
      "stateLabel": "Local state",
      "stateDetail": "SQLite DB\n+ auth.json"
    },
    "threadsIntro": "The process also runs <strong>background threads</strong>. The examples below sample host data, update caches and SQLite, and dispatch notifications. Other workers also exist, and notification delivery depends on its enabled setting:",
    "headerThread": "Thread",
    "headerCadence": "Cadence",
    "headerJob": "Job",
    "rows": [
      {
        "thread": "_temperature_collector_loop",
        "cadence": "60 s",
        "job": "Records CPU temperature and a network-latency sample into the history DB so the dashboard graphs have data even when no client is connected."
      },
      {
        "thread": "_health_collector_loop",
        "cadence": "5 min",
        "job": "Runs the full Health Monitor cycle (10 categories), persists active errors, dismissals and disk observations, and feeds new events into the notification engine."
      },
      {
        "thread": "_vital_signs_sampler",
        "cadence": "CPU 5 s; temperature 15 s; memory 30 s",
        "job": "Samples CPU usage, memory and CPU temperature for live metrics and sustained-condition detection, independently of the five-minute health collector."
      },
      {
        "thread": "notification_manager.start()",
        "cadence": "event-driven",
        "job": "Spawns the journal / task / hook watchers (<code>JournalWatcher</code>, <code>TaskWatcher</code>, <code>ProxmoxHookWatcher</code>) and dispatches to configured channels with optional AI rewriting."
      }
    ]
  },
  "systemd": {
    "heading": "systemd unit",
    "intro": "The installer drops a unit at <code>/etc/systemd/system/proxmenux-monitor.service</code>. Default content:",
    "items": [
      "<strong><code>User=root</code></strong> — the supported service runs with root privileges for host operations such as SMART access, <code>pvesh</code> and ZFS management. Its web-terminal processes inherit those privileges.",
      "<strong><code>Restart=on-failure</code></strong> with a 10-second back-off — non-zero exits relaunch automatically.",
      "<strong><code>After=network.target</code></strong> — orders startup after that target; it does not guarantee an assigned IP, DNS or external connectivity."
    ],
    "inspectTitle": "Inspect the live unit"
  },
  "appimage": {
    "heading": "What the AppImage contains",
    "intro": "The AppImage is a self-mounting filesystem. <code>AppRun</code> at the root sets up the environment and execs <code>flask_server.py</code>:",
    "consequencesIntro": "Two consequences of this layout:",
    "consequences": [
      "<strong>Bundled Python packages.</strong> Python packages ship inside the AppImage and run on the host's own Python 3.11 (PVE 8) or 3.13 (PVE 9) interpreter, without installing anything into the host's system Python. The application still uses host kernel interfaces and external host tools.",
      "<strong>Hardware clients are bundled too.</strong> <code>ipmitool</code>, <code>lm-sensors</code> and <code>upsc</code> are included. Readings still depend on supported hardware, access to sensors and, for UPS data, a configured NUT server."
    ]
  },
  "flask": {
    "heading": "Flask app structure",
    "intro": "<code>flask_server.py</code> creates one <code>Flask(__name__)</code> instance with restricted CORS origins, registers ten blueprints directly and initializes terminal routes. The table highlights the main modules; PBS, Monitor links, audit and VM-backup blueprints are also registered. PVE-specific prewarmers run only on PVE hosts.",
    "headerBlueprint": "Blueprint / module",
    "headerPrefix": "Routes prefix",
    "headerOwns": "Owns",
    "rows": [
      {
        "blueprint": "flask_server.py",
        "prefix": [
          "/api/system",
          "/api/storage",
          "/api/network",
          "/api/vms",
          "/api/hardware",
          "/api/logs",
          "/api/prometheus"
        ],
        "owns": "Core data endpoints + static dashboard serving + optional Fail2Ban app-level check (active only when Fail2Ban is installed on the host with the <code>proxmenux</code> jail)."
      },
      {
        "blueprint": "flask_auth_routes.py",
        "prefix": [
          "/api/auth/*"
        ],
        "owns": "Login, JWT issuing, TOTP setup/verify, password change, API token generation."
      },
      {
        "blueprint": "flask_health_routes.py",
        "prefix": [
          "/api/health/*"
        ],
        "owns": "Health probes, detailed status, active and dismissed errors, and suppression settings. Protected routes require authentication when it is enabled."
      },
      {
        "blueprint": "flask_terminal_routes.py",
        "prefix": [
          "/api/terminal/* + WS"
        ],
        "owns": "PTY allocation per session and WebSocket pipe to <code>xterm.js</code> in the browser."
      },
      {
        "blueprint": "flask_notification_routes.py",
        "prefix": [
          "/api/notifications/*"
        ],
        "owns": "Channel CRUD, test-send, AI provider config, history, manual sends."
      },
      {
        "blueprint": "flask_security_routes.py",
        "prefix": [
          "/api/security/*"
        ],
        "owns": "Authentication failures and, when Fail2Ban is installed, jail status, ban events and manual unban."
      },
      {
        "blueprint": "flask_proxmenux_routes.py",
        "prefix": [
          "/api/proxmenux/*"
        ],
        "owns": "Reads the ProxMenux post-install optimizations recorded in the host registry."
      },
      {
        "blueprint": "flask_oci_routes.py",
        "prefix": [
          "/api/oci/*"
        ],
        "owns": "OCI / container app deployment helpers (Proxmox VE 9.1+)."
      }
    ],
    "endpointsLink": "Integration endpoints and their request/response formats are described in <link>API Reference</link>."
  },
  "dataSources": {
    "heading": "Data sources",
    "intro": "Host data comes mainly from standard libraries, files and administrative commands. Some guest information also depends on the QEMU Guest Agent or commands executed inside a CT:",
    "headerSource": "Source",
    "headerUsedFor": "Used for",
    "rows": [
      {
        "source": "psutil",
        "usedFor": "CPU load, memory, swap, mountpoint usage, NIC counters, process list."
      },
      {
        "source": "pvesh / qm / pct",
        "usedFor": "Proxmox node info, VM and CT inventory and config, storage pools, task history."
      },
      {
        "source": "smartctl",
        "usedFor": "SATA / NVMe attributes, SMART health, wear / lifetime, model and serial."
      },
      {
        "source": "zpool / zfs",
        "usedFor": "Pool state (ONLINE / DEGRADED / FAULTED / UNAVAIL), scrub progress, dataset usage."
      },
      {
        "source": "journalctl",
        "usedFor": "System logs, OOM kills, ATA / NVMe / dm errors, security events, custom service units."
      },
      {
        "source": "ip / iproute2",
        "usedFor": "Interfaces, addresses, bridges, bonds, OVS-managed devices."
      },
      {
        "source": "nvidia-smi · intel_gpu_top",
        "usedFor": "Available GPU utilisation, VRAM, temperature and encoder/decoder measurements; fields depend on the vendor, tool and driver."
      },
      {
        "source": "lspci · lscpu · dmidecode",
        "usedFor": "PCIe topology, CPU model and topology, board and BIOS info."
      },
      {
        "source": "ipmitool · sensors",
        "usedFor": "Out-of-band sensors, fan speeds, board temperatures (when supported)."
      },
      {
        "source": "upsc (NUT)",
        "usedFor": "UPS battery state, load, runtime — when a NUT server is configured on the host."
      }
    ],
    "cacheTitle": "Output is cached — not every request hits the host",
    "cacheBody": "Expensive sources (<code>smartctl -a</code>, <code>pvesh get</code>) are wrapped in time-bound caches inside the Flask process so a busy dashboard tab doesn't hammer the disk or the cluster API. The cache TTLs are tuned per source (a few seconds for live metrics, several minutes for SMART)."
  },
  "persistence": {
    "heading": "Persistence",
    "intro": "Several state files have different contents and access requirements:",
    "headerPath": "Path",
    "headerOwner": "Owner",
    "headerContents": "Contents",
    "rows": [
      {
        "path": "/usr/local/share/proxmenux/health_monitor.db",
        "owner": "root:root",
        "contents": "SQLite DB. Tables: <code>errors</code>, <code>events</code>, <code>disk_registry</code>, <code>disk_observations</code>, <code>user_settings</code>, <code>notification_history</code>, <code>excluded_storages</code>, <code>excluded_interfaces</code>. WAL journal mode."
      },
      {
        "path": "/usr/local/share/proxmenux/.notification_key",
        "owner": "root <code>0600</code>",
        "contents": "32-byte master key for the ENC2 notification-secret format. The implementation uses a per-value random salt, derived keys and a MAC; old fixed-key XOR values are supported for migration."
      },
      {
        "path": "/root/.config/proxmenux-monitor/auth.json",
        "owner": "root:root",
        "contents": "Authentication configuration: enabled flag, username, salted PBKDF2-HMAC-SHA256 password hash, TOTP secret, backup codes, per-install JWT signing secret, API-token metadata and revocation hashes. Legacy SHA-256 password hashes are migrated on successful login."
      },
      {
        "path": "/var/log/proxmenux-auth.log",
        "owner": "root:root",
        "contents": "Plain-text authentication event log, written independently of Fail2Ban. When the <code>[proxmenux]</code> jail is installed and enabled, it reads this log to detect repeated failed logins. Without that jail, the log remains available for inspection."
      }
    ],
    "backupTitle": "Back up auth.json before reinstalling",
    "backupBody": "Replacing the AppImage normally preserves <code>/root/.config/proxmenux-monitor/auth.json</code> and <code>/usr/local/share/proxmenux/health_monitor.db</code>. Preserve <code>auth.json</code> with its per-install jwt_secret to retain token validity and metadata. The database stores health and notification state independently. Back up <code>.notification_key</code> with the database to restore encrypted channel settings; restoring the DB alone does not restore API authentication."
  },
  "health": {
    "heading": "Health Monitor cycle",
    "intro": "About every five minutes, the background collector invokes <code>health_monitor.py</code> to run the checks behind the dashboard’s ten categories:",
    "items": [
      "Critical PVE services (<code>pveproxy</code>, <code>pvedaemon</code>, <code>pvestatd</code>, <code>pve-cluster</code>).",
      "Proxmox storage pools (<code>pvesh get /storage</code> + per-storage availability).",
      "Disks and filesystems: SMART, dmesg I/O errors, ZFS pool health, mountpoint capacity.",
      "VMs and CTs: failed starts, crashed guests, QMP errors, shutdown failures.",
      "Network: bridge / bond status, link state, latency to the gateway.",
      "Updates: pending package upgrades and security patches.",
      "Logs: persistent / spike / cascade pattern detection in the system journal.",
      "Memory: OOM killer activity, sustained high pressure.",
      "CPU: usage and CPU temperature checked against the configurable thresholds; available sensors depend on the hardware.",
      "Security: authentication failures, ban events, fail2ban jail status."
    ],
    "afterIntro": "Findings use a stable <code>error_key</code>, category and severity. Existing records in <code>errors</code> are updated in place (<code>last_seen</code>, severity, reason) rather than duplicated. Notification delivery also has its own fingerprint and cooldown rules.",
    "cycleEnd": "Cleanup can auto-resolve stale errors using category <em>Suppression Duration</em> values and state-based rules, remove orphan records for resources that were removed, and prune <code>events</code> older than 30 days. The categories and dashboard are described in <link>Dashboard → Health Monitor</link>."
  },
  "notifications": {
    "heading": "Notification engine",
    "intro": "<code>notification_manager.py</code> is the orchestrator. It loads the configured channels, owns the delivery queue, and exposes both a Python API (for Flask routes and the Health Monitor cycle) and a CLI entrypoint (for the <code>.sh</code> hook scripts shipped with ProxMenux).",
    "items": [
      "<strong>Watchers</strong> push events: <code>JournalWatcher</code> tails the system journal, <code>TaskWatcher</code> polls the Proxmox task list, <code>ProxmoxHookWatcher</code> reacts to backup / replication / snapshot hooks, and <code>PollingCollector</code> handles slow data sources.",
      "<strong>Templates</strong> turn an event into a (title, body) pair. The same template can run through the configured AI provider (OpenAI / Anthropic / Gemini / Groq / Ollama / OpenRouter) to produce a plain-language rewrite; both versions are stored in <code>notification_history</code>.",
      "<strong>Channels</strong> deliver messages through services including Telegram, Discord, Email, Gotify, Pushover and Apprise. Implementations in <code>notification_channels.py</code> use the shared <code>create_channel()</code> / <code>send()</code> interface.",
      "<strong>Encryption.</strong> Sensitive settings (<code>telegram.bot_token</code>, <code>discord.webhook_url</code>, <code>pushover.user_key</code>, <code>pushover.api_token</code>, <code>ai_api_key_*</code>, <code>email.password</code>) are encrypted with the key in <code>.notification_key</code> before being written to the DB and are masked in the interface."
    ],
    "linksFooter": "Per-event toggles, channel overrides and AI configuration are surfaced in <notifLink>Settings → Notifications</notifLink> and <aiLink>Settings → AI Assistant</aiLink>."
  },
  "websocket": {
    "heading": "WebSocket terminal",
    "intro": "The <em>Terminal</em> tab in the dashboard is a thin <code>xterm.js</code> client wired to a server-side PTY through a WebSocket. Two transport modes:",
    "items": [
      "<strong>HTTP mode (default):</strong> the built-in Flask server with <code>flask-sock</code> handles WebSocket upgrade requests.",
      "<strong>HTTPS / WSS mode:</strong> the TLS server uses <code>gevent.pywsgi.WSGIServer</code> with its default handler; <code>flask-sock</code> handles WebSocket connections over TLS."
    ],
    "outro": "Terminal child processes inherit <code>User=root</code> from the service. Opening a WebSocket requires a short-lived, single-use ticket obtained from the authenticated terminal-ticket endpoint with administrative scope.",
    "proxyNote": "If you access the Monitor through a reverse proxy, make sure WebSocket forwarding is enabled (the <code>Upgrade</code> and <code>Connection</code> headers). Without it the terminal won't work."
  },
  "proxy": {
    "heading": "Reverse proxy & Fail2Ban",
    "intro": "Client-IP selection and optional application-level blocking depend on these settings and components:",
    "items": [
      "<strong>Client IP selection.</strong> The <code>before_request</code> ban check and auth logging trust <code>X-Forwarded-For</code>, then <code>X-Real-IP</code>, only when PROXMENUX_TRUST_PROXY=1. Otherwise they use <code>request.remote_addr</code>. Forwarded headers are not accepted automatically.",
      "<strong>Optional application-level Fail2Ban.</strong> A proxy connection can present the proxy as the TCP peer. When the trusted client address is available and the <code>proxmenux</code> jail is active, Flask rejects listed IPs with HTTP 403. The ban list is cached for 30 seconds and refreshed by request processing after expiry, rather than polled by a dedicated timer."
    ],
    "calloutTitle": "Fail2Ban is not bundled",
    "calloutBody": "Fail2Ban is <strong>not</strong> bundled or installed automatically. It can be installed through the optional Monitor Security workflow or <link>Security → Fail2Ban</link> in ProxMenux Scripts. If <code>fail2ban-client</code> or the <code>proxmenux</code> jail is absent, this application hook does not block requests. The configured authentication policy still applies.",
    "outro": "Reverse-proxy snippets (Nginx / Caddy / Traefik) and the Fail2Ban jail walkthrough are in <accessLink>Access & Authentication</accessLink> and <fail2banLink>Security → Fail2Ban</fail2banLink>."
  },
  "whereNext": {
    "heading": "Where to next",
    "items": [
      {
        "label": "Access & Authentication",
        "href": "/docs/monitor/access-auth",
        "tail": " — first-launch setup, password + TOTP 2FA, reverse-proxy snippets, Fail2Ban jail."
      },
      {
        "label": "API Reference",
        "href": "/docs/monitor/api",
        "tail": " — integration endpoints, token management and authentication conventions."
      },
      {
        "label": "Settings → ProxMenux Monitor",
        "href": "/docs/settings/proxmenux-monitor",
        "tail": " — the in-menu service toggle and status verification flow inside the ProxMenux TUI."
      }
    ]
  }
}