"twitterDescription":"AppImage, Flask, SQLite, WebSocket, AI providers and notification channels — inside the Monitor."
},
"header":{
"title":"Architecture",
"description":"How ProxMenux Monitor is packaged, what runs inside the AppImage, and how requests flow from the browser through the Flask backend to the host's tooling and SQLite store.",
"section":"ProxMenux Monitor"
},
"intro":{
"title":"One process, many responsibilities",
"body":"A single Python process listens on TCP 8008. It serves the static Next.js build, exposes the REST API, handles the WebSocket terminal, runs the periodic Health Monitor, and dispatches notifications. There is no separate web server, no message broker, no external database."
"intro":"A typical dashboard data request follows this path; browser-only preferences can be handled locally:",
"diagramCaption":"Protected routes validate JWT when authentication is enabled. Core routes and blueprints return cached or newly collected host data. The optional Fail2Ban hook checks its cached jail ban list. Forwarded client-IP headers are trusted only with PROXMENUX_TRUST_PROXY=1; otherwise the peer address is used. Persistent state includes SQLite databases and JSON files, including auth.json.",
"threadsIntro":"The process also runs <strong>background threads</strong>. The examples below sample host data, update caches and SQLite, and dispatch notifications. Other workers also exist, and notification delivery depends on its enabled setting:",
"job":"Records CPU temperature and a network-latency sample into the history DB so the dashboard graphs have data even when no client is connected."
},
{
"thread":"_health_collector_loop",
"cadence":"5 min",
"job":"Runs the full Health Monitor cycle (10 categories), persists active errors, dismissals and disk observations, and feeds new events into the notification engine."
"cadence":"CPU 5 s; temperature 15 s; memory 30 s",
"job":"Samples CPU usage, memory and CPU temperature for live metrics and sustained-condition detection, independently of the five-minute health collector."
"job":"Spawns the journal / task / hook watchers (<code>JournalWatcher</code>, <code>TaskWatcher</code>, <code>ProxmoxHookWatcher</code>) and dispatches to configured channels with optional AI rewriting."
}
]
},
"systemd":{
"heading":"systemd unit",
"intro":"The installer drops a unit at <code>/etc/systemd/system/proxmenux-monitor.service</code>. Default content:",
"<strong><code>User=root</code></strong> — the supported service runs with root privileges for host operations such as SMART access, <code>pvesh</code> and ZFS management. Its web-terminal processes inherit those privileges.",
"<strong><code>After=network.target</code></strong> — orders startup after that target; it does not guarantee an assigned IP, DNS or external connectivity."
"<strong>Bundled Python packages.</strong> Python packages ship inside the AppImage and run on the host's own Python 3.11 (PVE 8) or 3.13 (PVE 9) interpreter, without installing anything into the host's system Python. The application still uses host kernel interfaces and external host tools.",
"<strong>Hardware clients are bundled too.</strong> <code>ipmitool</code>, <code>lm-sensors</code> and <code>upsc</code> are included. Readings still depend on supported hardware, access to sensors and, for UPS data, a configured NUT server."
"intro":"<code>flask_server.py</code> creates one <code>Flask(__name__)</code> instance with restricted CORS origins, registers ten blueprints directly and initializes terminal routes. The table highlights the main modules; PBS, Monitor links, audit and VM-backup blueprints are also registered. PVE-specific prewarmers run only on PVE hosts.",
"owns":"Core data endpoints + static dashboard serving + optional Fail2Ban app-level check (active only when Fail2Ban is installed on the host with the <code>proxmenux</code> jail)."
},
{
"blueprint":"flask_auth_routes.py",
"prefix":[
"/api/auth/*"
],
"owns":"Login, JWT issuing, TOTP setup/verify, password change, API token generation."
"owns":"Health probes, detailed status, active and dismissed errors, and suppression settings. Protected routes require authentication when it is enabled."
"intro":"Host data comes mainly from standard libraries, files and administrative commands. Some guest information also depends on the QEMU Guest Agent or commands executed inside a CT:",
"usedFor":"PCIe topology, CPU model and topology, board and BIOS info."
},
{
"source":"ipmitool · sensors",
"usedFor":"Out-of-band sensors, fan speeds, board temperatures (when supported)."
},
{
"source":"upsc (NUT)",
"usedFor":"UPS battery state, load, runtime — when a NUT server is configured on the host."
}
],
"cacheTitle":"Output is cached — not every request hits the host",
"cacheBody":"Expensive sources (<code>smartctl -a</code>, <code>pvesh get</code>) are wrapped in time-bound caches inside the Flask process so a busy dashboard tab doesn't hammer the disk or the cluster API. The cache TTLs are tuned per source (a few seconds for live metrics, several minutes for SMART)."
"contents":"32-byte master key for the ENC2 notification-secret format. The implementation uses a per-value random salt, derived keys and a MAC; old fixed-key XOR values are supported for migration."
"contents":"Plain-text authentication event log, written independently of Fail2Ban. When the <code>[proxmenux]</code> jail is installed and enabled, it reads this log to detect repeated failed logins. Without that jail, the log remains available for inspection."
"backupBody":"Replacing the AppImage normally preserves <code>/root/.config/proxmenux-monitor/auth.json</code> and <code>/usr/local/share/proxmenux/health_monitor.db</code>. Preserve <code>auth.json</code> with its per-install jwt_secret to retain token validity and metadata. The database stores health and notification state independently. Back up <code>.notification_key</code> with the database to restore encrypted channel settings; restoring the DB alone does not restore API authentication."
"intro":"About every five minutes, the background collector invokes <code>health_monitor.py</code> to run the checks behind the dashboard’s ten categories:",
"afterIntro":"Findings use a stable <code>error_key</code>, category and severity. Existing records in <code>errors</code> are updated in place (<code>last_seen</code>, severity, reason) rather than duplicated. Notification delivery also has its own fingerprint and cooldown rules.",
"cycleEnd":"Cleanup can auto-resolve stale errors using category <em>Suppression Duration</em> values and state-based rules, remove orphan records for resources that were removed, and prune <code>events</code> older than 30 days. The categories and dashboard are described in <link>Dashboard → Health Monitor</link>."
"intro":"<code>notification_manager.py</code> is the orchestrator. It loads the configured channels, owns the delivery queue, and exposes both a Python API (for Flask routes and the Health Monitor cycle) and a CLI entrypoint (for the <code>.sh</code> hook scripts shipped with ProxMenux).",
"items":[
"<strong>Watchers</strong> push events: <code>JournalWatcher</code> tails the system journal, <code>TaskWatcher</code> polls the Proxmox task list, <code>ProxmoxHookWatcher</code> reacts to backup / replication / snapshot hooks, and <code>PollingCollector</code> handles slow data sources.",
"<strong>Templates</strong> turn an event into a (title, body) pair. The same template can run through the configured AI provider (OpenAI / Anthropic / Gemini / Groq / Ollama / OpenRouter) to produce a plain-language rewrite; both versions are stored in <code>notification_history</code>.",
"<strong>Channels</strong> deliver messages through services including Telegram, Discord, Email, Gotify, Pushover and Apprise. Implementations in <code>notification_channels.py</code> use the shared <code>create_channel()</code> / <code>send()</code> interface.",
"<strong>Encryption.</strong> Sensitive settings (<code>telegram.bot_token</code>, <code>discord.webhook_url</code>, <code>pushover.user_key</code>, <code>pushover.api_token</code>, <code>ai_api_key_*</code>, <code>email.password</code>) are encrypted with the key in <code>.notification_key</code> before being written to the DB and are masked in the interface."
"linksFooter":"Per-event toggles, channel overrides and AI configuration are surfaced in <notifLink>Settings → Notifications</notifLink> and <aiLink>Settings → AI Assistant</aiLink>."
},
"websocket":{
"heading":"WebSocket terminal",
"intro":"The <em>Terminal</em> tab in the dashboard is a thin <code>xterm.js</code> client wired to a server-side PTY through a WebSocket. Two transport modes:",
"<strong>HTTPS / WSS mode:</strong> the TLS server uses <code>gevent.pywsgi.WSGIServer</code> with its default handler; <code>flask-sock</code> handles WebSocket connections over TLS."
"outro":"Terminal child processes inherit <code>User=root</code> from the service. Opening a WebSocket requires a short-lived, single-use ticket obtained from the authenticated terminal-ticket endpoint with administrative scope.",
"proxyNote":"If you access the Monitor through a reverse proxy, make sure WebSocket forwarding is enabled (the <code>Upgrade</code> and <code>Connection</code> headers). Without it the terminal won't work."
"<strong>Client IP selection.</strong> The <code>before_request</code> ban check and auth logging trust <code>X-Forwarded-For</code>, then <code>X-Real-IP</code>, only when PROXMENUX_TRUST_PROXY=1. Otherwise they use <code>request.remote_addr</code>. Forwarded headers are not accepted automatically.",
"<strong>Optional application-level Fail2Ban.</strong> A proxy connection can present the proxy as the TCP peer. When the trusted client address is available and the <code>proxmenux</code> jail is active, Flask rejects listed IPs with HTTP 403. The ban list is cached for 30 seconds and refreshed by request processing after expiry, rather than polled by a dedicated timer."
"calloutBody":"Fail2Ban is <strong>not</strong> bundled or installed automatically. It can be installed through the optional Monitor Security workflow or <link>Security → Fail2Ban</link> in ProxMenux Scripts. If <code>fail2ban-client</code> or the <code>proxmenux</code> jail is absent, this application hook does not block requests. The configured authentication policy still applies.",
"outro":"Reverse-proxy snippets (Nginx / Caddy / Traefik) and the Fail2Ban jail walkthrough are in <accessLink>Access & Authentication</accessLink> and <fail2banLink>Security → Fail2Ban</fail2banLink>."
