feat(oci): community validation record, report form and generated list

2026-10-07 14:06:40 +00:00 · 2026-09-28 17:30:29 +02:00
parent 3d2aff6c66
commit adc42caae3
13 changed files with 999 additions and 10 deletions
@@ -228,10 +228,17 @@ Generated templates start as `generated-unvalidated`. Promotion requires:
 6. Review of every platform adaptation and unsupported feature.

 Real tests are recorded in `catalog/verification.json`, which the catalog
-applies on top of the generated templates and the overlays, so a regeneration
-keeps them: `"status": "laboratory-validated"` for an application tested in
-the ProxMenux lab, or `"community_tested": {"by": "<GitHub user>", "date":
-"<YYYY-MM-DD>"}` for one tested by the community.
+reads when it loads and applies on top of the generated templates and the
+overlays, so a new entry shows without regenerating and a regeneration keeps
+it: `"status": "laboratory-validated"` for an application tested in the
+ProxMenux lab, or `"community_tested": {"by": "<GitHub user>", "date":
+"<YYYY-MM-DD>", "report": "<link to the report>"}` for one tested by the
+community. Both show as verified in the OCI installer.
+
+[VALIDATION.md](VALIDATION.md) lists the verified applications and explains
+how to take part. It is generated from `catalog/verification.json` with
+`python3 .github/scripts/oci_validation.py render`, and CI checks that it is
+current.

 The mini changelog comes from the LinuxServer README `Versions` section. At
 installation, the architecture-specific registry digest and image labels are
@@ -0,0 +1,69 @@
+# OCI application validation
+
+<!-- Generated from oci/catalog/verification.json by .github/scripts/oci_validation.py. Do not edit by hand. -->
+
+38 of 335 applications in the OCI catalog have been tested for real: 36 in the ProxMenux lab and 2 by the community. The OCI installer shows them as verified, with a ✓ in the lists.
+
+Each test describes the scenario that was run and names the image it ran on. It does not cover every possible configuration of the application, and a newer image has not been tested until someone reports it.
+
+## Taking part
+
+Any application that is not listed here is open for testing.
+
+1. Check the [ProxMenux Roadmap](https://github.com/users/MacRimi/projects/1) to see which applications someone is already testing. With access to the board, create a card for the application and move it to In progress while you test it.
+2. Test the application with the OCI manager and fill in the [OCI validation report](https://github.com/MacRimi/ProxMenux/issues/new?template=oci-validation.yml). The report records the image and the exact scenario: install mode, storage, network and GPU.
+3. A reviewer reads the report and comments `/validated` on it. The application is then added to this list with your GitHub user and shown as verified in the OCI installer, and the report is closed.
+
+Reviewers are listed in [.github/oci-validation-reviewers](../.github/oci-validation-reviewers), and a reviewer does not validate their own report. A validation can also be added with a pull request that adds the entry to `oci/catalog/verification.json` and regenerates this file with `python3 .github/scripts/oci_validation.py render`; CI checks that the entry credits the author of the pull request.
+
+An application that cannot be validated for a reason outside the tester's hands — hardware nobody has, something only Docker provides, a fix still pending in ProxMenux — moves to Blocked on the board, with a line saying why and what would make it worth trying again.
+
+Questions and ideas about OCI testing go in [discussion #360](https://github.com/MacRimi/ProxMenux/discussions/360).
+
+## Tested by the community
+
+| Application | Tested by | Date | Image | Scenario | Report |
+|---|---|---|---|---|---|
+| Glances | [@Vaso73](https://github.com/Vaso73) | 2026-09-27 | — | — | [Report](https://github.com/MacRimi/ProxMenux/discussions/392#discussioncomment-18623868) |
+| PocketBase | [@Vaso73](https://github.com/Vaso73) | 2026-09-27 | — | — | [Report](https://github.com/MacRimi/ProxMenux/discussions/392#discussioncomment-18623868) |
+
+## Tested in the ProxMenux lab
+
+| Application | Category |
+|---|---|
+| 2FAuth | Authentication & Security |
+| Adguardhome Sync | Adblock & DNS |
+| Alby Hub ✨ | Finance & Budgeting |
+| Alist | Productivity & Workflows |
+| aMule | Files & Downloads |
+| CodeProject.AI Server | AI |
+| CopyParty | Files & Downloads |
+| Crafty | Gaming & Leisure |
+| Ddclient | Adblock & DNS |
+| Duplicati | Backup & Recovery |
+| Etherpad | Documents & Notes |
+| FileBrowser Quantum | Tools |
+| FlareSolverr | *Arr Suite |
+| Flexget | *Arr Suite |
+| Frigate | NVR & Cameras |
+| Grafana | Monitoring & Analytics |
+| Immich | Media |
+| JDownloader | Files & Downloads |
+| Jenkins CI/CD | AI / Coding & Dev-Tools |
+| Linkwarden | Documents & Notes |
+| Memos | Documents & Notes |
+| MineOS | Gaming & Leisure |
+| Motioneye | NVR & Cameras |
+| Nextcloud | Productivity & Workflows |
+| OpenList | Productivity & Workflows |
+| Openssh Server | Remote Access & VPN |
+| Paperless-ngx | Productivity & Workflows |
+| Phpmyadmin | Databases |
+| Qbittorrent | Files & Downloads |
+| Rclone WebUI | Backup & Recovery |
+| Real-Debrid Torrent Client | Files & Downloads |
+| SnapOtter | Media & Streaming |
+| Thelounge | Communication & Community |
+| Trilium | Documents & Notes |
+| Wireguard | Remote Access & VPN |
+| WireGuard Easy | Remote Access & VPN |
@@ -1,5 +1,5 @@
 {
-  "_comment": "Applications tested for real. \"status\": \"laboratory-validated\" when tested in the ProxMenux lab; \"community_tested\" with the tester's GitHub user and the date when tested by the community. Applied on top of the generated templates and the overlays, so a catalog regeneration keeps it.",
+  "_comment": "Applications tested for real. \"status\": \"laboratory-validated\" when tested in the ProxMenux lab; \"community_tested\" with the tester's GitHub user, the date and a link to the report when tested by the community. Read when the catalog loads and applied on top of the generated templates and the overlays, so a new entry shows without regenerating and a regeneration keeps it. oci/VALIDATION.md is generated from this file.",
  "applications": {
    "2fauth": {
      "status": "laboratory-validated"
@@ -49,7 +49,8 @@
    "glances": {
      "community_tested": {
        "by": "Vaso73",
-        "date": "2026-09-27"
+        "date": "2026-09-27",
+        "report": "https://github.com/MacRimi/ProxMenux/discussions/392#discussioncomment-18623868"
      }
    },
    "grafana": {
@@ -94,7 +95,8 @@
    "pocketbase": {
      "community_tested": {
        "by": "Vaso73",
-        "date": "2026-09-27"
+        "date": "2026-09-27",
+        "report": "https://github.com/MacRimi/ProxMenux/discussions/392#discussioncomment-18623868"
      }
    },
    "qbittorrent": {
@@ -30,7 +30,9 @@
      "properties": {
        "by": {"type": "string", "pattern": "^[A-Za-z0-9][A-Za-z0-9-]{0,38}$"},
        "date": {"type": "string", "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}$"},
-        "report": {"type": "string", "pattern": "^https://github\\.com/MacRimi/ProxMenux/(issues/[0-9]+|discussions/[0-9]+(#discussioncomment-[0-9]+)?)$"}
+        "report": {"type": "string", "pattern": "^https://github\\.com/MacRimi/ProxMenux/(issues/[0-9]+|discussions/[0-9]+(#discussioncomment-[0-9]+)?)$"},
+        "digest": {"type": "string", "pattern": "^[a-f0-9]{8,64}$"},
+        "scenario": {"type": "string", "maxLength": 300}
      }
    },
    "catalog_ui": {
@@ -341,6 +341,7 @@ class Catalog:
            return self.sync_index()
        payload = json.loads(self.index_path.read_text(encoding="utf-8"))
        self._enrich_index_from_templates(payload)
+        self._apply_verification_to_index(payload)
        return payload

    def categories(self) -> dict[str, Any]:
@@ -816,13 +817,29 @@ class Catalog:
        if isinstance(entry.get("community_tested"), dict):
            template["community_tested"] = dict(entry["community_tested"])

+    def _apply_verification_to_index(self, payload: dict[str, Any]) -> None:
+        """verification.json is read at load time, so a new entry shows without
+        regenerating the templates."""
+        path = self.catalog_dir / "verification.json"
+        try:
+            entries = json.loads(path.read_text(encoding="utf-8")).get("applications", {})
+        except (OSError, json.JSONDecodeError, AttributeError):
+            return
+        for item in payload.get("applications", []):
+            entry = entries.get(item.get("id"))
+            if not isinstance(entry, dict):
+                entry = {}
+            if entry.get("status") == "laboratory-validated" and item.get("automatic_install_candidate"):
+                item["template_status"] = "laboratory-validated"
+            self._index_community_tested(item, entry)
+
    @staticmethod
    def _index_community_tested(item: dict[str, Any], template: dict[str, Any]) -> None:
        """Who tested the application for real outside the ProxMenux lab, and
-        when, as recorded in its overlay."""
+        when, as recorded in verification.json."""
        tested = template.get("community_tested")
        if isinstance(tested, dict) and tested.get("by"):
-            item["community_tested"] = {"by": str(tested["by"]), "date": str(tested.get("date") or "")}
+            item["community_tested"] = {key: str(tested[key]) for key in ("by", "date", "report") if tested.get(key)}
        else:
            item.pop("community_tested", None)

@@ -0,0 +1,54 @@
+"""verification.json applies when the catalog loads, without regenerating templates."""
+
+import json
+from pathlib import Path
+import sys
+import tempfile
+import unittest
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT / "src"))
+
+from proxmenux_oci import cli
+from proxmenux_oci.catalog import Catalog
+
+
+class VerificationAtLoadTests(unittest.TestCase):
+    def catalog(self, applications, verification):
+        tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(tmp.cleanup)
+        root = Path(tmp.name)
+        (root / "catalog").mkdir()
+        files = {"index.json": {"applications": applications},
+                 "categories.json": {"applications": {}, "labels": {}},
+                 "verification.json": {"applications": verification}}
+        for name, data in files.items():
+            (root / "catalog" / name).write_text(json.dumps(data), encoding="utf-8")
+        return {item["id"]: item for item in Catalog(root).load_index()["applications"]}
+
+    def test_new_entries_show_as_verified(self):
+        items = self.catalog(
+            [{"id": "glances", "automatic_install_candidate": True},
+             {"id": "2fauth", "automatic_install_candidate": True},
+             {"id": "legacy", "automatic_install_candidate": False},
+             {"id": "plex", "automatic_install_candidate": True}],
+            {"glances": {"community_tested": {"by": "Vaso73", "date": "2026-09-27",
+                                              "report": "https://github.com/MacRimi/ProxMenux/issues/400"}},
+             "2fauth": {"status": "laboratory-validated"},
+             "legacy": {"status": "laboratory-validated"}})
+        self.assertTrue(cli.is_tested(items["glances"]))
+        self.assertEqual(items["glances"]["community_tested"]["report"],
+                         "https://github.com/MacRimi/ProxMenux/issues/400")
+        self.assertTrue(cli.is_tested(items["2fauth"]))
+        self.assertFalse(cli.is_tested(items["legacy"]))
+        self.assertFalse(cli.is_tested(items["plex"]))
+
+    def test_removed_community_entry_no_longer_shows(self):
+        items = self.catalog([{"id": "glances", "automatic_install_candidate": True,
+                               "community_tested": {"by": "Vaso73", "date": "2026-09-27"}}], {})
+        self.assertNotIn("community_tested", items["glances"])
+        self.assertFalse(cli.is_tested(items["glances"]))
+
+
+if __name__ == "__main__":
+    unittest.main()