feat(ems): notify a URL after every completed optimization (#1383)

* feat(devices): add load-dependent DC-to-AC efficiency curve for GENETIC inverters Real inverters are much less efficient at low load than at rated power (e.g. 0.80 at 2 % vs. 0.93 at 7 % load). A constant dc_to_ac_efficiency overrates night-time battery discharge into a small base load and, with it, arbitrage and grid charging. Add two optional inverter settings for the GENETIC optimizer: - dc_to_ac_efficiency_curve: list of (load_fraction, efficiency) points, load fraction = AC power / max_power_w. Points must be strictly increasing in [0, 1] with efficiencies in (0, 1]; at least two points. Linear interpolation, clamped at both ends. When set, the curve replaces dc_to_ac_efficiency. - dc_to_ac_efficiency_reference_load_fraction (default 0.06): load at which the curve values stored energy without a specific conversion. Battery discharges in Inverter.process_energy() use the efficiency at the AC energy of each conversion. The end-of-horizon value, the terminal and tail value curves and the AC charge break-even penalty use Inverter.reference_dc_to_ac_efficiency. Without a curve, dc_to_ac_efficiency_at() and the reference efficiency return dc_to_ac_efficiency itself, so results are bit-identical to the previous model. GENETIC0 is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): make the fitness cache size configurable The per-run fitness cache of the GENETIC optimizer is unbounded. It grows with individuals x generations (~110k keys at 300 x 400, about 1.1 kB each, 1.6 kB with an EV) and adds 120-170 MB to the peak memory of a run, while only 6-15 % of the lookups hit. On 512 MB boards this is the largest part of the optimizer's memory. Add optimization.genetic.fitness_cache_max_entries: - None (default): unbounded, the previous behaviour. - 0: no cache; every genome is evaluated. - N > 0: at most N keys. The cache is then an OrderedDict and evicts the oldest key first (FIFO, O(1)); LRU would cost bookkeeping on every hit for a cache that rarely hits. A cache hit returns exactly what the deterministic evaluation would return, so the plan for a fixed seed is identical with any setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf: import heavy optional dependencies on first use A GENETIC server with external (import) providers and EOSdash disabled loads several large packages at start that it never uses, which costs a noticeable part of the memory on 512 MB boards: - matplotlib via the GENETIC/GENETIC0 PDF report modules (server/eos.py) - statsmodels via the Holt-Winters fallback in prediction/priceabc.py - pvlib via the weather base class and the pvlib-based providers - bs4/lxml via the ClearOutside provider - scipy via the self-consumption interpolator singletons, which were loaded at import time - the GENETIC0 optimizer, imported by core/ems.py - tzfpy, whose polygon data the computed general.timezone loads Import these where they are used. The interpolator tables are loaded on first use (thread-safe). Nothing changes in what is computed. general.timezone is computed from latitude/longitude and cannot be set. Add general.timezone_override (IANA name, validated with pendulum): when set, general.timezone returns it and the lookup is skipped. When unset (default) the lookup is unchanged. Tests that patched the moved names now patch them at their source. A new test starts the server module in a subprocess and checks that none of these modules is loaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(prediction): evaluate self-consumption tables without SciPy The GENETIC inverter (and the GENETIC0 one) evaluates a pickled scipy.interpolate.RegularGridInterpolator for the self-consumption probabilities. SciPy is used nowhere else in EOS, yet loading it costs about 35 MB RSS (21 MB anonymous) on Linux arm64, which every optimization run pays. Add utils/gridinterpolator.py: - LinearGridInterpolator: 2-D linear interpolation with the same steps as SciPy's linear 2-D path (interval search, normalised distances, corner weights summed in the same order, fill value outside the grid, NaN for NaN input). - load_grid_interpolator(): reads the shipped pickles with a restricted unpickler that maps the SciPy class to a stand-in and accepts only the NumPy globals needed for the arrays. SciPy is not imported, and the data files no longer depend on SciPy's private module paths. Results are bit-identical to SciPy on Linux (aarch64, checked on 3.6M points per table including the query pattern of the inverter). The macOS arm64 SciPy wheel fuses multiply-add, so there SciPy itself differs by <= 1.1e-16; GENETIC plans for fixed seeds are unchanged on both platforms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(prediction): make the SciPy-free table evaluation opt-in Add optimization.self_consumption_interpolator (scipy | numpy, default scipy). The default keeps evaluating the self-consumption tables with SciPy's RegularGridInterpolator, built from the same grid, values and fill value the pickle holds, so results are unchanged. With numpy the tables are evaluated by LinearGridInterpolator and SciPy is never imported. The backend is read on every evaluation, so a config change applies to the next run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): evaluate candidates in parallel worker processes A GENETIC run evaluated every candidate in the server process, so a run used one CPU core however many the machine had. On small ARM boards (4x Cortex-A55, thermally limited) a 300x200 run took 12-15 minutes. Candidates are now evaluated in up to two worker processes, forked once per run after the run is fully prepared. Each worker owns a copy of the prepared optimizer; only genomes and results cross the process boundary. The fitness cache stays in the server process: hits are answered there, and identical genomes in a batch are evaluated once, as a serial run would hit the cache for the repeats. Results are applied only after the whole batch returned. If a batch fails or times out, the run continues serially. For a fixed seed the result is identical to a serial run. optimization.genetic.workers sets the number of processes. Unset (default) is automatic: one CPU core always stays free and at most two are used (1-2 cores -> 1, 3 or more -> 2), capped by a cgroup v2 CPU limit of the process or any parent (container limit, systemd CPUQuota). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * feat(ems): notify a URL after every completed optimization A client that consumes EOS plans (here DVhub) had to poll /v1/energy-management/optimization/solution to notice a new plan. That delays the plan by up to the poll interval and makes EOS serialize the complete solution on every poll, also while it is computing. ems.notify_url (default None = off) receives an HTTP POST with a small JSON event ({"event": "optimization_completed", "algorithm", "completed_at", "duration_s"}) when a run has published its solution. The POST runs in a daemon thread with a 5 s timeout, so a slow or unreachable receiver never delays energy management; a failure is logged once per failure streak. Only http(s) URLs are accepted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(genetic): freeze the optimization configuration for the duration of a run The genetic optimization reads horizon_hours, tail_horizon_hours, interval_sec and prediction.hours from the live configuration on every access (control_slots, tail_slots, ... are properties). A client that updates the configuration while a run is in progress - DVhub shrinks or grows the control horizon as the known price horizon moves during the night - changed these values mid-run. Genomes and device arrays sized at the start then no longer matched the simulation loop and the final evaluation failed: IndexError: index 84 is out of bounds for axis 0 with size 84 Long runs on slow hardware hit this almost every night. optimize_ems() now takes a deep copy of the genetic configuration and the prediction hours at the start of a run, all reads in genetic.py go through it, and it is released when the run ends (also on errors). New values apply to the next run. Tests: tests/test_genetic_config_frozen_per_run.py reproduces the error without the fix (horizon 20 -> 24 mid-run: index 20 out of bounds) and passes with it; genetic/optimization tests 439 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): pin evaluation workers, keep the server responsive Workers are pinned to the last `workers` CPU cores (new setting optimization.genetic.pin_workers, default on); the server process stays unpinned. The run never spreads beyond `workers` cores without a CPU quota, and the server and the rest of the system keep the other cores. No pinning without a spare core. With a CPU limit (container cpu.max, systemd CPUQuota) the automatic worker count now keeps a quarter core of it for the server process (limit 2 -> 1 worker, 2.5 -> 2): two workers used a 2-core limit fully. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(ems): tolerate settings without notify_url Test doubles and older configuration objects have no ems.notify_url. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(devices): evaluate the efficiency curve at the delivered AC load When the battery limits a discharge (low state of charge or its power limit), the conversion ran with the efficiency of the requested AC energy. With a 1000 Wh battery at 10 % SoC, a 1000 W inverter and the curve [(0, .8), (.5, .95), (1, .95)], a request of 500 Wh AC returned 95 Wh AC from the available 100 Wh DC, although the curve gives 82.5 % at that load. The delivered AC energy now solves a = dc * efficiency(a / rated) exactly on the piecewise-linear curve (solve_ac_for_dc_energy), bounded by the request. Unlimited discharges and inverters without a curve are unchanged. Regression tests use a real Battery: low SoC and a power-limited discharge, each with 60- and 15-minute slots, battery discharge losses, the load path through process_energy, and constant efficiency. * docs: regenerate OpenAPI for the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * test(genetic): cover serial fallback on platforms without fork --------- Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Andreas <drbacke@gmx.de>
2026-10-13 09:31:56 +00:00 · 2026-10-11 10:48:04 +02:00
parent 1f8742a54b
commit 70b72bfbce
7 changed files with 197 additions and 3 deletions
@@ -33,6 +33,8 @@ compatibility changes and a short test procedure. These changes are not yet a ta
 - Forecast continuation and AUTO/FIXED valuation of energy left in the battery, separate
  from battery wear costs. Continuation helps choose the plan; it does not add commands
  beyond the configured control horizon.
+- Optional `ems.notify_url`: EOS sends an HTTP POST with a small JSON event after every
+  completed optimization, so a client can fetch the new plan right away instead of polling.
 - Configuration-driven `POST /v1/optimize` and a PDF report of the stored GENETIC result.
 - Measurement sample, energy, household-balance and battery-capacity APIs with data-quality
  and coverage information. A capacity estimate does not overwrite configured capacity.
@@ -10,6 +10,7 @@
 | interval | `EOS_EMS__INTERVAL` | `float` | `rw` | `300.0` | Intervall between EOS energy management runs [seconds]. |
 | mode | `EOS_EMS__MODE` | `<enum 'EnergyManagementMode'>` | `rw` | `required` | Energy management mode [DISABLED | PREDICTION | OPTIMIZATION]. Defaults to DISABLED. |
 | modes | | `list[str]` | `ro` | `N/A` | Available energy management modes. |
+| notify_url | `EOS_EMS__NOTIFY_URL` | `Optional[str]` | `rw` | `None` | URL that receives an HTTP POST with a small JSON event after every completed optimization, so a client can fetch the new solution right away instead of polling for it. None = off. |
 | startup_delay | `EOS_EMS__STARTUP_DELAY` | `float` | `rw` | `5` | Startup delay in seconds for EOS energy management runs. |
 :::
 <!-- pyml enable line-length -->
@@ -24,7 +25,8 @@
       "ems": {
           "startup_delay": 5.0,
           "interval": 300.0,
-           "mode": "OPTIMIZATION"
+           "mode": "OPTIMIZATION",
+           "notify_url": null
       }
   }
 ```
@@ -41,6 +43,7 @@
           "startup_delay": 5.0,
           "interval": 300.0,
           "mode": "OPTIMIZATION",
+           "notify_url": null,
           "modes": [
               "DISABLED",
               "PREDICTION",
@@ -180,7 +180,8 @@
       "ems": {
           "startup_delay": 5.0,
           "interval": 300.0,
-           "mode": "OPTIMIZATION"
+           "mode": "OPTIMIZATION",
+           "notify_url": null
       },
       "feedintariff": {
           "direct_marketing_enabled": false,
@@ -5947,6 +5947,22 @@
            "examples": [
              "OPTIMIZATION"
            ]
+          },
+          "notify_url": {
+            "anyOf": [
+              {
+                "type": "string"
+              },
+              {
+                "type": "null"
+              }
+            ],
+            "title": "Notify Url",
+            "description": "URL that receives an HTTP POST with a small JSON event after every completed optimization, so a client can fetch the new solution right away instead of polling for it. None = off.",
+            "examples": [
+              null,
+              "http://127.0.0.1:8080/api/eos/solution-ready"
+            ]
          }
        },
        "type": "object",
@@ -5979,6 +5995,22 @@
              "OPTIMIZATION"
            ]
          },
+          "notify_url": {
+            "anyOf": [
+              {
+                "type": "string"
+              },
+              {
+                "type": "null"
+              }
+            ],
+            "title": "Notify Url",
+            "description": "URL that receives an HTTP POST with a small JSON event after every completed optimization, so a client can fetch the new solution right away instead of polling for it. None = off.",
+            "examples": [
+              null,
+              "http://127.0.0.1:8080/api/eos/solution-ready"
+            ]
+          },
          "modes": {
            "items": {
              "type": "string"
@@ -1,4 +1,7 @@
+import json
+import threading
 import traceback
+import urllib.request
 from asyncio import Lock, get_running_loop
 from concurrent.futures import ThreadPoolExecutor
 from enum import StrEnum
@@ -56,6 +59,48 @@ async def ems_manage_energy() -> None:
    await EnergyManagement().run()


+NOTIFY_TIMEOUT_SECONDS = 5.0
+_notify_failing = False
+
+
+def _post_notification(url: str, event: dict) -> None:
+    """POST one event; log a failure once per failure streak, not every run."""
+    global _notify_failing
+    # The scheme is restricted to http(s) by the notify_url setting.
+    request = urllib.request.Request(  # noqa: S310
+        url,
+        data=json.dumps(event).encode("utf-8"),
+        headers={"Content-Type": "application/json"},
+        method="POST",
+    )
+    try:
+        with urllib.request.urlopen(request, timeout=NOTIFY_TIMEOUT_SECONDS) as response:  # noqa: S310
+            response.read()
+    except Exception as exc:
+        if not _notify_failing:
+            logger.warning("Optimization notification to {} failed: {}", url, exc)
+        _notify_failing = True
+        return
+    if _notify_failing:
+        logger.info("Optimization notification to {} works again.", url)
+    _notify_failing = False
+
+
+def notify_optimization_completed(url: Optional[str], event: dict) -> Optional[threading.Thread]:
+    """Tell ``ems.notify_url`` that a new solution is available.
+
+    Runs in a background thread so a slow or unreachable receiver never delays
+    the energy management run.
+    """
+    if not url:
+        return None
+    thread = threading.Thread(
+        target=_post_notification, args=(url, event), name="eos-notify", daemon=True
+    )
+    thread.start()
+    return thread
+
+
 class EnergyManagement(
    SingletonMixin, ConfigMixin, PredictionMixin, AdapterMixin, PydanticBaseModel
 ):
@@ -492,6 +537,15 @@ class EnergyManagement(
            logger.debug("{}: Energy management plan:\n{}", algorithm, EnergyManagement._plan)

            logger.info("{}: Energy management run done (optimization updated)", algorithm)
+            notify_optimization_completed(
+                getattr(self.config.ems, "notify_url", None),
+                {
+                    "event": "optimization_completed",
+                    "algorithm": str(algorithm),
+                    "completed_at": to_datetime().to_iso8601_string(),
+                    "duration_s": round(optimization_duration.total_seconds(), 1),
+                },
+            )

            # --- Dispatch control by adapters ---
            EnergyManagement._stage = EnergyManagementStage.CONTROL_DISPATCH
@@ -4,8 +4,9 @@ Kept in an extra module to avoid cyclic dependencies on package import.
 """

 from enum import StrEnum
+from typing import Optional

-from pydantic import Field, computed_field
+from pydantic import Field, computed_field, field_validator

 from akkudoktoreos.config.configabc import SettingsBaseModel, is_home_assistant_addon

@@ -60,6 +61,28 @@ class EnergyManagementCommonSettings(SettingsBaseModel):
        },
    )

+    notify_url: Optional[str] = Field(
+        default=None,
+        json_schema_extra={
+            "description": (
+                "URL that receives an HTTP POST with a small JSON event after every "
+                "completed optimization, so a client can fetch the new solution right "
+                "away instead of polling for it. None = off."
+            ),
+            "examples": [None, "http://127.0.0.1:8080/api/eos/solution-ready"],
+        },
+    )
+
+    @field_validator("notify_url")
+    @classmethod
+    def _validate_notify_url(cls, value: Optional[str]) -> Optional[str]:
+        if value is None or value.strip() == "":
+            return None
+        value = value.strip()
+        if not value.startswith(("http://", "https://")):
+            raise ValueError("notify_url must be an http:// or https:// URL")
+        return value
+
    @computed_field  # type: ignore[prop-decorator]
    @property
    def modes(self) -> list[str]:
@@ -0,0 +1,79 @@
+"""EOS notifies ``ems.notify_url`` after a completed optimization."""
+
+import json
+import threading
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+import pytest
+from pydantic import ValidationError
+
+from akkudoktoreos.core import ems as ems_module
+from akkudoktoreos.core.emsettings import EnergyManagementCommonSettings
+
+
+def _receiver():
+    received: list[dict] = []
+
+    class Handler(BaseHTTPRequestHandler):
+        def do_POST(self):  # noqa: N802
+            length = int(self.headers.get("Content-Length", 0))
+            received.append(
+                {
+                    "path": self.path,
+                    "type": self.headers.get("Content-Type"),
+                    "body": json.loads(self.rfile.read(length)),
+                }
+            )
+            self.send_response(204)
+            self.end_headers()
+
+        def log_message(self, *args):
+            pass
+
+    server = HTTPServer(("127.0.0.1", 0), Handler)
+    threading.Thread(target=server.serve_forever, daemon=True).start()
+    return server, received
+
+
+def test_completed_optimization_is_posted():
+    server, received = _receiver()
+    try:
+        url = f"http://127.0.0.1:{server.server_port}/api/eos/solution-ready"
+        event = {"event": "optimization_completed", "duration_s": 12.5}
+        thread = ems_module.notify_optimization_completed(url, event)
+        assert thread is not None
+        thread.join(timeout=10)
+        assert received == [
+            {"path": "/api/eos/solution-ready", "type": "application/json", "body": event}
+        ]
+    finally:
+        server.shutdown()
+
+
+def test_no_url_no_notification():
+    assert ems_module.notify_optimization_completed(None, {"event": "x"}) is None
+    assert ems_module.notify_optimization_completed("", {"event": "x"}) is None
+
+
+def test_unreachable_receiver_is_logged_once(monkeypatch):
+    warnings: list[str] = []
+    monkeypatch.setattr(ems_module.logger, "warning", lambda *a, **k: warnings.append(a[0]))
+    monkeypatch.setattr(ems_module, "_notify_failing", False)
+    server, _ = _receiver()
+    port = server.server_port
+    server.shutdown()
+    server.server_close()
+    for _ in range(3):
+        ems_module._post_notification(f"http://127.0.0.1:{port}/x", {"event": "x"})
+    assert len(warnings) == 1
+
+
+def test_notify_url_setting():
+    assert EnergyManagementCommonSettings().notify_url is None
+    assert EnergyManagementCommonSettings(notify_url="  ").notify_url is None
+    assert (
+        EnergyManagementCommonSettings(notify_url=" http://127.0.0.1:8080/a ").notify_url
+        == "http://127.0.0.1:8080/a"
+    )
+    with pytest.raises(ValidationError):
+        EnergyManagementCommonSettings(notify_url="file:///etc/passwd")