mirror of
https://github.com/Akkudoktor-EOS/EOS.git
synced 2026-10-29 09:11:57 +00:00
8e34b1779b974bb627918bc43609df590499125d
1123
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
8e34b1779b |
fix(genetic): stop parallel evaluation workers reliably at the end of a run (#1473)
Follow-up to #1382. Under the EOS server the end of a GENETIC run with parallel evaluation (workers >= 2, which is the automatic setting on machines with 3 or more cores) could hang forever in Pool.terminate()/join(), so /v1/optimize never answered. Cause: uvicorn installs a Python SIGTERM/SIGINT handler in the server process that only asks the server to shut down. The forked evaluation workers inherit it and survive the SIGTERM of Pool.terminate(). terminate() keeps the task queue's read lock (Pool._help_stuff_finish), so a surviving worker that has not yet taken its stop sentinel waits for that lock forever, and join() with it. This always happened after a batch timeout (busy workers survive) and by chance after a normal run when a worker was not scheduled in time (several EOS instances or other load; also seen with a single instance on a 2-vCPU VM). A worker that got SIGTERM before its initializer ran hit the same path. - Workers ignore SIGTERM and SIGINT. Sent to the whole process group (Ctrl-C, systemd stopping the service) they are meant for the server, which finishes or aborts the run and stops the workers itself. Restoring the default action instead was measured to be worse: a group SIGTERM killed the workers mid-batch, the batch ran into its 120 s timeout and the server did not exit within 120 s (12 s before). - _EvaluationPool stops its workers with SIGKILL at the point where Pool.terminate() would send SIGTERM (it already holds the read lock). This does not depend on handlers, initialization state or whether a worker is busy. _stop_evaluation_pool() is unchanged. Measured (8 EOS servers x 2 workers on 8 vCPU, real configuration, seed 42): 11 hangs in 160 runs before, 0 in 440 after; pool only under load: 107/300 before, 0/600 after. Plans and costs are identical, the stop takes no longer. Ctrl-C, SIGTERM to the server and SIGTERM to the group behave as before (run completes with HTTP 200, exit after ~12 s, no orphaned workers). Tests (tests/test_genetic_parallel_evaluation.py, under uvicorn's real signal handling): inherited handler, batch timeout, late worker start, late idle worker (SIGSTOP, Linux) and SIGTERM to the group during a batch. They fail without the fix. Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
 NormannandGitHub
|
aa97e677b6 | build(deps): bump dnspython to 2.9.0 (#1472) | ||
  |
bb9c47ea90 |
build(deps): bump dnspython from 2.8.0 to 2.9.0 (#1471)
Bumps [dnspython](https://github.com/rthalley/dnspython) from 2.8.0 to 2.9.0. - [Release notes](https://github.com/rthalley/dnspython/releases) - [Changelog](https://github.com/rthalley/dnspython/blob/main/doc/whatsnew.rst) - [Commits](https://github.com/rthalley/dnspython/compare/v2.8.0...v2.9.0) --- updated-dependencies: - dependency-name: dnspython dependency-version: 2.9.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Normann <github@koldrack.com> |
||
|
NormannandGitHub
|
5d7198d090 |
test: consolidate suites and remove redundant cases (#1459)
Regression tests were scattered across feature- and bug-specific files, and some suites repeated the same checks. Group tests by responsibility, isolate feature fixtures in test classes, and remove duplicate or obsolete cases. The review covered all 127 collected test modules on main. This reduces them and removes or combines definitions:. No application code changes. |
||
|
NormannandGitHub
|
2863373b17 |
build(deps): combine FastAPI and three patch updates (#1467)
python-fasthtml 0.14.13 → 0.14.14 pydantic-settings 2.16.0 → 2.16.1 FastAPI 0.143.3 → 0.162.0 Commitizen (dev) 4.19.2 → 4.19.3 |
||
 Wu ShuwenandGitHub
|
727f20fdf0 |
fix(homeassistant): resolve legacy device measurement keys
Resolve legacy underscore spellings for Home Assistant device measurements only when they uniquely match a registered EOS key; preserve canonical, custom, and ambiguous keys. Correct the schema examples to show canonical hyphenated keys. Add a synthetic regression test that exercises data acquisition without contacting Home Assistant. |
||
 NormannandBobby Noelte
|
63e7e7e729 |
fix: honor DST boundaries in appliance scheduling
Compare elapsed offsets when checking and merging appliance windows so both occurrences of the autumn hour obey the configured boundaries. Preserve calendar durations when finding the earliest valid start. Add optimizer and calendar-day regressions, and consolidate the existing time-window regression files into test_configabc classes. |
||
|
NormannandBobby Noelte
|
7ca66c7dee | fix: compare time window boundaries by elapsed time across DST | ||
|
NormannandBobby Noelte
|
d22628b312 | fix: honor effective time window boundaries in cycle scheduling | ||
|
Bobby Noelte
|
c02d250639 |
feat: Add month and end_time to time window configuration
Extend TimeWindow with two new optional fields: - month (1–12): restricts the window to a start date in the given calendar month. Useful for seasonal schedules. - end_time: explicit exclusive end-of-window clock time. The effective end is the later of end_time and start_time + duration. Primary motivation is DST handling — a wall-clock end_time stays correct across DST transitions, whereas a fixed duration does not. Rework window semantics so date, weekday, and month restrictions apply to the window's start date, not to every timestamp covered by the window. This makes overnight windows (e.g. Friday 22:00 → Saturday 06:00) behave intuitively under calendar constraints; contains() now checks both today's and yesterday's candidate start. duration becomes optional (default 0). All start/end/duration computations route through _window_start_end and a new _matches_start_date helper, so contains, earliest_start_time, latest_start_time, and available_duration stay consistent. Dashboard: add end_time and month inputs to the time-window editor, render the collapsed window summary as icons + pills. Docs, OpenAPI spec, and generated config docs are regenerated for the new fields. Tests cover month filtering, end_time boundary cases (incl. midnight and overnight), and start-date semantics for overnight windows. Bump version to support automatic updates in Home Assistant. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
Bobby NoelteandGitHub
|
8a8cd43a19 | chore: development version update (incl. home assistant config) (#1456) | ||
 sarankumarandGitHub
|
5ceb21819b | fix(genetic0): keep parameter arrays aligned with the start hour on DST days (#1455) | ||
 YTM-21andGitHub
|
8f4fa6fe94 |
fix(config): report computed fields as read-only on update (#1454)
PUT /v1/config/{path} answered "does not exist in model" when the
target was a computed field such as eos_device_instruction_entity_ids.
Raise an explicit read-only error instead, so API clients can tell
read-only fields from unknown ones.
Closes #913
|
||
|
|
58b968562e |
fix(adapter): stamp Home Assistant device states with run time, not slot start (#1452)
* fix(adapter): stamp Home Assistant device states with run time, not slot start The Home Assistant adapter stored every reading with ems_start_datetime, which set_start_datetime() rounds down to the optimization slot (e.g. 14:00 for a run at 14:11). The GENETIC configuration request checks SoC freshness against the unrounded observation_datetime with measurement_max_age_seconds (default 300 s). Any run starting more than five minutes into a slot therefore discarded the freshly read SoC and optimized with 0 % initial SoC. Stamp device state measurements (device_measurement_entity_ids, e.g. SoC) with observation_datetime instead. Not to_datetime(): a reading taken a few seconds after the observation time would be rejected as a future sample. Meter readings (load, grid import/export, PV) keep the slot start, because their energy is aggregated per slot. With device states at their real time, a record holding only some keys can now be the newest one (e.g. the Home Assistant battery SoC while the EV SoC or a cycle count comes through the measurement API). Readers therefore skip records without the value they need: - GENETIC SoC and completed appliance cycles: such a record no longer hides a fresh value (the cycle count case aborted the optimization). This also covers a meter reading at the slot start that is newer than a SoC supplied through the measurement API. Values stored as pandas NA count as missing instead of raising TypeError; an explicit NaN stays an invalid sample. - GENETIC0 SoC: looks up the reading nearest to observation_datetime instead of the slot start, skipping records without SoC and converting only the selected sample. Before, a run late in the slot (14:40 in the 14:00 slot) would pick the previous run's reading (13:40). This also fixes SoC values PUT via the REST API with their real time. LoadAkkudoktorAdjusted anchored its hourly comparison window on the newest record of any key. An off-hour device state would shift every bucket (e.g. to :11) against hour-aligned meter readings; the window now starts and ends on full hours, and less than one full hour of data yields no adjustment instead of an error. (15-minute slot starts and SoC values PUT with their real time had the same effect before.) Regenerated openapi.json and docs/_generated/openapi.md (version only). Fixes #1447 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * test(genetic0): use pv_battery instead of deprecated pv_akku mypy flags attribute access on the pv_akku computed field (attr-defined), which failed the pre-commit check of the PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
NormannandGitHub
|
5249b389f5 | build(deps): combine pypdf and certifi updates (#1451) | ||
|
NormannandGitHub
|
5a0763deec |
build(deps): combine four Dependabot updates and refresh uv.lock (#1444)
Consolidates four currently open Dependabot updates. Only pyproject.toml and uv.lock change. Included sources Source Dependency Change #1436 platformdirs 4.13.3 → 4.13.5 #1437 Pillow (via matplotlib and bokeh) 12.3.0 → 13.0.0 #1442 filelock (development tools) 3.32.6 → 4.2.1 #1443 virtualenv (via pre-commit) 21.7.13 → 21.14.13 |
||
|
sarankumarandGitHub
|
cdbaae45ba |
fix(elecprice): handle mixed UTC offsets in Tibber prices around DST (#1448)
Once Tibber publishes prices across a DST change, the series holds timestamps with two UTC offsets and pd.to_datetime() raises "Mixed timezones detected", so every optimization run is canceled. Parse via UTC and convert back to the configured timezone. Fixes #1446 |
||
|
NormannandGitHub
|
baafe570ff |
build(deps): combine seven Dependabot updates and synchronize uv.lock (#1433)
* build(deps): combine five Dependabot updates pydantic-settings 2.15.0 → 2.16.0 FastAPI 0.143.0 → 0.143.3 GitPython 3.2.0 → 3.2.1 platformdirs 4.13.0 → 4.13.3 pypdf 6.20.0 → 6.21.1 * build(deps): add Pendulum and pre-commit updates Pendulum 3.2.0 → 3.3.0 pre-commit 4.6.2 → 4.7.0 |
||
|
Bobby NoelteandGitHub
|
024706b467 |
fix: use of inverter config dict in genetic0 (#1432)
Adapt GENETIC0 to inverters configuration in dict - used to be a list. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
 AndreasandGitHub
|
46a89a6f39 | fix(devices): let EVs finish charging within the final slot (#1423) | ||
 
|
08178b9f68 |
perf(docker): preserve dependency caches across release versions (#1425)
Preserve dependency caches across release versions and preserve build argument comments. Changing BUILD_VERSION invalidated OS-package and Python-dependency installation layers in both Docker stages, even when the dependencies are unchanged. Docker makes an in-scope build argument available to subsequent RUN instructions, so declaring it early makes a release-version change affect those cache keys. With the change, the second build reuses the builder OS-package, locked Python-dependency and runtime OS-package layers. Without the change, all three installation steps would run again. Co-authored-by: David Siegel <dsiegel@MacBook-Air-von-David.local> |
||
|
Bobby NoelteandGitHub
|
5ad109cd22 |
chore: development version update (incl. home assistant config) (#1424)
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
NormannandGitHub
|
fb2184c23a |
build(deps): combine eleven Dependabot updates and synchronize uv.lock (#1415)
Consolidates all eleven Dependabot updates in this batch into one reviewable change with a synchronized uv.lock:
build(deps-dev): bump mypy from 2.3.1 to 2.4.0 #1410 — mypy 2.3.1 → 2.4.0
build(deps-dev): bump gitpython from 3.1.62 to 3.2.0 #1411 — GitPython 3.1.62 → 3.2.0
build(deps-dev): bump commitizen from 4.19.1 to 4.19.2 #1412 — Commitizen 4.19.1 → 4.19.2
build(deps): bump fastapi from 0.142.2 to 0.143.0 #1413 — FastAPI 0.142.2 → 0.143.0
build(deps-dev): bump pypdf from 6.19.0 to 6.20.0 #1414 — pypdf 6.19.0 → 6.20.0
build(deps): bump platformdirs from 4.12.2 to 4.13.0 #1416 — platformdirs 4.12.2 → 4.13.0
build(deps): bump pydantic from 2.13.5 to 2.14.1 #1417 — Pydantic 2.13.5 → 2.14.1
build(deps): bump tzfpy from 2.1.0 to 2.1.1 #1418 — tzfpy 2.1.0 → 2.1.1
build(deps): bump cachebox from 6.2.7 to 6.2.8 #1419 — cachebox 6.2.7 → 6.2.8
build(deps-dev): bump types-requests from 2.33.0.20260906 to 2.33.0.20261006 #1420 — types-requests 2.33.0.20260906 → 2.33.0.20261006
build(deps): bump rich-toolkit from 0.20.5 to 0.20.6 #1421 — rich-toolkit 0.20.5 → 0.20.6
|
||
|
AndreasandGitHub
|
cab088f09b |
fix(planning): accept full SoC targets and clarify unlimited EV SoC age (#1407)
* fix(devices): allow charging targets equal to the maximum SoC * test(planning): protect EV SoC without age limits |
||
|
|
1f2faf3f7e |
Potential fix for code scanning alert no. 6: URL redirection from remote source (#1406)
* Potential fix for code scanning alert no. 6: URL redirection from remote source Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * fix: reject scheme-bearing EOSdash redirect paths --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> |
||
|
|
812aa3dda9 |
fix: prevent exception diagnostics from leaking in API problem responses (#1405)
* Potential fix for code scanning alert no. 11: Information exposure through an exception traceback should not be in client side code Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * fix: keep unexpected API error details private and test problem responses --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> |
||
|
AndreasandGitHub
|
49ea062dc8 | fix(genetic): allow optimization without fresh SoC (#1403) | ||
 
|
fab3e17446 |
feat(server): report the process start time in /v1/health (#1386)
* feat(devices): add load-dependent DC-to-AC efficiency curve for GENETIC inverters Real inverters are much less efficient at low load than at rated power (e.g. 0.80 at 2 % vs. 0.93 at 7 % load). A constant dc_to_ac_efficiency overrates night-time battery discharge into a small base load and, with it, arbitrage and grid charging. Add two optional inverter settings for the GENETIC optimizer: - dc_to_ac_efficiency_curve: list of (load_fraction, efficiency) points, load fraction = AC power / max_power_w. Points must be strictly increasing in [0, 1] with efficiencies in (0, 1]; at least two points. Linear interpolation, clamped at both ends. When set, the curve replaces dc_to_ac_efficiency. - dc_to_ac_efficiency_reference_load_fraction (default 0.06): load at which the curve values stored energy without a specific conversion. Battery discharges in Inverter.process_energy() use the efficiency at the AC energy of each conversion. The end-of-horizon value, the terminal and tail value curves and the AC charge break-even penalty use Inverter.reference_dc_to_ac_efficiency. Without a curve, dc_to_ac_efficiency_at() and the reference efficiency return dc_to_ac_efficiency itself, so results are bit-identical to the previous model. GENETIC0 is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): make the fitness cache size configurable The per-run fitness cache of the GENETIC optimizer is unbounded. It grows with individuals x generations (~110k keys at 300 x 400, about 1.1 kB each, 1.6 kB with an EV) and adds 120-170 MB to the peak memory of a run, while only 6-15 % of the lookups hit. On 512 MB boards this is the largest part of the optimizer's memory. Add optimization.genetic.fitness_cache_max_entries: - None (default): unbounded, the previous behaviour. - 0: no cache; every genome is evaluated. - N > 0: at most N keys. The cache is then an OrderedDict and evicts the oldest key first (FIFO, O(1)); LRU would cost bookkeeping on every hit for a cache that rarely hits. A cache hit returns exactly what the deterministic evaluation would return, so the plan for a fixed seed is identical with any setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf: import heavy optional dependencies on first use A GENETIC server with external (import) providers and EOSdash disabled loads several large packages at start that it never uses, which costs a noticeable part of the memory on 512 MB boards: - matplotlib via the GENETIC/GENETIC0 PDF report modules (server/eos.py) - statsmodels via the Holt-Winters fallback in prediction/priceabc.py - pvlib via the weather base class and the pvlib-based providers - bs4/lxml via the ClearOutside provider - scipy via the self-consumption interpolator singletons, which were loaded at import time - the GENETIC0 optimizer, imported by core/ems.py - tzfpy, whose polygon data the computed general.timezone loads Import these where they are used. The interpolator tables are loaded on first use (thread-safe). Nothing changes in what is computed. general.timezone is computed from latitude/longitude and cannot be set. Add general.timezone_override (IANA name, validated with pendulum): when set, general.timezone returns it and the lookup is skipped. When unset (default) the lookup is unchanged. Tests that patched the moved names now patch them at their source. A new test starts the server module in a subprocess and checks that none of these modules is loaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(prediction): evaluate self-consumption tables without SciPy The GENETIC inverter (and the GENETIC0 one) evaluates a pickled scipy.interpolate.RegularGridInterpolator for the self-consumption probabilities. SciPy is used nowhere else in EOS, yet loading it costs about 35 MB RSS (21 MB anonymous) on Linux arm64, which every optimization run pays. Add utils/gridinterpolator.py: - LinearGridInterpolator: 2-D linear interpolation with the same steps as SciPy's linear 2-D path (interval search, normalised distances, corner weights summed in the same order, fill value outside the grid, NaN for NaN input). - load_grid_interpolator(): reads the shipped pickles with a restricted unpickler that maps the SciPy class to a stand-in and accepts only the NumPy globals needed for the arrays. SciPy is not imported, and the data files no longer depend on SciPy's private module paths. Results are bit-identical to SciPy on Linux (aarch64, checked on 3.6M points per table including the query pattern of the inverter). The macOS arm64 SciPy wheel fuses multiply-add, so there SciPy itself differs by <= 1.1e-16; GENETIC plans for fixed seeds are unchanged on both platforms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(prediction): make the SciPy-free table evaluation opt-in Add optimization.self_consumption_interpolator (scipy | numpy, default scipy). The default keeps evaluating the self-consumption tables with SciPy's RegularGridInterpolator, built from the same grid, values and fill value the pickle holds, so results are unchanged. With numpy the tables are evaluated by LinearGridInterpolator and SciPy is never imported. The backend is read on every evaluation, so a config change applies to the next run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(genetic): cache the load distribution, clone genomes directly Two hot spots of a GENETIC run, found by profiling a 15 minute run with EV and two appliances. The solution is unchanged: both repository reference inputs give a byte-identical GeneticSolution at 400 generations. - SelfConsumptionProbabilityInterpolator._load_distribution depends on the mean load only, but ran the table interpolation over all minute-load bins for every new (load, PV) pair - 27 % of the run. It is now kept per (backend, bounded mean load); a run asks for few loads (one per slot, plus appliance combinations) with ever-changing PV values. The table is static, so entries never go stale; the cache is bounded. Scalar bounds use min/max instead of np.clip. - _make_offspring cloned parents with copy.deepcopy (8 % of the run), although the child's fitness, extra_data and immigrant protection are dropped right after. A genome is a flat list of ints, so the child is created from the parent's genes directly. Measured (1 process, Apple M5): eHive run with EV + 2 appliances, 200 generations 33.7 s -> 25.1 s; EV only 34.1 s -> 24.3 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): cheaper slot simulation, lean lookup cache, batch duplicates Four hot spots of a GENETIC run, found by profiling a 15 minute run with EV and two appliances. The solution is unchanged: both repository reference inputs (optimize_input_1/2, seed 42, 400 generations) give a byte-identical solution before and after, as do the fixed-seed runs with 1, 2 and 3 evaluation processes. * Expected direct consumption: the lookup went through the shared cache_energy_management store on every slot of every candidate - a key built from the callable, a callback per call and room for the last 100 results only (18 % misses). It is now a dictionary of the interpolator, dropped together with that store at the start of an energy management run (CacheEnergyManagementStore.generation), bounded to 8192 entries. * Slot simulation: per-slot inputs are read as plain Python numbers and per-slot results are collected in lists, turned into arrays after the loop. Element access on NumPy arrays and NumPy scalar arithmetic were the largest share of the loop. process_energy() skips the charge/export block when a slot has no PV surplus and bounds values without min()/max() calls (exact equivalents, NaN and signed zero included). * Genome keys: bytes() does the 0-255 range check in C, and a genome that is relevant as a whole is packed without copying it first. * Identical genomes within a generation are evaluated once also when the fitness cache is off (fitness_cache_max_entries = 0), by grouping the batch. It needs no memory beyond the batch; the cache costs 85-95 MB. Same run, one process, 60 generations (Apple M-series): 7.8 s -> 5.9 s with appliances, 7.1 s -> 5.0 s without. 200 generations with 3 evaluation processes: 11.4 s -> 8.2 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): keep the battery charge bookkeeping in plain numbers Battery.charge_energy() runs for every slot of every candidate plan. The per-slot charge limit and charged energy were NumPy arrays read and written one element at a time, and the NumPy scalars they return made the state of charge arithmetic slow from then on. They are plain lists now. The lower charge rates tried when a request does not fit are taken from a tuple built once per charge_rates array instead of a searchsorted() plus generator per request. The results are unchanged (same IEEE doubles, same order of operations). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): check the discharge release first, plain discharge bookkeeping Inverter._discharge_battery_to_ac() looked up the conversion efficiency before the battery told it that the plan does not release discharge in this slot. Battery.discharge_released() answers that first; the result of such a slot was (0.0, 0.0) before and is now returned directly. The discharged energy per slot is a plain list instead of a NumPy array read and written one element at a time. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): find the efficiency curve segment by bisection Inverter.dc_to_ac_efficiency_at() runs for every battery discharge of every candidate plan and walked the curve points one by one. The load fractions are split off once per curve and the segment is found by bisection; the segment and the interpolation are those of interpolate_efficiency_curve(). Compared bit for bit against that function for 200000 load values, the curve points themselves, zero, infinities and NaN. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): sum appliance loads once per candidate, read the SoC directly The slot loop asked every appliance for its load in every slot (one method call with a range check each) and converted the state of charge to percent through a method call twice per slot. The aggregate appliance load is now summed once per candidate, in the order the loop added the appliances, and the state of charge is read with the Battery formula in place. Appliances of another class or with a curve that does not cover the simulated slots, and batteries with their own current_soc_percentage(), keep the calls. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): compute the control end slot once per run control_end_slot is a property that reads the interval and the horizon from the configuration on every access - several hundred thousand times per run. The configuration is frozen for the duration of a run, so the value is computed once when the snapshot is taken and dropped with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): work out the energy-shift targets once per source slot _energy_shift_target_slots() runs for every export slot of every energy-shift mutation, in the server process while the evaluation workers wait. Which later slots are worth the shift, and in which order, depends on the forecasts and the source slot only; it is now worked out once per source slot and run, and the individual only filters the ordered slots. Filtering the sorted superset gives the same list as sorting the filtered slots (the sort is stable and its key does not depend on the individual), so mutations and the random number stream are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): compute the battery discharge of a slot in one go Serving load or exporting from the battery went through three nested calls per slot: Inverter._discharge_battery_to_ac() -> dc_to_ac_efficiency_at() -> Battery.discharge_energy(), with the release of the slot checked twice and the efficiency function called also without a curve. For the plain Battery the inverter now computes the release check, the efficiency and the discharge in one go; other battery implementations keep the step-by-step path. The arithmetic is that of Battery.discharge_energy(). The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): decode appliance starts once per gene combination Every candidate decoded its appliance start genes (feasibility repair over cycle windows and minimum gaps) and rebuilt each appliance's load curve over the whole horizon. The decoded starts, the canonical gene values and the curves depend on the appliance genes only, and a run sees few different combinations among tens of thousands of candidates. Each combination is now decoded and built once per layout (bounded to 4096 combinations) and looked up afterwards. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): compute only the terminal credit per candidate Every evaluation called _terminal_value(), which returns the credit for the energy left in the battery together with a TerminalValueResult report (component values, diagnostics, curves). The evaluation used the credit and dropped the report. _terminal_value_credit() computes the same credit without building the report; the report is still built for the final solution. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): decode battery states through a lookup table decode_charge_discharge() decodes every slot on its own from its state, with about twenty array operations per candidate. The four results per state are now worked out once per state layout and looked up for the slots of a candidate. States outside the table keep the explicit decoding. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): check for a full EV before splitting the genome _repair_ev_charge_at_full_soc() split every candidate's genome and walked all slots in Python to find EV charging in slots that begin at full SoC. Nearly no candidate has such a slot. The full slots are now found on the simulated SoC array first; the genome is split and only those slots are looked at when there are any. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * feat(genetic): evaluate candidates in parallel worker processes A GENETIC run evaluated every candidate in the server process, so a run used one CPU core however many the machine had. On small ARM boards (4x Cortex-A55, thermally limited) a 300x200 run took 12-15 minutes. Candidates are now evaluated in up to two worker processes, forked once per run after the run is fully prepared. Each worker owns a copy of the prepared optimizer; only genomes and results cross the process boundary. The fitness cache stays in the server process: hits are answered there, and identical genomes in a batch are evaluated once, as a serial run would hit the cache for the repeats. Results are applied only after the whole batch returned. If a batch fails or times out, the run continues serially. For a fixed seed the result is identical to a serial run. optimization.genetic.workers sets the number of processes. Unset (default) is automatic: one CPU core always stays free and at most two are used (1-2 cores -> 1, 3 or more -> 2), capped by a cgroup v2 CPU limit of the process or any parent (container limit, systemd CPUQuota). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * feat(ems): notify a URL after every completed optimization A client that consumes EOS plans (here DVhub) had to poll /v1/energy-management/optimization/solution to notice a new plan. That delays the plan by up to the poll interval and makes EOS serialize the complete solution on every poll, also while it is computing. ems.notify_url (default None = off) receives an HTTP POST with a small JSON event ({"event": "optimization_completed", "algorithm", "completed_at", "duration_s"}) when a run has published its solution. The POST runs in a daemon thread with a 5 s timeout, so a slow or unreachable receiver never delays energy management; a failure is logged once per failure streak. Only http(s) URLs are accepted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): fewer calls per slot in the simulation loop Three calls ran for every slot of every candidate although most slots do not need them: the AC charge factor was capped by the inverter also when the slot has no AC charging, the battery was asked for its discharged energy through a method, and process_energy() got its export arguments by keyword. Slots without AC charging skip the cap, the discharged energy is read with the Battery formula in place, and the arguments are positional. reset() of a battery creates its two control arrays with zeros() instead of full(). The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(genetic): freeze the optimization configuration for the duration of a run The genetic optimization reads horizon_hours, tail_horizon_hours, interval_sec and prediction.hours from the live configuration on every access (control_slots, tail_slots, ... are properties). A client that updates the configuration while a run is in progress - DVhub shrinks or grows the control horizon as the known price horizon moves during the night - changed these values mid-run. Genomes and device arrays sized at the start then no longer matched the simulation loop and the final evaluation failed: IndexError: index 84 is out of bounds for axis 0 with size 84 Long runs on slow hardware hit this almost every night. optimize_ems() now takes a deep copy of the genetic configuration and the prediction hours at the start of a run, all reads in genetic.py go through it, and it is released when the run ends (also on errors). New values apply to the next run. Tests: tests/test_genetic_config_frozen_per_run.py reproduces the error without the fix (horizon 20 -> 24 mid-run: index 20 out of bounds) and passes with it; genetic/optimization tests 439 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): pin evaluation workers, keep the server responsive Workers are pinned to the last `workers` CPU cores (new setting optimization.genetic.pin_workers, default on); the server process stays unpinned. The run never spreads beyond `workers` cores without a CPU quota, and the server and the rest of the system keep the other cores. No pinning without a spare core. With a CPU limit (container cpu.max, systemd CPUQuota) the automatic worker count now keeps a quarter core of it for the server process (limit 2 -> 1 worker, 2.5 -> 2): two workers used a 2-core limit fully. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(ems): tolerate settings without notify_url Test doubles and older configuration objects have no ems.notify_url. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): bound the PV surplus and export values without min()/max() The rest of Inverter.process_energy() - PV surplus, curtailment and battery export - bounded its values with min()/max() calls. They are conditional statements now, as the first half of the method already is: 'if x < 0.0: x = 0.0' is max(x, 0.0) and 'b if b < a else a' is min(a, b), NaN and signed zero included. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * feat(server): report the process start time in /v1/health A client that pushes forecasts and configuration to EOS has to notice a restart and push again: a restarted server has lost the imported series and falls back to its stored configuration. The pid cannot tell - in a container the server is always pid 1, so a client watching the pid never saw a container restart and the first runs were cancelled ("Missing or invalid PV within the control horizon"). /v1/health now carries "started_at", fixed for the life of the process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(devices): evaluate the efficiency curve at the delivered AC load When the battery limits a discharge (low state of charge or its power limit), the conversion ran with the efficiency of the requested AC energy. With a 1000 Wh battery at 10 % SoC, a 1000 W inverter and the curve [(0, .8), (.5, .95), (1, .95)], a request of 500 Wh AC returned 95 Wh AC from the available 100 Wh DC, although the curve gives 82.5 % at that load. The delivered AC energy now solves a = dc * efficiency(a / rated) exactly on the piecewise-linear curve (solve_ac_for_dc_energy), bounded by the request. Unlimited discharges and inverters without a curve are unchanged. Regression tests use a real Battery: low SoC and a power-limited discharge, each with 60- and 15-minute slots, battery discharge losses, the load path through process_energy, and constant efficiency. * docs: regenerate OpenAPI for the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix --------- Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Andreas <drbacke@gmx.de> |
||
|
|
7dc1ee92fa |
perf(genetic): faster runs with unchanged results (#1385)
* feat(devices): add load-dependent DC-to-AC efficiency curve for GENETIC inverters Real inverters are much less efficient at low load than at rated power (e.g. 0.80 at 2 % vs. 0.93 at 7 % load). A constant dc_to_ac_efficiency overrates night-time battery discharge into a small base load and, with it, arbitrage and grid charging. Add two optional inverter settings for the GENETIC optimizer: - dc_to_ac_efficiency_curve: list of (load_fraction, efficiency) points, load fraction = AC power / max_power_w. Points must be strictly increasing in [0, 1] with efficiencies in (0, 1]; at least two points. Linear interpolation, clamped at both ends. When set, the curve replaces dc_to_ac_efficiency. - dc_to_ac_efficiency_reference_load_fraction (default 0.06): load at which the curve values stored energy without a specific conversion. Battery discharges in Inverter.process_energy() use the efficiency at the AC energy of each conversion. The end-of-horizon value, the terminal and tail value curves and the AC charge break-even penalty use Inverter.reference_dc_to_ac_efficiency. Without a curve, dc_to_ac_efficiency_at() and the reference efficiency return dc_to_ac_efficiency itself, so results are bit-identical to the previous model. GENETIC0 is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): make the fitness cache size configurable The per-run fitness cache of the GENETIC optimizer is unbounded. It grows with individuals x generations (~110k keys at 300 x 400, about 1.1 kB each, 1.6 kB with an EV) and adds 120-170 MB to the peak memory of a run, while only 6-15 % of the lookups hit. On 512 MB boards this is the largest part of the optimizer's memory. Add optimization.genetic.fitness_cache_max_entries: - None (default): unbounded, the previous behaviour. - 0: no cache; every genome is evaluated. - N > 0: at most N keys. The cache is then an OrderedDict and evicts the oldest key first (FIFO, O(1)); LRU would cost bookkeeping on every hit for a cache that rarely hits. A cache hit returns exactly what the deterministic evaluation would return, so the plan for a fixed seed is identical with any setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf: import heavy optional dependencies on first use A GENETIC server with external (import) providers and EOSdash disabled loads several large packages at start that it never uses, which costs a noticeable part of the memory on 512 MB boards: - matplotlib via the GENETIC/GENETIC0 PDF report modules (server/eos.py) - statsmodels via the Holt-Winters fallback in prediction/priceabc.py - pvlib via the weather base class and the pvlib-based providers - bs4/lxml via the ClearOutside provider - scipy via the self-consumption interpolator singletons, which were loaded at import time - the GENETIC0 optimizer, imported by core/ems.py - tzfpy, whose polygon data the computed general.timezone loads Import these where they are used. The interpolator tables are loaded on first use (thread-safe). Nothing changes in what is computed. general.timezone is computed from latitude/longitude and cannot be set. Add general.timezone_override (IANA name, validated with pendulum): when set, general.timezone returns it and the lookup is skipped. When unset (default) the lookup is unchanged. Tests that patched the moved names now patch them at their source. A new test starts the server module in a subprocess and checks that none of these modules is loaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(prediction): evaluate self-consumption tables without SciPy The GENETIC inverter (and the GENETIC0 one) evaluates a pickled scipy.interpolate.RegularGridInterpolator for the self-consumption probabilities. SciPy is used nowhere else in EOS, yet loading it costs about 35 MB RSS (21 MB anonymous) on Linux arm64, which every optimization run pays. Add utils/gridinterpolator.py: - LinearGridInterpolator: 2-D linear interpolation with the same steps as SciPy's linear 2-D path (interval search, normalised distances, corner weights summed in the same order, fill value outside the grid, NaN for NaN input). - load_grid_interpolator(): reads the shipped pickles with a restricted unpickler that maps the SciPy class to a stand-in and accepts only the NumPy globals needed for the arrays. SciPy is not imported, and the data files no longer depend on SciPy's private module paths. Results are bit-identical to SciPy on Linux (aarch64, checked on 3.6M points per table including the query pattern of the inverter). The macOS arm64 SciPy wheel fuses multiply-add, so there SciPy itself differs by <= 1.1e-16; GENETIC plans for fixed seeds are unchanged on both platforms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(prediction): make the SciPy-free table evaluation opt-in Add optimization.self_consumption_interpolator (scipy | numpy, default scipy). The default keeps evaluating the self-consumption tables with SciPy's RegularGridInterpolator, built from the same grid, values and fill value the pickle holds, so results are unchanged. With numpy the tables are evaluated by LinearGridInterpolator and SciPy is never imported. The backend is read on every evaluation, so a config change applies to the next run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(genetic): cache the load distribution, clone genomes directly Two hot spots of a GENETIC run, found by profiling a 15 minute run with EV and two appliances. The solution is unchanged: both repository reference inputs give a byte-identical GeneticSolution at 400 generations. - SelfConsumptionProbabilityInterpolator._load_distribution depends on the mean load only, but ran the table interpolation over all minute-load bins for every new (load, PV) pair - 27 % of the run. It is now kept per (backend, bounded mean load); a run asks for few loads (one per slot, plus appliance combinations) with ever-changing PV values. The table is static, so entries never go stale; the cache is bounded. Scalar bounds use min/max instead of np.clip. - _make_offspring cloned parents with copy.deepcopy (8 % of the run), although the child's fitness, extra_data and immigrant protection are dropped right after. A genome is a flat list of ints, so the child is created from the parent's genes directly. Measured (1 process, Apple M5): eHive run with EV + 2 appliances, 200 generations 33.7 s -> 25.1 s; EV only 34.1 s -> 24.3 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): cheaper slot simulation, lean lookup cache, batch duplicates Four hot spots of a GENETIC run, found by profiling a 15 minute run with EV and two appliances. The solution is unchanged: both repository reference inputs (optimize_input_1/2, seed 42, 400 generations) give a byte-identical solution before and after, as do the fixed-seed runs with 1, 2 and 3 evaluation processes. * Expected direct consumption: the lookup went through the shared cache_energy_management store on every slot of every candidate - a key built from the callable, a callback per call and room for the last 100 results only (18 % misses). It is now a dictionary of the interpolator, dropped together with that store at the start of an energy management run (CacheEnergyManagementStore.generation), bounded to 8192 entries. * Slot simulation: per-slot inputs are read as plain Python numbers and per-slot results are collected in lists, turned into arrays after the loop. Element access on NumPy arrays and NumPy scalar arithmetic were the largest share of the loop. process_energy() skips the charge/export block when a slot has no PV surplus and bounds values without min()/max() calls (exact equivalents, NaN and signed zero included). * Genome keys: bytes() does the 0-255 range check in C, and a genome that is relevant as a whole is packed without copying it first. * Identical genomes within a generation are evaluated once also when the fitness cache is off (fitness_cache_max_entries = 0), by grouping the batch. It needs no memory beyond the batch; the cache costs 85-95 MB. Same run, one process, 60 generations (Apple M-series): 7.8 s -> 5.9 s with appliances, 7.1 s -> 5.0 s without. 200 generations with 3 evaluation processes: 11.4 s -> 8.2 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): keep the battery charge bookkeeping in plain numbers Battery.charge_energy() runs for every slot of every candidate plan. The per-slot charge limit and charged energy were NumPy arrays read and written one element at a time, and the NumPy scalars they return made the state of charge arithmetic slow from then on. They are plain lists now. The lower charge rates tried when a request does not fit are taken from a tuple built once per charge_rates array instead of a searchsorted() plus generator per request. The results are unchanged (same IEEE doubles, same order of operations). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): check the discharge release first, plain discharge bookkeeping Inverter._discharge_battery_to_ac() looked up the conversion efficiency before the battery told it that the plan does not release discharge in this slot. Battery.discharge_released() answers that first; the result of such a slot was (0.0, 0.0) before and is now returned directly. The discharged energy per slot is a plain list instead of a NumPy array read and written one element at a time. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): find the efficiency curve segment by bisection Inverter.dc_to_ac_efficiency_at() runs for every battery discharge of every candidate plan and walked the curve points one by one. The load fractions are split off once per curve and the segment is found by bisection; the segment and the interpolation are those of interpolate_efficiency_curve(). Compared bit for bit against that function for 200000 load values, the curve points themselves, zero, infinities and NaN. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): sum appliance loads once per candidate, read the SoC directly The slot loop asked every appliance for its load in every slot (one method call with a range check each) and converted the state of charge to percent through a method call twice per slot. The aggregate appliance load is now summed once per candidate, in the order the loop added the appliances, and the state of charge is read with the Battery formula in place. Appliances of another class or with a curve that does not cover the simulated slots, and batteries with their own current_soc_percentage(), keep the calls. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): compute the control end slot once per run control_end_slot is a property that reads the interval and the horizon from the configuration on every access - several hundred thousand times per run. The configuration is frozen for the duration of a run, so the value is computed once when the snapshot is taken and dropped with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): work out the energy-shift targets once per source slot _energy_shift_target_slots() runs for every export slot of every energy-shift mutation, in the server process while the evaluation workers wait. Which later slots are worth the shift, and in which order, depends on the forecasts and the source slot only; it is now worked out once per source slot and run, and the individual only filters the ordered slots. Filtering the sorted superset gives the same list as sorting the filtered slots (the sort is stable and its key does not depend on the individual), so mutations and the random number stream are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): compute the battery discharge of a slot in one go Serving load or exporting from the battery went through three nested calls per slot: Inverter._discharge_battery_to_ac() -> dc_to_ac_efficiency_at() -> Battery.discharge_energy(), with the release of the slot checked twice and the efficiency function called also without a curve. For the plain Battery the inverter now computes the release check, the efficiency and the discharge in one go; other battery implementations keep the step-by-step path. The arithmetic is that of Battery.discharge_energy(). The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): decode appliance starts once per gene combination Every candidate decoded its appliance start genes (feasibility repair over cycle windows and minimum gaps) and rebuilt each appliance's load curve over the whole horizon. The decoded starts, the canonical gene values and the curves depend on the appliance genes only, and a run sees few different combinations among tens of thousands of candidates. Each combination is now decoded and built once per layout (bounded to 4096 combinations) and looked up afterwards. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): compute only the terminal credit per candidate Every evaluation called _terminal_value(), which returns the credit for the energy left in the battery together with a TerminalValueResult report (component values, diagnostics, curves). The evaluation used the credit and dropped the report. _terminal_value_credit() computes the same credit without building the report; the report is still built for the final solution. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): decode battery states through a lookup table decode_charge_discharge() decodes every slot on its own from its state, with about twenty array operations per candidate. The four results per state are now worked out once per state layout and looked up for the slots of a candidate. States outside the table keep the explicit decoding. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): check for a full EV before splitting the genome _repair_ev_charge_at_full_soc() split every candidate's genome and walked all slots in Python to find EV charging in slots that begin at full SoC. Nearly no candidate has such a slot. The full slots are now found on the simulated SoC array first; the genome is split and only those slots are looked at when there are any. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * feat(genetic): evaluate candidates in parallel worker processes A GENETIC run evaluated every candidate in the server process, so a run used one CPU core however many the machine had. On small ARM boards (4x Cortex-A55, thermally limited) a 300x200 run took 12-15 minutes. Candidates are now evaluated in up to two worker processes, forked once per run after the run is fully prepared. Each worker owns a copy of the prepared optimizer; only genomes and results cross the process boundary. The fitness cache stays in the server process: hits are answered there, and identical genomes in a batch are evaluated once, as a serial run would hit the cache for the repeats. Results are applied only after the whole batch returned. If a batch fails or times out, the run continues serially. For a fixed seed the result is identical to a serial run. optimization.genetic.workers sets the number of processes. Unset (default) is automatic: one CPU core always stays free and at most two are used (1-2 cores -> 1, 3 or more -> 2), capped by a cgroup v2 CPU limit of the process or any parent (container limit, systemd CPUQuota). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * feat(ems): notify a URL after every completed optimization A client that consumes EOS plans (here DVhub) had to poll /v1/energy-management/optimization/solution to notice a new plan. That delays the plan by up to the poll interval and makes EOS serialize the complete solution on every poll, also while it is computing. ems.notify_url (default None = off) receives an HTTP POST with a small JSON event ({"event": "optimization_completed", "algorithm", "completed_at", "duration_s"}) when a run has published its solution. The POST runs in a daemon thread with a 5 s timeout, so a slow or unreachable receiver never delays energy management; a failure is logged once per failure streak. Only http(s) URLs are accepted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): fewer calls per slot in the simulation loop Three calls ran for every slot of every candidate although most slots do not need them: the AC charge factor was capped by the inverter also when the slot has no AC charging, the battery was asked for its discharged energy through a method, and process_energy() got its export arguments by keyword. Slots without AC charging skip the cap, the discharged energy is read with the Battery formula in place, and the arguments are positional. reset() of a battery creates its two control arrays with zeros() instead of full(). The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(genetic): freeze the optimization configuration for the duration of a run The genetic optimization reads horizon_hours, tail_horizon_hours, interval_sec and prediction.hours from the live configuration on every access (control_slots, tail_slots, ... are properties). A client that updates the configuration while a run is in progress - DVhub shrinks or grows the control horizon as the known price horizon moves during the night - changed these values mid-run. Genomes and device arrays sized at the start then no longer matched the simulation loop and the final evaluation failed: IndexError: index 84 is out of bounds for axis 0 with size 84 Long runs on slow hardware hit this almost every night. optimize_ems() now takes a deep copy of the genetic configuration and the prediction hours at the start of a run, all reads in genetic.py go through it, and it is released when the run ends (also on errors). New values apply to the next run. Tests: tests/test_genetic_config_frozen_per_run.py reproduces the error without the fix (horizon 20 -> 24 mid-run: index 20 out of bounds) and passes with it; genetic/optimization tests 439 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): pin evaluation workers, keep the server responsive Workers are pinned to the last `workers` CPU cores (new setting optimization.genetic.pin_workers, default on); the server process stays unpinned. The run never spreads beyond `workers` cores without a CPU quota, and the server and the rest of the system keep the other cores. No pinning without a spare core. With a CPU limit (container cpu.max, systemd CPUQuota) the automatic worker count now keeps a quarter core of it for the server process (limit 2 -> 1 worker, 2.5 -> 2): two workers used a 2-core limit fully. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(ems): tolerate settings without notify_url Test doubles and older configuration objects have no ems.notify_url. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): bound the PV surplus and export values without min()/max() The rest of Inverter.process_energy() - PV surplus, curtailment and battery export - bounded its values with min()/max() calls. They are conditional statements now, as the first half of the method already is: 'if x < 0.0: x = 0.0' is max(x, 0.0) and 'b if b < a else a' is min(a, b), NaN and signed zero included. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(devices): evaluate the efficiency curve at the delivered AC load When the battery limits a discharge (low state of charge or its power limit), the conversion ran with the efficiency of the requested AC energy. With a 1000 Wh battery at 10 % SoC, a 1000 W inverter and the curve [(0, .8), (.5, .95), (1, .95)], a request of 500 Wh AC returned 95 Wh AC from the available 100 Wh DC, although the curve gives 82.5 % at that load. The delivered AC energy now solves a = dc * efficiency(a / rated) exactly on the piecewise-linear curve (solve_ac_for_dc_energy), bounded by the request. Unlimited discharges and inverters without a curve are unchanged. Regression tests use a real Battery: low SoC and a power-limited discharge, each with 60- and 15-minute slots, battery discharge losses, the load path through process_energy, and constant efficiency. * docs: regenerate OpenAPI for the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix --------- Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Andreas <drbacke@gmx.de> |
||
|
|
70b72bfbce |
feat(ems): notify a URL after every completed optimization (#1383)
* feat(devices): add load-dependent DC-to-AC efficiency curve for GENETIC inverters Real inverters are much less efficient at low load than at rated power (e.g. 0.80 at 2 % vs. 0.93 at 7 % load). A constant dc_to_ac_efficiency overrates night-time battery discharge into a small base load and, with it, arbitrage and grid charging. Add two optional inverter settings for the GENETIC optimizer: - dc_to_ac_efficiency_curve: list of (load_fraction, efficiency) points, load fraction = AC power / max_power_w. Points must be strictly increasing in [0, 1] with efficiencies in (0, 1]; at least two points. Linear interpolation, clamped at both ends. When set, the curve replaces dc_to_ac_efficiency. - dc_to_ac_efficiency_reference_load_fraction (default 0.06): load at which the curve values stored energy without a specific conversion. Battery discharges in Inverter.process_energy() use the efficiency at the AC energy of each conversion. The end-of-horizon value, the terminal and tail value curves and the AC charge break-even penalty use Inverter.reference_dc_to_ac_efficiency. Without a curve, dc_to_ac_efficiency_at() and the reference efficiency return dc_to_ac_efficiency itself, so results are bit-identical to the previous model. GENETIC0 is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): make the fitness cache size configurable The per-run fitness cache of the GENETIC optimizer is unbounded. It grows with individuals x generations (~110k keys at 300 x 400, about 1.1 kB each, 1.6 kB with an EV) and adds 120-170 MB to the peak memory of a run, while only 6-15 % of the lookups hit. On 512 MB boards this is the largest part of the optimizer's memory. Add optimization.genetic.fitness_cache_max_entries: - None (default): unbounded, the previous behaviour. - 0: no cache; every genome is evaluated. - N > 0: at most N keys. The cache is then an OrderedDict and evicts the oldest key first (FIFO, O(1)); LRU would cost bookkeeping on every hit for a cache that rarely hits. A cache hit returns exactly what the deterministic evaluation would return, so the plan for a fixed seed is identical with any setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf: import heavy optional dependencies on first use A GENETIC server with external (import) providers and EOSdash disabled loads several large packages at start that it never uses, which costs a noticeable part of the memory on 512 MB boards: - matplotlib via the GENETIC/GENETIC0 PDF report modules (server/eos.py) - statsmodels via the Holt-Winters fallback in prediction/priceabc.py - pvlib via the weather base class and the pvlib-based providers - bs4/lxml via the ClearOutside provider - scipy via the self-consumption interpolator singletons, which were loaded at import time - the GENETIC0 optimizer, imported by core/ems.py - tzfpy, whose polygon data the computed general.timezone loads Import these where they are used. The interpolator tables are loaded on first use (thread-safe). Nothing changes in what is computed. general.timezone is computed from latitude/longitude and cannot be set. Add general.timezone_override (IANA name, validated with pendulum): when set, general.timezone returns it and the lookup is skipped. When unset (default) the lookup is unchanged. Tests that patched the moved names now patch them at their source. A new test starts the server module in a subprocess and checks that none of these modules is loaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(prediction): evaluate self-consumption tables without SciPy The GENETIC inverter (and the GENETIC0 one) evaluates a pickled scipy.interpolate.RegularGridInterpolator for the self-consumption probabilities. SciPy is used nowhere else in EOS, yet loading it costs about 35 MB RSS (21 MB anonymous) on Linux arm64, which every optimization run pays. Add utils/gridinterpolator.py: - LinearGridInterpolator: 2-D linear interpolation with the same steps as SciPy's linear 2-D path (interval search, normalised distances, corner weights summed in the same order, fill value outside the grid, NaN for NaN input). - load_grid_interpolator(): reads the shipped pickles with a restricted unpickler that maps the SciPy class to a stand-in and accepts only the NumPy globals needed for the arrays. SciPy is not imported, and the data files no longer depend on SciPy's private module paths. Results are bit-identical to SciPy on Linux (aarch64, checked on 3.6M points per table including the query pattern of the inverter). The macOS arm64 SciPy wheel fuses multiply-add, so there SciPy itself differs by <= 1.1e-16; GENETIC plans for fixed seeds are unchanged on both platforms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(prediction): make the SciPy-free table evaluation opt-in Add optimization.self_consumption_interpolator (scipy | numpy, default scipy). The default keeps evaluating the self-consumption tables with SciPy's RegularGridInterpolator, built from the same grid, values and fill value the pickle holds, so results are unchanged. With numpy the tables are evaluated by LinearGridInterpolator and SciPy is never imported. The backend is read on every evaluation, so a config change applies to the next run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): evaluate candidates in parallel worker processes A GENETIC run evaluated every candidate in the server process, so a run used one CPU core however many the machine had. On small ARM boards (4x Cortex-A55, thermally limited) a 300x200 run took 12-15 minutes. Candidates are now evaluated in up to two worker processes, forked once per run after the run is fully prepared. Each worker owns a copy of the prepared optimizer; only genomes and results cross the process boundary. The fitness cache stays in the server process: hits are answered there, and identical genomes in a batch are evaluated once, as a serial run would hit the cache for the repeats. Results are applied only after the whole batch returned. If a batch fails or times out, the run continues serially. For a fixed seed the result is identical to a serial run. optimization.genetic.workers sets the number of processes. Unset (default) is automatic: one CPU core always stays free and at most two are used (1-2 cores -> 1, 3 or more -> 2), capped by a cgroup v2 CPU limit of the process or any parent (container limit, systemd CPUQuota). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * feat(ems): notify a URL after every completed optimization A client that consumes EOS plans (here DVhub) had to poll /v1/energy-management/optimization/solution to notice a new plan. That delays the plan by up to the poll interval and makes EOS serialize the complete solution on every poll, also while it is computing. ems.notify_url (default None = off) receives an HTTP POST with a small JSON event ({"event": "optimization_completed", "algorithm", "completed_at", "duration_s"}) when a run has published its solution. The POST runs in a daemon thread with a 5 s timeout, so a slow or unreachable receiver never delays energy management; a failure is logged once per failure streak. Only http(s) URLs are accepted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(genetic): freeze the optimization configuration for the duration of a run The genetic optimization reads horizon_hours, tail_horizon_hours, interval_sec and prediction.hours from the live configuration on every access (control_slots, tail_slots, ... are properties). A client that updates the configuration while a run is in progress - DVhub shrinks or grows the control horizon as the known price horizon moves during the night - changed these values mid-run. Genomes and device arrays sized at the start then no longer matched the simulation loop and the final evaluation failed: IndexError: index 84 is out of bounds for axis 0 with size 84 Long runs on slow hardware hit this almost every night. optimize_ems() now takes a deep copy of the genetic configuration and the prediction hours at the start of a run, all reads in genetic.py go through it, and it is released when the run ends (also on errors). New values apply to the next run. Tests: tests/test_genetic_config_frozen_per_run.py reproduces the error without the fix (horizon 20 -> 24 mid-run: index 20 out of bounds) and passes with it; genetic/optimization tests 439 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): pin evaluation workers, keep the server responsive Workers are pinned to the last `workers` CPU cores (new setting optimization.genetic.pin_workers, default on); the server process stays unpinned. The run never spreads beyond `workers` cores without a CPU quota, and the server and the rest of the system keep the other cores. No pinning without a spare core. With a CPU limit (container cpu.max, systemd CPUQuota) the automatic worker count now keeps a quarter core of it for the server process (limit 2 -> 1 worker, 2.5 -> 2): two workers used a 2-core limit fully. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(ems): tolerate settings without notify_url Test doubles and older configuration objects have no ems.notify_url. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(devices): evaluate the efficiency curve at the delivered AC load When the battery limits a discharge (low state of charge or its power limit), the conversion ran with the efficiency of the requested AC energy. With a 1000 Wh battery at 10 % SoC, a 1000 W inverter and the curve [(0, .8), (.5, .95), (1, .95)], a request of 500 Wh AC returned 95 Wh AC from the available 100 Wh DC, although the curve gives 82.5 % at that load. The delivered AC energy now solves a = dc * efficiency(a / rated) exactly on the piecewise-linear curve (solve_ac_for_dc_energy), bounded by the request. Unlimited discharges and inverters without a curve are unchanged. Regression tests use a real Battery: low SoC and a power-limited discharge, each with 60- and 15-minute slots, battery discharge losses, the load path through process_energy, and constant efficiency. * docs: regenerate OpenAPI for the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * test(genetic): cover serial fallback on platforms without fork --------- Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Andreas <drbacke@gmx.de> |
||
|
|
1f8742a54b |
feat(genetic): evaluate candidates in parallel worker processes (#1382)
* feat(devices): add load-dependent DC-to-AC efficiency curve for GENETIC inverters Real inverters are much less efficient at low load than at rated power (e.g. 0.80 at 2 % vs. 0.93 at 7 % load). A constant dc_to_ac_efficiency overrates night-time battery discharge into a small base load and, with it, arbitrage and grid charging. Add two optional inverter settings for the GENETIC optimizer: - dc_to_ac_efficiency_curve: list of (load_fraction, efficiency) points, load fraction = AC power / max_power_w. Points must be strictly increasing in [0, 1] with efficiencies in (0, 1]; at least two points. Linear interpolation, clamped at both ends. When set, the curve replaces dc_to_ac_efficiency. - dc_to_ac_efficiency_reference_load_fraction (default 0.06): load at which the curve values stored energy without a specific conversion. Battery discharges in Inverter.process_energy() use the efficiency at the AC energy of each conversion. The end-of-horizon value, the terminal and tail value curves and the AC charge break-even penalty use Inverter.reference_dc_to_ac_efficiency. Without a curve, dc_to_ac_efficiency_at() and the reference efficiency return dc_to_ac_efficiency itself, so results are bit-identical to the previous model. GENETIC0 is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): make the fitness cache size configurable The per-run fitness cache of the GENETIC optimizer is unbounded. It grows with individuals x generations (~110k keys at 300 x 400, about 1.1 kB each, 1.6 kB with an EV) and adds 120-170 MB to the peak memory of a run, while only 6-15 % of the lookups hit. On 512 MB boards this is the largest part of the optimizer's memory. Add optimization.genetic.fitness_cache_max_entries: - None (default): unbounded, the previous behaviour. - 0: no cache; every genome is evaluated. - N > 0: at most N keys. The cache is then an OrderedDict and evicts the oldest key first (FIFO, O(1)); LRU would cost bookkeeping on every hit for a cache that rarely hits. A cache hit returns exactly what the deterministic evaluation would return, so the plan for a fixed seed is identical with any setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf: import heavy optional dependencies on first use A GENETIC server with external (import) providers and EOSdash disabled loads several large packages at start that it never uses, which costs a noticeable part of the memory on 512 MB boards: - matplotlib via the GENETIC/GENETIC0 PDF report modules (server/eos.py) - statsmodels via the Holt-Winters fallback in prediction/priceabc.py - pvlib via the weather base class and the pvlib-based providers - bs4/lxml via the ClearOutside provider - scipy via the self-consumption interpolator singletons, which were loaded at import time - the GENETIC0 optimizer, imported by core/ems.py - tzfpy, whose polygon data the computed general.timezone loads Import these where they are used. The interpolator tables are loaded on first use (thread-safe). Nothing changes in what is computed. general.timezone is computed from latitude/longitude and cannot be set. Add general.timezone_override (IANA name, validated with pendulum): when set, general.timezone returns it and the lookup is skipped. When unset (default) the lookup is unchanged. Tests that patched the moved names now patch them at their source. A new test starts the server module in a subprocess and checks that none of these modules is loaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(prediction): evaluate self-consumption tables without SciPy The GENETIC inverter (and the GENETIC0 one) evaluates a pickled scipy.interpolate.RegularGridInterpolator for the self-consumption probabilities. SciPy is used nowhere else in EOS, yet loading it costs about 35 MB RSS (21 MB anonymous) on Linux arm64, which every optimization run pays. Add utils/gridinterpolator.py: - LinearGridInterpolator: 2-D linear interpolation with the same steps as SciPy's linear 2-D path (interval search, normalised distances, corner weights summed in the same order, fill value outside the grid, NaN for NaN input). - load_grid_interpolator(): reads the shipped pickles with a restricted unpickler that maps the SciPy class to a stand-in and accepts only the NumPy globals needed for the arrays. SciPy is not imported, and the data files no longer depend on SciPy's private module paths. Results are bit-identical to SciPy on Linux (aarch64, checked on 3.6M points per table including the query pattern of the inverter). The macOS arm64 SciPy wheel fuses multiply-add, so there SciPy itself differs by <= 1.1e-16; GENETIC plans for fixed seeds are unchanged on both platforms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(prediction): make the SciPy-free table evaluation opt-in Add optimization.self_consumption_interpolator (scipy | numpy, default scipy). The default keeps evaluating the self-consumption tables with SciPy's RegularGridInterpolator, built from the same grid, values and fill value the pickle holds, so results are unchanged. With numpy the tables are evaluated by LinearGridInterpolator and SciPy is never imported. The backend is read on every evaluation, so a config change applies to the next run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): evaluate candidates in parallel worker processes A GENETIC run evaluated every candidate in the server process, so a run used one CPU core however many the machine had. On small ARM boards (4x Cortex-A55, thermally limited) a 300x200 run took 12-15 minutes. Candidates are now evaluated in up to two worker processes, forked once per run after the run is fully prepared. Each worker owns a copy of the prepared optimizer; only genomes and results cross the process boundary. The fitness cache stays in the server process: hits are answered there, and identical genomes in a batch are evaluated once, as a serial run would hit the cache for the repeats. Results are applied only after the whole batch returned. If a batch fails or times out, the run continues serially. For a fixed seed the result is identical to a serial run. optimization.genetic.workers sets the number of processes. Unset (default) is automatic: one CPU core always stays free and at most two are used (1-2 cores -> 1, 3 or more -> 2), capped by a cgroup v2 CPU limit of the process or any parent (container limit, systemd CPUQuota). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(genetic): freeze the optimization configuration for the duration of a run The genetic optimization reads horizon_hours, tail_horizon_hours, interval_sec and prediction.hours from the live configuration on every access (control_slots, tail_slots, ... are properties). A client that updates the configuration while a run is in progress - DVhub shrinks or grows the control horizon as the known price horizon moves during the night - changed these values mid-run. Genomes and device arrays sized at the start then no longer matched the simulation loop and the final evaluation failed: IndexError: index 84 is out of bounds for axis 0 with size 84 Long runs on slow hardware hit this almost every night. optimize_ems() now takes a deep copy of the genetic configuration and the prediction hours at the start of a run, all reads in genetic.py go through it, and it is released when the run ends (also on errors). New values apply to the next run. Tests: tests/test_genetic_config_frozen_per_run.py reproduces the error without the fix (horizon 20 -> 24 mid-run: index 20 out of bounds) and passes with it; genetic/optimization tests 439 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): pin evaluation workers, keep the server responsive Workers are pinned to the last `workers` CPU cores (new setting optimization.genetic.pin_workers, default on); the server process stays unpinned. The run never spreads beyond `workers` cores without a CPU quota, and the server and the rest of the system keep the other cores. No pinning without a spare core. With a CPU limit (container cpu.max, systemd CPUQuota) the automatic worker count now keeps a quarter core of it for the server process (limit 2 -> 1 worker, 2.5 -> 2): two workers used a 2-core limit fully. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(devices): evaluate the efficiency curve at the delivered AC load When the battery limits a discharge (low state of charge or its power limit), the conversion ran with the efficiency of the requested AC energy. With a 1000 Wh battery at 10 % SoC, a 1000 W inverter and the curve [(0, .8), (.5, .95), (1, .95)], a request of 500 Wh AC returned 95 Wh AC from the available 100 Wh DC, although the curve gives 82.5 % at that load. The delivered AC energy now solves a = dc * efficiency(a / rated) exactly on the piecewise-linear curve (solve_ac_for_dc_energy), bounded by the request. Unlimited discharges and inverters without a curve are unchanged. Regression tests use a real Battery: low SoC and a power-limited discharge, each with 60- and 15-minute slots, battery discharge losses, the load path through process_energy, and constant efficiency. * docs: regenerate OpenAPI for the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * test(genetic): cover serial fallback on platforms without fork --------- Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Andreas <drbacke@gmx.de> |
||
|
|
720809397c |
fix(genetic): freeze the optimization configuration for the duration of a run (#1381)
* feat(devices): add load-dependent DC-to-AC efficiency curve for GENETIC inverters Real inverters are much less efficient at low load than at rated power (e.g. 0.80 at 2 % vs. 0.93 at 7 % load). A constant dc_to_ac_efficiency overrates night-time battery discharge into a small base load and, with it, arbitrage and grid charging. Add two optional inverter settings for the GENETIC optimizer: - dc_to_ac_efficiency_curve: list of (load_fraction, efficiency) points, load fraction = AC power / max_power_w. Points must be strictly increasing in [0, 1] with efficiencies in (0, 1]; at least two points. Linear interpolation, clamped at both ends. When set, the curve replaces dc_to_ac_efficiency. - dc_to_ac_efficiency_reference_load_fraction (default 0.06): load at which the curve values stored energy without a specific conversion. Battery discharges in Inverter.process_energy() use the efficiency at the AC energy of each conversion. The end-of-horizon value, the terminal and tail value curves and the AC charge break-even penalty use Inverter.reference_dc_to_ac_efficiency. Without a curve, dc_to_ac_efficiency_at() and the reference efficiency return dc_to_ac_efficiency itself, so results are bit-identical to the previous model. GENETIC0 is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): make the fitness cache size configurable The per-run fitness cache of the GENETIC optimizer is unbounded. It grows with individuals x generations (~110k keys at 300 x 400, about 1.1 kB each, 1.6 kB with an EV) and adds 120-170 MB to the peak memory of a run, while only 6-15 % of the lookups hit. On 512 MB boards this is the largest part of the optimizer's memory. Add optimization.genetic.fitness_cache_max_entries: - None (default): unbounded, the previous behaviour. - 0: no cache; every genome is evaluated. - N > 0: at most N keys. The cache is then an OrderedDict and evicts the oldest key first (FIFO, O(1)); LRU would cost bookkeeping on every hit for a cache that rarely hits. A cache hit returns exactly what the deterministic evaluation would return, so the plan for a fixed seed is identical with any setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf: import heavy optional dependencies on first use A GENETIC server with external (import) providers and EOSdash disabled loads several large packages at start that it never uses, which costs a noticeable part of the memory on 512 MB boards: - matplotlib via the GENETIC/GENETIC0 PDF report modules (server/eos.py) - statsmodels via the Holt-Winters fallback in prediction/priceabc.py - pvlib via the weather base class and the pvlib-based providers - bs4/lxml via the ClearOutside provider - scipy via the self-consumption interpolator singletons, which were loaded at import time - the GENETIC0 optimizer, imported by core/ems.py - tzfpy, whose polygon data the computed general.timezone loads Import these where they are used. The interpolator tables are loaded on first use (thread-safe). Nothing changes in what is computed. general.timezone is computed from latitude/longitude and cannot be set. Add general.timezone_override (IANA name, validated with pendulum): when set, general.timezone returns it and the lookup is skipped. When unset (default) the lookup is unchanged. Tests that patched the moved names now patch them at their source. A new test starts the server module in a subprocess and checks that none of these modules is loaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(prediction): evaluate self-consumption tables without SciPy The GENETIC inverter (and the GENETIC0 one) evaluates a pickled scipy.interpolate.RegularGridInterpolator for the self-consumption probabilities. SciPy is used nowhere else in EOS, yet loading it costs about 35 MB RSS (21 MB anonymous) on Linux arm64, which every optimization run pays. Add utils/gridinterpolator.py: - LinearGridInterpolator: 2-D linear interpolation with the same steps as SciPy's linear 2-D path (interval search, normalised distances, corner weights summed in the same order, fill value outside the grid, NaN for NaN input). - load_grid_interpolator(): reads the shipped pickles with a restricted unpickler that maps the SciPy class to a stand-in and accepts only the NumPy globals needed for the arrays. SciPy is not imported, and the data files no longer depend on SciPy's private module paths. Results are bit-identical to SciPy on Linux (aarch64, checked on 3.6M points per table including the query pattern of the inverter). The macOS arm64 SciPy wheel fuses multiply-add, so there SciPy itself differs by <= 1.1e-16; GENETIC plans for fixed seeds are unchanged on both platforms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(prediction): make the SciPy-free table evaluation opt-in Add optimization.self_consumption_interpolator (scipy | numpy, default scipy). The default keeps evaluating the self-consumption tables with SciPy's RegularGridInterpolator, built from the same grid, values and fill value the pickle holds, so results are unchanged. With numpy the tables are evaluated by LinearGridInterpolator and SciPy is never imported. The backend is read on every evaluation, so a config change applies to the next run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(genetic): freeze the optimization configuration for the duration of a run The genetic optimization reads horizon_hours, tail_horizon_hours, interval_sec and prediction.hours from the live configuration on every access (control_slots, tail_slots, ... are properties). A client that updates the configuration while a run is in progress - DVhub shrinks or grows the control horizon as the known price horizon moves during the night - changed these values mid-run. Genomes and device arrays sized at the start then no longer matched the simulation loop and the final evaluation failed: IndexError: index 84 is out of bounds for axis 0 with size 84 Long runs on slow hardware hit this almost every night. optimize_ems() now takes a deep copy of the genetic configuration and the prediction hours at the start of a run, all reads in genetic.py go through it, and it is released when the run ends (also on errors). New values apply to the next run. Tests: tests/test_genetic_config_frozen_per_run.py reproduces the error without the fix (horizon 20 -> 24 mid-run: index 20 out of bounds) and passes with it; genetic/optimization tests 439 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(devices): evaluate the efficiency curve at the delivered AC load When the battery limits a discharge (low state of charge or its power limit), the conversion ran with the efficiency of the requested AC energy. With a 1000 Wh battery at 10 % SoC, a 1000 W inverter and the curve [(0, .8), (.5, .95), (1, .95)], a request of 500 Wh AC returned 95 Wh AC from the available 100 Wh DC, although the curve gives 82.5 % at that load. The delivered AC energy now solves a = dc * efficiency(a / rated) exactly on the piecewise-linear curve (solve_ac_for_dc_energy), bounded by the request. Unlimited discharges and inverters without a curve are unchanged. Regression tests use a real Battery: low SoC and a power-limited discharge, each with 60- and 15-minute slots, battery discharge losses, the load path through process_energy, and constant efficiency. * docs: regenerate OpenAPI for the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix --------- Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Andreas <drbacke@gmx.de> |
||
|
|
7f26ea3f3f |
feat(devices): add load-dependent DC-to-AC efficiency curve for GENETIC inverters (#1375)
* feat(devices): add load-dependent DC-to-AC efficiency curve for GENETIC inverters Real inverters are much less efficient at low load than at rated power (e.g. 0.80 at 2 % vs. 0.93 at 7 % load). A constant dc_to_ac_efficiency overrates night-time battery discharge into a small base load and, with it, arbitrage and grid charging. Add two optional inverter settings for the GENETIC optimizer: - dc_to_ac_efficiency_curve: list of (load_fraction, efficiency) points, load fraction = AC power / max_power_w. Points must be strictly increasing in [0, 1] with efficiencies in (0, 1]; at least two points. Linear interpolation, clamped at both ends. When set, the curve replaces dc_to_ac_efficiency. - dc_to_ac_efficiency_reference_load_fraction (default 0.06): load at which the curve values stored energy without a specific conversion. Battery discharges in Inverter.process_energy() use the efficiency at the AC energy of each conversion. The end-of-horizon value, the terminal and tail value curves and the AC charge break-even penalty use Inverter.reference_dc_to_ac_efficiency. Without a curve, dc_to_ac_efficiency_at() and the reference efficiency return dc_to_ac_efficiency itself, so results are bit-identical to the previous model. GENETIC0 is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(devices): evaluate the efficiency curve at the delivered AC load When the battery limits a discharge (low state of charge or its power limit), the conversion ran with the efficiency of the requested AC energy. With a 1000 Wh battery at 10 % SoC, a 1000 W inverter and the curve [(0, .8), (.5, .95), (1, .95)], a request of 500 Wh AC returned 95 Wh AC from the available 100 Wh DC, although the curve gives 82.5 % at that load. The delivered AC energy now solves a = dc * efficiency(a / rated) exactly on the piecewise-linear curve (solve_ac_for_dc_energy), bounded by the request. Unlimited discharges and inverters without a curve are unchanged. Regression tests use a real Battery: low SoC and a power-limited discharge, each with 60- and 15-minute slots, battery discharge losses, the load path through process_energy, and constant efficiency. * docs: regenerate OpenAPI for the efficiency-curve fix --------- Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Andreas <drbacke@gmx.de> |
||
|
|
560370281f |
perf(prediction): optional SciPy-free evaluation of the self-consumption tables (#1378)
* perf(prediction): evaluate self-consumption tables without SciPy The GENETIC inverter (and the GENETIC0 one) evaluates a pickled scipy.interpolate.RegularGridInterpolator for the self-consumption probabilities. SciPy is used nowhere else in EOS, yet loading it costs about 35 MB RSS (21 MB anonymous) on Linux arm64, which every optimization run pays. Add utils/gridinterpolator.py: - LinearGridInterpolator: 2-D linear interpolation with the same steps as SciPy's linear 2-D path (interval search, normalised distances, corner weights summed in the same order, fill value outside the grid, NaN for NaN input). - load_grid_interpolator(): reads the shipped pickles with a restricted unpickler that maps the SciPy class to a stand-in and accepts only the NumPy globals needed for the arrays. SciPy is not imported, and the data files no longer depend on SciPy's private module paths. Results are bit-identical to SciPy on Linux (aarch64, checked on 3.6M points per table including the query pattern of the inverter). The macOS arm64 SciPy wheel fuses multiply-add, so there SciPy itself differs by <= 1.1e-16; GENETIC plans for fixed seeds are unchanged on both platforms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(prediction): make the SciPy-free table evaluation opt-in Add optimization.self_consumption_interpolator (scipy | numpy, default scipy). The default keeps evaluating the self-consumption tables with SciPy's RegularGridInterpolator, built from the same grid, values and fill value the pickle holds, so results are unchanged. With numpy the tables are evaluated by LinearGridInterpolator and SciPy is never imported. The backend is read on every evaluation, so a config change applies to the next run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Andreas <drbacke@gmx.de> |
||
|
|
14d8587c9c |
feat(genetic): make the fitness cache size configurable (#1376)
The per-run fitness cache of the GENETIC optimizer is unbounded. It grows with individuals x generations (~110k keys at 300 x 400, about 1.1 kB each, 1.6 kB with an EV) and adds 120-170 MB to the peak memory of a run, while only 6-15 % of the lookups hit. On 512 MB boards this is the largest part of the optimizer's memory. Add optimization.genetic.fitness_cache_max_entries: - None (default): unbounded, the previous behaviour. - 0: no cache; every genome is evaluated. - N > 0: at most N keys. The cache is then an OrderedDict and evicts the oldest key first (FIFO, O(1)); LRU would cost bookkeeping on every hit for a cache that rarely hits. A cache hit returns exactly what the deterministic evaluation would return, so the plan for a fixed seed is identical with any setting. Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Andreas <drbacke@gmx.de> |
||
|
|
ecec39fcd8 |
docs: add a guide for running EOS on small devices (#1379)
Collect the settings that bring EOS from about 290 MB idle / 400-450 MB during a GENETIC run down to about 140 MB, with the measurements behind them: fitness cache, timezone override and SciPy-free self-consumption tables (#1376, #1377, #1378), autosave for SD/eMMC storage, and the glibc allocator and BLAS environment. Includes run times on a Raspberry Pi 4 and an x86 VM with data of a real installation. Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3651d33915 |
perf: import heavy optional dependencies on first use (#1377)
A GENETIC server with external (import) providers and EOSdash disabled loads several large packages at start that it never uses, which costs a noticeable part of the memory on 512 MB boards: - matplotlib via the GENETIC/GENETIC0 PDF report modules (server/eos.py) - statsmodels via the Holt-Winters fallback in prediction/priceabc.py - pvlib via the weather base class and the pvlib-based providers - bs4/lxml via the ClearOutside provider - scipy via the self-consumption interpolator singletons, which were loaded at import time - the GENETIC0 optimizer, imported by core/ems.py - tzfpy, whose polygon data the computed general.timezone loads Import these where they are used. The interpolator tables are loaded on first use (thread-safe). Nothing changes in what is computed. general.timezone is computed from latitude/longitude and cannot be set. Add general.timezone_override (IANA name, validated with pendulum): when set, general.timezone returns it and the lookup is skipped. When unset (default) the lookup is unchanged. Tests that patched the moved names now patch them at their source. A new test starts the server module in a subprocess and checks that none of these modules is loaded. Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
NormannandGitHub
|
6d26252c6a |
build(deps): combine fifteen Dependabot updates and synchronize uv.lock (#1367)
* build(deps): combine pending Dependabot updates * build(deps): add platformdirs and pvlib updates * build(deps): consolidate urllib3 tornado oauthlib and virtualenv updates * build(deps): add five dependency updates * build(deps): bump virtualenv to 21.7.13 |
||
|
NormannandGitHub
|
b7610e8274 |
fix: detect recent Energy-Charts cadence for feed-in coverage (#1362)
* fix: use recent original feed-in cadence for publication coverage * test: cover mixed Energy-Charts feed-in source cadence |
||
|
NormannandGitHub
|
d072fec3f2 |
test: resolve server temporary path aliases before containment checks (#1361)
* test: canonicalize temporary server directory * test: compare resolved server configuration paths * test: check canonical system-test configuration path * test: expose path containment through pytest fixture * test: request containment fixture in server checks * test: use injected fixture in system path check |
||
|
NormannandGitHub
|
090ade790e |
test: isolate electricity price resampling tests from live APIs (#1360)
* test: isolate Akkudoktor price resampling from live API * test: isolate Energy-Charts price resampling from live API |
||
|
NormannandGitHub
|
9cf20c6ef1 |
fix: parse naive date/time strings in to_time's target timezone (#1359)
* fix: interpret naive time date strings in target timezone * test: cover naive and offset time date strings |
||
|
NormannandGitHub
|
7d772abd54 | fix: ignore commit-derived OpenAPI snapshot version suffix (#1358) | ||
|
NormannandGitHub
|
e203a81666 |
fix: surface DVhub feed-in tariff update failures (#1357)
* fix: surface DVhub feed-in tariff update failures * test: annotate DVhub sample records for mypy * fix: require real feed-in tariff coverage in GENETIC0 * test: satisfy GENETIC0 tariff coverage checks * test: patch DVhub storage method on its class |
||
|
NormannandGitHub
|
fca07a0c62 |
fix: show Home Assistant add-on version in EOSdash (#1356)
Align add-on and EOSdash version. |
||
|
NormannandGitHub
|
4fe879ddd2 |
fix: allow EOSdash through an external reverse proxy (#1355)
Support externally proxied EOSdash (#1320) * test: verify optional dashboard port stays unpublished * docs: describe optional proxy access and public URL |
||
|
NormannandGitHub
|
3d2daa694b |
fix(dash): use reachable host in footer API docs link (#1354)
link to EOS API using reachable host |
||
|
|
efab8cd0a0 |
fix(genetic): keep fitness-cache memory within pymalloc and release arena after each run (#1353)
* fix(genetic): keep fitness-cache memory in pymalloc and release arena after each run At fine time resolution (interval_sec=900, ~192 control slots over a multi-day horizon) the fitness-cache keys are ~1.6 KB int tuples, above CPython's 512-byte pymalloc threshold, so they are served by glibc malloc in the optimization worker thread's arena and are not returned to the OS on `self._fitness_cache.clear()`. With re-optimization every 15 min, RSS stair-steps up to the memory limit within about a day (OOM / forced restart). At hourly resolution the tuples stay < 512 B, so pymalloc reclaims them and the effect is negligible. See #1352. - Store the cache key/genome compactly as bytes (1 byte per gene, 8-byte fallback for larger state spaces) so entries stay within pymalloc regardless of resolution. - After each optimization run, gc.collect() + malloc_trim(0) (guarded, glibc-only) to return freed arena pages to the OS. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(genetic): pack fitness-cache genome as bounded chunks Storing the genome as a single bytes object still exceeds pymalloc's 512-byte threshold once the genome grows: at 15-min resolution over a 60 h horizon with EV genes the key is ~480 genes, so even the one-byte encoding is 481 bytes (514 with the object header) and any value >255 switches the whole genome to 8 bytes per gene. Such keys land in glibc malloc, which does not reliably return the pages (malloc_trim is glibc-only, absent on musl) — the platform-independent guarantee did not actually hold for supported settings. Encode the genome (key and FitnessCacheEntry.genome) as a tuple of bounded byte chunks instead — 256 one-byte genes or 32 signed-64-bit genes per chunk, 256 bytes each — built per chunk so the encoder never materialises an oversized temporary. Every object then stays inside pymalloc regardless of horizon, on every platform. malloc_trim after a run is kept as a secondary release for the rest of the run's heap. Tests: round-trips incl. 480-gene narrow/wide and negative genes, and sys.getsizeof for the key AND every chunk at 480 genes in both encodings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
NormannandGitHub
|
986b4eb0d4 |
build(deps): combine eleven Dependabot updates and synchronize uv.lock (#1340)
* build(deps): consolidate Dependabot updates with synchronized uv lock * build(deps): add four pending Dependabot updates and refresh uv lock * build(deps): add platformdirs and Commitizen updates |
||
 Hoffis-WeltandGitHub
|
38915a620b |
fix(docker): bind EOSdash to 0.0.0.0 so HA ingress can reach it (#1346)
server.eosdash_host defaults to 127.0.0.1 (not None), so the fallback in run_eos() that inherits server.host never triggers. The --host 0.0.0.0 CLI argument only sets server.host. As a result EOSdash binds to 127.0.0.1 inside the container and the Home Assistant Supervisor ingress (ingress_port 8504) cannot reach it (502). Set EOS_SERVER__EOSDASH_HOST=0.0.0.0 in the image. |