mirror of
https://github.com/Akkudoktor-EOS/EOS.git
synced 2026-10-13 01:21:58 +00:00
7dc1ee92fab26a378cc0af906c31f4611fabf45b
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
    
|
7dc1ee92fa |
perf(genetic): faster runs with unchanged results (#1385)
* feat(devices): add load-dependent DC-to-AC efficiency curve for GENETIC inverters Real inverters are much less efficient at low load than at rated power (e.g. 0.80 at 2 % vs. 0.93 at 7 % load). A constant dc_to_ac_efficiency overrates night-time battery discharge into a small base load and, with it, arbitrage and grid charging. Add two optional inverter settings for the GENETIC optimizer: - dc_to_ac_efficiency_curve: list of (load_fraction, efficiency) points, load fraction = AC power / max_power_w. Points must be strictly increasing in [0, 1] with efficiencies in (0, 1]; at least two points. Linear interpolation, clamped at both ends. When set, the curve replaces dc_to_ac_efficiency. - dc_to_ac_efficiency_reference_load_fraction (default 0.06): load at which the curve values stored energy without a specific conversion. Battery discharges in Inverter.process_energy() use the efficiency at the AC energy of each conversion. The end-of-horizon value, the terminal and tail value curves and the AC charge break-even penalty use Inverter.reference_dc_to_ac_efficiency. Without a curve, dc_to_ac_efficiency_at() and the reference efficiency return dc_to_ac_efficiency itself, so results are bit-identical to the previous model. GENETIC0 is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): make the fitness cache size configurable The per-run fitness cache of the GENETIC optimizer is unbounded. It grows with individuals x generations (~110k keys at 300 x 400, about 1.1 kB each, 1.6 kB with an EV) and adds 120-170 MB to the peak memory of a run, while only 6-15 % of the lookups hit. On 512 MB boards this is the largest part of the optimizer's memory. Add optimization.genetic.fitness_cache_max_entries: - None (default): unbounded, the previous behaviour. - 0: no cache; every genome is evaluated. - N > 0: at most N keys. The cache is then an OrderedDict and evicts the oldest key first (FIFO, O(1)); LRU would cost bookkeeping on every hit for a cache that rarely hits. A cache hit returns exactly what the deterministic evaluation would return, so the plan for a fixed seed is identical with any setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf: import heavy optional dependencies on first use A GENETIC server with external (import) providers and EOSdash disabled loads several large packages at start that it never uses, which costs a noticeable part of the memory on 512 MB boards: - matplotlib via the GENETIC/GENETIC0 PDF report modules (server/eos.py) - statsmodels via the Holt-Winters fallback in prediction/priceabc.py - pvlib via the weather base class and the pvlib-based providers - bs4/lxml via the ClearOutside provider - scipy via the self-consumption interpolator singletons, which were loaded at import time - the GENETIC0 optimizer, imported by core/ems.py - tzfpy, whose polygon data the computed general.timezone loads Import these where they are used. The interpolator tables are loaded on first use (thread-safe). Nothing changes in what is computed. general.timezone is computed from latitude/longitude and cannot be set. Add general.timezone_override (IANA name, validated with pendulum): when set, general.timezone returns it and the lookup is skipped. When unset (default) the lookup is unchanged. Tests that patched the moved names now patch them at their source. A new test starts the server module in a subprocess and checks that none of these modules is loaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(prediction): evaluate self-consumption tables without SciPy The GENETIC inverter (and the GENETIC0 one) evaluates a pickled scipy.interpolate.RegularGridInterpolator for the self-consumption probabilities. SciPy is used nowhere else in EOS, yet loading it costs about 35 MB RSS (21 MB anonymous) on Linux arm64, which every optimization run pays. Add utils/gridinterpolator.py: - LinearGridInterpolator: 2-D linear interpolation with the same steps as SciPy's linear 2-D path (interval search, normalised distances, corner weights summed in the same order, fill value outside the grid, NaN for NaN input). - load_grid_interpolator(): reads the shipped pickles with a restricted unpickler that maps the SciPy class to a stand-in and accepts only the NumPy globals needed for the arrays. SciPy is not imported, and the data files no longer depend on SciPy's private module paths. Results are bit-identical to SciPy on Linux (aarch64, checked on 3.6M points per table including the query pattern of the inverter). The macOS arm64 SciPy wheel fuses multiply-add, so there SciPy itself differs by <= 1.1e-16; GENETIC plans for fixed seeds are unchanged on both platforms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(prediction): make the SciPy-free table evaluation opt-in Add optimization.self_consumption_interpolator (scipy | numpy, default scipy). The default keeps evaluating the self-consumption tables with SciPy's RegularGridInterpolator, built from the same grid, values and fill value the pickle holds, so results are unchanged. With numpy the tables are evaluated by LinearGridInterpolator and SciPy is never imported. The backend is read on every evaluation, so a config change applies to the next run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(genetic): cache the load distribution, clone genomes directly Two hot spots of a GENETIC run, found by profiling a 15 minute run with EV and two appliances. The solution is unchanged: both repository reference inputs give a byte-identical GeneticSolution at 400 generations. - SelfConsumptionProbabilityInterpolator._load_distribution depends on the mean load only, but ran the table interpolation over all minute-load bins for every new (load, PV) pair - 27 % of the run. It is now kept per (backend, bounded mean load); a run asks for few loads (one per slot, plus appliance combinations) with ever-changing PV values. The table is static, so entries never go stale; the cache is bounded. Scalar bounds use min/max instead of np.clip. - _make_offspring cloned parents with copy.deepcopy (8 % of the run), although the child's fitness, extra_data and immigrant protection are dropped right after. A genome is a flat list of ints, so the child is created from the parent's genes directly. Measured (1 process, Apple M5): eHive run with EV + 2 appliances, 200 generations 33.7 s -> 25.1 s; EV only 34.1 s -> 24.3 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): cheaper slot simulation, lean lookup cache, batch duplicates Four hot spots of a GENETIC run, found by profiling a 15 minute run with EV and two appliances. The solution is unchanged: both repository reference inputs (optimize_input_1/2, seed 42, 400 generations) give a byte-identical solution before and after, as do the fixed-seed runs with 1, 2 and 3 evaluation processes. * Expected direct consumption: the lookup went through the shared cache_energy_management store on every slot of every candidate - a key built from the callable, a callback per call and room for the last 100 results only (18 % misses). It is now a dictionary of the interpolator, dropped together with that store at the start of an energy management run (CacheEnergyManagementStore.generation), bounded to 8192 entries. * Slot simulation: per-slot inputs are read as plain Python numbers and per-slot results are collected in lists, turned into arrays after the loop. Element access on NumPy arrays and NumPy scalar arithmetic were the largest share of the loop. process_energy() skips the charge/export block when a slot has no PV surplus and bounds values without min()/max() calls (exact equivalents, NaN and signed zero included). * Genome keys: bytes() does the 0-255 range check in C, and a genome that is relevant as a whole is packed without copying it first. * Identical genomes within a generation are evaluated once also when the fitness cache is off (fitness_cache_max_entries = 0), by grouping the batch. It needs no memory beyond the batch; the cache costs 85-95 MB. Same run, one process, 60 generations (Apple M-series): 7.8 s -> 5.9 s with appliances, 7.1 s -> 5.0 s without. 200 generations with 3 evaluation processes: 11.4 s -> 8.2 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): keep the battery charge bookkeeping in plain numbers Battery.charge_energy() runs for every slot of every candidate plan. The per-slot charge limit and charged energy were NumPy arrays read and written one element at a time, and the NumPy scalars they return made the state of charge arithmetic slow from then on. They are plain lists now. The lower charge rates tried when a request does not fit are taken from a tuple built once per charge_rates array instead of a searchsorted() plus generator per request. The results are unchanged (same IEEE doubles, same order of operations). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): check the discharge release first, plain discharge bookkeeping Inverter._discharge_battery_to_ac() looked up the conversion efficiency before the battery told it that the plan does not release discharge in this slot. Battery.discharge_released() answers that first; the result of such a slot was (0.0, 0.0) before and is now returned directly. The discharged energy per slot is a plain list instead of a NumPy array read and written one element at a time. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): find the efficiency curve segment by bisection Inverter.dc_to_ac_efficiency_at() runs for every battery discharge of every candidate plan and walked the curve points one by one. The load fractions are split off once per curve and the segment is found by bisection; the segment and the interpolation are those of interpolate_efficiency_curve(). Compared bit for bit against that function for 200000 load values, the curve points themselves, zero, infinities and NaN. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): sum appliance loads once per candidate, read the SoC directly The slot loop asked every appliance for its load in every slot (one method call with a range check each) and converted the state of charge to percent through a method call twice per slot. The aggregate appliance load is now summed once per candidate, in the order the loop added the appliances, and the state of charge is read with the Battery formula in place. Appliances of another class or with a curve that does not cover the simulated slots, and batteries with their own current_soc_percentage(), keep the calls. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): compute the control end slot once per run control_end_slot is a property that reads the interval and the horizon from the configuration on every access - several hundred thousand times per run. The configuration is frozen for the duration of a run, so the value is computed once when the snapshot is taken and dropped with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): work out the energy-shift targets once per source slot _energy_shift_target_slots() runs for every export slot of every energy-shift mutation, in the server process while the evaluation workers wait. Which later slots are worth the shift, and in which order, depends on the forecasts and the source slot only; it is now worked out once per source slot and run, and the individual only filters the ordered slots. Filtering the sorted superset gives the same list as sorting the filtered slots (the sort is stable and its key does not depend on the individual), so mutations and the random number stream are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): compute the battery discharge of a slot in one go Serving load or exporting from the battery went through three nested calls per slot: Inverter._discharge_battery_to_ac() -> dc_to_ac_efficiency_at() -> Battery.discharge_energy(), with the release of the slot checked twice and the efficiency function called also without a curve. For the plain Battery the inverter now computes the release check, the efficiency and the discharge in one go; other battery implementations keep the step-by-step path. The arithmetic is that of Battery.discharge_energy(). The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): decode appliance starts once per gene combination Every candidate decoded its appliance start genes (feasibility repair over cycle windows and minimum gaps) and rebuilt each appliance's load curve over the whole horizon. The decoded starts, the canonical gene values and the curves depend on the appliance genes only, and a run sees few different combinations among tens of thousands of candidates. Each combination is now decoded and built once per layout (bounded to 4096 combinations) and looked up afterwards. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): compute only the terminal credit per candidate Every evaluation called _terminal_value(), which returns the credit for the energy left in the battery together with a TerminalValueResult report (component values, diagnostics, curves). The evaluation used the credit and dropped the report. _terminal_value_credit() computes the same credit without building the report; the report is still built for the final solution. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): decode battery states through a lookup table decode_charge_discharge() decodes every slot on its own from its state, with about twenty array operations per candidate. The four results per state are now worked out once per state layout and looked up for the slots of a candidate. States outside the table keep the explicit decoding. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): check for a full EV before splitting the genome _repair_ev_charge_at_full_soc() split every candidate's genome and walked all slots in Python to find EV charging in slots that begin at full SoC. Nearly no candidate has such a slot. The full slots are now found on the simulated SoC array first; the genome is split and only those slots are looked at when there are any. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * feat(genetic): evaluate candidates in parallel worker processes A GENETIC run evaluated every candidate in the server process, so a run used one CPU core however many the machine had. On small ARM boards (4x Cortex-A55, thermally limited) a 300x200 run took 12-15 minutes. Candidates are now evaluated in up to two worker processes, forked once per run after the run is fully prepared. Each worker owns a copy of the prepared optimizer; only genomes and results cross the process boundary. The fitness cache stays in the server process: hits are answered there, and identical genomes in a batch are evaluated once, as a serial run would hit the cache for the repeats. Results are applied only after the whole batch returned. If a batch fails or times out, the run continues serially. For a fixed seed the result is identical to a serial run. optimization.genetic.workers sets the number of processes. Unset (default) is automatic: one CPU core always stays free and at most two are used (1-2 cores -> 1, 3 or more -> 2), capped by a cgroup v2 CPU limit of the process or any parent (container limit, systemd CPUQuota). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * feat(ems): notify a URL after every completed optimization A client that consumes EOS plans (here DVhub) had to poll /v1/energy-management/optimization/solution to notice a new plan. That delays the plan by up to the poll interval and makes EOS serialize the complete solution on every poll, also while it is computing. ems.notify_url (default None = off) receives an HTTP POST with a small JSON event ({"event": "optimization_completed", "algorithm", "completed_at", "duration_s"}) when a run has published its solution. The POST runs in a daemon thread with a 5 s timeout, so a slow or unreachable receiver never delays energy management; a failure is logged once per failure streak. Only http(s) URLs are accepted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): fewer calls per slot in the simulation loop Three calls ran for every slot of every candidate although most slots do not need them: the AC charge factor was capped by the inverter also when the slot has no AC charging, the battery was asked for its discharged energy through a method, and process_energy() got its export arguments by keyword. Slots without AC charging skip the cap, the discharged energy is read with the Battery formula in place, and the arguments are positional. reset() of a battery creates its two control arrays with zeros() instead of full(). The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(genetic): freeze the optimization configuration for the duration of a run The genetic optimization reads horizon_hours, tail_horizon_hours, interval_sec and prediction.hours from the live configuration on every access (control_slots, tail_slots, ... are properties). A client that updates the configuration while a run is in progress - DVhub shrinks or grows the control horizon as the known price horizon moves during the night - changed these values mid-run. Genomes and device arrays sized at the start then no longer matched the simulation loop and the final evaluation failed: IndexError: index 84 is out of bounds for axis 0 with size 84 Long runs on slow hardware hit this almost every night. optimize_ems() now takes a deep copy of the genetic configuration and the prediction hours at the start of a run, all reads in genetic.py go through it, and it is released when the run ends (also on errors). New values apply to the next run. Tests: tests/test_genetic_config_frozen_per_run.py reproduces the error without the fix (horizon 20 -> 24 mid-run: index 20 out of bounds) and passes with it; genetic/optimization tests 439 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): pin evaluation workers, keep the server responsive Workers are pinned to the last `workers` CPU cores (new setting optimization.genetic.pin_workers, default on); the server process stays unpinned. The run never spreads beyond `workers` cores without a CPU quota, and the server and the rest of the system keep the other cores. No pinning without a spare core. With a CPU limit (container cpu.max, systemd CPUQuota) the automatic worker count now keeps a quarter core of it for the server process (limit 2 -> 1 worker, 2.5 -> 2): two workers used a 2-core limit fully. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(ems): tolerate settings without notify_url Test doubles and older configuration objects have no ems.notify_url. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): bound the PV surplus and export values without min()/max() The rest of Inverter.process_energy() - PV surplus, curtailment and battery export - bounded its values with min()/max() calls. They are conditional statements now, as the first half of the method already is: 'if x < 0.0: x = 0.0' is max(x, 0.0) and 'b if b < a else a' is min(a, b), NaN and signed zero included. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(devices): evaluate the efficiency curve at the delivered AC load When the battery limits a discharge (low state of charge or its power limit), the conversion ran with the efficiency of the requested AC energy. With a 1000 Wh battery at 10 % SoC, a 1000 W inverter and the curve [(0, .8), (.5, .95), (1, .95)], a request of 500 Wh AC returned 95 Wh AC from the available 100 Wh DC, although the curve gives 82.5 % at that load. The delivered AC energy now solves a = dc * efficiency(a / rated) exactly on the piecewise-linear curve (solve_ac_for_dc_energy), bounded by the request. Unlimited discharges and inverters without a curve are unchanged. Regression tests use a real Battery: low SoC and a power-limited discharge, each with 60- and 15-minute slots, battery discharge losses, the load path through process_energy, and constant efficiency. * docs: regenerate OpenAPI for the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix --------- Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Andreas <drbacke@gmx.de> |
||
   
|
3c862543a1 |
feat(devices): add bounded slot physics for GENETIC (#1327)
* feat: adapt configuration for multi optimization algorithms
Decouple configuration from optimization algorithm parameters. Add to_[algorithm]_param() methods
to the configuration that derive optimization algorithm specific parameters from the configuration.
Add x-scope tags to the configuration options that describe for which specific algorithms the
configuration option is for.
The whole device settings are restructured. There are now general settings for the device classes
with the afore mentioned to_[algorithm]_param() methods. The general device settings got their own
directory `devices/settings`. By this the parameter class also does not have to be a pydantic model
which can be used for future optimization/ simulations speed up.
Also the parameter class for a device is now part of the device module. This better decouples and
also is the natural place for parameters of a device.
Besides this feature there are also fixes and improvements:
* feat: extend home appliance time window settings and simulation
Home appliance can now be configured for multiple runs with per-cycle allowed time windows. The
number of remaining cycles to plan is determined at runtime by reading the
``cycles_completed_measurement_key`` from the measurement store.
* feat: specialiced CycleTimeWindowSequence for time window sequences
Sequence of time windows associated to cycles.
This model specializes ``ValueTimeWindowSequence`` so that the ``value``
field of each ``ValueTimeWindow`` encodes the **cycle index** (0-based
integer) the window belongs to.
Typical use: an appliance that must run ``n`` times per day, each run
constrained to a distinct time window. Assign ``value=0`` to windows
for the first cycle, ``value=1`` for the second, and so on. Multiple
windows may share the same cycle index (their allowed regions are unioned).
Windows with ``value=None`` are silently ignored by all cycle-aware methods.
* fix: Make test_configmigrate also regard the _ANY_SENTENIEL in key values
* chore: Make devices configurations a map instead of a list
This makes config paths stable regardless of declaration order and lets each device settings
class build its own config path from ``self.device_id`` without needing an external index.
Tests are adapted likewise.
Devices configurations are automatically migrated from lists to maps.
* chore: rename levelized_cost_of_storage_kwh to levelized_cost_of_storage_amt kwh
This better fits in the naming scheme and also makes clear the costs are money.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* fix: runtime config update ignored by config file
Runtime settings were handed back to pydantic-settings as init settings,
which rank below the config file and the environment. Any key already
present in EOS.config.json or in the environment silently discarded the
update, so a bulk PUT /v1/config returned 200 without applying anything,
while the granular PUT /v1/config/{path} endpoint kept working.
Add a dedicated runtime settings source ranked directly below the command
line arguments and record granular updates there as well, so both
endpoints share one store that survives re-evaluation of the settings
sources. Environment variables keep precedence over the config file for
all keys that were not set at runtime.
Also repairs revert_settings() and update(), which passed their data
through the same init settings.
Closes #1303
* fix: env vars ignored on first config build
ConfigEOS.__init__ passed self as first positional argument to _setup,
which forwards it to pydantic_settings.BaseSettings.__init__. Its first
positional parameter is _case_sensitive, so the environment source
matched the upper case variable names against the lower case field names
and returned nothing. Environment settings only took effect after the
next configuration setup.
* docs: changelog for config priority fixes
* fix(config): preserve device identities and storage costs during migration
* fix(devices): preserve charge-rate typing and public import compatibility
* ruff format fix
* feat(devices): port slot-aware battery export and direct-use physics
Port scoped device changes from
|