* perf(prediction): evaluate self-consumption tables without SciPy
The GENETIC inverter (and the GENETIC0 one) evaluates a pickled
scipy.interpolate.RegularGridInterpolator for the self-consumption
probabilities. SciPy is used nowhere else in EOS, yet loading it costs
about 35 MB RSS (21 MB anonymous) on Linux arm64, which every
optimization run pays.
Add utils/gridinterpolator.py:
- LinearGridInterpolator: 2-D linear interpolation with the same steps
as SciPy's linear 2-D path (interval search, normalised distances,
corner weights summed in the same order, fill value outside the grid,
NaN for NaN input).
- load_grid_interpolator(): reads the shipped pickles with a restricted
unpickler that maps the SciPy class to a stand-in and accepts only
the NumPy globals needed for the arrays. SciPy is not imported, and
the data files no longer depend on SciPy's private module paths.
Results are bit-identical to SciPy on Linux (aarch64, checked on 3.6M
points per table including the query pattern of the inverter). The
macOS arm64 SciPy wheel fuses multiply-add, so there SciPy itself
differs by <= 1.1e-16; GENETIC plans for fixed seeds are unchanged on
both platforms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(prediction): make the SciPy-free table evaluation opt-in
Add optimization.self_consumption_interpolator (scipy | numpy, default scipy).
The default keeps evaluating the self-consumption tables with SciPy's
RegularGridInterpolator, built from the same grid, values and fill value the
pickle holds, so results are unchanged. With numpy the tables are evaluated by
LinearGridInterpolator and SciPy is never imported. The backend is read on
every evaluation, so a config change applies to the next run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Andreas <drbacke@gmx.de>
The per-run fitness cache of the GENETIC optimizer is unbounded. It
grows with individuals x generations (~110k keys at 300 x 400, about
1.1 kB each, 1.6 kB with an EV) and adds 120-170 MB to the peak memory
of a run, while only 6-15 % of the lookups hit. On 512 MB boards this
is the largest part of the optimizer's memory.
Add optimization.genetic.fitness_cache_max_entries:
- None (default): unbounded, the previous behaviour.
- 0: no cache; every genome is evaluated.
- N > 0: at most N keys. The cache is then an OrderedDict and evicts
the oldest key first (FIFO, O(1)); LRU would cost bookkeeping on
every hit for a cache that rarely hits.
A cache hit returns exactly what the deterministic evaluation would
return, so the plan for a fixed seed is identical with any setting.
Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Andreas <drbacke@gmx.de>
Collect the settings that bring EOS from about 290 MB idle / 400-450 MB
during a GENETIC run down to about 140 MB, with the measurements behind
them: fitness cache, timezone override and SciPy-free self-consumption
tables (#1376, #1377, #1378), autosave for SD/eMMC storage, and the glibc
allocator and BLAS environment. Includes run times on a Raspberry Pi 4 and
an x86 VM with data of a real installation.
Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A GENETIC server with external (import) providers and EOSdash disabled
loads several large packages at start that it never uses, which costs
a noticeable part of the memory on 512 MB boards:
- matplotlib via the GENETIC/GENETIC0 PDF report modules (server/eos.py)
- statsmodels via the Holt-Winters fallback in prediction/priceabc.py
- pvlib via the weather base class and the pvlib-based providers
- bs4/lxml via the ClearOutside provider
- scipy via the self-consumption interpolator singletons, which were
loaded at import time
- the GENETIC0 optimizer, imported by core/ems.py
- tzfpy, whose polygon data the computed general.timezone loads
Import these where they are used. The interpolator tables are loaded
on first use (thread-safe). Nothing changes in what is computed.
general.timezone is computed from latitude/longitude and cannot be set.
Add general.timezone_override (IANA name, validated with pendulum):
when set, general.timezone returns it and the lookup is skipped. When
unset (default) the lookup is unchanged.
Tests that patched the moved names now patch them at their source.
A new test starts the server module in a subprocess and checks that
none of these modules is loaded.
Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(genetic): keep fitness-cache memory in pymalloc and release arena after each run
At fine time resolution (interval_sec=900, ~192 control slots over a multi-day
horizon) the fitness-cache keys are ~1.6 KB int tuples, above CPython's 512-byte
pymalloc threshold, so they are served by glibc malloc in the optimization worker
thread's arena and are not returned to the OS on `self._fitness_cache.clear()`.
With re-optimization every 15 min, RSS stair-steps up to the memory limit within
about a day (OOM / forced restart). At hourly resolution the tuples stay < 512 B,
so pymalloc reclaims them and the effect is negligible. See #1352.
- Store the cache key/genome compactly as bytes (1 byte per gene, 8-byte fallback
for larger state spaces) so entries stay within pymalloc regardless of resolution.
- After each optimization run, gc.collect() + malloc_trim(0) (guarded, glibc-only)
to return freed arena pages to the OS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(genetic): pack fitness-cache genome as bounded chunks
Storing the genome as a single bytes object still exceeds pymalloc's
512-byte threshold once the genome grows: at 15-min resolution over a
60 h horizon with EV genes the key is ~480 genes, so even the one-byte
encoding is 481 bytes (514 with the object header) and any value >255
switches the whole genome to 8 bytes per gene. Such keys land in glibc
malloc, which does not reliably return the pages (malloc_trim is
glibc-only, absent on musl) — the platform-independent guarantee did
not actually hold for supported settings.
Encode the genome (key and FitnessCacheEntry.genome) as a tuple of
bounded byte chunks instead — 256 one-byte genes or 32 signed-64-bit
genes per chunk, 256 bytes each — built per chunk so the encoder never
materialises an oversized temporary. Every object then stays inside
pymalloc regardless of horizon, on every platform. malloc_trim after a
run is kept as a secondary release for the rest of the run's heap.
Tests: round-trips incl. 480-gene narrow/wide and negative genes, and
sys.getsizeof for the key AND every chunk at 480 genes in both
encodings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
server.eosdash_host defaults to 127.0.0.1 (not None), so the fallback in run_eos() that inherits server.host never triggers. The --host 0.0.0.0 CLI argument only sets server.host. As a result EOSdash binds to 127.0.0.1 inside the container and the Home Assistant Supervisor ingress (ingress_port 8504) cannot reach it (502). Set EOS_SERVER__EOSDASH_HOST=0.0.0.0 in the image.
* fix(optimization): let the AC setpoint cap the total charge where the inverter does
Some hybrid inverters limit the battery's whole charge current to the grid
charge setpoint while grid charging is on. A Deye 12K in time-of-use grid
charging with max_grid_charge_current = 75 A charges at ~80 A even with 8 kW
of PV, and exports the rest - including the surplus of micro inverters on
the grid side. GENETIC modelled an AC slot as "PV surplus first, grid adds
ac_charge x the remaining charge power", so under PV surplus an AC slot
looked at least as good as a DC slot and the optimizer picked a low AC
factor, while the real system exported what the plan meant to store.
New inverter config option devices.inverters[].ac_charge_limits_total_charge
(GENETIC scope, default False, so the existing model is unchanged). When
True, an AC slot caps the battery's raw charge at ac_charge x
max_charge_power_w from all sources: PV surplus above the cap is exported,
the grid only fills what PV leaves of it. The option reaches the optimizer
through InverterCommonSettings.to_genetic_param, i.e. POST /v1/optimize and
the EMS run. GENETIC0 (POST /optimize) is left as it is.
The AC slot logic moves into Inverter (ac_charge_factor,
begin_ac_charge_slot, charge_battery_from_grid) so the simulation and the
tail value curve share it; Battery gets a per-slot charge cap that reset()
lifts again.
* test(optimization): narrow optional simulation arrays for mypy
The annotated fixture indexes GeneticSimulation arrays typed as Optional; assert them first so the locked mypy hook passes.
* docs: explain energy planner upgrade and compatibility changes
* docs: add upgrade guide to documentation navigation
* docs: resolve changelog upgrade link in generated site
* feat: adapt configuration for multi optimization algorithms
Decouple configuration from optimization algorithm parameters. Add to_[algorithm]_param() methods
to the configuration that derive optimization algorithm specific parameters from the configuration.
Add x-scope tags to the configuration options that describe for which specific algorithms the
configuration option is for.
The whole device settings are restructured. There are now general settings for the device classes
with the afore mentioned to_[algorithm]_param() methods. The general device settings got their own
directory `devices/settings`. By this the parameter class also does not have to be a pydantic model
which can be used for future optimization/ simulations speed up.
Also the parameter class for a device is now part of the device module. This better decouples and
also is the natural place for parameters of a device.
Besides this feature there are also fixes and improvements:
* feat: extend home appliance time window settings and simulation
Home appliance can now be configured for multiple runs with per-cycle allowed time windows. The
number of remaining cycles to plan is determined at runtime by reading the
``cycles_completed_measurement_key`` from the measurement store.
* feat: specialiced CycleTimeWindowSequence for time window sequences
Sequence of time windows associated to cycles.
This model specializes ``ValueTimeWindowSequence`` so that the ``value``
field of each ``ValueTimeWindow`` encodes the **cycle index** (0-based
integer) the window belongs to.
Typical use: an appliance that must run ``n`` times per day, each run
constrained to a distinct time window. Assign ``value=0`` to windows
for the first cycle, ``value=1`` for the second, and so on. Multiple
windows may share the same cycle index (their allowed regions are unioned).
Windows with ``value=None`` are silently ignored by all cycle-aware methods.
* fix: Make test_configmigrate also regard the _ANY_SENTENIEL in key values
* chore: Make devices configurations a map instead of a list
This makes config paths stable regardless of declaration order and lets each device settings
class build its own config path from ``self.device_id`` without needing an external index.
Tests are adapted likewise.
Devices configurations are automatically migrated from lists to maps.
* chore: rename levelized_cost_of_storage_kwh to levelized_cost_of_storage_amt kwh
This better fits in the naming scheme and also makes clear the costs are money.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* fix: runtime config update ignored by config file
Runtime settings were handed back to pydantic-settings as init settings,
which rank below the config file and the environment. Any key already
present in EOS.config.json or in the environment silently discarded the
update, so a bulk PUT /v1/config returned 200 without applying anything,
while the granular PUT /v1/config/{path} endpoint kept working.
Add a dedicated runtime settings source ranked directly below the command
line arguments and record granular updates there as well, so both
endpoints share one store that survives re-evaluation of the settings
sources. Environment variables keep precedence over the config file for
all keys that were not set at runtime.
Also repairs revert_settings() and update(), which passed their data
through the same init settings.
Closes#1303
* fix: env vars ignored on first config build
ConfigEOS.__init__ passed self as first positional argument to _setup,
which forwards it to pydantic_settings.BaseSettings.__init__. Its first
positional parameter is _case_sensitive, so the environment source
matched the upper case variable names against the lower case field names
and returned nothing. Environment settings only took effect after the
next configuration setup.
* docs: changelog for config priority fixes
* fix(config): preserve device identities and storage costs during migration
* fix(measurement): restore JSON records into the existing singleton
* fix(devices): preserve charge-rate typing and public import compatibility
* feat(measurement): integrate typed energy quality and capacity APIs
Port the locally backed-up measurement extensions to main async storage and PR #1256 device maps. Preserve runtime capacity estimates across #1305 bulk updates. Confirm JSON singleton restore defect on unchanged main and add regression. No production configuration or measurements included.
Co-authored-by: Andreas <drbacke@gmx.de>
* docs(measurement): describe household settings and consolidate regression coverage
* docs(measurement): regenerate configuration and API contracts
* test(measurement): isolate capacity database state between tests
* ruff format fix
* fix(measurement): restore JSON records into the existing singleton
* test(measurement): assert restored timestamps before timezone conversion
* test(measurement): assert restored timestamps before timezone conversion
* fix: preserve imported feed-in revenue during parameter preparation
Cancel GENETIC preparation when imported revenue cannot be read or contains invalid values, preserving the chosen provider instead of replacing it with demo tariffs. Keep valid positive, zero and negative amount/Wh series unchanged.
Adapt the revenue-preservation regressions from PRs #1224 and #1304 to the async main API, including real timestamped imports and simulation repricing. The feature-only direct-marketing override remains outside this main fix.
Co-authored-by: Christin <info@bikinibottom.capital>
Co-authored-by: Normann <github@koldrack.com>
* feat(devices): port slot-aware battery export and direct-use physics
Port scoped device changes from d2e2d58237. Keep PR #1256 parameter conversion structure and separate GENETIC0 devices. Validate physical flows and reprice changed simulation results.
Co-authored-by: Andreas <drbacke@gmx.de>
Co-authored-by: Christin <info@bikinibottom.capital>
* docs(measurement): align API version with refreshed prerequisites
* fix: return only completed optimization results per run
* feat(pvforecast): add calibrated local Akkudoktor backend
Port local PV modeling and outage calibration from feature commits f976335, 6dc58c3 and faed0fd by Andreas. Keep PVForecastAkkudoktor identity and remote default, adapt to async storage, and migrate legacy provider settings.
* fix(cache): distinguish callables in the shared EMS cache
Include the function object in cache keys so methods of one interpolator cannot reuse a probability as a power value. Cover both call orders, keyword arguments, cache hits and separate closures with identical qualified names.
* fix(devices): constrain the physics port and validate export levels
Defer inactive EV deadline fields to the optimizer port, reject nonfinite export rates, and document the hourly Optimize boundary. Verify converter IDs, rates and LCOS, separate GENETIC0 interpolation, physical boundary flows and independent GENETIC repricing.
* docs(pvforecast): regenerate local backend configuration schema
* docs(devices): regenerate slot-physics configuration and OpenAPI schemas
* test: type dynamic Optimize regression arguments
* style: wrap imported tariff test parameter import
* style(pvforecast): apply CI import formatting
* docs(pvforecast): refresh API version after CI formatting
* fix(config): satisfy typed device conversion and migration contracts
* docs(config): refresh validated configuration prerequisite schemas
* fix(measurement): enforce typed capacity and sample validation
* test(devices): align physics regressions with strict type checking
* style(measurement): normalize imports for CI
* docs(measurement): refresh typed measurement API schemas
* docs(devices): refresh API version after prerequisite merge
* test: make optimization dispatch timezones explicit
* docs(interpolator): use portable reStructuredText markup
* docs(devices): refresh API version after docstring compatibility fix
* feat: complete configuration-driven GENETIC optimization and reports (#1329)
* feat(devices): port slot-aware battery export and direct-use physics
Port scoped device changes from d2e2d58237. Keep PR #1256 parameter conversion structure and separate GENETIC0 devices. Validate physical flows and reprice changed simulation results.
Co-authored-by: Andreas <drbacke@gmx.de>
Co-authored-by: Christin <info@bikinibottom.capital>
* feat(optimization): port tested terminal and tail value primitives
Source d2e2d58237. 22 primitive tests pass; integration with the optimizer, forecast horizon and API is still pending.
Co-authored-by: Andreas <drbacke@gmx.de>
Co-authored-by: Christin <info@bikinibottom.capital>
* fix(devices): preserve charge-rate typing and public import compatibility
* feat(measurement): integrate typed energy quality and capacity APIs
Port the locally backed-up measurement extensions to main async storage and PR #1256 device maps. Preserve runtime capacity estimates across #1305 bulk updates. Confirm JSON singleton restore defect on unchanged main and add regression. No production configuration or measurements included.
Co-authored-by: Andreas <drbacke@gmx.de>
* test(integration): validate optimizer economics and document measurement settings
* docs(integration): record tested checkpoint and remaining consolidation work
* docs(development): define isolated PR packages and remaining porting gates
* docs(integration): refresh API version after measurement reconciliation
* docs(integration): record PR readiness verification results
* docs(development): record publication and verification of PR 1322
* test(measurement): assert restored timestamps before timezone conversion
* docs(development): record corrected PR head and CI progress
* docs(integration): refresh API version after prerequisite alignment
* docs(integration): define parallel packages and Optimize compatibility gates
* fix: preserve imported feed-in revenue during parameter preparation
Cancel GENETIC preparation when imported revenue cannot be read or contains invalid values, preserving the chosen provider instead of replacing it with demo tariffs. Keep valid positive, zero and negative amount/Wh series unchanged.
Adapt the revenue-preservation regressions from PRs #1224 and #1304 to the async main API, including real timestamped imports and simulation repricing. The feature-only direct-marketing override remains outside this main fix.
Co-authored-by: Christin <info@bikinibottom.capital>
Co-authored-by: Normann <github@koldrack.com>
* test(integration): verify tariff protection with mapped device physics
* fix: return only completed optimization results per run
* test(integration): verify algorithm aliases and mapped-device contracts
* fix(cache): distinguish callables in the shared EMS cache
Include the function object in cache keys so methods of one interpolator cannot reuse a probability as a power value. Cover both call orders, keyword arguments, cache hits and separate closures with identical qualified names.
* fix(devices): constrain the physics port and validate export levels
Defer inactive EV deadline fields to the optimizer port, reject nonfinite export rates, and document the hourly Optimize boundary. Verify converter IDs, rates and LCOS, separate GENETIC0 interpolation, physical boundary flows and independent GENETIC repricing.
* feat(pvforecast): add calibrated local Akkudoktor backend
Port local PV modeling and outage calibration from feature commits f976335, 6dc58c3 and faed0fd by Andreas. Keep PVForecastAkkudoktor identity and remote default, adapt to async storage, and migrate legacy provider settings.
* docs(integration): record combined compatibility checks and green JSON PR CI
* test: type dynamic Optimize regression arguments
* docs(integration): record Optimize fix PR publication
* docs(integration): record imported tariff protection PR
* style(pvforecast): apply CI import formatting
* style(integration): align combined regression imports
* test: make optimization dispatch timezones explicit
* docs(interpolator): use portable reStructuredText markup
* chore: validate combined integration with locked mypy
* docs: hand off six validated pull requests for manual review
* feat: report genetic interval and terminal value diagnostics
* feat(devices): reconcile flexible profiles and EV deadlines with cycle scheduling
Adapt the flexible consumer primitives from d2e2d582 while retaining the keyed settings and per-cycle scheduling introduced by #1256. Preserve slot battery physics and GENETIC0 flat-load conversion. Cover energy conservation, deadlines, window intersections, DST, completed cycles and EV converters.
* test: satisfy typed genetic PDF chart contracts
* feat(optimization): resolve quarter-hour GENETIC requests from configuration
* test(genetic): verify real device scheduling, measurement and export contracts
Register appliance completed-cycle measurement keys so the real store accepts both default and custom counters. Exercise complete low-budget optimizer runs, persisted measurements, generic solution output and instructions, including zero-power phases, EV departure boundaries, per-cycle windows and LCOS.
* fix: bound genetic report forecasts to executable horizon
* feat: complete native genetic scheduling and retained result contracts
* fix: retain missing raw samples when dropna is disabled
* fix: align local optimization slots and measurement instants
* docs: explain complete genetic rollout and PR dependencies
* feat: expose retained GENETIC report through the versioned API
* docs: regenerate complete genetic configuration and API schema
* docs: format consolidation and review handoff markdown
* test: align isolated EMS fixture with native genetic run options
* test(genetic): clean up singleton measurements after device integration tests
* test: freeze the clock without replacing timestamp conversion
* fix: preserve explicit warmstart timezones in runtime requests
* test(genetic): validate device schedules in UTC and Berlin
Use explicit Berlin origins for Berlin wall-clock windows, compare absolute deadline instants correctly, and run all real device optimizer scenarios under both UTC and Europe/Berlin. Compare exported starts in the run timezone instead of assuming the output timezone matches the host.
* fix: start automatic genetic runs in the site timezone
* Preserve aware GENETIC snapshot times across host timezones
* docs: specify site clock and rehearsed merge resolutions
* test: isolate invalid measurement records and refresh API version
* fix: render single-slot genetic tail diagnostics
* docs: refresh schema version after report fix
* fix: preserve configuration-only Optimize API contract
* docs: refresh configuration request schema
---------
Co-authored-by: Christin <info@bikinibottom.capital>
Co-authored-by: Normann <github@koldrack.com>
---------
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: r0b2g1t <r0b2g1t@users.noreply.github.com>
Co-authored-by: Normann <github@koldrack.com>
Co-authored-by: Christin <info@bikinibottom.capital>
* feat: adapt configuration for multi optimization algorithms
Decouple configuration from optimization algorithm parameters. Add to_[algorithm]_param() methods
to the configuration that derive optimization algorithm specific parameters from the configuration.
Add x-scope tags to the configuration options that describe for which specific algorithms the
configuration option is for.
The whole device settings are restructured. There are now general settings for the device classes
with the afore mentioned to_[algorithm]_param() methods. The general device settings got their own
directory `devices/settings`. By this the parameter class also does not have to be a pydantic model
which can be used for future optimization/ simulations speed up.
Also the parameter class for a device is now part of the device module. This better decouples and
also is the natural place for parameters of a device.
Besides this feature there are also fixes and improvements:
* feat: extend home appliance time window settings and simulation
Home appliance can now be configured for multiple runs with per-cycle allowed time windows. The
number of remaining cycles to plan is determined at runtime by reading the
``cycles_completed_measurement_key`` from the measurement store.
* feat: specialiced CycleTimeWindowSequence for time window sequences
Sequence of time windows associated to cycles.
This model specializes ``ValueTimeWindowSequence`` so that the ``value``
field of each ``ValueTimeWindow`` encodes the **cycle index** (0-based
integer) the window belongs to.
Typical use: an appliance that must run ``n`` times per day, each run
constrained to a distinct time window. Assign ``value=0`` to windows
for the first cycle, ``value=1`` for the second, and so on. Multiple
windows may share the same cycle index (their allowed regions are unioned).
Windows with ``value=None`` are silently ignored by all cycle-aware methods.
* fix: Make test_configmigrate also regard the _ANY_SENTENIEL in key values
* chore: Make devices configurations a map instead of a list
This makes config paths stable regardless of declaration order and lets each device settings
class build its own config path from ``self.device_id`` without needing an external index.
Tests are adapted likewise.
Devices configurations are automatically migrated from lists to maps.
* chore: rename levelized_cost_of_storage_kwh to levelized_cost_of_storage_amt kwh
This better fits in the naming scheme and also makes clear the costs are money.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* fix: runtime config update ignored by config file
Runtime settings were handed back to pydantic-settings as init settings,
which rank below the config file and the environment. Any key already
present in EOS.config.json or in the environment silently discarded the
update, so a bulk PUT /v1/config returned 200 without applying anything,
while the granular PUT /v1/config/{path} endpoint kept working.
Add a dedicated runtime settings source ranked directly below the command
line arguments and record granular updates there as well, so both
endpoints share one store that survives re-evaluation of the settings
sources. Environment variables keep precedence over the config file for
all keys that were not set at runtime.
Also repairs revert_settings() and update(), which passed their data
through the same init settings.
Closes#1303
* fix: env vars ignored on first config build
ConfigEOS.__init__ passed self as first positional argument to _setup,
which forwards it to pydantic_settings.BaseSettings.__init__. Its first
positional parameter is _case_sensitive, so the environment source
matched the upper case variable names against the lower case field names
and returned nothing. Environment settings only took effect after the
next configuration setup.
* docs: changelog for config priority fixes
* fix(config): preserve device identities and storage costs during migration
* fix(devices): preserve charge-rate typing and public import compatibility
* ruff format fix
* feat(devices): port slot-aware battery export and direct-use physics
Port scoped device changes from d2e2d58237. Keep PR #1256 parameter conversion structure and separate GENETIC0 devices. Validate physical flows and reprice changed simulation results.
Co-authored-by: Andreas <drbacke@gmx.de>
Co-authored-by: Christin <info@bikinibottom.capital>
* fix(cache): distinguish callables in the shared EMS cache
Include the function object in cache keys so methods of one interpolator cannot reuse a probability as a power value. Cover both call orders, keyword arguments, cache hits and separate closures with identical qualified names.
* fix(devices): constrain the physics port and validate export levels
Defer inactive EV deadline fields to the optimizer port, reject nonfinite export rates, and document the hourly Optimize boundary. Verify converter IDs, rates and LCOS, separate GENETIC0 interpolation, physical boundary flows and independent GENETIC repricing.
* docs(devices): regenerate slot-physics configuration and OpenAPI schemas
* fix(config): satisfy typed device conversion and migration contracts
* docs(config): refresh validated configuration prerequisite schemas
* test(devices): align physics regressions with strict type checking
* docs(devices): refresh API version after prerequisite merge
* docs(interpolator): use portable reStructuredText markup
* docs(devices): refresh API version after docstring compatibility fix
---------
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: r0b2g1t <r0b2g1t@users.noreply.github.com>
Co-authored-by: Christin <info@bikinibottom.capital>
* feat: adapt configuration for multi optimization algorithms
Decouple configuration from optimization algorithm parameters. Add to_[algorithm]_param() methods
to the configuration that derive optimization algorithm specific parameters from the configuration.
Add x-scope tags to the configuration options that describe for which specific algorithms the
configuration option is for.
The whole device settings are restructured. There are now general settings for the device classes
with the afore mentioned to_[algorithm]_param() methods. The general device settings got their own
directory `devices/settings`. By this the parameter class also does not have to be a pydantic model
which can be used for future optimization/ simulations speed up.
Also the parameter class for a device is now part of the device module. This better decouples and
also is the natural place for parameters of a device.
Besides this feature there are also fixes and improvements:
* feat: extend home appliance time window settings and simulation
Home appliance can now be configured for multiple runs with per-cycle allowed time windows. The
number of remaining cycles to plan is determined at runtime by reading the
``cycles_completed_measurement_key`` from the measurement store.
* feat: specialiced CycleTimeWindowSequence for time window sequences
Sequence of time windows associated to cycles.
This model specializes ``ValueTimeWindowSequence`` so that the ``value``
field of each ``ValueTimeWindow`` encodes the **cycle index** (0-based
integer) the window belongs to.
Typical use: an appliance that must run ``n`` times per day, each run
constrained to a distinct time window. Assign ``value=0`` to windows
for the first cycle, ``value=1`` for the second, and so on. Multiple
windows may share the same cycle index (their allowed regions are unioned).
Windows with ``value=None`` are silently ignored by all cycle-aware methods.
* fix: Make test_configmigrate also regard the _ANY_SENTENIEL in key values
* chore: Make devices configurations a map instead of a list
This makes config paths stable regardless of declaration order and lets each device settings
class build its own config path from ``self.device_id`` without needing an external index.
Tests are adapted likewise.
Devices configurations are automatically migrated from lists to maps.
* chore: rename levelized_cost_of_storage_kwh to levelized_cost_of_storage_amt kwh
This better fits in the naming scheme and also makes clear the costs are money.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* fix: runtime config update ignored by config file
Runtime settings were handed back to pydantic-settings as init settings,
which rank below the config file and the environment. Any key already
present in EOS.config.json or in the environment silently discarded the
update, so a bulk PUT /v1/config returned 200 without applying anything,
while the granular PUT /v1/config/{path} endpoint kept working.
Add a dedicated runtime settings source ranked directly below the command
line arguments and record granular updates there as well, so both
endpoints share one store that survives re-evaluation of the settings
sources. Environment variables keep precedence over the config file for
all keys that were not set at runtime.
Also repairs revert_settings() and update(), which passed their data
through the same init settings.
Closes#1303
* fix: env vars ignored on first config build
ConfigEOS.__init__ passed self as first positional argument to _setup,
which forwards it to pydantic_settings.BaseSettings.__init__. Its first
positional parameter is _case_sensitive, so the environment source
matched the upper case variable names against the lower case field names
and returned nothing. Environment settings only took effect after the
next configuration setup.
* docs: changelog for config priority fixes
* fix(config): preserve device identities and storage costs during migration
* fix(measurement): restore JSON records into the existing singleton
* fix(devices): preserve charge-rate typing and public import compatibility
* feat(measurement): integrate typed energy quality and capacity APIs
Port the locally backed-up measurement extensions to main async storage and PR #1256 device maps. Preserve runtime capacity estimates across #1305 bulk updates. Confirm JSON singleton restore defect on unchanged main and add regression. No production configuration or measurements included.
Co-authored-by: Andreas <drbacke@gmx.de>
* docs(measurement): describe household settings and consolidate regression coverage
* docs(measurement): regenerate configuration and API contracts
* test(measurement): isolate capacity database state between tests
* ruff format fix
* test(measurement): assert restored timestamps before timezone conversion
* docs(measurement): align API version with refreshed prerequisites
* fix(config): satisfy typed device conversion and migration contracts
* docs(config): refresh validated configuration prerequisite schemas
* fix(measurement): enforce typed capacity and sample validation
* style(measurement): normalize imports for CI
* docs(measurement): refresh typed measurement API schemas
---------
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: r0b2g1t <r0b2g1t@users.noreply.github.com>
* feat: adapt configuration for multi optimization algorithms
Decouple configuration from optimization algorithm parameters. Add to_[algorithm]_param() methods
to the configuration that derive optimization algorithm specific parameters from the configuration.
Add x-scope tags to the configuration options that describe for which specific algorithms the
configuration option is for.
The whole device settings are restructured. There are now general settings for the device classes
with the afore mentioned to_[algorithm]_param() methods. The general device settings got their own
directory `devices/settings`. By this the parameter class also does not have to be a pydantic model
which can be used for future optimization/ simulations speed up.
Also the parameter class for a device is now part of the device module. This better decouples and
also is the natural place for parameters of a device.
Besides this feature there are also fixes and improvements:
* feat: extend home appliance time window settings and simulation
Home appliance can now be configured for multiple runs with per-cycle allowed time windows. The
number of remaining cycles to plan is determined at runtime by reading the
``cycles_completed_measurement_key`` from the measurement store.
* feat: specialiced CycleTimeWindowSequence for time window sequences
Sequence of time windows associated to cycles.
This model specializes ``ValueTimeWindowSequence`` so that the ``value``
field of each ``ValueTimeWindow`` encodes the **cycle index** (0-based
integer) the window belongs to.
Typical use: an appliance that must run ``n`` times per day, each run
constrained to a distinct time window. Assign ``value=0`` to windows
for the first cycle, ``value=1`` for the second, and so on. Multiple
windows may share the same cycle index (their allowed regions are unioned).
Windows with ``value=None`` are silently ignored by all cycle-aware methods.
* fix: Make test_configmigrate also regard the _ANY_SENTENIEL in key values
* chore: Make devices configurations a map instead of a list
This makes config paths stable regardless of declaration order and lets each device settings
class build its own config path from ``self.device_id`` without needing an external index.
Tests are adapted likewise.
Devices configurations are automatically migrated from lists to maps.
* chore: rename levelized_cost_of_storage_kwh to levelized_cost_of_storage_amt kwh
This better fits in the naming scheme and also makes clear the costs are money.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* fix: runtime config update ignored by config file
Runtime settings were handed back to pydantic-settings as init settings,
which rank below the config file and the environment. Any key already
present in EOS.config.json or in the environment silently discarded the
update, so a bulk PUT /v1/config returned 200 without applying anything,
while the granular PUT /v1/config/{path} endpoint kept working.
Add a dedicated runtime settings source ranked directly below the command
line arguments and record granular updates there as well, so both
endpoints share one store that survives re-evaluation of the settings
sources. Environment variables keep precedence over the config file for
all keys that were not set at runtime.
Also repairs revert_settings() and update(), which passed their data
through the same init settings.
Closes#1303
* fix: env vars ignored on first config build
ConfigEOS.__init__ passed self as first positional argument to _setup,
which forwards it to pydantic_settings.BaseSettings.__init__. Its first
positional parameter is _case_sensitive, so the environment source
matched the upper case variable names against the lower case field names
and returned nothing. Environment settings only took effect after the
next configuration setup.
* docs: changelog for config priority fixes
* fix(config): preserve device identities and storage costs during migration
* fix(devices): preserve charge-rate typing and public import compatibility
* ruff format fix
* fix(config): satisfy typed device conversion and migration contracts
* docs(config): refresh validated configuration prerequisite schemas
---------
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: r0b2g1t <r0b2g1t@users.noreply.github.com>
Port local PV modeling and outage calibration from feature commits f976335, 6dc58c3 and faed0fd by Andreas. Keep PVForecastAkkudoktor identity and remote default, adapt to async storage, and migrate legacy provider settings.
Cancel GENETIC preparation when imported revenue cannot be read or contains invalid values, preserving the chosen provider instead of replacing it with demo tariffs. Keep valid positive, zero and negative amount/Wh series unchanged.
Adapt the revenue-preservation regressions from PRs #1224 and #1304 to the async main API, including real timestamped imports and simulation repricing. The feature-only direct-marketing override remains outside this main fix.
Co-authored-by: Christin <info@bikinibottom.capital>
Co-authored-by: Normann <github@koldrack.com>
Infer interval coverage from consistent recent original price spacings while preserving history and forecasting resolution policies. Reuse source data until a successful refresh and cover cadence transitions, fallback, and history repair.
Fixes#1279
Consolidates seven compatible Dependabot updates into one reviewable change:
Runtime dependencies
platformdirs 4.11.7 → 4.11.8 (build(deps): bump platformdirs from 4.11.7 to 4.11.8 #1307)
rich-toolkit 0.20.4 → 0.20.5 (build(deps): bump rich-toolkit from 0.20.4 to 0.20.5 #1313)
Development, documentation, and CI dependencies
types-requests 2.33.0.20260712 → 2.33.0.20260906 (build(deps-dev): bump types-requests from 2.33.0.20260712 to 2.33.0.20260906 #1306)
pypdf 6.17.0 → 6.18.0 (build(deps-dev): bump pypdf from 6.17.0 to 6.18.0 #1308)
types-PyYaml 6.0.12.20260815 → 6.0.12.20260906 (build(deps-dev): bump types-pyyaml from 6.0.12.20260815 to 6.0.12.20260906 #1309)
sphinxcontrib-mermaid 2.1.0 → 2.1.1 (build(deps-dev): bump sphinxcontrib-mermaid from 2.1.0 to 2.1.1 #1310)
GitPython 3.1.61 → 3.1.62 (build(deps-dev): bump gitpython from 3.1.61 to 3.1.62 #1312)
The committed uv.lock file is regenerated for all seven updates. The pre-commit mypy hook already executes the locked project development environment, so the type-stub pins have a single source of truth in pyproject.toml and uv.lock.
to_datetime() previously interpreted naive strings differently depending on their precision. For example, 2026-01-15 23:45 in Europe/Berlin became 2026-01-16 00:45+01:00, while the equivalent string with seconds kept the intended date and time. Minutes, seconds, and fractional seconds now consistently represent 2026-01-15 23:45+01:00.
Pass the resolved target/local timezone to Pendulum's fallback parser. Explicit Z and numeric offsets continue to determine the input instant, and Unix timestamps retain their UTC semantics. Update the affected docstring examples to distinguish local wall time from UTC conversion.
Regression coverage includes space and T separators, explicit and default Europe/Berlin timezones, winter/summer dates near midnight, fractional-second precision, explicit offsets, and Unix timestamps. The Open-Meteo integration test now independently checks the timestamps and all three irradiance series for all 72 fixture hours, with both UTC and Europe/Berlin as the host timezone. This replaces an expectation that incorrectly associated the 08:00 values with 09:00. The previous parser misplaces all 72 source hours; the corrected parser maps all 72 correctly.
Track owned test processes and restrict fallback cleanup to verified EOS servers using the test configuration. Add regression coverage for denied inspection and cleanup failures.
Fixes#1297
* fix: check port availability without process inspection
Probe local TCP addresses and use process inspection only for optional diagnostics. Preserve occupied-port detection, bounded waits, and port reuse after closed connections.
Fixes#1298
* fix: preserve IPv6 scope IDs in port availability probes
The mypy regression-test command in CONTRIBUTING.md points to the nonexistent tests/test_typingmypytoolchain.py, causing copied commands to fail.
Change the path to tests/test_typingmypytooling.py, the existing integration test.
The isolated pre-commit mypy hook previously omitted runtime type information that
make mypy used, hiding errors involving dependencies such as Pydantic and Pendulum.
Makefile, pre-commit and CI now run the same full-project typing policy in the
development environment defined by uv.lock.
- Use uv run --locked --exact --extra dev and the same mypy arguments for Makefile
and the local hook. Check all of src and tests, including on configuration-only
changes.
- Pin Python 3.13 for local development and the pre-commit CI job, and install the
locked pre-commit version in CI.
- Disable incremental analysis because existing Pendulum cache state changes mypy 2.3.1
diagnostics. Document the policy, the performance tradeoff and the existing typing debt.
- Add a regression test that exercises Makefile, the hook and the CI command in a
temporary project, accepting valid dependency types and detecting deliberate
Pydantic/Pendulum assignment errors.
Resolve the newly detected mypy diagnostics.
- Enable the numpydantic and Pydantic mypy plugins, retaining strict Pydantic
constructor typing with init_typed = true. Validate raw/coercible payloads through model_validate.
- Propagate concrete record, provider and time-window types through generic collections,
factories and lookup methods. Preserve runtime field inspection and generated time-window
documentation.
- Align Pendulum annotations with actual factory/arithmetic results while retaining Pydantic
validation adapters at runtime. Correct optional values, array boundaries, REST handlers
and plotting interfaces.
- Add pinned scipy-stubs and types-psutil, update uv.lock, and supply the plugins' dependencies.
- Add runtime regression coverage for validated path defaults, normalized time-series metadata,
generic field inspection, invalid timestamps and unsupported provider imports.
Runtime and compatibility details:
- Validate path defaults as Path objects while retaining raw string defaults needed by
migration serialization with exclude_defaults.
- Normalize feed-in tariff lists and default charge rates to NumPy arrays; reject missing
timestamps/uninitialized values explicitly. Importing into a provider without import support
returns HTTP 400.
- Public JSON schemas and OpenAPI structure match main (excluding the generated version).
Signed-off-by: dr-dimitry
Signed-off-by: dr-dimitry
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: dr-dimitri <87113560+dr-dimitri@users.noreply.github.com>
Co-authored-by: Normann <github@koldrack.com>
test_configmigrate leaves a <fixture>.new working copy behind whenever a
migration test fails, test_visualize writes example_report.pdf into the current
directory, and test_config creates tests/testdata/docs/. All three showed up as
untracked noise in every status and invited accidental commits.
Co-authored-by: Andreas <drbacke@gmx.de>
* build: multi-stage Docker image with uv and BuildKit caching
Builder/runtime split keeps the toolchain out of the runtime image
(standalone ~1.8GB -> ~1.2GB). Dependencies install with uv from
uv.lock in a cache-mounted, source-independent layer. Runtime keeps
the editable install because akkudoktoreos.core.version needs the
src/ layout. Also fixes the io.hass.version label and adds a
HEALTHCHECK.
* build: address Docker image review feedback
- Healthcheck honours EOS_SERVER__PORT instead of hardcoding 8503.
- BUILD_VERSION defaults to "dev" rather than the literal "VERSION";
docker-compose passes the real version, and the redundant
org.opencontainers.image.version label is dropped (CI sets it via
docker/metadata-action).
* fix: use resolved server port for container healthcheck
---------
Co-authored-by: Normann <github@koldrack.com>
Co-authored-by: Normann Koldrack <normann.koldrack@desy.de>
NormannandGitHub
Hoffis-WeltandGitHub
AndreasandGitHub
dr-dimitriandGitHub
