From 04f28997ea8ccf38682bb942a3a114e10c151df3 Mon Sep 17 00:00:00 2001 From: Andreas <35328755+drbacke@users.noreply.github.com> Date: Thu, 17 Sep 2026 20:14:24 +0200 Subject: [PATCH] feat: deliver complete GENETIC optimization to main (#1330) * feat: adapt configuration for multi optimization algorithms Decouple configuration from optimization algorithm parameters. Add to_[algorithm]_param() methods to the configuration that derive optimization algorithm specific parameters from the configuration. Add x-scope tags to the configuration options that describe for which specific algorithms the configuration option is for. The whole device settings are restructured. There are now general settings for the device classes with the afore mentioned to_[algorithm]_param() methods. The general device settings got their own directory `devices/settings`. By this the parameter class also does not have to be a pydantic model which can be used for future optimization/ simulations speed up. Also the parameter class for a device is now part of the device module. This better decouples and also is the natural place for parameters of a device. Besides this feature there are also fixes and improvements: * feat: extend home appliance time window settings and simulation Home appliance can now be configured for multiple runs with per-cycle allowed time windows. The number of remaining cycles to plan is determined at runtime by reading the ``cycles_completed_measurement_key`` from the measurement store. * feat: specialiced CycleTimeWindowSequence for time window sequences Sequence of time windows associated to cycles. This model specializes ``ValueTimeWindowSequence`` so that the ``value`` field of each ``ValueTimeWindow`` encodes the **cycle index** (0-based integer) the window belongs to. Typical use: an appliance that must run ``n`` times per day, each run constrained to a distinct time window. Assign ``value=0`` to windows for the first cycle, ``value=1`` for the second, and so on. Multiple windows may share the same cycle index (their allowed regions are unioned). Windows with ``value=None`` are silently ignored by all cycle-aware methods. * fix: Make test_configmigrate also regard the _ANY_SENTENIEL in key values * chore: Make devices configurations a map instead of a list This makes config paths stable regardless of declaration order and lets each device settings class build its own config path from ``self.device_id`` without needing an external index. Tests are adapted likewise. Devices configurations are automatically migrated from lists to maps. * chore: rename levelized_cost_of_storage_kwh to levelized_cost_of_storage_amt kwh This better fits in the naming scheme and also makes clear the costs are money. Signed-off-by: Bobby Noelte * fix: runtime config update ignored by config file Runtime settings were handed back to pydantic-settings as init settings, which rank below the config file and the environment. Any key already present in EOS.config.json or in the environment silently discarded the update, so a bulk PUT /v1/config returned 200 without applying anything, while the granular PUT /v1/config/{path} endpoint kept working. Add a dedicated runtime settings source ranked directly below the command line arguments and record granular updates there as well, so both endpoints share one store that survives re-evaluation of the settings sources. Environment variables keep precedence over the config file for all keys that were not set at runtime. Also repairs revert_settings() and update(), which passed their data through the same init settings. Closes #1303 * fix: env vars ignored on first config build ConfigEOS.__init__ passed self as first positional argument to _setup, which forwards it to pydantic_settings.BaseSettings.__init__. Its first positional parameter is _case_sensitive, so the environment source matched the upper case variable names against the lower case field names and returned nothing. Environment settings only took effect after the next configuration setup. * docs: changelog for config priority fixes * fix(config): preserve device identities and storage costs during migration * fix(measurement): restore JSON records into the existing singleton * fix(devices): preserve charge-rate typing and public import compatibility * feat(measurement): integrate typed energy quality and capacity APIs Port the locally backed-up measurement extensions to main async storage and PR #1256 device maps. Preserve runtime capacity estimates across #1305 bulk updates. Confirm JSON singleton restore defect on unchanged main and add regression. No production configuration or measurements included. Co-authored-by: Andreas * docs(measurement): describe household settings and consolidate regression coverage * docs(measurement): regenerate configuration and API contracts * test(measurement): isolate capacity database state between tests * ruff format fix * fix(measurement): restore JSON records into the existing singleton * test(measurement): assert restored timestamps before timezone conversion * test(measurement): assert restored timestamps before timezone conversion * fix: preserve imported feed-in revenue during parameter preparation Cancel GENETIC preparation when imported revenue cannot be read or contains invalid values, preserving the chosen provider instead of replacing it with demo tariffs. Keep valid positive, zero and negative amount/Wh series unchanged. Adapt the revenue-preservation regressions from PRs #1224 and #1304 to the async main API, including real timestamped imports and simulation repricing. The feature-only direct-marketing override remains outside this main fix. Co-authored-by: Christin Co-authored-by: Normann * feat(devices): port slot-aware battery export and direct-use physics Port scoped device changes from d2e2d58237339454dd8bb226f92677c5987f8b27. Keep PR #1256 parameter conversion structure and separate GENETIC0 devices. Validate physical flows and reprice changed simulation results. Co-authored-by: Andreas Co-authored-by: Christin * docs(measurement): align API version with refreshed prerequisites * fix: return only completed optimization results per run * feat(pvforecast): add calibrated local Akkudoktor backend Port local PV modeling and outage calibration from feature commits f976335, 6dc58c3 and faed0fd by Andreas. Keep PVForecastAkkudoktor identity and remote default, adapt to async storage, and migrate legacy provider settings. * fix(cache): distinguish callables in the shared EMS cache Include the function object in cache keys so methods of one interpolator cannot reuse a probability as a power value. Cover both call orders, keyword arguments, cache hits and separate closures with identical qualified names. * fix(devices): constrain the physics port and validate export levels Defer inactive EV deadline fields to the optimizer port, reject nonfinite export rates, and document the hourly Optimize boundary. Verify converter IDs, rates and LCOS, separate GENETIC0 interpolation, physical boundary flows and independent GENETIC repricing. * docs(pvforecast): regenerate local backend configuration schema * docs(devices): regenerate slot-physics configuration and OpenAPI schemas * test: type dynamic Optimize regression arguments * style: wrap imported tariff test parameter import * style(pvforecast): apply CI import formatting * docs(pvforecast): refresh API version after CI formatting * fix(config): satisfy typed device conversion and migration contracts * docs(config): refresh validated configuration prerequisite schemas * fix(measurement): enforce typed capacity and sample validation * test(devices): align physics regressions with strict type checking * style(measurement): normalize imports for CI * docs(measurement): refresh typed measurement API schemas * docs(devices): refresh API version after prerequisite merge * test: make optimization dispatch timezones explicit * docs(interpolator): use portable reStructuredText markup * docs(devices): refresh API version after docstring compatibility fix * feat: complete configuration-driven GENETIC optimization and reports (#1329) * feat(devices): port slot-aware battery export and direct-use physics Port scoped device changes from d2e2d58237339454dd8bb226f92677c5987f8b27. Keep PR #1256 parameter conversion structure and separate GENETIC0 devices. Validate physical flows and reprice changed simulation results. Co-authored-by: Andreas Co-authored-by: Christin * feat(optimization): port tested terminal and tail value primitives Source d2e2d58237339454dd8bb226f92677c5987f8b27. 22 primitive tests pass; integration with the optimizer, forecast horizon and API is still pending. Co-authored-by: Andreas Co-authored-by: Christin * fix(devices): preserve charge-rate typing and public import compatibility * feat(measurement): integrate typed energy quality and capacity APIs Port the locally backed-up measurement extensions to main async storage and PR #1256 device maps. Preserve runtime capacity estimates across #1305 bulk updates. Confirm JSON singleton restore defect on unchanged main and add regression. No production configuration or measurements included. Co-authored-by: Andreas * test(integration): validate optimizer economics and document measurement settings * docs(integration): record tested checkpoint and remaining consolidation work * docs(development): define isolated PR packages and remaining porting gates * docs(integration): refresh API version after measurement reconciliation * docs(integration): record PR readiness verification results * docs(development): record publication and verification of PR 1322 * test(measurement): assert restored timestamps before timezone conversion * docs(development): record corrected PR head and CI progress * docs(integration): refresh API version after prerequisite alignment * docs(integration): define parallel packages and Optimize compatibility gates * fix: preserve imported feed-in revenue during parameter preparation Cancel GENETIC preparation when imported revenue cannot be read or contains invalid values, preserving the chosen provider instead of replacing it with demo tariffs. Keep valid positive, zero and negative amount/Wh series unchanged. Adapt the revenue-preservation regressions from PRs #1224 and #1304 to the async main API, including real timestamped imports and simulation repricing. The feature-only direct-marketing override remains outside this main fix. Co-authored-by: Christin Co-authored-by: Normann * test(integration): verify tariff protection with mapped device physics * fix: return only completed optimization results per run * test(integration): verify algorithm aliases and mapped-device contracts * fix(cache): distinguish callables in the shared EMS cache Include the function object in cache keys so methods of one interpolator cannot reuse a probability as a power value. Cover both call orders, keyword arguments, cache hits and separate closures with identical qualified names. * fix(devices): constrain the physics port and validate export levels Defer inactive EV deadline fields to the optimizer port, reject nonfinite export rates, and document the hourly Optimize boundary. Verify converter IDs, rates and LCOS, separate GENETIC0 interpolation, physical boundary flows and independent GENETIC repricing. * feat(pvforecast): add calibrated local Akkudoktor backend Port local PV modeling and outage calibration from feature commits f976335, 6dc58c3 and faed0fd by Andreas. Keep PVForecastAkkudoktor identity and remote default, adapt to async storage, and migrate legacy provider settings. * docs(integration): record combined compatibility checks and green JSON PR CI * test: type dynamic Optimize regression arguments * docs(integration): record Optimize fix PR publication * docs(integration): record imported tariff protection PR * style(pvforecast): apply CI import formatting * style(integration): align combined regression imports * test: make optimization dispatch timezones explicit * docs(interpolator): use portable reStructuredText markup * chore: validate combined integration with locked mypy * docs: hand off six validated pull requests for manual review * feat: report genetic interval and terminal value diagnostics * feat(devices): reconcile flexible profiles and EV deadlines with cycle scheduling Adapt the flexible consumer primitives from d2e2d582 while retaining the keyed settings and per-cycle scheduling introduced by #1256. Preserve slot battery physics and GENETIC0 flat-load conversion. Cover energy conservation, deadlines, window intersections, DST, completed cycles and EV converters. * test: satisfy typed genetic PDF chart contracts * feat(optimization): resolve quarter-hour GENETIC requests from configuration * test(genetic): verify real device scheduling, measurement and export contracts Register appliance completed-cycle measurement keys so the real store accepts both default and custom counters. Exercise complete low-budget optimizer runs, persisted measurements, generic solution output and instructions, including zero-power phases, EV departure boundaries, per-cycle windows and LCOS. * fix: bound genetic report forecasts to executable horizon * feat: complete native genetic scheduling and retained result contracts * fix: retain missing raw samples when dropna is disabled * fix: align local optimization slots and measurement instants * docs: explain complete genetic rollout and PR dependencies * feat: expose retained GENETIC report through the versioned API * docs: regenerate complete genetic configuration and API schema * docs: format consolidation and review handoff markdown * test: align isolated EMS fixture with native genetic run options * test(genetic): clean up singleton measurements after device integration tests * test: freeze the clock without replacing timestamp conversion * fix: preserve explicit warmstart timezones in runtime requests * test(genetic): validate device schedules in UTC and Berlin Use explicit Berlin origins for Berlin wall-clock windows, compare absolute deadline instants correctly, and run all real device optimizer scenarios under both UTC and Europe/Berlin. Compare exported starts in the run timezone instead of assuming the output timezone matches the host. * fix: start automatic genetic runs in the site timezone * Preserve aware GENETIC snapshot times across host timezones * docs: specify site clock and rehearsed merge resolutions * test: isolate invalid measurement records and refresh API version * fix: render single-slot genetic tail diagnostics * docs: refresh schema version after report fix * fix: preserve configuration-only Optimize API contract * docs: refresh configuration request schema --------- Co-authored-by: Christin Co-authored-by: Normann --------- Signed-off-by: Bobby Noelte Co-authored-by: Bobby Noelte Co-authored-by: r0b2g1t Co-authored-by: Normann Co-authored-by: Christin --- docs/_generated/configdevices.md | 35 +- docs/_generated/configexample.md | 23 +- docs/_generated/configfeedintariff.md | 3 + docs/_generated/configoptimization.md | 27 +- docs/_generated/openapi.md | 64 +- docs/development/eos-consolidation.md | 270 ++ docs/development/eos-ha-handoff.md | 67 + docs/development/genetic-rollout.md | 93 + .../pr-drafts/imported-feedin-main.md | 25 + .../measurement-energy-quality-capacity.md | 38 + .../pr-drafts/measurement-json-reload.md | 41 + .../pr-drafts/optimize-run-result.md | 24 + docs/development/pr-integration-matrix.md | 125 + docs/development/pr-workflow.md | 134 + docs/development/review-handoff.md | 60 + openapi.json | 1426 +++++++- src/akkudoktoreos/config/config.py | 31 + src/akkudoktoreos/config/configmigrate.py | 6 + src/akkudoktoreos/core/dataabc.py | 2 +- src/akkudoktoreos/core/emplan.py | 22 +- src/akkudoktoreos/core/ems.py | 70 +- src/akkudoktoreos/devices/devicesabc.py | 111 + src/akkudoktoreos/devices/genetic/battery.py | 36 +- .../devices/genetic/homeappliance.py | 529 ++- .../devices/settings/batterysettings.py | 39 + .../devices/settings/homeappliancesettings.py | 161 +- .../optimization/genetic/configrequest.py | 241 ++ .../optimization/genetic/forecast.py | 61 + .../optimization/genetic/genetic.py | 3085 +++++++++++++++-- .../optimization/genetic/geneticparams.py | 746 +--- .../optimization/genetic/geneticsettings.py | 110 +- .../optimization/genetic/geneticsolution.py | 657 ++-- .../optimization/genetic/geneticvisualize.py | 246 +- .../optimization/genetic/tailvalue.py | 305 ++ .../optimization/genetic/terminalvalue.py | 359 ++ .../optimization/optimization.py | 15 +- src/akkudoktoreos/prediction/feedintariff.py | 8 + src/akkudoktoreos/server/eos.py | 71 + tests/test_config_optimization_request.py | 398 +++ tests/test_consolidation_file_restore.py | 29 + tests/test_dataabcsequence.py | 26 +- tests/test_emplan_dst_instants.py | 38 + tests/test_genetic_complete_devices.py | 293 ++ tests/test_genetic_complete_optimization.py | 385 ++ tests/test_genetic_complete_simulation.py | 528 +++ tests/test_genetic_complete_solution.py | 188 + tests/test_genetic_complete_tail.py | 438 +++ tests/test_genetic_complete_timegrid.py | 165 + tests/test_genetic_end_to_end_devices.py | 417 +++ tests/test_genetic_forecast_coverage.py | 66 + tests/test_genetic_seeding.py | 476 +++ tests/test_genetic_warm_start_alignment.py | 146 + tests/test_geneticoptimize.py | 87 +- tests/test_geneticparams_feedin.py | 30 +- tests/test_geneticvisualize_intervals.py | 217 ++ tests/test_optimization_compatibility.py | 115 + tests/test_optimization_pdf_api.py | 78 + tests/test_optimize_run_result.py | 55 +- tests/test_tailvalue_physics.py | 199 ++ tests/test_terminalvalue.py | 129 + tests/test_typingmodels.py | 10 +- .../testdata/docs/_generated/configdevices.md | 50 +- .../testdata/docs/_generated/configexample.md | 23 +- .../docs/_generated/configmeasurement.md | 19 +- 64 files changed, 12540 insertions(+), 1431 deletions(-) create mode 100644 docs/development/eos-consolidation.md create mode 100644 docs/development/eos-ha-handoff.md create mode 100644 docs/development/genetic-rollout.md create mode 100644 docs/development/pr-drafts/imported-feedin-main.md create mode 100644 docs/development/pr-drafts/measurement-energy-quality-capacity.md create mode 100644 docs/development/pr-drafts/measurement-json-reload.md create mode 100644 docs/development/pr-drafts/optimize-run-result.md create mode 100644 docs/development/pr-integration-matrix.md create mode 100644 docs/development/pr-workflow.md create mode 100644 docs/development/review-handoff.md create mode 100644 src/akkudoktoreos/optimization/genetic/configrequest.py create mode 100644 src/akkudoktoreos/optimization/genetic/forecast.py create mode 100644 src/akkudoktoreos/optimization/genetic/tailvalue.py create mode 100644 src/akkudoktoreos/optimization/genetic/terminalvalue.py create mode 100644 tests/test_config_optimization_request.py create mode 100644 tests/test_consolidation_file_restore.py create mode 100644 tests/test_emplan_dst_instants.py create mode 100644 tests/test_genetic_complete_devices.py create mode 100644 tests/test_genetic_complete_optimization.py create mode 100644 tests/test_genetic_complete_simulation.py create mode 100644 tests/test_genetic_complete_solution.py create mode 100644 tests/test_genetic_complete_tail.py create mode 100644 tests/test_genetic_complete_timegrid.py create mode 100644 tests/test_genetic_end_to_end_devices.py create mode 100644 tests/test_genetic_forecast_coverage.py create mode 100644 tests/test_genetic_seeding.py create mode 100644 tests/test_genetic_warm_start_alignment.py create mode 100644 tests/test_geneticvisualize_intervals.py create mode 100644 tests/test_optimization_compatibility.py create mode 100644 tests/test_optimization_pdf_api.py create mode 100644 tests/test_tailvalue_physics.py create mode 100644 tests/test_terminalvalue.py diff --git a/docs/_generated/configdevices.md b/docs/_generated/configdevices.md index ce09b543..a975835f 100644 --- a/docs/_generated/configdevices.md +++ b/docs/_generated/configdevices.md @@ -63,7 +63,9 @@ config path from ``self.device_id`` without needing an external index. 0.5, 0.75, 1.0 - ] + ], + "min_soc_deadline_datetime": null, + "min_soc_max_duration_h": null } }, "max_batteries": 1, @@ -98,7 +100,9 @@ config path from ``self.device_id`` without needing an external index. 0.5, 0.75, 1.0 - ] + ], + "min_soc_deadline_datetime": null, + "min_soc_max_duration_h": null } }, "max_electric_vehicles": 1, @@ -112,7 +116,14 @@ config path from ``self.device_id`` without needing an external index. "num_cycles": 1, "cycle_time_windows": null, "min_cycle_gap_h": 0, - "cycles_completed_measurement_key": null + "cycles_completed_measurement_key": null, + "load_profile_power_w": null, + "load_profile_interval_seconds": null, + "schedule_mode": "ONCE", + "time_windows": null, + "earliest_start_datetime": null, + "deadline_datetime": null, + "deadline_policy": "BEST_EFFORT" } }, "max_home_appliances": 3 @@ -161,6 +172,8 @@ config path from ``self.device_id`` without needing an external index. 0.75, 1.0 ], + "min_soc_deadline_datetime": null, + "min_soc_max_duration_h": null, "measurement_key_soc_factor": "bat0-soc-factor", "measurement_key_power_l1_w": "bat0-power-l1-w", "measurement_key_power_l2_w": "bat0-power-l2-w", @@ -208,6 +221,8 @@ config path from ``self.device_id`` without needing an external index. 0.75, 1.0 ], + "min_soc_deadline_datetime": null, + "min_soc_max_duration_h": null, "measurement_key_soc_factor": "ev0-soc-factor", "measurement_key_power_l1_w": "ev0-power-l1-w", "measurement_key_power_l2_w": "ev0-power-l2-w", @@ -234,8 +249,17 @@ config path from ``self.device_id`` without needing an external index. "cycle_time_windows": null, "min_cycle_gap_h": 0, "cycles_completed_measurement_key": null, + "load_profile_power_w": null, + "load_profile_interval_seconds": null, + "schedule_mode": "ONCE", + "time_windows": null, + "earliest_start_datetime": null, + "deadline_datetime": null, + "deadline_policy": "BEST_EFFORT", "effective_num_cycles": 1, - "measurement_keys": [] + "measurement_keys": [ + "dishwasher.cycles_completed" + ] } }, "max_home_appliances": 3, @@ -249,7 +273,8 @@ config path from ``self.device_id`` without needing an external index. "ev0-power-l1-w", "ev0-power-l2-w", "ev0-power-l3-w", - "ev0-power-3-phase-sym-w" + "ev0-power-3-phase-sym-w", + "dishwasher.cycles_completed" ] } } diff --git a/docs/_generated/configexample.md b/docs/_generated/configexample.md index d926c3c4..1d46008d 100644 --- a/docs/_generated/configexample.md +++ b/docs/_generated/configexample.md @@ -67,7 +67,9 @@ 0.5, 0.75, 1.0 - ] + ], + "min_soc_deadline_datetime": null, + "min_soc_max_duration_h": null } }, "max_batteries": 1, @@ -102,7 +104,9 @@ 0.5, 0.75, 1.0 - ] + ], + "min_soc_deadline_datetime": null, + "min_soc_max_duration_h": null } }, "max_electric_vehicles": 1, @@ -116,7 +120,14 @@ "num_cycles": 1, "cycle_time_windows": null, "min_cycle_gap_h": 0, - "cycles_completed_measurement_key": null + "cycles_completed_measurement_key": null, + "load_profile_power_w": null, + "load_profile_interval_seconds": null, + "schedule_mode": "ONCE", + "time_windows": null, + "earliest_start_datetime": null, + "deadline_datetime": null, + "deadline_policy": "BEST_EFFORT" } }, "max_home_appliances": 3 @@ -172,6 +183,7 @@ "mode": "OPTIMIZATION" }, "feedintariff": { + "direct_marketing_enabled": false, "provider": "FeedInTariffFixed", "feedintarifffixed": { "feed_in_tariff_amt_kwh": { @@ -245,6 +257,11 @@ "individuals": 400, "generations": 400, "seed": null, + "measurement_max_age_seconds": 300, + "tail_horizon_hours": 48, + "terminal_value_mode": "AUTO", + "terminal_value_euro_per_kwh": 0.0, + "terminal_value_window_hours": 24, "penalties": { "ev_soc_miss": 10 } diff --git a/docs/_generated/configfeedintariff.md b/docs/_generated/configfeedintariff.md index b8ce90bd..2a855cd5 100644 --- a/docs/_generated/configfeedintariff.md +++ b/docs/_generated/configfeedintariff.md @@ -7,6 +7,7 @@ | Name | Environment Variable | Type | Read-Only | Default | Description | | ---- | -------------------- | ---- | --------- | ------- | ----------- | +| direct_marketing_enabled | `EOS_FEEDINTARIFF__DIRECT_MARKETING_ENABLED` | `bool` | `rw` | `False` | Enable export-aware GENETIC optimization. Sale revenues remain those of the configured feed-in provider or explicit forecast; purchase prices never replace them. | | dvhubonline | `EOS_FEEDINTARIFF__DVHUBONLINE` | `FeedInTariffDvhubOnlineCommonSettings` | `rw` | `required` | DvhubOnline feed in tariff provider settings. | | energycharts | `EOS_FEEDINTARIFF__ENERGYCHARTS` | `FeedInTariffEnergyChartsCommonSettings` | `rw` | `required` | EnergyCharts feed in tariff provider settings. | | feedintarifffixed | `EOS_FEEDINTARIFF__FEEDINTARIFFFIXED` | `FeedInTariffFixedCommonSettings` | `rw` | `required` | Fixed feed in tariff provider settings. | @@ -25,6 +26,7 @@ ```json { "feedintariff": { + "direct_marketing_enabled": false, "provider": "FeedInTariffFixed", "feedintarifffixed": { "feed_in_tariff_amt_kwh": { @@ -56,6 +58,7 @@ ```json { "feedintariff": { + "direct_marketing_enabled": false, "provider": "FeedInTariffFixed", "feedintarifffixed": { "feed_in_tariff_amt_kwh": { diff --git a/docs/_generated/configoptimization.md b/docs/_generated/configoptimization.md index ec44f827..7ddf1101 100644 --- a/docs/_generated/configoptimization.md +++ b/docs/_generated/configoptimization.md @@ -30,6 +30,11 @@ "individuals": 400, "generations": 400, "seed": null, + "measurement_max_age_seconds": 300, + "tail_horizon_hours": 48, + "terminal_value_mode": "AUTO", + "terminal_value_euro_per_kwh": 0.0, + "terminal_value_window_hours": 24, "penalties": { "ev_soc_miss": 10 } @@ -63,6 +68,11 @@ "individuals": 400, "generations": 400, "seed": null, + "measurement_max_age_seconds": 300, + "tail_horizon_hours": 48, + "terminal_value_mode": "AUTO", + "terminal_value_euro_per_kwh": 0.0, + "terminal_value_window_hours": 24, "penalties": { "ev_soc_miss": 10 }, @@ -167,9 +177,14 @@ | horizon | `int` | `ro` | `N/A` | Number of optimization steps. | | horizon_hours | `int` | `rw` | `24` | The general time window within which the energy optimization goal shall be achieved [h]. Defaults to 24 hours. | | individuals | `Optional[int]` | `rw` | `300` | Number of individuals (solutions) in the population [>= 10]. Defaults to 300. | -| interval_sec | `int` | `rw` | `3600` | The optimization interval [sec]. Defaults to 3600 seconds (1 hour) | +| interval_sec | `Literal[900, 3600]` | `rw` | `3600` | The optimization interval [sec]. Defaults to 3600 seconds (1 hour) | +| measurement_max_age_seconds | `int` | `rw` | `300` | Maximum age of SoC measurements for configuration-based optimization [s]. | | penalties | `dict[str, Union[float, int, str]]` | `rw` | `required` | Penalty parameters used in fitness evaluation. | | seed | `Optional[int]` | `rw` | `None` | Random seed for reproducibility. None = random. | +| tail_horizon_hours | `int` | `rw` | `48` | Forecast lookahead after the control horizon [h]. No tail commands are issued. Set 0 to disable. | +| terminal_value_euro_per_kwh | `float` | `rw` | `0.0` | Value assigned to usable battery energy remaining at the end of the optimization horizon [EUR/kWh]. This terminal value is independent of the battery LCOS. Only used with terminal_value_mode = FIXED. Defaults to 0 EUR/kWh. | +| terminal_value_mode | `` | `rw` | `AUTO` | How to value the energy left in the battery at the end of the control horizon. AUTO solves the forecast tail with an AUTO continuation proxy at its end (or only the proxy if tail is zero); FIXED uses 'terminal_value_euro_per_kwh'. Defaults to AUTO. | +| terminal_value_window_hours | `int` | `rw` | `24` | Length of the trailing window at the effective tail end the AUTO continuation curve is derived from [h]. One day covers a full load and PV cycle. Defaults to 24 hours. | ::: @@ -187,6 +202,11 @@ "individuals": 300, "generations": 400, "seed": null, + "measurement_max_age_seconds": 300, + "tail_horizon_hours": 48, + "terminal_value_mode": "AUTO", + "terminal_value_euro_per_kwh": 0.0, + "terminal_value_window_hours": 24, "penalties": { "ev_soc_miss": 10 } @@ -210,6 +230,11 @@ "individuals": 300, "generations": 400, "seed": null, + "measurement_max_age_seconds": 300, + "tail_horizon_hours": 48, + "terminal_value_mode": "AUTO", + "terminal_value_euro_per_kwh": 0.0, + "terminal_value_window_hours": 24, "penalties": { "ev_soc_miss": 10 }, diff --git a/docs/_generated/openapi.md b/docs/_generated/openapi.md index 056e6de5..ce21c091 100644 --- a/docs/_generated/openapi.md +++ b/docs/_generated/openapi.md @@ -1,6 +1,6 @@ # Akkudoktor-EOS -**Version**: `v0.3.0.dev2609171651094774` +**Version**: `v0.3.0.dev2609171762020752` **Description**: This project provides a comprehensive solution for simulating and optimizing an energy system based on renewable energy sources. With a focus on photovoltaic (PV) systems, battery storage (batteries), load management (consumer requirements), heat pumps, electric vehicles, and consideration of electricity price data, this system enables forecasting and optimization of energy flow and costs over a specified period. @@ -750,6 +750,32 @@ Get the latest solution of the optimization. --- +## GET /v1/energy-management/optimization/solution/GENETIC/pdf + + +**Links**: [local](http://localhost:8503/docs#/default/fastapi_energy_management_optimization_solution_genetic_pdf_get_v1_energy-management_optimization_solution_genetic_pdf_get), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_energy_management_optimization_solution_genetic_pdf_get_v1_energy-management_optimization_solution_genetic_pdf_get) + + +Fastapi Energy Management Optimization Solution Genetic Pdf Get + + +```python +""" +Render the retained GENETIC result without rerunning optimization. + +Rendering runs outside the event loop. Copy the result before offloading; +its recorded timestamp, interval and inputs own the report's time grid. +The legacy /visualization_results.pdf route continues to serve GENETIC0. +""" +``` + + +**Responses**: + +- **200**: Successful Response + +--- + ## GET /v1/energy-management/optimization/solution/{algorithm} @@ -1343,6 +1369,42 @@ Merge the measurement of given key and value into EOS measurements at given date --- +## POST /v1/optimize + + +**Links**: [local](http://localhost:8503/docs#/default/fastapi_optimize_config_v1_optimize_post), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_optimize_config_v1_optimize_post) + + +Fastapi Optimize Config + + +```python +""" +Optimize GENETIC using configured devices and optional fresh runtime inputs. + +Static settings belong in configuration; query overrides are rejected. +Forecast arrays start at local +midnight and contain Wh per configured GENETIC slot; prices are currency/Wh. +An empty body uses configured providers and fresh measured states of charge. +The deprecated /optimize endpoint continues to run hourly GENETIC0. +""" +``` + + +**Request Body**: + +- `application/json`: { + "$ref": "#/components/schemas/ConfigOptimizationRequest" +} + +**Responses**: + +- **200**: Successful Response + +- **422**: Validation Error + +--- + ## GET /v1/prediction/dataframe diff --git a/docs/development/eos-consolidation.md b/docs/development/eos-consolidation.md new file mode 100644 index 00000000..2a0fb815 --- /dev/null +++ b/docs/development/eos-consolidation.md @@ -0,0 +1,270 @@ +# EOS consolidation — integration record + +Historical integration record, updated 2026-09-16. The completed optimizer port and +current merge instructions are in [review handoff](review-handoff.md). Pending-work +statements below describe earlier checkpoints. + +Current PR-ready packages, dependency order and development guidance: +[PR workflow](pr-workflow.md). The first independent fix is ready locally on main; +the complete feature consolidation remains open. + +## Pinned sources + +- Official main: `4a3724424f98b5ce814d0340ddf3b6b5f1364752` (refreshed after initial checkpoint). +- Feature branch: `d2e2d58237339454dd8bb226f92677c5987f8b27`. +- PR #1256: `6ebd343047b87819b57778359784a616452b5f76`, open, conflicts. +- PR #1305: `60b77f6da2da3a0f59518873fe7fbe9a92e733f4`, open, main target. Latest added commit is formatting only. +- PR #1224: `0b12a34c11685822d97c3bc9f56c6292774cd930`, open, feature target. +- PR #1304: `4b4b49f29902579a8deb34999d71eb2142c0620d`, open, feature target. +- PR #1190 merged into main: GENETIC0 retained separately. + +Sources: [issue #1192](https://github.com/Akkudoktor-EOS/EOS/issues/1192) and linked PRs. +Issue body, all issue comments (none), and PR metadata were retrieved from the GitHub API. +Use `refs/remotes/origin/main`: local `refs/heads/origin/main` is ambiguous. + +## Isolation and recovery + +Original EOS worktree remains on the feature branch. No reset, stash or checkout there. +Integration branch: `integration/eos-consolidation-20260916`, sibling directory +`EOS-integration-20260916`. Unmodified main comparison worktree: `EOS-reference-20260916`. + +Private backup: `%USERPROFILE%/.codex/backups/eos-20260916-120324`. +102 modified/untracked files copied byte-for-byte with SHA256 verification; binary +tracked and index patches, original status/refs, verified `repository.bundle`. +See private `RESTORE.md` and `manifest.json` for recovery into a NEW checkout. +Ignored runtime files stay in the original worktree; this is not an archive of its +virtual environment, caches or ignored production datasets. Backup is private and +may include credentials or real measurements; never stage or publish it. +Only explicitly reviewed source/test/document paths are staged for new commits. + +## Functional matrix + + +| Area | Evidence | Integration state | +| --- | --- | --- | +| Providers, SMARD, fee model | #1192 checked; main prediction modules and #1235 | Keep main implementation; provider equivalence not yet established | +| GENETIC0 / legacy API | #1190; `core/ems.py`, `server/eos.py`, `optimization/genetic0` | Preserved; device regressions pass, full endpoint validation pending | +| Device maps / algorithm conversion | #1256; `devices/settings`, `config/configmigrate.py` | Merged with original history; four merge conflicts resolved | +| Runtime updates | #1305; `config/config.py`, `core/pydantic.py` | Merged with original history; configuration tests pass | +| Stable IDs / LCOS migration | New `test_consolidation_config.py` | Four initial regression failures fixed; 55 configuration tests pass | +| 15-minute battery/inverter physics | Feature `devices/genetic`, `prediction/interpolator.py` | Ported in c0796e1; parameter classes retained in device modules, export/slot/efficiency tests pass | +| New GENETIC, adaptive evolution, export states | Feature `optimization/genetic/genetic.py` | Pending; current optimizer is not the complete feature optimizer | +| Warm start alignment | Feature d2e2d58, `test_genetic_warm_start_alignment.py` | Pending | +| Control horizon / forecast tail / terminal value | Feature a2f4ef6, `tailvalue.py`, `terminalvalue.py` | Primitives ported in f70a786; 22 tests pass. Optimizer/horizon/API integration pending | +| EV deadlines / flexible consumers | Feature f24d9ea; #1256 cycle windows | Pending reconciliation: preserve both per-cycle windows and profile/deadline semantics | +| Imported feed-in revenue | #1224 + #1304, 38 feature regression cases | Neither PR targets main; adapt after GENETIC preparation port, keep author credit | +| Local calibrated PV | Feature f976335/6dc58c3/faed0fd | Pending; keep public provider ID PVForecastAkkudoktor per #1192 | +| Algorithm-specific PDF | Main #1205 vs feature utils/visualize.py | Keep on-demand API; feature slot/tail fields pending | +| Measurement channels/energy/quality/capacity/household | Original uncommitted source + five new tests | Ported in 7338baf to async storage and keyed devices; 127 related tests passed | +| Request configuration learning | Original uncommitted `genetic/configrequest.py` | Secured only; pending | +| HA-private core differences | Separate private repo | Not ported; no HA files modified, no deployment | + + +## Completed packages and tests + +1. `d3f96c4`: merge #1256 onto pinned main. Conflicts in devices.py and three generated + documents; preserve maps/settings and main evolution. Regenerate OpenAPI from + merged code. Do not adopt PR build version bumps. 244 configuration/time-window + tests + 81 device/simulation tests passed before subsequent packages. +2. `b43c8df`: merge #1305 without conflicts. 275 config/configabc/configmigrate/pydantic + tests passed. +3. `e3987b2`: stable map identities, reject key/ID mismatch, preserve renamed LCOS, + repair legacy EV charge-rate migration. 4 regression cases failed before fix; + 55 related tests passed afterwards. These are integration/PR defects, not claimed + as unchanged-main defects. + +Tests currently use the existing Python 3.11.9 environment read-only; pytest adds +this worktree's src. No mutation of the original venv. Missing pypdf 6.19.0 installed into the private +backup test-packages directory and exposed only through test-process PYTHONPATH. +This is not yet validation against every pinned dependency / CI Python version. +Synthetic pytest fixtures; no real devices, server deployment or production config. + +## HA handoff and acceptance + +Do not relocate active feature development to this integration branch yet. +Device collections are keyed maps, with identical map key and device_id. New settings +live in devices/settings; conversion methods derive algorithm parameters. LCOS +configuration is `levelized_cost_of_storage_amt_kwh`; old files migrate. +Runtime config bulk/granular precedence follows #1305. Both algorithms remain +separate; `/optimize` uses GENETIC0 and cannot demonstrate the new GENETIC port. +Pin a final tested commit for HA only after all remaining packages and end-to-end +API/forecast/output tests. No public push, PR, comment or main update is authorized. + +1. `c0796e1`: slot-duration-aware battery/inverter simulation, bounded per-slot + discharge/export, probabilistic direct-use energy model and GENETIC converter + support for LCOS/export levels. Original author co-authorship retained. + Fixed monetary goldens from the previous model are intentionally replaced with + independent grid-flow repricing. Original simulation test passed on unchanged + main; the changed result is not labelled a baseline failure. +2. `f70a786`: bounded forecast reader, concave terminal value and deterministic + forecast-tail primitives. 22 direct primitive tests passed. Does NOT activate + the new optimization algorithm, warmstart, horizon handling or diagnostic API. +3. `d9c434f`: keep main's charge-rate ndarray contract and exported constant after + `#1256`. The first broad collection found this missed overlap. +4. `7338baf`: typed measurement channels, sample quality, energy integration, + household accounting and battery capacity estimation. Adapt all storage calls + and endpoints to main's async API; map battery IDs through device collections. + Capacity estimates survive subsequent runtime config updates and never change + the active capacity_wh. 127 new/existing measurement tests passed, including + JSON/SQLite/LMDB restart, partial coverage and real FastAPI routes with no lifespan. + JSON measurement reload regression reproduced on unchanged afb7bcb (0 records + after reload); applying the previously local singleton fix resolves it. +5. `876756b`: replace stochastic GENETIC output equality with schema, independent + per-slot cost/revenue accounting and physical-range assertions. GENETIC0 goldens + remain unchanged. Short optimizer/PDF runs: 4 GENETIC0 and 4 GENETIC pass; one + 400-generation case for each algorithm skipped by the existing --finalize rule. + New household configuration field has documentation metadata. + +### Remaining integration risks and concrete next package + +The current GENETIC orchestration is still the main-era optimizer with the newly +ported device physics. It must NOT be described as the complete feature optimizer. +Next port `optimization/genetic/genetic.py`, its parameter preparation and solution +model together, carrying feature d2e2d58 warmstart and #1304 revenue fixes. Translate +old top-level optimization settings to `optimization.genetic`. Check the actual +prediction record units: current `elecprice_marketprice_wh` and `feed_in_tariff_wh` +arrays already contain amount/Wh; only amount/kWh configuration is divided by 1000. +Do not apply a second conversion to the existing *_wh arrays. Use #1256 `to_genetic_*` converters +instead of reintroducing parallel settings. Reconcile flexible profiles/deadlines +with #1256 per-cycle windows/completed cycles before exposing the combined API. +Both porting sides must retain their regression cases. Follow with local PV and +algorithm-specific on-demand PDF; do not copy feature's synchronous/automatic-PDF +server paths over main. Then adapt local configrequest.py to the current API. + +No feature-provider robustness fixes are claimed ported merely because a provider +with the same name exists on main. HA private-core differences remain unaudited. +No full 2030-test suite run, --finalize optimization run, dependency-pin CI matrix, +or physical installation validation has been completed. + +### Backup verification + +An independent clone of repository.bundle at the original feature HEAD was restored +using the backup files. All 102 restored SHA256 hashes match. The original worktree +HEAD and all 102 file hashes were also rechecked unchanged after the source ports. + +## Reproducing the focused acceptance run + +Use a disposable environment with the repository dependencies plus pytest, +pytest-asyncio, pytest-xprocess, pytest-cov and pypdf. The private backup contains +`test-environment.json`, `integration-final.xml` and the complete collection log. +Run from the integration worktree (not the original feature worktree): + +```powershell +python -m pytest ` + tests/test_typingmodels.py ` + tests/test_config.py ` + tests/test_configabc.py ` + tests/test_configmigrate.py ` + tests/test_configfile.py ` + tests/test_pydantic.py ` + tests/test_consolidation_config.py ` + tests/test_genetichomeappliance.py ` + tests/test_genetic0battery.py ` + tests/test_genetic0inverterefficiency.py ` + tests/test_genetic0simulation.py ` + tests/test_battery.py ` + tests/test_inverter.py ` + tests/test_inverter_efficiency.py ` + tests/test_geneticsimulation.py ` + tests/test_geneticsimulation2.py ` + tests/test_terminalvalue.py ` + tests/test_tailvalue_physics.py ` + tests/test_interpolator.py ` + tests/test_measurement_channels.py ` + tests/test_measurement_energy.py ` + tests/test_measurement_household.py ` + tests/test_battery_capacity.py ` + tests/test_measurement_file_restore.py ` + tests/test_measurement.py ` + tests/test_genetic0optimize.py ` + tests/test_geneticoptimize.py ` + tests/test_doc.py ` + -q ` + --tb=short +``` + +Do not inherit EOS_DIR/EOS_CONFIG_DIR from documentation generation when running +pytest: these conflict with fixture-controlled temporary config directories. +Generate OpenAPI after committing source changes; source-dirty version timestamps +otherwise make exact documentation comparisons nondeterministic. + +## Final verified checkpoint for this work session + +Focused combined run: **633 passed, 2 skipped in 65.46 s**. The skips are the +existing 400-generation --finalize cases. Zero remaining failures in that run. +All 2030 collected tests were collectable after supplying pypdf and resolving the +charge-rate import overlap; collection alone is not a pass of the full suite. +Ruff passed for the ported measurement/device/interpolator/primitive source files; +git diff --check passed. OpenAPI/config documentation generation and the documentation +comparison tests passed. The first combined run caught fixture state leakage in the +new battery test; reset-before/after fixture isolation resolved it on the repeated run. + +This is a tested partial integration checkpoint, NOT completion of the consolidation. +The source packages are locally committed; release, full feature acceptance and +upstream submission remain pending. Do not move active development or HA deployment +here until the remaining optimizer/prognosis/output/request packages are integrated. + +## PR-readiness verification, 2026-09-16 + +Current main `7ebe6d7` is incorporated. Two reviewable local packages now exist: +`fix/measurement-json-reload` (`dba0c9c`, independently based on main) and +`feat/measurement-energy-quality-capacity` (`ea3383e`, depends on the configuration +integration base `d546f08`). The latter excludes the new optimizer/device physics. +See [PR workflow](pr-workflow.md) and its concrete draft descriptions. + +The expanded integration run completed with 764 passed, 3 skipped and 2 documentation +failures in 453.99 seconds. Both failures were solely stale generated OpenAPI version +metadata, not schema or functional differences. Regenerated the two files; the focused +rerun of all 5 documentation tests passed. No remaining failure from that selection. +The complete expanded selection was not rerun after this documentation-only correction. +Three skips: the two existing long --finalize optimizer cases and the development-only +Energy-Charts forecast case. All 128 active Energy-Charts regressions passed. + +Independent package checks: JSON fix 49 tests; isolated measurement package 453 tests +plus 5 documentation tests; capacity fixture isolation followed by 74 passing tests. +Ruff and source formatting passed for the independent fix; measurement source Ruff +passed. Full pinned Linux/Python 3.13 CI remains outstanding. + +All original 102 saved file hashes and original feature HEAD were checked unchanged. +Both PR worktrees are clean, locally committed, and unpublished. This enables small +independent PRs now; it does not complete the still-open optimizer/PV/output port. + +## First upstream PR published, 2026-09-16 + +Explicit user approval received for publishing the standalone JSON restore fix. +Fetched main `4a37244` (dependency update #1321) and rebased the standalone branch; +its new head is `bdc754d12fd08e0da18d3156642c695f4bf67bd2`. All 49 measurement tests, +source Ruff and formatting checks passed again. Pushed only +`fix/measurement-json-reload` and created [PR #1322](https://github.com/Akkudoktor-EOS/EOS/pull/1322) +against main. Verified the remote head and PR patch: one commit, exactly two files. +The other integration/measurement branches were not pushed. The PR is conflict-free, +not merged; GitHub CI was started and is being checked. Original HEAD and all 102 +backup hashes were rechecked unchanged. Earlier notes saying all packages are +unpublished are historical checkpoints; this section supersedes them for this fix. + +## Parallel package preparation and compatibility + +On the user's explicit request, independent PV and tariff ports and Optimize-mode +compatibility coverage are being prepared in separate worktrees. The main/integration +configuration prerequisite was refreshed to main4a37244 and PR1305head60b77f6; +285 configuration/migration/Pydantic tests pass. The latest 1305 change only formats +three files. See [PR integration matrix](pr-integration-matrix.md) for the seven +remaining functional packages, existing upstream prerequisites, and combined +Optimize acceptance gates. Parallel work is not authorization to publish every lane. + +## Completed parallel checkpoint and first green CI + +Source checkpoint b684748 now combines local PV, imported tariff protection, +device physics/cache corrections and the confirmed Optimize failure-path fix. +The combined selection passed 277 tests (3 regular skips); measurement APIs and +persistence then passed 132 tests; regenerated documentation passed all 5 checks. +These are targeted local checks, not a full integration CI run. The complete new +GENETIC orchestration, config-owned request and result/PDF port remain outstanding. +See the updated integration matrix for exact dependencies and limitations. + +PR #1322 now has two commits through ce132ea. Its initial mypy error in a test's +optional timestamp assertion was fixed; all current checks are green: 1,884 passed, +16 skipped in pinned Linux/Python 3.13 pytest, plus pre-commit/mypy, CodeQL and +Docker build. PR remains open and mergeable. Only this branch has been published; +no merge or HA deployment occurred. The finite CI follow-up was paused after +verification. Original feature HEAD and all 102 backed-up file hashes remain intact. diff --git a/docs/development/eos-ha-handoff.md b/docs/development/eos-ha-handoff.md new file mode 100644 index 00000000..2d467fe2 --- /dev/null +++ b/docs/development/eos-ha-handoff.md @@ -0,0 +1,67 @@ +# EOS interface handoff to Home Assistant + +2026-09-16 — complete EOS feature port prepared for review in eight PRs. + +The combined branch is `feat/genetic-complete`, based on main `4a37244` and the +prerequisite PRs. Use [review handoff](review-handoff.md) for merge order and current +PR links, and [GENETIC rollout](genetic-rollout.md) for configuration and manual +acceptance. Pin the final merged EOS commit after CI and installation acceptance; +the earlier `integration/eos-consolidation-20260916` is a historical checkpoint. + +- Configuration collections `devices.batteries`, `electric_vehicles`, `inverters`, + `home_appliances` are maps keyed by stable device ID. A supplied `device_id` must + match its map key; omitted IDs use that key. Old lists migrate. LCOS configuration + uses `levelized_cost_of_storage_amt_kwh`; old files retain their values. +- Algorithm settings are separate under `optimization.genetic` / `genetic0`. + Existing main configuration uses `interval_sec` and `horizon_hours` there. + Do not send feature-branch top-level interval/horizon settings without migration. +- `#1305` runtime bulk/granular changes are retained above file/environment sources; + config persistence is still an explicit existing save operation. +- New measurement routes: PUT/GET `/v1/measurement/samples`, GET + `/v1/measurement/energy`, GET `/v1/measurement/household`, POST + `/v1/measurement/battery-capacity/{battery_id}`. OpenAPI describes request schemas. + Sample times require timezone; values remain in declared raw units, derived energy + is Wh and includes coverage/quality. Missing data is not implicitly zero energy. +- Capacity estimates are separate evidence; even store_estimate=true never replaces + devices.batteries[id].capacity_wh. Explicitly stored estimates survive runtime + bulk changes; existing config save controls disk persistence. +- Python measurement storage-facing methods are async and must be awaited. The HTTP + schema remains ordinary JSON; HA does not need to mirror EOS internals. +- Legacy POST `/optimize` remains GENETIC0. New POST `/v1/optimize` runs GENETIC + from the typed EOS device configuration. Its body accepts runtime `soc`, + `forecasts`, `start_solution` and `start_solution_datetime`; hardware overrides + and query-string overrides are rejected. Request SoC is an integer percentage + keyed by device ID; automatic + measurement lookup uses recent `-soc-factor` values from 0 to 1. + Missing, stale, future or invalid observations fail instead of implying zero. +- GENETIC supports 900/3600-second slots. Provider power in W is converted once to + Wh per slot; runtime forecast arrays are already Wh per slot and prices per Wh. + Runtime arrays begin at local midnight. Missing control forecasts fail the run; + shorter forecast tails are clipped with diagnostics. Explicit/imported sale + prices, including zero and negative values, remain authoritative. +- The new optimizer includes timestamp-aligned warmstarts, opt-in battery export, + EV deadlines, flexible consumer profiles and per-cycle windows/completed cycles. + AUTO/FIXED terminal value and forecast-tail diagnostics affect scoring, never + extend the executable control horizon, and remain separate from battery wear. +- Automatic GENETIC runs use the site's timezone derived from its coordinates; + explicit run starts and warmstart timestamps retain their timezone/instant. + Configure device IDs and schedules against this timezone, including DST. +- GET `/v1/energy-management/optimization/solution/{algorithm}` returns the stored + algorithm-specific result. Native/generic results and execution plans publish + atomically for the successful run. Failed Optimize calls return errors rather + than a previous successful result. +- GET `/v1/energy-management/optimization/solution/GENETIC/pdf` renders the stored + GENETIC snapshot on demand (404 before a result exists). The legacy PDF endpoint + and GENETIC0 implementation remain separate. The local calibrated Akkudoktor PV + backend and the measurement APIs are included in the prerequisite PRs. + +No HA repository changes, lab deployment, real device control or production config +were performed. Retire the private HA core only after its differences are audited +and a final EOS commit passes the full feature/API acceptance scenarios. + +GENETIC currently supports one inverter, one stationary battery, one EV and multiple +household appliances. Unsupported device counts or inconsistent inverter/battery +links are rejected. Verify both 15- and 60-minute Optimize runs, fresh measurements, +tariff units, result/plan timestamps and reports in the actual HA installation. +EOS code and synthetic scenarios are automated-test coverage; they do not validate +HA entities, physical hardware or the remaining private HA-core differences. diff --git a/docs/development/genetic-rollout.md b/docs/development/genetic-rollout.md new file mode 100644 index 00000000..c02441f2 --- /dev/null +++ b/docs/development/genetic-rollout.md @@ -0,0 +1,93 @@ +# GENETIC configuration and manual acceptance + +Apply the dependencies in [review handoff](review-handoff.md) before the complete +GENETIC PR. These changes implement the feature branch on current EOS interfaces; +they do not migrate a running Home Assistant installation automatically. + +## Configuration + +Hardware and schedules belong in the typed `devices` maps, keyed by stable device +IDs. Configure an inverter and link its `battery_id` to the configured stationary +battery, or leave it empty when there is no battery. GENETIC currently supports one +inverter, one stationary battery, one EV and multiple household appliances. Explicit +validation rejects unsupported device counts and inconsistent links. + +Example optimization settings (merge into an otherwise complete configuration): + +```json +{ + "optimization": { + "algorithm": "GENETIC", + "genetic": { + "interval_sec": 900, + "horizon_hours": 24, + "tail_horizon_hours": 48, + "terminal_value_mode": "AUTO", + "measurement_max_age_seconds": 300 + } + } +} +``` + +Use `interval_sec: 3600` for hourly operation. AUTO evaluates available forecast +continuation; FIXED uses `terminal_value_euro_per_kwh`. Battery wear/LCOS remains a +separate cost. The tail affects scoring, never the emitted control horizon. +`prediction.hours` must cover the control horizon; shorter available tails are +clipped with diagnostics. Missing control forecasts cancel the run. + +Old feature settings such as `optimization.interval` and flat terminal-value fields +migrate into `optimization.genetic`; explicit nested settings win. Check the saved +configuration after migration. GENETIC0 retains its separate settings and `/optimize` API. + +Automatic GENETIC runs use the site's timezone derived from its coordinates, even +when the server runs in UTC. An explicitly supplied run start retains its timezone. +Stored results and warmstarts retain explicit timezones and absolute instants. + +## Requests and measurements + +`POST /v1/optimize` runs GENETIC from configuration. An empty JSON object uses the +configured forecasts and fresh measurements. Runtime input may contain `soc`, +`forecasts`, `start_solution` and `start_solution_datetime`. Hardware overrides in +this body and query-string overrides are rejected. For example, with a configured +device ID `storage`: + +```json +{"soc": {"storage": 42}} +``` + +Request SoC values are integer percentages. Without an override, the measurement +key `-soc-factor` must contain a recent factor from 0 to 1. Invalid, +future or stale measurements never become an assumed zero SoC. Completed appliance +cycles use their configured measurement key, default `.cycles_completed`. + +Optional forecast arrays begin at local midnight, with one value per configured +slot: `pv_forecast_wh` and `total_load` are Wh per slot; `electricity_price_per_wh` +and `feed_in_tariff_per_wh` are currency per Wh. Provider PV/load power in W is +converted by the resolver. Supplied/imported sale tariffs are never replaced by +purchase prices. Missing optional arrays come from the selected providers. + +Successful requests return the native result for that run. A failed run returns an +error and cannot masquerade as a cached success. Generic solutions, plans and reports +are derived from the same stored result. Warmstarts retain their source time so a +later run shifts the previous controls to the new slot origin. + +Retrieve the new GENETIC report with +`GET /v1/energy-management/optimization/solution/GENETIC/pdf`. It renders the stored +snapshot on demand; a missing result returns 404. The legacy PDF URL remains unchanged. + +## Manual acceptance after merge + +1. Back up the running configuration and update all merged EOS packages together. +2. Confirm device IDs, inverter linkage, capacities, efficiencies, import/export + limits, tariff units and the battery-export setting against your actual hardware. +3. Check fresh SoC measurements and forecast coverage, then run Optimize at 60 and + 15 minutes. Inspect timestamps, slot energies, native/generic results and the PDF. +4. Compare AUTO and FIXED terminal value with identical inputs. Tail diagnostics + must not appear as extra commands; sale prices must retain their configured signs. +5. Exercise an EV departure and a flexible consumer schedule, including a completed + cycle. Check permitted windows, required energy and device commands in your setup. +6. Make a measurement stale or omit control forecast coverage and confirm the request + fails explicitly. Verify the legacy GENETIC0 endpoint if your installation uses it. + +Automated checks cover modeled behavior. The final manual test validates the actual +HA entities, provider data and physical installation. diff --git a/docs/development/pr-drafts/imported-feedin-main.md b/docs/development/pr-drafts/imported-feedin-main.md new file mode 100644 index 00000000..aa61fec6 --- /dev/null +++ b/docs/development/pr-drafts/imported-feedin-main.md @@ -0,0 +1,25 @@ +# fix(optimization): reject invalid imported feed-in tariffs without fallback + +Published with user approval as [PR #1324](https://github.com/Akkudoktor-EOS/EOS/pull/1324). +Branch `fix/imported-feedin-main`, head `2a3b961`, base main `4a37244`. +Two commits, two changed files. Original contributions from #1224/#1304 are credited. + +GENETIC preparation now cancels if FeedInTariffImport cannot supply a finite +one-dimensional tariff array matching the forecast length. It keeps the configured +provider instead of switching to demo tariffs. Positive, zero and negative valid +amount/Wh revenues remain unchanged. Other providers keep their fallback behavior. + +Publication check: 68 passed, 2 regular long-running tests skipped; source Ruff, +format and diff checks passed. Scoped mypy passed for the two changed files with +transitive imports/untyped dependency diagnostics excluded. Full pinned Linux CI: 1921 passed, 16 skipped; +pre-commit including full mypy, CodeQL and Docker passed. +Regression coverage includes seven provider identities, simulation arithmetic and +actual timestamped imports. Eleven invalid-input regressions fail on unchanged main. + +The existing forward-fill path is unchanged: validation of its resulting array +does not establish raw timestamp coverage or freshness. Feature direct-marketing +overrides and quarter-hour orchestration remain part of the later GENETIC port. +Legacy /optimize and GENETIC0 are unchanged. The source fix is already integrated +with PV, device/configuration and Optimize-result packages in local combined tests. + +No merge or deployment. All current-head checks completed successfully. diff --git a/docs/development/pr-drafts/measurement-energy-quality-capacity.md b/docs/development/pr-drafts/measurement-energy-quality-capacity.md new file mode 100644 index 00000000..8def7c23 --- /dev/null +++ b/docs/development/pr-drafts/measurement-energy-quality-capacity.md @@ -0,0 +1,38 @@ +# feat(measurement): add typed energy, quality and capacity APIs + +Local branch: `feat/measurement-energy-quality-capacity`. +Review base: `feat/config-integration-base` (`9038b65`). +Eventual target: official main, after its configuration prerequisites land. +Status: published as [PR #1326](https://github.com/Akkudoktor-EOS/EOS/pull/1326), head `635ff2d`, against the +prerequisite branch. + +## Proposed PR body + +Add typed measurement channels with explicit units and timestamp semantics, sample +quality, energy integration, household energy balances and battery capacity estimates. +Adapt persistence and REST access to main's asynchronous storage. Battery estimates +use keyed device identities and remain separate from the configured active capacity; +storing an estimate requires an explicit request and preserves runtime config updates. + +Expose `/v1/measurement/samples`, `/v1/measurement/energy`, +`/v1/measurement/household` and `/v1/measurement/battery-capacity/{battery_id}`. +Regenerate the configuration and OpenAPI contracts. Use synthetic data only. + +Validation: 453 configuration/measurement/device simulation tests and 5 documentation +tests pass. A further 74 measurement/capacity tests pass after carrying over fixture +isolation. Ruff passes for measurement source and the measurement REST module. +Final pinned Linux/Python 3.13 CI: 2046 passed, 16 skipped; pre-commit with full mypy and Docker passed. +CodeQL becomes applicable after retargeting to main. + +Depends on the device maps/converters in #1256, runtime configuration in #1305 and +local fixes for stable device IDs, LCOS migration and charge-rate compatibility. +Includes the independently prepared JSON restore fix; it should land separately first. +No new GENETIC orchestration or battery/inverter physics is included in this branch. + +## Submission gate + +Do not open this whole branch against main now: its ancestry still includes the +unmerged configuration PRs. Preserve those contributors' existing PRs and credit. +Once prerequisites are merged, rebuild/rebase the measurement package onto that +main, inspect the resulting diff and rerun combined tests and CodeQL before merging. +The comparison to `feat/config-integration-base` isolates today's measurement work. diff --git a/docs/development/pr-drafts/measurement-json-reload.md b/docs/development/pr-drafts/measurement-json-reload.md new file mode 100644 index 00000000..a867bf0e --- /dev/null +++ b/docs/development/pr-drafts/measurement-json-reload.md @@ -0,0 +1,41 @@ +# fix(measurement): restore JSON records into the existing singleton + +Target: `Akkudoktor-EOS/EOS:main`. +Local branch: `fix/measurement-json-reload`. +Local head: `ce132ea` (based on main `4a37244`). +Status: published with explicit user approval as [PR #1322](https://github.com/Akkudoktor-EOS/EOS/pull/1322). +Two commits, two files; published head and diff verified. Not merged. +The second commit adds an explicit non-null timestamp assertion after CI mypy +flagged the test. All 49 local tests still pass. Pre-commit (including mypy), +CodeQL and Docker build passed on ce132ea. Full pinned Linux/Python 3.13 CI pytest +also passed: 1,884 passed, 16 skipped, including --finalize and config side-effect +checks (run 35118801197). PR remains open and mergeable; no merge performed. + +## Proposed PR body + +When measurement persistence falls back to JSON, loading a saved file reports success +but does not restore its records: validating a second `Measurement` returns the +existing singleton. Parse and validate the individual records before inserting them +into that singleton. Invalid files now return `False` instead of reporting success. + +Regression coverage checks round trips, timestamp preservation, repeated loading, +merging with existing timestamps, malformed files without partial validation writes, +and database-provider precedence. Four regression cases fail on unchanged main. +With the fix, all 49 measurement tests pass. + +Validation: `python -m pytest tests/test_measurement_file_restore.py tests/test_measurement.py -q`; +Ruff check and format check of the changed source; `git diff --check`. +Run locally on Windows/Python 3.11.9, followed by successful pinned Linux/Python 3.13 +CI and its complete test suite as recorded above. + +No settings or API schema changes. This fix is independent of #1256, #1305 and the +GENETIC port. + +## Exact review scope + +- `src/akkudoktoreos/measurement/measurement.py` +- `tests/test_measurement_file_restore.py` + +Only these two files differ from the pinned main. Do not publish the integration +branch as part of this PR. Publication approval was received on 2026-09-16. Only this named branch was pushed. +Inspect CI and review before any merge; merging was not part of this publication request. diff --git a/docs/development/pr-drafts/optimize-run-result.md b/docs/development/pr-drafts/optimize-run-result.md new file mode 100644 index 00000000..f30fd611 --- /dev/null +++ b/docs/development/pr-drafts/optimize-run-result.md @@ -0,0 +1,24 @@ +# fix(optimization): return only the current completed Optimize result + +Published with user approval as [PR #1323](https://github.com/Akkudoktor-EOS/EOS/pull/1323). +Branch: `fix/optimize-run-result`; base main `4a37244`; head `ef8d913`. +Three commits, three changed files. No merge or deployment. + +A failed explicit optimization previously returned an old cached solution as HTTP +200. Conversion failures could publish a new native result alongside the previous +generic result and plan. Return this run's completed solution directly to the route; +build all three representations before publishing them together. Preserve previous +consistent results after optimizer/conversion errors and skip the failed run's +control dispatch. Legacy `/optimize` remains GENETIC0; automatic mode keeps its +configured algorithm selection. No optimizer mathematics or slot behavior changes. + +Validation: 38 tests passed, 2 regular long-running tests skipped. All 22 new +regression cases passed again after typing dynamic test keyword arguments. Source +Ruff, formatting and diff checks passed. Scoped mypy passed for all three changed +files (transitive imports/untyped dependency diagnostics excluded locally). +Pinned Linux CI: 1908 passed, 16 skipped; pre-commit including full mypy, CodeQL and Docker passed. Thirty +regressions also passed separately in UTC and Europe/Berlin; timezone expectations are now explicit. + +The fix is already combined with PV, tariff and device packages in the integration +branch. Integration also carries the test annotation correction. Independent of +`#1322`, #1256 and #1305. Original working copy remains untouched. diff --git a/docs/development/pr-integration-matrix.md b/docs/development/pr-integration-matrix.md new file mode 100644 index 00000000..f976a258 --- /dev/null +++ b/docs/development/pr-integration-matrix.md @@ -0,0 +1,125 @@ +# Remaining EOS packages and Optimize compatibility + +Historical planning snapshot. The remaining packages below are now implemented; +see [review handoff](review-handoff.md) for current dependencies and acceptance. + +Snapshot: 2026-09-16. Official main: `4a37244`. Feature source: `d2e2d58`, plus +separately backed-up local work. This is a functional package estimate, not a claim +that every differing commit requires its own PR. + +## What is still missing on main + + +| Package | Missing behavior relative to feature/local work | Local state | Dependency | +| --- | --- | --- | --- | +| Device physics | Slot-duration-aware battery/inverter flows, export control, efficiency and limits | Published as #1327 against the configuration prerequisite branch; combined in integration | #1256 device settings/converters | +| Complete GENETIC | Quarter-hour orchestration, adaptive evolution, export states, warmstart alignment, forecast tail/terminal value, EV deadlines and flexible consumer profiles | Physics and primitives exist; orchestration/parameter/output integration remains open | Device physics, #1256, tariff contract | +| Imported tariff protection | Preserve supplied/imported revenue; avoid silent demo or market-price replacement | Published as #1324 and combined in integration; feature-specific override still required inside GENETIC port | Main patch independent; second part belongs with GENETIC | +| Local calibrated PV | Local Akkudoktor PV calculation, measurement calibration and outage handling under existing provider ID | Published as #1325 against main; combined in integration | Provider-specific settings; combined forecast/optimizer acceptance later | +| Measurement APIs | Typed channels, quality, energy integration, household balance and capacity estimate APIs | Published as #1326 against the configuration prerequisite branch | #1256, #1305, configuration corrections; JSON fix #1322 | +| Config-owned Optimize request | Local ConfigOptimizationRequest, /v1/optimize, runtime observations and common parameter resolver | Backed up; async/maps/converters adaptation pending | New GENETIC and #1305 | +| Result/PDF output | Quarter-hour, flexible-consumer, export, tail/rest-value diagnostics in main's on-demand algorithm-specific output | Pending | Final GENETIC result contract | + + +Four of the seven feature packages are now published as #1324–#1327. Three broader +packages remain unported: complete GENETIC, the config-owned Optimize API, and result/PDF +output. Their final PR split depends on the resulting contracts. #1256 and #1305 are existing +foundation PRs, not two newly invented replacements. #1224/#1304 already address +parts of the tariff work against the old feature branch; preserve/reconcile those +contributions rather than count duplicate implementations as separate deliverables. +PR #1322 is already published and is additional to this feature table. + +One additional standalone defect fix is published as PR #1323, fix/optimize-run-result: +an explicit Optimize request must return only its own completed result. Cache +method identity was also corrected as a prerequisite within the device package; +it does not currently add another planned feature PR. + +Local helper scripts and private HA-core divergence are not silently included in this +count. They remain separately secured/to be audited. Changelog/release work follows +acceptance; it is not another optimizer implementation. + +## Parallel lanes + +- Local PV: `feat/local-pv-main-port`, sibling worktree `EOS-pr-local-pv`. +- Tariff preparation: `fix/imported-feedin-main`, sibling `EOS-pr-feedin-main`. +- Device physics: `feat/slot-device-physics`, sibling `EOS-pr-device-physics`. +- Optimize failure correction: `fix/optimize-run-result`, sibling + `EOS-reference-optimize-main` (now a named review branch). +- Optimize compatibility tests: `test/optimize-pr-contracts`, sibling + `EOS-pr-optimize-contracts`. These are shared acceptance coverage, not necessarily + an extra standalone public PR. +- Root/integration: configuration prerequisites, shared API/algorithm contracts and + eventual combination of the tested packages. + +Do not have multiple lanes independently rewrite geneticparams.py, EMS.run or the +same configuration structure. A tested tariff patch will be carried into the core +port; the core port must keep its regression cases. PV keeps the public +PVForecastAkkudoktor ID and existing remote behavior unless explicitly selected. +Parallel preparation does not imply parallel unreviewed merges or publishing all +branches. The JSON fix #1322 and subsequently the Optimize fix #1323 have publication +approval. Tariff fix #1324 was subsequently authorized and published. The user then +authorized all prepared packages: PV #1325, measurement #1326 and device physics +`#1327` are also published. The latter two target the explicit configuration comparison +branch 9038b65 pending their prerequisites. No upstream merges were performed. + +## Three different Optimize contracts + +1. Legacy POST `/optimize` on main explicitly runs GENETIC0, hourly. Existing payload + and legacy response aliases must keep working regardless of configured default. +2. Automatic EMS with `ems.mode=OPTIMIZATION` selects `optimization.algorithm`: + GENETIC or GENETIC0. Preparation and solution conversion are asynchronous. + PREDICTION and DISABLED must not accidentally run optimization or dispatch controls. +3. The feature worktree's LOCAL POST `/v1/optimize` and ConfigOptimizationRequest are + not present on main/integration yet. They must be adapted to current maps, + converters and algorithm-specific settings. Old feature `/optimize` meant a + different optimizer; clients need explicit migration, not a silent route switch. + +Current blockers: GENETIC.prepare still forces interval_sec to 3600 and EMS start +alignment floors to the hour. Fifteen-minute device tests do not prove quarter-hour +Optimize-mode support. The core package must change preparation, slot alignment, +optimization and response metadata together. + +Confirmed on unchanged main 4a37244: after a failed explicit optimization, the HTTP +route returned the previous stored solution with HTTP 200. Conversion errors could +also publish a new native result alongside an old generic result/plan. The local +fix prepares all three before publishing and returns the successful current-run +result directly to the route. Errors retain the previous coherent trio, restore +IDLE and do not dispatch controls for that failed optimization. Strict passing +regression tests replace the initial expected-failure audit cases. + +## Shared acceptance before dependent PRs can land + + +| Area | Required combined check | +| --- | --- | +| Routing | Legacy /optimize remains GENETIC0; automatic mode selects the configured algorithm; the new explicit API selects GENETIC deliberately | +| Configuration | Stable device IDs/maps and converters, runtime changes retained, old aliases migrated without losing values | +| Time | 900/3600-second slots, non-hour-aligned start and advancing warmstart, timezone/DST boundaries, matching output timestamps | +| Units | PV/load predictions are W; integrate once to slot Wh. `feed_in_tariff_wh` and `elecprice_marketprice_wh` already yield amount/Wh; convert amount/kWh configuration exactly once | +| Tariffs | Positive/zero/negative/imported revenue remains distinct from purchase cost; missing or invalid imported data does not silently become demo data | +| Devices | SOC/energy balance, charge/discharge and inverter limits, allowed/blocked battery export | +| Consumers | Feature profiles/deadlines and #1256 per-cycle windows/gaps both preserved; impossible schedules fail explicitly | +| Results | Raw algorithm result, generic optimization solution, execution plan and on-demand PDF agree on slots and device IDs | +| Errors | Failed preparation/optimization produces no new control dispatch and no misleading fresh-success response using old results | + + +A passing individual PR is insufficient: after combining dependent packages, run +these synthetic end-to-end API cases together with the GENETIC0 regression suite. +No device control, HA deployment or production configuration is part of this work. + +## Completed combined check + +Integration source checkpoint b684748 combines tariff b2a4e2f, Optimize fix 8ed65ec, +device/cache corrections be184a6 and 38eb4ad, and local PV 84abe05, on the refreshed +configuration base. The combined API failure/algorithm-selection, tariff, cache, +battery/inverter, PV, configuration/migration and both optimizer test selection +passed: **277 passed, 3 regular skips**. No expected-failure markers were used to +hide Optimize defects. XML: private backup integration-parallel-compatibility.xml. +The complete measurement/channel/energy/household/capacity selection additionally +passed **132 tests** on this combined source checkpoint, including JSON persistence. + +This is local Windows/Python 3.11 verification, not full pinned CI or acceptance of +the pending new GENETIC. The wider standalone PV suite separately reproduced an +existing Windows file-timestamp test failure on unchanged main; that test was +explicitly excluded from its reported 164-passing selection. Forecast retention +does not establish complete horizon coverage; that remains a core-port gate. diff --git a/docs/development/pr-workflow.md b/docs/development/pr-workflow.md new file mode 100644 index 00000000..7ec5cd98 --- /dev/null +++ b/docs/development/pr-workflow.md @@ -0,0 +1,134 @@ +# EOS: Arbeitsstand und Weg zu kleinen PRs + +Historischer Zwischenstand. Die damals offenen GENETIC-Pakete sind inzwischen +implementiert. Maßgeblich sind [Review-Handoff](review-handoff.md) für die acht PRs, +Abhängigkeiten und Merge-Reihenfolge sowie [GENETIC-Rollout](genetic-rollout.md) +für Konfiguration und manuelle Abnahme. Die Statusangaben unten dokumentieren +frühere Arbeitsschritte. + +Stand: 16.09.2026. Offizielles main für PR #1322: `4a37244`. +Die lokale Integration enthält ebenfalls diesen main-Stand und den aktualisierten +Stand von #1305 (`60b77f6`). + +## Was jetzt möglich ist + +Ein unabhängiger, lokal getesteter PR ist veröffentlicht: `fix/measurement-json-reload` +auf aktuellem main. Sein Worktree ist `../EOS-pr-measurement-json`; er enthält nur +zwei geänderte Dateien. Der veröffentlichte [PR #1322](https://github.com/Akkudoktor-EOS/EOS/pull/1322) +enthält zwei Commits bis `ce132ea`. Der PR-Text steht in +[measurement-json-reload.md](pr-drafts/measurement-json-reload.md). + +Das ist ein konkreter Einstieg in den PR-Workflow. Die vollständige Übernahme aller +Funktionen aus dem alten Feature-Branch ist noch NICHT abgeschlossen. + +## Branches und ihre Aufgaben + + +| Branch | Zweck | Freigabezustand | +| --- | --- | --- | +| `fix/measurement-json-reload` | Kleiner JSON-Ladefehler direkt auf main | Veröffentlicht als #1322; gesamte CI grün, noch nicht gemergt | +| `feat/config-integration-base` | Zusammengeführte #1256/#1305 plus Integrationskorrekturen | Veröffentlichter Vergleichsbranch 9038b65; kein konkurrierender Sammel-PR | +| `feat/measurement-energy-quality-capacity` | Messdatenfunktionen ohne neue Optimiererphysik | PR #1326 gegen Konfigurationsbasis; später auf main umstellen | +| `fix/imported-feedin-main` | Importierte Einspeisetarife erhalten und prüfen | Veröffentlicht als #1324; CI läuft | +| `feat/local-pv-main-port` | Lokale PV-Prognose und Kalibrierung | PR #1325 gegen main | +| `feat/slot-device-physics` | Slotphysik und getrennte Cache-Methoden | PR #1327 gegen Konfigurationsbasis; später auf main umstellen | +| `fix/optimize-run-result` | Nur das Ergebnis des erfolgreichen aktuellen Laufs zurückgeben | Veröffentlicht als #1323; CI läuft | +| `integration/eos-consolidation-20260916` | Zusammenführung und Prüfung aller Portierungspakete | Unvollständig; kein Gesamt-PR und kein HA-Release | +| `feat/direct-marketing-battery-grid-export` | Ursprüngliche Entwicklung mit lokalen Änderungen | Unverändert erhalten und gesichert | + + +Der Worktree `../EOS-pr-measurement` gehört zum Messdatenpaket. +`../EOS-integration-20260916` bleibt der zeitlich begrenzte Portierungsarbeitsplatz. +`../EOS-reference-20260916` bleibt der unveränderte Vergleichsstand für Basisfehler. +Ein Worktree ist nur ein Arbeitsverzeichnis; Gegenstand eines PRs ist der Branch. + +## Umgesetzt und geprüft + +- Geräte-Maps, Algorithmuskonvertierung und Laufzeitkonfiguration aus #1256/#1305. +- Korrekturen für stabile Geräte-IDs, LCOS-Migration und Ladeleistungslisten. +- Messkanäle, Qualität, Energieintegration, Haushaltsbilanz und Kapazitätsschätzung, + einschließlich asynchroner Speicherung und REST-Schnittstellen. +- Im Integrationsbranch außerdem Viertelstunden-Gerätephysik, begrenzter + Batterieexport und Wirkungsgrade; Restwert-/Prognose-Nachlauf-Bausteine. +- GENETIC0 bleibt separat. Sein `/optimize`-Endpunkt beweist keine vollständige + Portierung des neuen GENETIC. + +## Noch offen + +1. Neues GENETIC vollständig auf main-Strukturen anpassen: Viertelstundenplanung, + Exportzustände, adaptive Evolution und zeitlich korrekter Warmstart. +2. Flexible Lastprofile/EV-Fristen mit den Mehrfachzyklen und Zeitfenstern aus + `#1256` verbinden. Beide vorhandenen Funktionssätze müssen erhalten bleiben. +3. Horizont, Prognoselücken, Nachlauf und Restwert mit Optimierer und Ergebnissen + verdrahten; bisher sind nur die Bausteine übernommen. +4. Den bereits portierten Tarifschutz auch in der neuen GENETIC-Parametervorbereitung + erhalten; dort Prognosegrenzen und Lücken verbindlich prüfen. +5. Algorithmusspezifische PDF-Ausgabe und die lokale Konfigurationslern-Anfrage + integrieren. Die lokale kalibrierte PV-Prognose ist inzwischen portiert. +6. Gesamtabnahme einschließlich API-Weg des neuen GENETIC und gepinnter CI. + Danach erst Übergabe eines festen EOS-Commits an HA und Release-Arbeiten. + +## PR-Reihenfolge + +1. JSON-Fix als PR #1322 veröffentlicht: CI und Review prüfen, danach über Merge entscheiden. +2. `#1256/#1305` über ihre vorhandenen PRs zusammenführen; lokale Korrekturen dort + zuordnen. Keine pauschale Veröffentlichung der kombinierten Integrationsbasis. +3. Das isolierte Messdatenpaket auf diesen main-Stand setzen, Diff prüfen und + nochmals testen; dann als eigenen PR einreichen. +4. Optimierer, Tarifschutz, PV und Ausgabe jeweils als abgegrenzte Pakete fertigstellen. + Abhängige PRs ausdrücklich als solche behandeln. + +Für jede neue unabhängige Änderung: aktuellen `refs/remotes/origin/main` holen, +einen Themenbranch mit eigenem Worktree starten, lokal testen, den Diff prüfen, +dann PR gegen main. Nach Review und grüner CI mergen. Alte Worktrees erst nach +abgeschlossener Übernahme und Prüfung lokaler Änderungen aufräumen. + +Wegen der vorhandenen gleichnamigen lokalen Branch-Referenz ausdrücklich +`refs/remotes/origin/main` verwenden. Keine neuen unabhängigen Funktionen auf den +alten großen Feature-Branch oder die Integrationsbasis stapeln. + +## Kann die laufende Entwicklung schon umziehen? + +Unabhängige Fehlerkorrekturen und neue Funktionen können ab jetzt in Themenbranches +auf main erfolgen. Für Entwicklung, die den vollständigen neuen GENETIC oder die +noch fehlenden Funktionen benötigt, ist der Integrationsstand noch nicht abgenommen. +Die ursprüngliche Arbeitskopie bleibt erhalten. JSON-Fix #1322, Optimize-Fix #1323 +und Tarifschutz #1324 sind veröffentlicht. Auf weitere Freigabe folgten PV #1325, +Messdaten #1326 und Gerätephysik #1327; nichts wurde gemergt oder deployt. +Der aktuelle Review-Überblick steht in [review-handoff.md](review-handoff.md). + +## HA-Übergabe + +Noch keinen neuen Gesamtstand pinnen oder deployen. Die geplante Schnittstelle nutzt +Geräte-Maps mit stabilen IDs, `levelized_cost_of_storage_amt_kwh`, asynchrone +Messdatenzugriffe und getrennte GENETIC/GENETIC0-Pfade. Details stehen in +[eos-ha-handoff.md](eos-ha-handoff.md). HA-Dateien wurden nicht verändert. + +## Nachweise und Grenzen + +Der JSON-PR: 49 bestandene Tests, Ruff und Formatprüfung. +Das isolierte Messdatenpaket: 453 bestandene Tests plus 5 Dokumentationstests; +74 Tests nach Übernahme der Fixture-Isolation nochmals erfolgreich. +XML-Protokolle liegen in der privaten Sicherung `eos-20260916-120324`. +Für PR #1322 ist die gepinnte Linux/Python-3.13-CI inzwischen bestätigt: +1.884 Tests bestanden, 16 übersprungen; Pre-commit/Mypy, CodeQL und Docker-Build +erfolgreich auf `ce132ea`. Die aktuellen CI-Ergebnisse aller sechs PRs stehen im Review-Handoff. + +Zusätzlicher Integrationslauf: 764 Tests bestanden, 3 übersprungen; zwei zunächst +fehlgeschlagene Dokumentationsvergleiche betrafen ausschließlich die Versionsangabe. +Nach Neugenerierung bestanden alle 5 Dokumentationstests. Darunter sind außerdem +128 bestandene Energy-Charts-Regressionen zum neuen main-Commit dokumentiert. + +Früherer gemeinsamer Source-Stand `b684748`: 277 Tests bestanden, 3 regulär +übersprungen, für PV, Tarifschutz, Gerätephysik, Cache, Konfiguration und beide +bisherigen Optimierer einschließlich API-Fehlerbehandlung. Anschließend bestanden +alle 132 Messdaten-/Haushalts-/Kapazitätsprüfungen auf diesem gemeinsamen Stand. +Die vollständige neue GENETIC-Orchestrierung bleibt offen. Pakete und +Kompatibilitätsbedingungen stehen in [pr-integration-matrix.md](pr-integration-matrix.md). +Die neu generierte gemeinsame Dokumentation besteht ebenfalls alle fünf Prüfungen. + +Aktueller Abschluss: Alle sechs veröffentlichten PRs haben ihre vorgesehenen +GitHub-Prüfungen bestanden. Gemeinsamer Stand: 253 Dateien ohne Mypy-Fehler, +37 gezielte Nachprüfungen und vollständiger Sphinx-Build erfolgreich. Die Grenzen +des Windows-Gesamtlaufs und die nachgewiesenen main-Baselinefehler sind im +[Review-Handoff](review-handoff.md) dokumentiert. Nichts wurde gemergt oder deployt. diff --git a/docs/development/review-handoff.md b/docs/development/review-handoff.md new file mode 100644 index 00000000..6395f145 --- /dev/null +++ b/docs/development/review-handoff.md @@ -0,0 +1,60 @@ +# EOS review and merge handoff + +The consolidation implements the remaining GENETIC optimizer, configuration-owned +Optimize request and result/PDF output. See [GENETIC rollout](genetic-rollout.md) +for configuration and manual acceptance. Historical planning documents in this +directory describe earlier checkpoints; this handoff supersedes their pending-work lists. + +## Integration status + +The seven prerequisite packages #1322-#1328 are merged into main. PR #1329 was +merged into `integration/genetic-prerequisites`, not into main. The final delivery +branch `feat/genetic-complete-main-port` brings that complete implementation directly +to main. Until that final PR is merged, main lacks the complete GENETIC port. + +The foundation preserves the original #1256/#1305 contributions and compatibility +corrections; do not merge those original PRs again as extra prerequisites. + +Main `3c86254` has exactly the production source tree of the original prerequisite +integration `5eacd54`. Only generated API version strings and one corrected test +import differ. The final delivery preserves the complete production source and +tests from #1329, including that corrected import, and regenerates API/configuration +documents from the combined code. Its merge conflicts arise from the rewritten +squash ancestry, not from additional production changes on main. + +Review and merge the final PR with **main** as its target and all four current-head +checks green: pytest, pre-commit, Docker and CodeQL. Squash and merge is supported. +Do not use a merge into the comparison branch as a release. Keep the original +working copy and backup until manual installation acceptance is complete. + +## Review and validation + +- Optimize: preparation or conversion failure never returns a previous successful + result; native result, generic solution and execution plan publish atomically. +- Forecasts: provider power is converted from W to slot Wh exactly once. Raw missing + records stay missing, control coverage is mandatory, and a shorter tail is clipped. +- Economics: explicit/imported sale prices remain authoritative, including zero and + negative values. Battery export is opt-in; terminal value is separate from LCOS. +- Time: 15/60-minute slots, repeated DST hours, local-midnight forecast origins and + non-integer timezone offsets are covered. Old GENETIC reports use saved timestamps. +- Devices: EV deadlines, flexible power profiles, crossed per-cycle windows, completed + cycles and minimum gaps are tested through the optimizer and result conversion. +- Compatibility: GENETIC0 keeps its legacy request and device implementation. Both + algorithms retain finite solution validity and existing persistent plan instructions. + +Exact workflow results belong to each PR's current head; superseded green heads do +not prove a later revision. The combined checks cover pinned mypy, generated OpenAPI +and configuration, physics, native HTTP, automatic preparation, PDF generation and +the 400-generation optimizer regression. + +Local Windows server-PID tests and Docker builds have known environment failures +that also reproduce on unchanged main. Linux CI remains the full-suite gate. Local +PDF pixel comparison needs an optional converter; report semantics and PDF bytes +are tested independently. Production HA/device behavior still needs manual acceptance. + +## Preservation and release boundary + +The original feature working copy is read-only. Its original HEAD and 102 saved files +are checked against the private preservation manifest. Runtime configuration, secrets +and real measurements are excluded from the PRs. No remote PR is merged and no HA +configuration, production server or physical device is changed by this preparation. diff --git a/openapi.json b/openapi.json index 7eabfef0..6cf3fb12 100644 --- a/openapi.json +++ b/openapi.json @@ -8,7 +8,7 @@ "name": "Apache 2.0", "url": "https://www.apache.org/licenses/LICENSE-2.0.html" }, - "version": "v0.3.0.dev2609171651094774" + "version": "v0.3.0.dev2609171762020752" }, "paths": { "/v1/measurement/battery-capacity/{battery_id}": { @@ -2834,6 +2834,24 @@ } } }, + "/v1/energy-management/optimization/solution/GENETIC/pdf": { + "get": { + "tags": [ + "energy-management" + ], + "summary": "Fastapi Energy Management Optimization Solution Genetic Pdf Get", + "description": "Render the retained GENETIC result without rerunning optimization.\n\nRendering runs outside the event loop. Copy the result before offloading;\nits recorded timestamp, interval and inputs own the report's time grid.\nThe legacy /visualization_results.pdf route continues to serve GENETIC0.", + "operationId": "fastapi_energy_management_optimization_solution_genetic_pdf_get_v1_energy_management_optimization_solution_GENETIC_pdf_get", + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/pdf": {} + } + } + } + } + }, "/v1/energy-management/plan": { "get": { "tags": [ @@ -3000,6 +3018,47 @@ } } }, + "/v1/optimize": { + "post": { + "tags": [ + "optimize" + ], + "summary": "Fastapi Optimize Config", + "description": "Optimize GENETIC using configured devices and optional fresh runtime inputs.\n\nStatic settings belong in configuration; query overrides are rejected.\nForecast arrays start at local\nmidnight and contain Wh per configured GENETIC slot; prices are currency/Wh.\nAn empty body uses configured providers and fresh measured states of charge.\nThe deprecated /optimize endpoint continues to run hourly GENETIC0.", + "operationId": "fastapi_optimize_config_v1_optimize_post", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ConfigOptimizationRequest" + } + } + } + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/GeneticSolution" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, "/optimize": { "post": { "tags": [ @@ -3393,6 +3452,46 @@ ], null ] + }, + "min_soc_deadline_datetime": { + "anyOf": [ + { + "type": "string", + "format": "date-time" + }, + { + "type": "null" + } + ], + "title": "Min Soc Deadline Datetime", + "description": "Absolute moment by which 'min_soc_percentage' has to be reached (departure time). A date time without timezone is read as local time. None means end of the optimization horizon.", + "examples": [ + null, + "2026-07-16T07:00:00+02:00" + ], + "x-scope": [ + "GENETIC" + ] + }, + "min_soc_max_duration_h": { + "anyOf": [ + { + "type": "number", + "exclusiveMinimum": 0.0 + }, + { + "type": "null" + } + ], + "title": "Min Soc Max Duration H", + "description": "Maximum time from the start of the optimization until 'min_soc_percentage' has to be reached [h]. Combined with 'min_soc_deadline_datetime' the earlier of the two applies.", + "examples": [ + null, + 6.0 + ], + "x-scope": [ + "GENETIC" + ] } }, "type": "object", @@ -3601,6 +3700,46 @@ null ] }, + "min_soc_deadline_datetime": { + "anyOf": [ + { + "type": "string", + "format": "date-time" + }, + { + "type": "null" + } + ], + "title": "Min Soc Deadline Datetime", + "description": "Absolute moment by which 'min_soc_percentage' has to be reached (departure time). A date time without timezone is read as local time. None means end of the optimization horizon.", + "examples": [ + null, + "2026-07-16T07:00:00+02:00" + ], + "x-scope": [ + "GENETIC" + ] + }, + "min_soc_max_duration_h": { + "anyOf": [ + { + "type": "number", + "exclusiveMinimum": 0.0 + }, + { + "type": "null" + } + ], + "title": "Min Soc Max Duration H", + "description": "Maximum time from the start of the optimization until 'min_soc_percentage' has to be reached [h]. Combined with 'min_soc_deadline_datetime' the earlier of the two applies.", + "examples": [ + null, + 6.0 + ], + "x-scope": [ + "GENETIC" + ] + }, "measurement_key_soc_factor": { "type": "string", "title": "Measurement Key Soc Factor", @@ -3980,6 +4119,52 @@ "title": "ConfigEOS", "description": "Singleton configuration handler for the EOS application.\n\nConfigEOS extends `SettingsEOS` with support for default configuration paths and automatic\ninitialization.\n\n`ConfigEOS` ensures that only one instance of the class is created throughout the application,\nallowing consistent access to EOS configuration settings. This singleton instance loads\nconfiguration data from a predefined set of directories or creates a default configuration if\nnone is found.\n\nInitialization Process:\n - Upon instantiation, the singleton instance attempts to load a configuration file in this order:\n 1. The directory specified by the `EOS_CONFIG_DIR` environment variable\n 2. The directory specified by the `EOS_DIR` environment variable.\n 3. A platform specific default directory for EOS.\n 4. The current working directory.\n - The first available configuration file found in these directories is loaded.\n - If no configuration file is found, a default configuration file is created in the platform\n specific default directory, and default settings are loaded into it.\n\nAttributes from the loaded configuration are accessible directly as instance attributes of\n`ConfigEOS`, providing a centralized, shared configuration object for EOS.\n\nSingleton Behavior:\n - This class uses the `SingletonMixin` to ensure that all requests for `ConfigEOS` return\n the same instance, which contains the most up-to-date configuration. Modifying the configuration\n in one part of the application reflects across all references to this class.\n\nRaises:\n FileNotFoundError: If no configuration file is found, and creating a default configuration fails.\n\nExample:\n To initialize and access configuration attributes (only one instance is created):\n .. code-block:: python\n\n config_eos = ConfigEOS() # Always returns the same instance\n print(config_eos.prediction.hours) # Access a setting from the loaded configuration" }, + "ConfigOptimizationRequest": { + "properties": { + "soc": { + "additionalProperties": { + "type": "integer", + "maximum": 100.0, + "minimum": 0.0 + }, + "type": "object", + "title": "Soc" + }, + "forecasts": { + "$ref": "#/components/schemas/RuntimeForecasts" + }, + "start_solution": { + "anyOf": [ + { + "items": { + "type": "number" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Start Solution" + }, + "start_solution_datetime": { + "anyOf": [ + { + "type": "string", + "format": "date-time" + }, + { + "type": "null" + } + ], + "title": "Start Solution Datetime" + } + }, + "additionalProperties": false, + "type": "object", + "title": "ConfigOptimizationRequest", + "description": "Supply observations while hardware, tariffs and scheduling remain in config." + }, "ConfigSaveMode": { "type": "string", "enum": [ @@ -3989,6 +4174,24 @@ "title": "ConfigSaveMode", "description": "Configuration file save mode." }, + "ConsumerDeadlinePolicy": { + "type": "string", + "enum": [ + "BEST_EFFORT", + "STRICT" + ], + "title": "ConsumerDeadlinePolicy", + "description": "Behaviour when a flexible consumer's deadline cannot be met.\n\nA deadline (``deadline_datetime``) demands that a complete run has *finished*\nbefore that moment. Depending on \"now\", the run duration, the optimization\nhorizon and the allowed time windows, no such start may exist.\n\nPolicies\n--------\n- BEST_EFFORT:\n Run as early as the remaining constraints allow, i.e. minimize the\n delay instead of the cost (\"it should have been done by 03:00, so\n start now\"). A warning is logged. This keeps an optimization request\n answerable instead of failing it - the usual choice for home\n automation.\n\n- STRICT:\n Keep the deadline. A ONCE consumer without a feasible start makes the\n optimization fail; a DAILY consumer is simply not scheduled on days\n without a feasible start." + }, + "ConsumerScheduleMode": { + "type": "string", + "enum": [ + "ONCE", + "DAILY" + ], + "title": "ConsumerScheduleMode", + "description": "Schedule mode of a flexible consumer (home appliance).\n\nDetermines how often a consumer's load profile is scheduled within the\noptimization horizon.\n\nModes\n-----\n- ONCE:\n The consumer runs exactly once somewhere within the optimization\n horizon (\"fire and forget\"). The optimizer picks the start.\n\n- DAILY:\n The consumer runs once per local calendar day, but only on days for\n which at least one complete, allowed run still fits into the remaining\n horizon. The optimizer picks one start per eligible day." + }, "CycleTimeWindowSequence-Input": { "properties": { "windows": { @@ -5471,6 +5674,40 @@ ], null ] + }, + "min_soc_deadline_datetime": { + "anyOf": [ + { + "type": "string", + "format": "date-time" + }, + { + "type": "null" + } + ], + "title": "Min Soc Deadline Datetime", + "description": "Absolute moment by which 'min_soc_percentage' has to be reached (departure time). A date time without timezone is read as local time. None means end of the optimization horizon.", + "examples": [ + null, + "2026-07-16T07:00:00+02:00" + ] + }, + "min_soc_max_duration_h": { + "anyOf": [ + { + "type": "number", + "exclusiveMinimum": 0.0 + }, + { + "type": "null" + } + ], + "title": "Min Soc Max Duration H", + "description": "Maximum time from the start of the optimization until 'min_soc_percentage' has to be reached [h]. Combined with 'min_soc_deadline_datetime' the earlier of the two applies.", + "examples": [ + null, + 6.0 + ] } }, "additionalProperties": false, @@ -6129,6 +6366,12 @@ }, "FeedInTariffCommonSettings-Input": { "properties": { + "direct_marketing_enabled": { + "type": "boolean", + "title": "Direct Marketing Enabled", + "description": "Enable export-aware GENETIC optimization. Sale revenues remain those of the configured feed-in provider or explicit forecast; purchase prices never replace them.", + "default": false + }, "provider": { "anyOf": [ { @@ -6172,6 +6415,12 @@ }, "FeedInTariffCommonSettings-Output": { "properties": { + "direct_marketing_enabled": { + "type": "boolean", + "title": "Direct Marketing Enabled", + "description": "Enable export-aware GENETIC optimization. Sale revenues remain those of the configured feed-in provider or explicit forecast; purchase prices never replace them.", + "default": false + }, "provider": { "anyOf": [ { @@ -8447,8 +8696,10 @@ "properties": { "interval_sec": { "type": "integer", - "maximum": 3600.0, - "minimum": 900.0, + "enum": [ + 900, + 3600 + ], "title": "Interval Sec", "description": "The optimization interval [sec]. Defaults to 3600 seconds (1 hour)", "default": 3600, @@ -8518,6 +8769,49 @@ 42 ] }, + "measurement_max_age_seconds": { + "type": "integer", + "exclusiveMinimum": 0.0, + "title": "Measurement Max Age Seconds", + "description": "Maximum age of SoC measurements for configuration-based optimization [s].", + "default": 300 + }, + "tail_horizon_hours": { + "type": "integer", + "minimum": 0.0, + "title": "Tail Horizon Hours", + "description": "Forecast lookahead after the control horizon [h]. No tail commands are issued. Set 0 to disable.", + "default": 48 + }, + "terminal_value_mode": { + "$ref": "#/components/schemas/TerminalValueMode", + "description": "How to value the energy left in the battery at the end of the control horizon. AUTO solves the forecast tail with an AUTO continuation proxy at its end (or only the proxy if tail is zero); FIXED uses 'terminal_value_euro_per_kwh'. Defaults to AUTO.", + "default": "AUTO", + "examples": [ + "AUTO", + "FIXED" + ] + }, + "terminal_value_euro_per_kwh": { + "type": "number", + "title": "Terminal Value Euro Per Kwh", + "description": "Value assigned to usable battery energy remaining at the end of the optimization horizon [EUR/kWh]. This terminal value is independent of the battery LCOS. Only used with terminal_value_mode = FIXED. Defaults to 0 EUR/kWh.", + "default": 0.0, + "examples": [ + 0.0, + 0.2 + ] + }, + "terminal_value_window_hours": { + "type": "integer", + "minimum": 1.0, + "title": "Terminal Value Window Hours", + "description": "Length of the trailing window at the effective tail end the AUTO continuation curve is derived from [h]. One day covers a full load and PV cycle. Defaults to 24 hours.", + "default": 24, + "examples": [ + 24 + ] + }, "penalties": { "additionalProperties": { "anyOf": [ @@ -8550,8 +8844,10 @@ "properties": { "interval_sec": { "type": "integer", - "maximum": 3600.0, - "minimum": 900.0, + "enum": [ + 900, + 3600 + ], "title": "Interval Sec", "description": "The optimization interval [sec]. Defaults to 3600 seconds (1 hour)", "default": 3600, @@ -8621,6 +8917,49 @@ 42 ] }, + "measurement_max_age_seconds": { + "type": "integer", + "exclusiveMinimum": 0.0, + "title": "Measurement Max Age Seconds", + "description": "Maximum age of SoC measurements for configuration-based optimization [s].", + "default": 300 + }, + "tail_horizon_hours": { + "type": "integer", + "minimum": 0.0, + "title": "Tail Horizon Hours", + "description": "Forecast lookahead after the control horizon [h]. No tail commands are issued. Set 0 to disable.", + "default": 48 + }, + "terminal_value_mode": { + "$ref": "#/components/schemas/TerminalValueMode", + "description": "How to value the energy left in the battery at the end of the control horizon. AUTO solves the forecast tail with an AUTO continuation proxy at its end (or only the proxy if tail is zero); FIXED uses 'terminal_value_euro_per_kwh'. Defaults to AUTO.", + "default": "AUTO", + "examples": [ + "AUTO", + "FIXED" + ] + }, + "terminal_value_euro_per_kwh": { + "type": "number", + "title": "Terminal Value Euro Per Kwh", + "description": "Value assigned to usable battery energy remaining at the end of the optimization horizon [EUR/kWh]. This terminal value is independent of the battery LCOS. Only used with terminal_value_mode = FIXED. Defaults to 0 EUR/kWh.", + "default": 0.0, + "examples": [ + 0.0, + 0.2 + ] + }, + "terminal_value_window_hours": { + "type": "integer", + "minimum": 1.0, + "title": "Terminal Value Window Hours", + "description": "Length of the trailing window at the effective tail end the AUTO continuation curve is derived from [h]. One day covers a full load and PV cycle. Defaults to 24 hours.", + "default": 24, + "examples": [ + 24 + ] + }, "penalties": { "additionalProperties": { "anyOf": [ @@ -8666,7 +9005,7 @@ }, "type": "array", "title": "Pv Forecast Wh", - "description": "An array of floats representing the forecasted photovoltaic output in watts for different time intervals." + "description": "An array of floats representing the forecasted photovoltaic energy in watt-hours per slot for different time intervals." }, "electricity_price_per_wh": { "items": { @@ -8702,7 +9041,7 @@ }, "type": "array", "title": "Total Load", - "description": "An array of floats representing the total load (consumption) in watts for different time intervals." + "description": "An array of floats representing the total load (consumption) in watt-hours per slot for different time intervals." }, "pv_prognose_wh": { "items": { @@ -8859,6 +9198,50 @@ "title": "Start Solution", "description": "Can be `null` or contain a previous solution (if available)." }, + "forecast_interval_seconds": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Forecast Interval Seconds", + "description": "Input interval: 3600 for hourly, or optimization interval for native slots." + }, + "home_appliances": { + "anyOf": [ + { + "items": { + "$ref": "#/components/schemas/HomeApplianceParameters" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Home Appliances", + "description": "List of flexible consumers (home appliances) to schedule." + }, + "start_solution_datetime": { + "anyOf": [ + { + "type": "string", + "format": "date-time" + }, + { + "type": "null" + } + ], + "title": "Start Solution Datetime", + "description": "Start of the slot that gene 0 of 'start_solution' controls, as returned with the previous solution. The warm start is shifted by the slots that have elapsed until this run. Without it, a 'start_solution' identical to the last solution of this server uses that solution's start; any other one is used unshifted.", + "examples": [ + null, + "2026-09-14T07:45:00+02:00" + ] + }, "pv_akku": { "anyOf": [ { @@ -9007,6 +9390,36 @@ "title": "Electricity Price", "description": "Used Electricity Price, including predictions" }, + "home_appliance_energy_wh": { + "additionalProperties": { + "items": { + "type": "number" + }, + "type": "array" + }, + "type": "object", + "title": "Home Appliance Energy Wh", + "description": "Per-device appliance energy in watt-hours per optimization slot, keyed by device_id." + }, + "feed_in_tariff": { + "items": { + "type": "number" + }, + "type": "array", + "title": "Feed In Tariff", + "description": "Used feed-in tariff in \u20ac/Wh per hour, including predictions" + }, + "home_appliance_running": { + "additionalProperties": { + "items": { + "type": "boolean" + }, + "type": "array" + }, + "type": "object", + "title": "Home Appliance Running", + "description": "Active run occupancy, including zero-power phases." + }, "Last_Wh_pro_Stunde": { "items": { "type": "number" @@ -9141,6 +9554,23 @@ "description": "Deprecated: Use electricity_price instead.", "deprecated": true, "readOnly": true + }, + "Feed_in_tariff": { + "anyOf": [ + { + "items": { + "type": "number" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Feed In Tariff", + "description": "Deprecated: use feed_in_tariff.", + "deprecated": true, + "readOnly": true } }, "type": "object", @@ -9172,7 +9602,8 @@ "Netzeinspeisung_Wh_pro_Stunde", "Verluste_Pro_Stunde", "akku_soc_pro_stunde", - "Electricity_price" + "Electricity_price", + "Feed_in_tariff" ], "title": "GeneticSimulationResult", "description": "This object contains the results of the simulation and provides insights into various parameters over the entire forecast period." @@ -9300,22 +9731,7 @@ "fitness_history": { "anyOf": [ { - "additionalProperties": { - "anyOf": [ - { - "items": { - "type": "integer" - }, - "type": "array" - }, - { - "items": { - "type": "number" - }, - "type": "array" - } - ] - }, + "additionalProperties": true, "type": "object" }, { @@ -9337,6 +9753,82 @@ "title": "Fixed Seed", "description": "Fixed seed." }, + "controls_start_at_now": { + "type": "boolean", + "title": "Controls Start At Now", + "description": "Control arrays start at the run timestamp instead of midnight.", + "default": false + }, + "battery_grid_export_allowed": { + "items": { + "type": "integer" + }, + "type": "array", + "title": "Battery Grid Export Allowed", + "description": "Array with battery-to-grid export values (1 for export discharge, 0 otherwise)." + }, + "terminal_value": { + "anyOf": [ + { + "$ref": "#/components/schemas/TerminalValueResult" + }, + { + "type": "null" + } + ], + "description": "The terminal value applied to the energy left in the battery at the end of the horizon, including the curve it was read from. None when no battery is part of the optimization." + }, + "battery_grid_export_factor": { + "items": { + "type": "number" + }, + "type": "array", + "title": "Battery Grid Export Factor", + "description": "Array with the battery-to-grid export level per slot as factor of the rated discharge power (0.0 for no export). Empty when direct marketing is disabled; a solution without this array exports at full power wherever 'battery_grid_export_allowed' is 1." + }, + "start_solution_datetime": { + "anyOf": [ + { + "type": "string", + "format": "date-time" + }, + { + "type": "null" + } + ], + "title": "Start Solution Datetime", + "description": "Start of the slot that gene 0 of 'start_solution' controls. Send it back together with 'start_solution' so the next run can shift the warm start by the slots that have elapsed since.", + "examples": [ + null, + "2026-09-14T07:45:00+02:00" + ] + }, + "appliance_starts": { + "additionalProperties": { + "items": { + "type": "string", + "format": "date-time" + }, + "type": "array" + }, + "type": "object", + "title": "Appliance Starts", + "description": "Scheduled run start times per appliance device_id as absolute local datetimes." + }, + "appliance_deadline_missed": { + "additionalProperties": { + "type": "boolean" + }, + "type": "object", + "title": "Appliance Deadline Missed", + "description": "Per appliance device_id with a 'deadline_datetime': whether the scheduled run misses that deadline (or was not scheduled at all). Appliances without a deadline are not listed." + }, + "interval_seconds": { + "type": "integer", + "title": "Interval Seconds", + "description": "Duration of one result/control slot in seconds.", + "default": 3600 + }, "eautocharge_hours_float": { "anyOf": [ { @@ -9437,8 +9929,15 @@ ] }, "consumption_wh": { - "type": "integer", - "exclusiveMinimum": 0.0, + "anyOf": [ + { + "type": "integer", + "exclusiveMinimum": 0.0 + }, + { + "type": "null" + } + ], "title": "Consumption Wh", "description": "Energy consumption per run cycle [Wh].", "default": 3000, @@ -9451,9 +9950,16 @@ ] }, "duration_h": { - "type": "integer", - "maximum": 24.0, - "exclusiveMinimum": 0.0, + "anyOf": [ + { + "type": "integer", + "maximum": 24.0, + "exclusiveMinimum": 0.0 + }, + { + "type": "null" + } + ], "title": "Duration H", "description": "Run duration per cycle [h] (1-24).", "default": 3, @@ -9544,6 +10050,138 @@ "x-scope": [ "GENETIC" ] + }, + "load_profile_power_w": { + "anyOf": [ + { + "items": { + "type": "number" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Load Profile Power W", + "description": "Explicit load profile describing a single complete run as a sequence of non-negative power values in watts. Each value covers 'load_profile_interval_seconds'. Mutually exclusive with consumption_wh/duration_h.", + "examples": [ + [ + 200.0, + 2000.0, + 1800.0, + 100.0 + ] + ], + "x-scope": [ + "GENETIC" + ] + }, + "load_profile_interval_seconds": { + "anyOf": [ + { + "type": "integer", + "exclusiveMinimum": 0.0 + }, + { + "type": "null" + } + ], + "title": "Load Profile Interval Seconds", + "description": "Duration of one 'load_profile_power_w' step in seconds. Defaults to the configured optimization interval when a profile is given.", + "examples": [ + 900, + 3600 + ], + "x-scope": [ + "GENETIC" + ] + }, + "schedule_mode": { + "$ref": "#/components/schemas/ConsumerScheduleMode", + "description": "Scheduling mode: ONCE (a single run within the horizon) or DAILY (one run per local calendar day with a feasible full run).", + "default": "ONCE", + "examples": [ + "ONCE", + "DAILY" + ], + "x-scope": [ + "GENETIC" + ] + }, + "time_windows": { + "anyOf": [ + { + "$ref": "#/components/schemas/TimeWindowSequence-Input" + }, + { + "type": "null" + } + ], + "description": "List of allowed time windows. Defaults to optimization general time window.", + "examples": [ + [ + { + "duration": "3 hours", + "start_time": "10:00" + } + ] + ], + "x-scope": [ + "GENETIC" + ] + }, + "earliest_start_datetime": { + "anyOf": [ + { + "type": "string", + "format": "date-time" + }, + { + "type": "null" + } + ], + "title": "Earliest Start Datetime", + "description": "Absolute earliest moment the run may start. Starts before it are dropped, in addition to 'time_windows' and the horizon. A date time without timezone is read as local time. This bound is never relaxed.", + "examples": [ + null, + "2026-07-15T20:00:00+02:00" + ], + "x-scope": [ + "GENETIC" + ] + }, + "deadline_datetime": { + "anyOf": [ + { + "type": "string", + "format": "date-time" + }, + { + "type": "null" + } + ], + "title": "Deadline Datetime", + "description": "Absolute deadline: the complete run must have *finished* at or before this moment (e.g. end of the day, or 03:00 tonight). A date time without timezone is read as local time. See 'deadline_policy' for what happens when no start can meet it.", + "examples": [ + null, + "2026-07-16T03:00:00+02:00" + ], + "x-scope": [ + "GENETIC" + ] + }, + "deadline_policy": { + "$ref": "#/components/schemas/ConsumerDeadlinePolicy", + "description": "What to do when 'deadline_datetime' cannot be met: BEST_EFFORT runs as early as possible instead (warning logged), STRICT keeps the deadline (a ONCE consumer then fails the optimization).", + "default": "BEST_EFFORT", + "examples": [ + "BEST_EFFORT", + "STRICT" + ], + "x-scope": [ + "GENETIC" + ] } }, "type": "object", @@ -9564,8 +10202,15 @@ ] }, "consumption_wh": { - "type": "integer", - "exclusiveMinimum": 0.0, + "anyOf": [ + { + "type": "integer", + "exclusiveMinimum": 0.0 + }, + { + "type": "null" + } + ], "title": "Consumption Wh", "description": "Energy consumption per run cycle [Wh].", "default": 3000, @@ -9578,9 +10223,16 @@ ] }, "duration_h": { - "type": "integer", - "maximum": 24.0, - "exclusiveMinimum": 0.0, + "anyOf": [ + { + "type": "integer", + "maximum": 24.0, + "exclusiveMinimum": 0.0 + }, + { + "type": "null" + } + ], "title": "Duration H", "description": "Run duration per cycle [h] (1-24).", "default": 3, @@ -9672,6 +10324,138 @@ "GENETIC" ] }, + "load_profile_power_w": { + "anyOf": [ + { + "items": { + "type": "number" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Load Profile Power W", + "description": "Explicit load profile describing a single complete run as a sequence of non-negative power values in watts. Each value covers 'load_profile_interval_seconds'. Mutually exclusive with consumption_wh/duration_h.", + "examples": [ + [ + 200.0, + 2000.0, + 1800.0, + 100.0 + ] + ], + "x-scope": [ + "GENETIC" + ] + }, + "load_profile_interval_seconds": { + "anyOf": [ + { + "type": "integer", + "exclusiveMinimum": 0.0 + }, + { + "type": "null" + } + ], + "title": "Load Profile Interval Seconds", + "description": "Duration of one 'load_profile_power_w' step in seconds. Defaults to the configured optimization interval when a profile is given.", + "examples": [ + 900, + 3600 + ], + "x-scope": [ + "GENETIC" + ] + }, + "schedule_mode": { + "$ref": "#/components/schemas/ConsumerScheduleMode", + "description": "Scheduling mode: ONCE (a single run within the horizon) or DAILY (one run per local calendar day with a feasible full run).", + "default": "ONCE", + "examples": [ + "ONCE", + "DAILY" + ], + "x-scope": [ + "GENETIC" + ] + }, + "time_windows": { + "anyOf": [ + { + "$ref": "#/components/schemas/TimeWindowSequence-Output" + }, + { + "type": "null" + } + ], + "description": "List of allowed time windows. Defaults to optimization general time window.", + "examples": [ + [ + { + "duration": "3 hours", + "start_time": "10:00" + } + ] + ], + "x-scope": [ + "GENETIC" + ] + }, + "earliest_start_datetime": { + "anyOf": [ + { + "type": "string", + "format": "date-time" + }, + { + "type": "null" + } + ], + "title": "Earliest Start Datetime", + "description": "Absolute earliest moment the run may start. Starts before it are dropped, in addition to 'time_windows' and the horizon. A date time without timezone is read as local time. This bound is never relaxed.", + "examples": [ + null, + "2026-07-15T20:00:00+02:00" + ], + "x-scope": [ + "GENETIC" + ] + }, + "deadline_datetime": { + "anyOf": [ + { + "type": "string", + "format": "date-time" + }, + { + "type": "null" + } + ], + "title": "Deadline Datetime", + "description": "Absolute deadline: the complete run must have *finished* at or before this moment (e.g. end of the day, or 03:00 tonight). A date time without timezone is read as local time. See 'deadline_policy' for what happens when no start can meet it.", + "examples": [ + null, + "2026-07-16T03:00:00+02:00" + ], + "x-scope": [ + "GENETIC" + ] + }, + "deadline_policy": { + "$ref": "#/components/schemas/ConsumerDeadlinePolicy", + "description": "What to do when 'deadline_datetime' cannot be met: BEST_EFFORT runs as early as possible instead (warning logged), STRICT keeps the deadline (a ONCE consumer then fails the optimization).", + "default": "BEST_EFFORT", + "examples": [ + "BEST_EFFORT", + "STRICT" + ], + "x-scope": [ + "GENETIC" + ] + }, "effective_num_cycles": { "type": "integer", "title": "Effective Num Cycles", @@ -9730,8 +10514,15 @@ ] }, "consumption_wh": { - "type": "integer", - "exclusiveMinimum": 0.0, + "anyOf": [ + { + "type": "integer", + "exclusiveMinimum": 0.0 + }, + { + "type": "null" + } + ], "title": "Consumption Wh", "description": "An integer representing the energy consumption of a household device in watt-hours.", "examples": [ @@ -9739,8 +10530,15 @@ ] }, "duration_h": { - "type": "integer", - "exclusiveMinimum": 0.0, + "anyOf": [ + { + "type": "integer", + "exclusiveMinimum": 0.0 + }, + { + "type": "null" + } + ], "title": "Duration H", "description": "An integer representing the usage duration of a household device in hours.", "examples": [ @@ -9757,6 +10555,13 @@ 2 ] }, + "completed_cycles": { + "type": "integer", + "minimum": 0.0, + "title": "Completed Cycles", + "description": "Cycles already completed on the first planning day.", + "default": 0 + }, "min_cycle_gap_h": { "type": "integer", "minimum": 0.0, @@ -9786,14 +10591,115 @@ } ] ] + }, + "shared_time_windows": { + "anyOf": [ + { + "$ref": "#/components/schemas/TimeWindowSequence-Output" + }, + { + "type": "null" + } + ], + "description": "Allowed recurring windows shared by every cycle; intersected with per-cycle windows." + }, + "load_profile_power_w": { + "anyOf": [ + { + "items": { + "type": "number" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Load Profile Power W", + "description": "Explicit load profile describing a single complete run as a sequence of non-negative power values in watts. Each value covers 'load_profile_interval_seconds'. Mutually exclusive with consumption_wh/duration_h.", + "examples": [ + [ + 200.0, + 2000.0, + 1800.0, + 100.0 + ] + ] + }, + "load_profile_interval_seconds": { + "anyOf": [ + { + "type": "integer", + "exclusiveMinimum": 0.0 + }, + { + "type": "null" + } + ], + "title": "Load Profile Interval Seconds", + "description": "Duration of one 'load_profile_power_w' step in seconds. Defaults to the configured optimization interval when a profile is given.", + "examples": [ + 900, + 3600 + ] + }, + "schedule_mode": { + "$ref": "#/components/schemas/ConsumerScheduleMode", + "description": "Scheduling mode: ONCE (a single run within the horizon) or DAILY (one run per local calendar day with a feasible full run).", + "default": "ONCE", + "examples": [ + "ONCE", + "DAILY" + ] + }, + "earliest_start_datetime": { + "anyOf": [ + { + "type": "string", + "format": "date-time" + }, + { + "type": "null" + } + ], + "title": "Earliest Start Datetime", + "description": "Absolute earliest moment the run may start. Starts before it are dropped, in addition to 'time_windows' and the horizon. A date time without timezone is read as local time. This bound is never relaxed.", + "examples": [ + null, + "2026-07-15T20:00:00+02:00" + ] + }, + "deadline_datetime": { + "anyOf": [ + { + "type": "string", + "format": "date-time" + }, + { + "type": "null" + } + ], + "title": "Deadline Datetime", + "description": "Absolute deadline: the complete run must have *finished* at or before this moment (e.g. end of the day, or 03:00 tonight). A date time without timezone is read as local time. See 'deadline_policy' for what happens when no start can meet it.", + "examples": [ + null, + "2026-07-16T03:00:00+02:00" + ] + }, + "deadline_policy": { + "$ref": "#/components/schemas/ConsumerDeadlinePolicy", + "description": "What to do when 'deadline_datetime' cannot be met: BEST_EFFORT runs as early as possible instead (warning logged), STRICT keeps the deadline (a ONCE consumer then fails the optimization).", + "default": "BEST_EFFORT", + "examples": [ + "BEST_EFFORT", + "STRICT" + ] } }, "additionalProperties": false, "type": "object", "required": [ - "device_id", - "consumption_wh", - "duration_h" + "device_id" ], "title": "HomeApplianceParameters", "description": "Configuration for a simulated home appliance device." @@ -13939,6 +14845,91 @@ "title": "PydanticDateTimeSeries", "description": "Pydantic model for validating pandas Series with datetime index in JSON format.\n\nThis model handles Series data serialized with orient='index', where the keys are\ndatetime strings and values are the series values. Provides validation and\nconversion between JSON and pandas Series with datetime index.\n\nAttributes:\n data (Dict[str, Any]): Dictionary mapping datetime strings to values.\n dtype (str): The data type of the series values.\n tz (str | None): Timezone name if the datetime index is timezone-aware." }, + "RuntimeForecasts": { + "properties": { + "pv_forecast_wh": { + "anyOf": [ + { + "items": { + "type": "number" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Pv Forecast Wh" + }, + "total_load": { + "anyOf": [ + { + "items": { + "type": "number" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Total Load" + }, + "electricity_price_per_wh": { + "anyOf": [ + { + "items": { + "type": "number" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Electricity Price Per Wh" + }, + "feed_in_tariff_per_wh": { + "anyOf": [ + { + "items": { + "type": "number" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Feed In Tariff Per Wh" + }, + "temperature_forecast": { + "anyOf": [ + { + "items": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ] + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Temperature Forecast" + } + }, + "additionalProperties": false, + "type": "object", + "title": "RuntimeForecasts", + "description": "External series from local midnight, in slot Wh and currency per Wh.\n\nThe interval is configured in optimization.genetic.interval_sec. Omitted\nseries are read from the configured providers, preserving their raw coverage." + }, "SampleQuality": { "properties": { "status": { @@ -14468,6 +15459,359 @@ "title": "SolarPanelBatteryParameters", "description": "PV battery device simulation configuration." }, + "TailDiagnostics": { + "properties": { + "slots": { + "type": "integer", + "title": "Slots", + "default": 0 + }, + "slot_hours": { + "type": "number", + "title": "Slot Hours", + "default": 0.0 + }, + "soc_grid_points": { + "type": "integer", + "title": "Soc Grid Points", + "default": 0 + }, + "min_import_price_euro_per_kwh": { + "type": "number", + "title": "Min Import Price Euro Per Kwh", + "default": 0.0 + }, + "max_import_price_euro_per_kwh": { + "type": "number", + "title": "Max Import Price Euro Per Kwh", + "default": 0.0 + }, + "min_feed_in_tariff_euro_per_kwh": { + "type": "number", + "title": "Min Feed In Tariff Euro Per Kwh", + "default": 0.0 + }, + "max_feed_in_tariff_euro_per_kwh": { + "type": "number", + "title": "Max Feed In Tariff Euro Per Kwh", + "default": 0.0 + }, + "negative_import_price_slots": { + "type": "integer", + "title": "Negative Import Price Slots", + "default": 0 + }, + "positive_battery_export_slots": { + "type": "integer", + "title": "Positive Battery Export Slots", + "default": 0 + } + }, + "type": "object", + "title": "TailDiagnostics", + "description": "Forecast summary used by the deterministic tail optimization." + }, + "TailPlanSlot": { + "properties": { + "slot": { + "type": "integer", + "title": "Slot" + }, + "hour_from_start": { + "type": "number", + "title": "Hour From Start" + }, + "action": { + "type": "string", + "title": "Action" + }, + "alternative_action": { + "type": "string", + "title": "Alternative Action", + "default": "" + }, + "decision_margin_euro": { + "type": "number", + "title": "Decision Margin Euro", + "default": 0.0 + }, + "soc_start_percentage": { + "type": "number", + "title": "Soc Start Percentage" + }, + "soc_end_percentage": { + "type": "number", + "title": "Soc End Percentage" + }, + "pv_wh": { + "type": "number", + "title": "Pv Wh" + }, + "load_wh": { + "type": "number", + "title": "Load Wh" + }, + "grid_import_wh": { + "type": "number", + "title": "Grid Import Wh" + }, + "grid_export_wh": { + "type": "number", + "title": "Grid Export Wh" + }, + "battery_charge_wh": { + "type": "number", + "title": "Battery Charge Wh" + }, + "battery_discharge_wh": { + "type": "number", + "title": "Battery Discharge Wh" + }, + "import_price_euro_per_kwh": { + "type": "number", + "title": "Import Price Euro Per Kwh" + }, + "feed_in_tariff_euro_per_kwh": { + "type": "number", + "title": "Feed In Tariff Euro Per Kwh" + }, + "slot_value_euro": { + "type": "number", + "title": "Slot Value Euro" + }, + "remaining_value_euro": { + "type": "number", + "title": "Remaining Value Euro" + }, + "ac_charge_factor": { + "type": "number", + "title": "Ac Charge Factor" + }, + "dc_charge_allowed": { + "type": "integer", + "title": "Dc Charge Allowed" + }, + "discharge_allowed": { + "type": "integer", + "title": "Discharge Allowed" + }, + "battery_grid_export_factor": { + "type": "number", + "title": "Battery Grid Export Factor" + } + }, + "type": "object", + "required": [ + "slot", + "hour_from_start", + "action", + "soc_start_percentage", + "soc_end_percentage", + "pv_wh", + "load_wh", + "grid_import_wh", + "grid_export_wh", + "battery_charge_wh", + "battery_discharge_wh", + "import_price_euro_per_kwh", + "feed_in_tariff_euro_per_kwh", + "slot_value_euro", + "remaining_value_euro", + "ac_charge_factor", + "dc_charge_allowed", + "discharge_allowed", + "battery_grid_export_factor" + ], + "title": "TailPlanSlot", + "description": "One diagnostic slot of the optimal tail path.\n\nThese values explain the lookahead used for fitness. They are diagnostics\nonly and are never copied into the executable control arrays." + }, + "TerminalValueCurve": { + "properties": { + "energy_wh": { + "items": { + "type": "number" + }, + "type": "array", + "title": "Energy Wh", + "description": "Breakpoints of usable AC energy left in the battery [Wh]." + }, + "value_euro": { + "items": { + "type": "number" + }, + "type": "array", + "title": "Value Euro", + "description": "Cumulative credit at each breakpoint [EUR]." + }, + "operating_value_euro": { + "items": { + "type": "number" + }, + "type": "array", + "title": "Operating Value Euro", + "description": "Tail operating component at each breakpoint [EUR]; empty for a proxy curve." + }, + "continuation_value_euro": { + "items": { + "type": "number" + }, + "type": "array", + "title": "Continuation Value Euro", + "description": "Continuation component at each breakpoint [EUR]; empty for a proxy curve." + }, + "marginal_euro_per_kwh": { + "items": { + "type": "number" + }, + "type": "array", + "title": "Marginal Euro Per Kwh", + "description": "Marginal value of the segment that starts at each breakpoint [EUR/kWh]. May be negative or non-monotone in TAIL mode." + }, + "residual_energy_wh": { + "type": "number", + "title": "Residual Energy Wh", + "description": "Energy up to which the curve is backed by residual load - the knee. Everything beyond it is only worth an export.", + "default": 0.0 + }, + "window_slots": { + "type": "integer", + "title": "Window Slots", + "description": "Number of trailing horizon slots the curve was derived from. Fewer slots than a full day mean a shorter proxy period.", + "default": 0 + } + }, + "type": "object", + "title": "TerminalValueCurve", + "description": "Piecewise linear, concave value of battery energy left at the horizon.\n\n``energy_wh`` and ``value_euro`` are the breakpoints of the cumulative\nvalue, ``marginal_euro_per_kwh`` the slope of each segment. Both arrays\nstart at the origin; the curve is flat beyond its last breakpoint." + }, + "TerminalValueMode": { + "type": "string", + "enum": [ + "AUTO", + "FIXED" + ], + "title": "TerminalValueMode", + "description": "How the energy left in the battery at the end of the horizon is valued.\n\nModes\n-----\n- AUTO:\n Solve the deterministic forecast tail and apply a conservative\n continuation proxy at its end. Tail values may decrease with SOC when\n empty capacity is valuable. With a zero tail, use the proxy directly.\n\n- FIXED:\n Credit every stored kWh with the configured\n ``terminal_value_euro_per_kwh`` (or ``preis_euro_pro_wh_akku`` of the\n request). The historical behaviour; a value of 0 makes the optimizer\n empty the battery towards the end of the horizon." + }, + "TerminalValueResult": { + "properties": { + "control_horizon_hours": { + "type": "number", + "title": "Control Horizon Hours", + "default": 0 + }, + "requested_tail_hours": { + "type": "number", + "title": "Requested Tail Hours", + "default": 0 + }, + "effective_tail_hours": { + "type": "number", + "title": "Effective Tail Hours", + "default": 0 + }, + "tail_end_hour": { + "type": "number", + "title": "Tail End Hour", + "default": 0 + }, + "continuation_mode": { + "type": "string", + "title": "Continuation Mode", + "default": "FIXED" + }, + "mode": { + "type": "string", + "title": "Mode", + "description": "Terminal value mode the run used: TAIL, AUTO or FIXED.", + "examples": [ + "TAIL", + "AUTO", + "FIXED" + ] + }, + "battery_energy_wh": { + "type": "number", + "title": "Battery Energy Wh", + "description": "Usable AC energy left in the battery at the end of the horizon [Wh].", + "default": 0.0 + }, + "credited_euro": { + "type": "number", + "title": "Credited Euro", + "description": "Credit applied to the total balance [EUR].", + "default": 0.0 + }, + "tail_operating_euro": { + "type": "number", + "title": "Tail Operating Euro", + "description": "Optimal net cash flow within the effective tail for the selected control-end battery state [EUR].", + "default": 0.0 + }, + "continuation_value_euro": { + "type": "number", + "title": "Continuation Value Euro", + "description": "Continuation credit remaining at the end of the optimal tail path [EUR].", + "default": 0.0 + }, + "curve": { + "anyOf": [ + { + "$ref": "#/components/schemas/TerminalValueCurve" + }, + { + "type": "null" + } + ], + "description": "Combined tail value curve (tail operation plus continuation) read by fitness; None in FIXED mode." + }, + "continuation_curve": { + "anyOf": [ + { + "$ref": "#/components/schemas/TerminalValueCurve" + }, + { + "type": "null" + } + ], + "description": "Conservative AUTO proxy constructed at the effective tail end." + }, + "tail_diagnostics": { + "anyOf": [ + { + "$ref": "#/components/schemas/TailDiagnostics" + }, + { + "type": "null" + } + ] + }, + "tail_plan": { + "items": { + "$ref": "#/components/schemas/TailPlanSlot" + }, + "type": "array", + "title": "Tail Plan", + "description": "Diagnostic optimal battery path inside the tail. It explains the lookahead but is never an executable control plan." + }, + "reason": { + "type": "string", + "title": "Reason", + "description": "Why this mode applied. Empty in AUTO mode; in FIXED mode it says whether FIXED was configured or whether AUTO fell back because no curve could be derived.", + "default": "", + "examples": [ + "", + "terminal_value_mode is FIXED" + ] + } + }, + "type": "object", + "required": [ + "mode" + ], + "title": "TerminalValueResult", + "description": "What the optimizer credited for the energy left in the battery." + }, "TimeWindow-Input": { "properties": { "start_time": { diff --git a/src/akkudoktoreos/config/config.py b/src/akkudoktoreos/config/config.py index 4e4dcd7e..6aacdcb1 100644 --- a/src/akkudoktoreos/config/config.py +++ b/src/akkudoktoreos/config/config.py @@ -339,6 +339,37 @@ class SettingsEOSDefaults(SettingsEOS): # This is mutable, so pydantic does not set a hash. return id(self) + def validate_optimization_horizons(self) -> "SettingsEOSDefaults": + """Report a forecast budget that cannot serve the optimization horizons. + + This never rejects a configuration. ``prediction.hours`` is a general + setting that also serves callers with nothing to do with optimization, + and refusing it here would stop EOS from starting over a horizon the + user may not even optimize on. A tail that does not fit is simply + shortened, and a control horizon that does not fit is caught by the + optimizer itself, which knows exactly which forecast series ran out. + """ + control = self.optimization.genetic.horizon_hours + tail = self.optimization.genetic.tail_horizon_hours + prediction = self.prediction.hours + if prediction is None or prediction < control: + logger.warning( + "Prediction horizon {} h is shorter than the {} h control horizon. Optimization " + "runs will fail until prediction.hours covers the control horizon.", + prediction, + control, + ) + elif prediction < control + tail: + logger.info( + "Prediction horizon {} h covers the {} h control horizon but not the requested " + "{} h tail. The tail is shortened to {} h; raise prediction.hours to use it fully.", + prediction, + control, + tail, + prediction - control, + ) + return self + class ConfigEOS(SingletonMixin, SettingsEOSDefaults): """Singleton configuration handler for the EOS application. diff --git a/src/akkudoktoreos/config/configmigrate.py b/src/akkudoktoreos/config/configmigrate.py index 252c32c8..aafc2f05 100644 --- a/src/akkudoktoreos/config/configmigrate.py +++ b/src/akkudoktoreos/config/configmigrate.py @@ -269,6 +269,12 @@ def migrate_config_data(config_data: Dict[str, Any]) -> "SettingsEOSDefaults": # Normalize this provider before the generic field-by-field transfer. Validate # coupled bounds together so a transient intermediate default cannot lose them. config_data = dict(config_data) + from akkudoktoreos.optimization.genetic.geneticsettings import ( + normalize_genetic_settings, + ) + + if "optimization" in config_data: + config_data["optimization"] = normalize_genetic_settings(config_data["optimization"]) pv_settings = normalize_akkudoktor_settings(config_data.get("pvforecast")) if isinstance(pv_settings, dict): config_data["pvforecast"] = pv_settings diff --git a/src/akkudoktoreos/core/dataabc.py b/src/akkudoktoreos/core/dataabc.py index 5407bdc5..753407be 100644 --- a/src/akkudoktoreos/core/dataabc.py +++ b/src/akkudoktoreos/core/dataabc.py @@ -1137,7 +1137,7 @@ class DataSequence(DataABC, DatabaseRecordProtocolMixin[DataRecordT], Generic[Da async for record in self.db_iterate_records(start_timestamp, end_timestamp): if ( record.date_time is None - or (getattr(record, key, None) is None) # key is not in record + or not hasattr(record, key) # key is not in record or (dropna and pd.isna(getattr(record, key, None))) ): continue diff --git a/src/akkudoktoreos/core/emplan.py b/src/akkudoktoreos/core/emplan.py index f1af9fa4..5c3f564f 100644 --- a/src/akkudoktoreos/core/emplan.py +++ b/src/akkudoktoreos/core/emplan.py @@ -2234,7 +2234,9 @@ class EnergyManagementPlan(PydanticBaseModel): self.valid_until = None return - self.valid_from = min(i.execution_time for i in self.instructions) + self.valid_from = min( + (i.execution_time for i in self.instructions), key=lambda value: value.timestamp() + ) end_times = [] for instr in self.instructions: @@ -2245,12 +2247,14 @@ class EnergyManagementPlan(PydanticBaseModel): return end_times.append(instr.execution_time + instr_duration) - self.valid_until = max(end_times) if end_times else None + self.valid_until = ( + max(end_times, key=lambda value: value.timestamp()) if end_times else None + ) def add_instruction(self, instruction: EnergyManagementInstruction) -> None: """Adds a new control instruction and updates time range.""" self.instructions.append(instruction) - self.instructions.sort(key=lambda i: i.execution_time) + self.instructions.sort(key=lambda i: i.execution_time.timestamp()) self._update_time_range() def clear(self) -> None: @@ -2289,14 +2293,14 @@ class EnergyManagementPlan(PydanticBaseModel): by_resource: dict[str, list["EnergyManagementInstruction"]] = defaultdict(list) for instr in self.instructions: # skip instructions scheduled in the future - if instr.execution_time <= now: + if instr.execution_time.timestamp() <= now.timestamp(): by_resource[instr.resource_id].append(instr) active: list["EnergyManagementInstruction"] = [] for resource_id, instrs in by_resource.items(): # pick latest instruction by execution_time - latest = max(instrs, key=lambda i: i.execution_time) + latest = max(instrs, key=lambda i: i.execution_time.timestamp()) if len(instrs) == 0: # No instructions, ther shall be at least one @@ -2310,7 +2314,7 @@ class EnergyManagementPlan(PydanticBaseModel): active.append(latest) else: # active only if now is strictly before execution_time + duration - if latest.execution_time + instr_duration > now: + if (latest.execution_time + instr_duration).timestamp() > now.timestamp(): active.append(latest) return active @@ -2320,9 +2324,11 @@ class EnergyManagementPlan(PydanticBaseModel): ) -> Optional[EnergyManagementInstruction]: """Finds the next instruction scheduled after the specified time.""" now = now or to_datetime() - future_instructions = [i for i in self.instructions if i.execution_time > now] + future_instructions = [ + i for i in self.instructions if i.execution_time.timestamp() > now.timestamp() + ] return ( - min(future_instructions, key=lambda i: i.execution_time) + min(future_instructions, key=lambda i: i.execution_time.timestamp()) if future_instructions else None ) diff --git a/src/akkudoktoreos/core/ems.py b/src/akkudoktoreos/core/ems.py index 53674757..b8e38888 100644 --- a/src/akkudoktoreos/core/ems.py +++ b/src/akkudoktoreos/core/ems.py @@ -22,6 +22,7 @@ from akkudoktoreos.optimization.genetic0.genetic0params import ( Genetic0OptimizationParameters, ) from akkudoktoreos.optimization.genetic0.genetic0solution import Genetic0Solution +from akkudoktoreos.optimization.genetic.configrequest import ConfigOptimizationRequest from akkudoktoreos.optimization.genetic.genetic import GeneticOptimization from akkudoktoreos.optimization.genetic.geneticparams import ( GeneticOptimizationParameters, @@ -63,6 +64,7 @@ class EnergyManagement( # Start datetime. _start_datetime: ClassVar[Optional[DateTime]] = None + _observation_datetime: ClassVar[Optional[DateTime]] = None # last run datetime. Used by energy management task _last_run_datetime: ClassVar[Optional[DateTime]] = None @@ -101,25 +103,47 @@ class EnergyManagement( """The datetime the last energy management was run.""" return EnergyManagement._last_run_datetime + @property + def observation_datetime(self) -> DateTime: + """Unrounded run time used to assess the freshness of runtime measurements.""" + return EnergyManagement._observation_datetime or self.start_datetime + @classmethod - def set_start_datetime(cls, start_datetime: Optional[DateTime] = None) -> DateTime: + def set_start_datetime( + cls, start_datetime: Optional[DateTime] = None, interval_seconds: Optional[int] = None + ) -> DateTime: """Set the start datetime for the next energy management run. If no datetime is provided, the current datetime is used. - The start datetime is always rounded down to the nearest hour - (i.e., setting minutes, seconds, and microseconds to zero). + The start is rounded down to the selected algorithm slot using elapsed + time, retaining the correct offset during repeated DST hours. Args: start_datetime (Optional[DateTime]): The datetime to set as the start. If None, the current datetime is used. + interval_seconds: Explicit slot duration; otherwise use the configured algorithm. Returns: DateTime: The adjusted start datetime. """ if start_datetime is None: start_datetime = to_datetime() - cls._start_datetime = start_datetime.set(minute=0, second=0, microsecond=0) + cls._observation_datetime = start_datetime + if interval_seconds is None: + interval_seconds = ( + cls.config.optimization.genetic.interval_sec + if cls.config.optimization.algorithm == OptimizationAlgorithm.GENETIC + else 3600 + ) + if interval_seconds not in (900, 3600): + raise ValueError("Optimization slots must be 900 or 3600 seconds.") + midnight = start_datetime.start_of("day").int_timestamp + elapsed = start_datetime.int_timestamp - midnight + cls._start_datetime = to_datetime( + midnight + (elapsed // interval_seconds) * interval_seconds, + in_timezone=start_datetime.timezone_name, + ) return cls._start_datetime @classmethod @@ -172,8 +196,11 @@ class EnergyManagement( start_datetime: Optional[DateTime] = None, mode: Optional[EnergyManagementMode] = None, algorithm: Optional[OptimizationAlgorithm] = None, - genetic_parameters: Optional[GeneticOptimizationParameters] = None, + genetic_parameters: Optional[ + GeneticOptimizationParameters | ConfigOptimizationRequest + ] = None, genetic_generations: Optional[int] = None, + genetic_individuals: Optional[int] = None, genetic_seed: Optional[int] = None, genetic0_parameters: Optional[Genetic0OptimizationParameters] = None, genetic0_generations: Optional[int] = None, @@ -207,6 +234,7 @@ class EnergyManagement( genetic_generations (int, optional): The number of generations for the `GENETIC` algorithm. Defaults to the algorithm's internal default (400) if not specified. + genetic_individuals (int, optional): Population size override for this run. genetic_seed (int, optional): The seed for the `GENETIC` algorithm. Defaults to the algorithm's internal random seed if not specified. genetic0_parameters (Genetic0OptimizationParameters, optional): The @@ -245,7 +273,20 @@ class EnergyManagement( # Remember/ set the start datetime of this energy management run. # None leads to current time as start datetime - self.set_start_datetime(start_datetime) + if algorithm is None: + algorithm = self.config.optimization.algorithm + if start_datetime is None and algorithm == OptimizationAlgorithm.GENETIC: + # Consumer windows and midnight-based forecasts use the site clock, + # which can differ from the server's local timezone. + start_datetime = to_datetime(in_timezone=self.config.general.timezone) + self.set_start_datetime( + start_datetime, + interval_seconds=( + self.config.optimization.genetic.interval_sec + if algorithm == OptimizationAlgorithm.GENETIC + else 3600 + ), + ) # Throw away any memory cached results of the last energy management run. CacheEnergyManagementStore().clear() @@ -296,15 +337,19 @@ class EnergyManagement( # Prepare optimization parameters # This also creates default configurations for missing values and updates the predictions logger.info(f"{algorithm}: Starting optimzation parameter preparation.") - if genetic_parameters is None: - genetic_parameters = await GeneticOptimizationParameters.prepare() + try: + if isinstance(genetic_parameters, ConfigOptimizationRequest): + genetic_parameters = await genetic_parameters.resolve() + elif genetic_parameters is None: + genetic_parameters = await GeneticOptimizationParameters.prepare() if genetic_parameters is None: - logger.error( - f"{algorithm}: Energy management run canceled. " - "Could not prepare optimisation parameters." - ) + logger.error("GENETIC: Parameter preparation failed; canceling run.") EnergyManagement._stage = EnergyManagementStage.IDLE return None + except Exception: + logger.exception("GENETIC: Parameter preparation failed.") + EnergyManagement._stage = EnergyManagementStage.IDLE + return None # Take values from config if not given if genetic_generations is None: @@ -332,6 +377,7 @@ class EnergyManagement( GeneticOptimizationParameters, genetic_parameters ), # cast for mypy ngen=genetic_generations, + individuals=genetic_individuals, ), ) diff --git a/src/akkudoktoreos/devices/devicesabc.py b/src/akkudoktoreos/devices/devicesabc.py index 42a945c5..144cf9a5 100644 --- a/src/akkudoktoreos/devices/devicesabc.py +++ b/src/akkudoktoreos/devices/devicesabc.py @@ -1,6 +1,8 @@ """Abstract and base classes for devices.""" +import math from enum import StrEnum +from typing import Optional class BatteryOperationMode(StrEnum): @@ -71,6 +73,115 @@ class BatteryOperationMode(StrEnum): FAULT = "FAULT" +def validate_home_appliance_load_definition( + *, + load_profile_power_w: Optional[list[float]], + load_profile_interval_seconds: Optional[int], + consumption_wh: Optional[float], + duration_h: Optional[float], +) -> None: + """Validate the load definition of a flexible consumer / home appliance. + + A consumer's load must be given **either** as a full explicit power profile + (``load_profile_power_w``) **or** as the complete flat fallback + (``consumption_wh`` together with ``duration_h``). Providing both, or only a + part of the fallback, is rejected. Profile values must be finite and + non-negative and the profile interval, if given, must be positive. + + Args: + load_profile_power_w: Explicit per-step power values [W], or None. + load_profile_interval_seconds: Duration of one profile step [s], or None. + consumption_wh: Fallback total energy of one run [Wh], or None. + duration_h: Fallback run duration [h], or None. + + Raises: + ValueError: If the definition is conflicting, incomplete, or contains + invalid profile values. + """ + profile_given = load_profile_power_w is not None + fallback_fields = (consumption_wh, duration_h) + fallback_partial = any(field is not None for field in fallback_fields) + fallback_given = all(field is not None for field in fallback_fields) + + if profile_given and fallback_partial: + raise ValueError( + "Conflicting home appliance load definition: provide either " + "load_profile_power_w or consumption_wh together with duration_h, " + "not both." + ) + + if load_profile_power_w is None: + if not fallback_given: + raise ValueError( + "Incomplete home appliance load definition: provide a full " + "load_profile_power_w or both consumption_wh and duration_h." + ) + # Value ranges of the fallback fields are enforced by their Field + # constraints (gt=0); nothing more to check here. + return + + # Explicit profile path. + if load_profile_interval_seconds is not None and load_profile_interval_seconds <= 0: + raise ValueError("load_profile_interval_seconds must be greater than zero.") + if len(load_profile_power_w) == 0: + raise ValueError("load_profile_power_w must not be empty.") + for value in load_profile_power_w: + if value is None or math.isnan(value) or math.isinf(value): + raise ValueError( + "load_profile_power_w must contain only finite values (no NaN or infinity)." + ) + if value < 0: + raise ValueError("load_profile_power_w must not contain negative values.") + + +class ConsumerScheduleMode(StrEnum): + """Schedule mode of a flexible consumer (home appliance). + + Determines how often a consumer's load profile is scheduled within the + optimization horizon. + + Modes + ----- + - ONCE: + The consumer runs exactly once somewhere within the optimization + horizon ("fire and forget"). The optimizer picks the start. + + - DAILY: + The consumer runs once per local calendar day, but only on days for + which at least one complete, allowed run still fits into the remaining + horizon. The optimizer picks one start per eligible day. + """ + + ONCE = "ONCE" + DAILY = "DAILY" + + +class ConsumerDeadlinePolicy(StrEnum): + """Behaviour when a flexible consumer's deadline cannot be met. + + A deadline (``deadline_datetime``) demands that a complete run has *finished* + before that moment. Depending on "now", the run duration, the optimization + horizon and the allowed time windows, no such start may exist. + + Policies + -------- + - BEST_EFFORT: + Run as early as the remaining constraints allow, i.e. minimize the + delay instead of the cost ("it should have been done by 03:00, so + start now"). A warning is logged. This keeps an optimization request + answerable instead of failing it - the usual choice for home + automation. + + - STRICT: + Keep the deadline. A ONCE consumer without a feasible start makes the + optimization fail; a DAILY consumer is simply not scheduled on days + without a feasible start. + """ + + BEST_EFFORT = "BEST_EFFORT" + STRICT = "STRICT" + + class ApplianceOperationMode(StrEnum): """Appliance operation modes. diff --git a/src/akkudoktoreos/devices/genetic/battery.py b/src/akkudoktoreos/devices/genetic/battery.py index bb45e52c..f6db6b12 100644 --- a/src/akkudoktoreos/devices/genetic/battery.py +++ b/src/akkudoktoreos/devices/genetic/battery.py @@ -1,10 +1,11 @@ from typing import Any, Iterator, Optional import numpy as np -from pydantic import Field +from pydantic import Field, field_validator from akkudoktoreos.devices.settings.batterysettings import BATTERY_DEFAULT_CHARGE_RATES from akkudoktoreos.optimization.genetic.geneticdevices import DeviceParameters +from akkudoktoreos.utils.datetimeutil import DateTime, to_datetime def max_charging_power_field(description: Optional[str] = None) -> float: @@ -122,6 +123,39 @@ class ElectricVehicleParameters(BaseBatteryParameters): "An integer representing the current state of charge (SOC) of the battery in percentage." ) + min_soc_deadline_datetime: Optional[DateTime] = Field( + default=None, + json_schema_extra={ + "description": ( + "Absolute moment by which 'min_soc_percentage' has to be " + "reached (departure time). A date time without timezone is read " + "as local time. None means end of the optimization horizon." + ), + "examples": [None, "2026-07-16T07:00:00+02:00"], + }, + ) + min_soc_max_duration_h: Optional[float] = Field( + default=None, + gt=0, + allow_inf_nan=False, + json_schema_extra={ + "description": ( + "Maximum time from the start of the optimization until " + "'min_soc_percentage' has to be reached [h]. Combined with " + "'min_soc_deadline_datetime' the earlier of the two applies." + ), + "examples": [None, 6.0], + }, + ) + + @field_validator("min_soc_deadline_datetime", mode="before") + @classmethod + def transform_deadline_to_datetime(cls, value: Any) -> Optional[DateTime]: + """Accept the usual date time representations, naive input is local time.""" + if value is None: + return None + return to_datetime(value) + class Battery: """Represents a battery device with methods to simulate energy charging and discharging.""" diff --git a/src/akkudoktoreos/devices/genetic/homeappliance.py b/src/akkudoktoreos/devices/genetic/homeappliance.py index ebbc1bdf..8d1f8dbb 100644 --- a/src/akkudoktoreos/devices/genetic/homeappliance.py +++ b/src/akkudoktoreos/devices/genetic/homeappliance.py @@ -5,7 +5,7 @@ machines that must run for a fixed duration, possibly multiple times (cycles), within one or more allowed time windows. Given a set of requested start times, `HomeAppliance` repairs them into a feasible, chronologically ordered schedule and produces the resulting -hourly load curve. +per-slot energy curve. Time windows are always expressed as a `CycleTimeWindowSequence` (see ``akkudoktoreos.config.configabc``): each contained window's @@ -29,22 +29,92 @@ Cycle start times are repaired according to the following rules: 5. Generate the combined hourly load curve from the final starts. """ -from typing import Optional +import math +from collections.abc import Sequence +from typing import Any, Optional, Self import numpy as np -from pydantic import Field +from loguru import logger +from pydantic import Field, field_validator, model_validator -from akkudoktoreos.config.configabc import CycleTimeWindowSequence, ValueTimeWindow +from akkudoktoreos.config.configabc import ( + CycleTimeWindowSequence, + TimeWindow, + TimeWindowSequence, + ValueTimeWindow, +) +from akkudoktoreos.devices.devicesabc import ( + ConsumerDeadlinePolicy, + ConsumerScheduleMode, + validate_home_appliance_load_definition, +) from akkudoktoreos.optimization.genetic.geneticdevices import DeviceParameters from akkudoktoreos.utils.datetimeutil import ( DateTime, Duration, + compare_datetimes, to_datetime, to_duration, to_time, ) +def resample_power_to_slot_energy( + power_w: list[float], + input_interval_seconds: float, + slot_interval_seconds: float, +) -> np.ndarray: + """Resample a piecewise-constant power profile to per-slot energy. + + Each input value ``power_w[i]`` is interpreted as a constant power [W] over + the interval ``[i * input_interval_seconds, (i + 1) * input_interval_seconds)``. + The energy of every output slot is the time-weighted integral of the input + power over that slot:: + + E_j = sum_i P_i * overlap(i, j) / 3600 [Wh] + + where ``overlap(i, j)`` is the temporal overlap (in seconds) between input + interval ``i`` and output slot ``j``. This is exact for arbitrary (including + non-integer) ratios such as 10 -> 15 or 20 -> 15 minutes and conserves + energy within numerical tolerance:: + + sum_j E_j == sum_i P_i * input_interval_seconds / 3600 + + Args: + power_w: Piecewise-constant power values [W] of a single run. + input_interval_seconds: Duration of one input step [s] (> 0). + slot_interval_seconds: Duration of one output slot [s] (> 0). + + Returns: + 1-D array of per-slot energy [Wh]; length is the number of slots the run + occupies (ceil of the total run duration divided by the slot duration). + """ + if not math.isfinite(input_interval_seconds) or input_interval_seconds <= 0: + raise ValueError("Input interval must be finite and positive.") + if not math.isfinite(slot_interval_seconds) or slot_interval_seconds <= 0: + raise ValueError("Slot interval must be finite and positive.") + if not power_w or any(not math.isfinite(p) or p < 0 for p in power_w): + raise ValueError("Power profile must contain finite non-negative values.") + n_in = len(power_w) + total_seconds = n_in * input_interval_seconds + n_slots = int(np.ceil(total_seconds / slot_interval_seconds - 1e-9)) + out = np.zeros(max(n_slots, 0), dtype=float) + for i, power in enumerate(power_w): + if power == 0.0: + continue + seg_start = i * input_interval_seconds + seg_end = seg_start + input_interval_seconds + first = int(seg_start // slot_interval_seconds) + last = int((seg_end - 1e-9) // slot_interval_seconds) + for j in range(first, last + 1): + slot_start = j * slot_interval_seconds + slot_end = slot_start + slot_interval_seconds + overlap = min(seg_end, slot_end) - max(seg_start, slot_start) + if overlap > 0: + out[j] += power * overlap / 3600.0 + return out + + class HomeApplianceParameters(DeviceParameters): """Configuration for a simulated home appliance device.""" @@ -54,7 +124,8 @@ class HomeApplianceParameters(DeviceParameters): "examples": ["dishwasher"], } ) - consumption_wh: int = Field( + consumption_wh: Optional[int] = Field( + default=None, gt=0, json_schema_extra={ "description": ( @@ -64,7 +135,8 @@ class HomeApplianceParameters(DeviceParameters): "examples": [2000], }, ) - duration_h: int = Field( + duration_h: Optional[int] = Field( + default=None, gt=0, json_schema_extra={ "description": ( @@ -81,6 +153,10 @@ class HomeApplianceParameters(DeviceParameters): "examples": [2], }, ) + completed_cycles: int = Field( + default=0, ge=0, description="Cycles already completed on the first planning day." + ) + min_cycle_gap_h: int = Field( default=0, ge=0, @@ -113,6 +189,112 @@ class HomeApplianceParameters(DeviceParameters): }, ) + shared_time_windows: Optional[TimeWindowSequence] = Field( + default=None, + description="Allowed recurring windows shared by every cycle; intersected with per-cycle windows.", + ) + + load_profile_power_w: Optional[list[float]] = Field( + default=None, + json_schema_extra={ + "description": ( + "Explicit load profile describing a single complete run as a " + "sequence of non-negative power values in watts. Each value " + "covers 'load_profile_interval_seconds'. Mutually exclusive with " + "consumption_wh/duration_h." + ), + "examples": [[200.0, 2000.0, 1800.0, 100.0]], + }, + ) + load_profile_interval_seconds: Optional[int] = Field( + default=None, + gt=0, + json_schema_extra={ + "description": ( + "Duration of one 'load_profile_power_w' step in seconds. Defaults " + "to the configured optimization interval when a profile is given." + ), + "examples": [900, 3600], + }, + ) + schedule_mode: ConsumerScheduleMode = Field( + default=ConsumerScheduleMode.ONCE, + json_schema_extra={ + "description": ( + "Scheduling mode: ONCE (a single run within the horizon) or DAILY " + "(one run per local calendar day with a feasible full run)." + ), + "examples": ["ONCE", "DAILY"], + }, + ) + earliest_start_datetime: Optional[DateTime] = Field( + default=None, + json_schema_extra={ + "description": ( + "Absolute earliest moment the run may start. Starts before it are " + "dropped, in addition to 'time_windows' and the horizon. A date " + "time without timezone is read as local time. This bound is never " + "relaxed." + ), + "examples": [None, "2026-07-15T20:00:00+02:00"], + }, + ) + deadline_datetime: Optional[DateTime] = Field( + default=None, + json_schema_extra={ + "description": ( + "Absolute deadline: the complete run must have *finished* at or " + "before this moment (e.g. end of the day, or 03:00 tonight). A " + "date time without timezone is read as local time. See " + "'deadline_policy' for what happens when no start can meet it." + ), + "examples": [None, "2026-07-16T03:00:00+02:00"], + }, + ) + deadline_policy: ConsumerDeadlinePolicy = Field( + default=ConsumerDeadlinePolicy.BEST_EFFORT, + json_schema_extra={ + "description": ( + "What to do when 'deadline_datetime' cannot be met: BEST_EFFORT " + "runs as early as possible instead (warning logged), STRICT keeps " + "the deadline (a ONCE consumer then fails the optimization)." + ), + "examples": ["BEST_EFFORT", "STRICT"], + }, + ) + + @field_validator("earliest_start_datetime", "deadline_datetime", mode="before") + @classmethod + def transform_to_datetime(cls, value: Any) -> Optional[DateTime]: + """Accept the usual date time representations, naive input is local time.""" + if value is None: + return None + return to_datetime(value) + + @model_validator(mode="after") + def validate_load_definition(self) -> Self: + """Ensure a complete load definition and valid completed-cycle count.""" + if self.completed_cycles > self.num_cycles: + raise ValueError("completed_cycles must not exceed num_cycles.") + validate_home_appliance_load_definition( + load_profile_power_w=self.load_profile_power_w, + load_profile_interval_seconds=self.load_profile_interval_seconds, + consumption_wh=self.consumption_wh, + duration_h=self.duration_h, + ) + return self + + @model_validator(mode="after") + def validate_schedule_bounds(self) -> Self: + """Reject an empty scheduling interval.""" + if self.earliest_start_datetime is not None and self.deadline_datetime is not None: + if compare_datetimes(self.deadline_datetime, self.earliest_start_datetime).le: + raise ValueError( + f"deadline_datetime {self.deadline_datetime} must be after " + f"earliest_start_datetime {self.earliest_start_datetime}." + ) + return self + class HomeAppliance: """Non-vectorized simulation of a multi-cycle home appliance. @@ -127,6 +309,7 @@ class HomeAppliance: parameters: HomeApplianceParameters, optimization_hours: int, prediction_hours: int, + slot_duration_h: float = 1.0, ) -> None: """Initializes the appliance and builds its allowed-start masks. @@ -135,18 +318,32 @@ class HomeAppliance: optimization_hours: Number of hours under active optimization. prediction_hours: Length of the simulation horizon, in - hours. + slots. + slot_duration_h: Physical duration of each slot, in hours. """ self.parameters = parameters self.optimization_hours = optimization_hours self.prediction_hours = prediction_hours - self.duration_h = parameters.duration_h - self.consumption_wh = parameters.consumption_wh + if not math.isfinite(slot_duration_h) or slot_duration_h <= 0: + raise ValueError("Slot duration must be finite and positive.") + self.total_slots = prediction_hours + self.slot_duration_h = slot_duration_h + self.slot_interval_seconds = slot_duration_h * 3600 + self.device_id = parameters.device_id + self.schedule_mode = parameters.schedule_mode + self.time_windows = parameters.shared_time_windows + self.earliest_start_datetime = parameters.earliest_start_datetime + self.deadline_datetime = parameters.deadline_datetime + self.deadline_policy = parameters.deadline_policy + self.deadline_relaxed = False + self._build_run_profile() + self.duration_h = self.run_slots * slot_duration_h + self.consumption_wh = float(self.run_energy_wh.sum()) self.num_cycles = parameters.num_cycles self.min_cycle_gap_h = parameters.min_cycle_gap_h - self.completed_cycles = 0 + self.completed_cycles = parameters.completed_cycles self.load_curve = np.zeros(prediction_hours) @@ -173,6 +370,280 @@ class HomeAppliance: self._setup() + def _build_run_profile(self) -> None: + """Build the per-slot energy [Wh] of a single complete run.""" + if self.parameters.load_profile_power_w is not None: + power = [float(value) for value in self.parameters.load_profile_power_w] + input_interval = ( + self.parameters.load_profile_interval_seconds or self.slot_interval_seconds + ) + else: + # Flat fallback: constant power over duration_h hours. Route it through + # the same resampling path so hourly and sub-hourly grids behave + # identically. Power [W] = energy per hour = consumption_wh / duration_h. + duration_h = self.parameters.duration_h + consumption_wh = self.parameters.consumption_wh + if duration_h is None or consumption_wh is None: + raise ValueError("Flat appliance load requires duration and consumption.") + power = [consumption_wh / duration_h] + input_interval = duration_h * 3600 + + self.run_energy_wh: np.ndarray = resample_power_to_slot_energy( + power, float(input_interval), float(self.slot_interval_seconds) + ) + self.run_slots: int = int(len(self.run_energy_wh)) + + def _slot_offset(self, moment: DateTime, slot0_datetime: DateTime, *, round_up: bool) -> int: + """Convert an absolute moment into a slot index relative to slot 0. + + Args: + moment: Absolute moment; converted into ``slot0_datetime``'s timezone. + slot0_datetime: Local, timezone-aware datetime of slot index 0. + round_up: ``True`` returns the first slot boundary at or after + ``moment`` (lower bounds), ``False`` the last one at or before + it (upper bounds). + + Returns: + Slot index (may be negative or beyond the grid; callers clamp). + """ + timezone = slot0_datetime.timezone + if timezone is None: + raise ValueError("The optimization slot origin must have a timezone.") + seconds = (moment.in_timezone(timezone) - slot0_datetime).total_seconds() + exact = seconds / self.slot_interval_seconds + # Tolerance absorbs float noise so a moment that sits exactly on a slot + # boundary is not pushed to the neighbouring slot. + return math.ceil(exact - 1e-9) if round_up else math.floor(exact + 1e-9) + + def allowed_start_slots( + self, + *, + slot0_datetime: DateTime, + earliest_slot: int, + horizon_end_slot: int, + cycle_index: Optional[int] = None, + ) -> list[int]: + """Return the sorted absolute start slots at which a full run is allowed. + + A start slot ``s`` is allowed when the complete run fits the optimization + horizon, both absolute bounds and (if configured) a single allowed time + window: + + - ``earliest_slot <= s`` and ``s + run_slots <= horizon_end_slot`` + - with ``earliest_start_datetime`` set, the run starts at or after it + - with ``deadline_datetime`` set, the run *ends* at or before it + - with ``time_windows`` set, the run's whole occupied span starting at + ``s`` is contained in one window (respecting weekday/date constraints) + + When a deadline leaves no start at all and the policy is + ``BEST_EFFORT``, the deadline is dropped and only the earliest still + possible start is offered (a warning is logged and ``deadline_relaxed`` + is set). Multiple cycles retain all relaxed choices for joint + earliest-start repair by the optimizer, respecting per-cycle windows + and minimum gaps. + + No snapping is performed: every returned slot is a genuinely valid start. + + Args: + slot0_datetime: Local, timezone-aware datetime of slot index 0. + earliest_slot: First slot the optimizer may schedule at ("now"). + horizon_end_slot: Exclusive upper bound; a run must end at or before. + cycle_index: Global configured cycle index, independent of completed cycles. + + Returns: + Sorted list of allowed absolute start slots (may be empty). + """ + self.deadline_relaxed = False + allowed = self._allowed_start_slots( + slot0_datetime=slot0_datetime, + earliest_slot=earliest_slot, + horizon_end_slot=horizon_end_slot, + apply_deadline=True, + cycle_index=cycle_index, + ) + if ( + allowed + or self.deadline_datetime is None + or self.deadline_policy == ConsumerDeadlinePolicy.STRICT + ): + return allowed + + relaxed = self._allowed_start_slots( + slot0_datetime=slot0_datetime, + earliest_slot=earliest_slot, + horizon_end_slot=horizon_end_slot, + apply_deadline=False, + cycle_index=cycle_index, + ) + if not relaxed: + return relaxed + self.deadline_relaxed = True + # Keep only the earliest possible start: the deadline is already missed, + # so the run is scheduled as soon as possible rather than as cheap as + # possible. Multiple cycles retain choices so the optimizer can find + # the earliest joint schedule that respects idle gaps. + earliest = relaxed[:1] if self.num_cycles == 1 else relaxed + logger.warning( + "Home appliance '{}': deadline {} can not be met - running as early as " + "possible instead (BEST_EFFORT). Run ends {}.", + self.device_id, + self.deadline_datetime, + self.run_end_datetime(earliest[0], slot0_datetime), + ) + return earliest + + def _allowed_start_slots( + self, + *, + slot0_datetime: DateTime, + earliest_slot: int, + horizon_end_slot: int, + cycle_index: Optional[int] = None, + apply_deadline: bool, + ) -> list[int]: + """Compute the allowed start slots for one set of constraints. + + Args: + slot0_datetime: Local, timezone-aware datetime of slot index 0. + earliest_slot: First slot the optimizer may schedule at ("now"). + horizon_end_slot: Exclusive upper bound; a run must end at or before. + cycle_index: Global configured cycle index, independent of completed cycles. + apply_deadline: Whether ``deadline_datetime`` restricts the run end. + + Returns: + Sorted list of allowed absolute start slots (may be empty). + """ + run_slots = self.run_slots + if run_slots <= 0: + return [] + + first_start = max(earliest_slot, 0) + if self.earliest_start_datetime is not None: + first_start = max( + first_start, + self._slot_offset(self.earliest_start_datetime, slot0_datetime, round_up=True), + ) + + end_bound = min(horizon_end_slot, self.total_slots) + if apply_deadline and self.deadline_datetime is not None: + end_bound = min( + end_bound, + self._slot_offset(self.deadline_datetime, slot0_datetime, round_up=False), + ) + + last_start = end_bound - run_slots + if last_start < first_start: + return [] + + if cycle_index is not None and not 0 <= cycle_index < self.num_cycles: + raise ValueError("Cycle index is outside the configured cycle range.") + cycle_windows = [ + window + for window in ( + self.parameters.time_windows.windows if self.parameters.time_windows else [] + ) + if window.value is not None and int(window.value) == cycle_index + ] + + run_duration = to_duration(f"{run_slots * self.slot_interval_seconds} seconds") + allowed: list[int] = [] + for slot in range(first_start, last_start + 1): + start_dt = slot0_datetime.add(seconds=slot * self.slot_interval_seconds) + if ( + self.time_windows is None + or self._windows_allow_run(self.time_windows.windows, start_dt, run_duration) + ) and ( + not cycle_windows + or self._windows_allow_run(cycle_windows, start_dt, run_duration, merge=True) + ): + allowed.append(slot) + return allowed + + @staticmethod + def _windows_allow_run( + windows: Sequence[TimeWindow], start: DateTime, duration: Duration, *, merge: bool = False + ) -> bool: + """Check full coverage including windows anchored on previous local dates. + + Date and weekday restrictions belong to the window's opening day. + Per-cycle windows retain their existing union semantics; shared windows + require one complete containing occurrence. + """ + intervals: list[tuple[DateTime, DateTime]] = [] + run_end = start + duration + for window in windows: + days_back = math.ceil(window.duration.total_seconds() / 86400) + 1 + for offset in range(days_back + 1): + anchor = start.subtract(days=offset) + if window.date is not None and anchor.date() != window.date: + continue + if window.day_of_week is not None and anchor.day_of_week != window.day_of_week: + continue + opening, closing = window._window_start_end(anchor) + if opening <= start and run_end <= closing: + return True + if merge and opening < run_end and closing > start: + intervals.append((opening, closing)) + covered_until = start + for opening, closing in sorted(intervals): + if opening > covered_until: + break + covered_until = max(covered_until, closing) + if covered_until >= run_end: + return True + return False + + def run_end_datetime(self, start_slot: int, slot0_datetime: DateTime) -> DateTime: + """Absolute local moment at which a run started at ``start_slot`` finishes. + + Args: + start_slot: Absolute start slot of the run. + slot0_datetime: Local, timezone-aware datetime of slot index 0. + + Returns: + End datetime of the run (exclusive, i.e. the first free moment). + """ + return slot0_datetime.add( + seconds=(start_slot + self.run_slots) * self.slot_interval_seconds + ) + + def deadline_missed(self, starts: list[int], slot0_datetime: DateTime) -> bool: + """Whether the scheduled runs violate the configured deadline. + + Without a deadline nothing can be missed. With one, a consumer that was + not scheduled at all, or whose run ends after the deadline (a relaxed + BEST_EFFORT deadline), counts as missed. + + Args: + starts: Absolute start slots of the scheduled runs. + slot0_datetime: Local, timezone-aware datetime of slot index 0. + + Returns: + True if the deadline is set and not met. + """ + if self.deadline_datetime is None: + return False + if not starts: + return True + timezone = slot0_datetime.timezone + if timezone is None: + raise ValueError("The optimization slot origin must have a timezone.") + deadline = self.deadline_datetime.in_timezone(timezone) + return any(self.run_end_datetime(start, slot0_datetime) > deadline for start in starts) + + def build_load_curve(self, starts: list[int]) -> None: + """Place the resampled run energy at each decoded start slot. + + Multiple runs may overlap; their per-slot energies are summed. + + Args: + starts: Absolute start slots of the scheduled runs. + """ + if any(start < 0 or start + self.run_slots > self.total_slots for start in starts): + raise ValueError("A complete appliance run must fit inside the slot horizon.") + self.start_hours = list(starts) + self._build_load_curve() + # ------------------------------------------------------------------ # Setup # ------------------------------------------------------------------ @@ -257,9 +728,9 @@ class HomeAppliance: microsecond=0, ) end_datetime = start_datetime.add( - hours=self.prediction_hours, + seconds=self.prediction_hours * self.slot_interval_seconds, ) - interval = to_duration("1 hour") + interval = to_duration(self.slot_interval_seconds) self._build_cycle_start_allowed( self.parameters.time_windows, @@ -293,7 +764,7 @@ class HomeAppliance: max_start = max( 0, - self.prediction_hours - self.duration_h, + self.prediction_hours - self.run_slots, ) # The matrix tells us which individual *steps* are inside @@ -348,7 +819,7 @@ class HomeAppliance: horizon = len(window_steps) max_start = max( 0, - horizon - self.duration_h, + horizon - self.run_slots, ) allowed = np.zeros( @@ -356,15 +827,15 @@ class HomeAppliance: dtype=bool, ) - if self.duration_h > horizon: + if self.run_slots > horizon: return allowed # Rolling sum of `duration_h` consecutive steps, aligned so # that window_sums[s] == sum(window_steps[s : s + duration_h]). cumulative = np.concatenate(([0.0], np.cumsum(window_steps))) - window_sums = cumulative[self.duration_h :] - cumulative[: -self.duration_h] + window_sums = cumulative[self.run_slots :] - cumulative[: -self.run_slots] - allowed[: max_start + 1] = window_sums[: max_start + 1] == float(self.duration_h) + allowed[: max_start + 1] = window_sums[: max_start + 1] == float(self.run_slots) return allowed @@ -404,7 +875,7 @@ class HomeAppliance: max_start = max( 0, - self.prediction_hours - self.duration_h, + self.prediction_hours - self.run_slots, ) # 1. Round and clip. @@ -442,7 +913,7 @@ class HomeAppliance: # 4. Enforce duration + minimum idle gap. Each cycle is # pushed forward, if needed, to the next start allowed by # its *own* window. - min_next_start = self.duration_h + self.min_cycle_gap_h + min_next_start = self.run_slots + math.ceil(self.min_cycle_gap_h / self.slot_duration_h) final_starts = [repaired[0][0]] @@ -582,25 +1053,17 @@ class HomeAppliance: """Builds the load curve from all scheduled cycles.""" self.reset_load_curve() - power_per_hour = self.consumption_wh / self.duration_h - - for start_hour in self.start_hours: - if start_hour >= self.prediction_hours: - continue - - end_hour = min( - start_hour + self.duration_h, - self.prediction_hours, - ) - - self.load_curve[start_hour:end_hour] += power_per_hour + for start in self.start_hours: + length = min(self.run_slots, self.total_slots - start) + if 0 <= start and length > 0: + self.load_curve[start : start + length] += self.run_energy_wh[:length] def reset_load_curve(self) -> None: """Resets the load curve to all zeros.""" self.load_curve = np.zeros(self.prediction_hours) def get_load_curve(self) -> np.ndarray: - """Returns the current hourly load curve, in watts.""" + """Returns the current per-slot load curve, in watt-hours.""" return self.load_curve def get_load_for_hour(self, hour: int) -> float: @@ -610,7 +1073,7 @@ class HomeAppliance: hour: Hour of the prediction horizon to look up. Returns: - The load, in watts, at ``hour``. + The energy, in watt-hours, at slot ``hour``. Raises: ValueError: If ``hour`` is outside diff --git a/src/akkudoktoreos/devices/settings/batterysettings.py b/src/akkudoktoreos/devices/settings/batterysettings.py index 84bd2f15..bdc26b65 100644 --- a/src/akkudoktoreos/devices/settings/batterysettings.py +++ b/src/akkudoktoreos/devices/settings/batterysettings.py @@ -10,11 +10,13 @@ import numpy as np from numpydantic import NDArray, Shape from pydantic import Field, computed_field, field_validator, model_validator +from akkudoktoreos.config.configabc import ConfigScope from akkudoktoreos.devices.settings.devicebasesettings import DevicesBaseSettings from akkudoktoreos.measurement.batterycapacity import ( BatteryCapacityEstimate, BatteryCapacityEstimationSettings, ) +from akkudoktoreos.utils.datetimeutil import DateTime, to_datetime if TYPE_CHECKING: from akkudoktoreos.devices.genetic0.genetic0battery import ( @@ -175,6 +177,41 @@ class BatteriesCommonSettings(DevicesBaseSettings): return arr.tolist() + min_soc_deadline_datetime: Optional[DateTime] = Field( + default=None, + json_schema_extra={ + "x-scope": [str(ConfigScope.GENETIC)], + "description": ( + "Absolute moment by which 'min_soc_percentage' has to be " + "reached (departure time). A date time without timezone is read " + "as local time. None means end of the optimization horizon." + ), + "examples": [None, "2026-07-16T07:00:00+02:00"], + }, + ) + min_soc_max_duration_h: Optional[float] = Field( + default=None, + gt=0, + allow_inf_nan=False, + json_schema_extra={ + "x-scope": [str(ConfigScope.GENETIC)], + "description": ( + "Maximum time from the start of the optimization until " + "'min_soc_percentage' has to be reached [h]. Combined with " + "'min_soc_deadline_datetime' the earlier of the two applies." + ), + "examples": [None, 6.0], + }, + ) + + @field_validator("min_soc_deadline_datetime", mode="before") + @classmethod + def transform_deadline_to_datetime(cls, value: Any) -> Optional[DateTime]: + """Accept the usual date time representations, naive input is local time.""" + if value is None: + return None + return to_datetime(value) + def to_genetic_pv_bat_param(self) -> "SolarPanelBatteryParameters": """Return SolarPanelBatteryParameters for the GENETIC optimizer.""" from akkudoktoreos.devices.genetic.battery import SolarPanelBatteryParameters @@ -205,6 +242,8 @@ class BatteriesCommonSettings(DevicesBaseSettings): max_charge_power_w=self.max_charge_power_w, min_soc_percentage=self.min_soc_percentage, max_soc_percentage=self.max_soc_percentage, + min_soc_deadline_datetime=self.min_soc_deadline_datetime, + min_soc_max_duration_h=self.min_soc_max_duration_h, ) # ------------------------------------------------------------------ diff --git a/src/akkudoktoreos/devices/settings/homeappliancesettings.py b/src/akkudoktoreos/devices/settings/homeappliancesettings.py index 81a6e448..22e8606e 100644 --- a/src/akkudoktoreos/devices/settings/homeappliancesettings.py +++ b/src/akkudoktoreos/devices/settings/homeappliancesettings.py @@ -3,12 +3,22 @@ Note: Used for the GENETIC and GENETIC0 algorithm. """ -from typing import TYPE_CHECKING, Optional +from typing import TYPE_CHECKING, Any, Optional, Self -from pydantic import Field, computed_field, model_validator +from pydantic import Field, computed_field, field_validator, model_validator -from akkudoktoreos.config.configabc import ConfigScope, CycleTimeWindowSequence +from akkudoktoreos.config.configabc import ( + ConfigScope, + CycleTimeWindowSequence, + TimeWindowSequence, +) +from akkudoktoreos.devices.devicesabc import ( + ConsumerDeadlinePolicy, + ConsumerScheduleMode, + validate_home_appliance_load_definition, +) from akkudoktoreos.devices.settings.devicebasesettings import DevicesBaseSettings +from akkudoktoreos.utils.datetimeutil import DateTime, compare_datetimes, to_datetime if TYPE_CHECKING: from akkudoktoreos.devices.genetic0.genetic0homeappliance import ( @@ -105,7 +115,7 @@ class HomeApplianceCommonSettings(DevicesBaseSettings): """ - consumption_wh: int = Field( + consumption_wh: Optional[int] = Field( default=3000, gt=0, json_schema_extra={ @@ -114,7 +124,7 @@ class HomeApplianceCommonSettings(DevicesBaseSettings): "x-scope": [str(ConfigScope.GENETIC), str(ConfigScope.GENETIC0)], }, ) - duration_h: int = Field( + duration_h: Optional[int] = Field( default=3, gt=0, le=24, @@ -193,6 +203,133 @@ class HomeApplianceCommonSettings(DevicesBaseSettings): }, ) + @model_validator(mode="before") + @classmethod + def _profile_replaces_flat_defaults(cls, value: Any) -> Any: + """Keep historical flat defaults unless an explicit profile is supplied.""" + if isinstance(value, dict) and value.get("load_profile_power_w") is not None: + value = dict(value) + value.setdefault("consumption_wh", None) + value.setdefault("duration_h", None) + return value + + load_profile_power_w: Optional[list[float]] = Field( + default=None, + json_schema_extra={ + "x-scope": [str(ConfigScope.GENETIC)], + "description": ( + "Explicit load profile describing a single complete run as a " + "sequence of non-negative power values in watts. Each value " + "covers 'load_profile_interval_seconds'. Mutually exclusive with " + "consumption_wh/duration_h." + ), + "examples": [[200.0, 2000.0, 1800.0, 100.0]], + }, + ) + load_profile_interval_seconds: Optional[int] = Field( + default=None, + gt=0, + json_schema_extra={ + "x-scope": [str(ConfigScope.GENETIC)], + "description": ( + "Duration of one 'load_profile_power_w' step in seconds. Defaults " + "to the configured optimization interval when a profile is given." + ), + "examples": [900, 3600], + }, + ) + schedule_mode: ConsumerScheduleMode = Field( + default=ConsumerScheduleMode.ONCE, + json_schema_extra={ + "x-scope": [str(ConfigScope.GENETIC)], + "description": ( + "Scheduling mode: ONCE (a single run within the horizon) or DAILY " + "(one run per local calendar day with a feasible full run)." + ), + "examples": ["ONCE", "DAILY"], + }, + ) + time_windows: Optional[TimeWindowSequence] = Field( + default=None, + json_schema_extra={ + "x-scope": [str(ConfigScope.GENETIC)], + "description": "List of allowed time windows. Defaults to optimization general time window.", + "examples": [ + [ + {"start_time": "10:00", "duration": "3 hours"}, + ], + ], + }, + ) + earliest_start_datetime: Optional[DateTime] = Field( + default=None, + json_schema_extra={ + "x-scope": [str(ConfigScope.GENETIC)], + "description": ( + "Absolute earliest moment the run may start. Starts before it are " + "dropped, in addition to 'time_windows' and the horizon. A date " + "time without timezone is read as local time. This bound is never " + "relaxed." + ), + "examples": [None, "2026-07-15T20:00:00+02:00"], + }, + ) + deadline_datetime: Optional[DateTime] = Field( + default=None, + json_schema_extra={ + "x-scope": [str(ConfigScope.GENETIC)], + "description": ( + "Absolute deadline: the complete run must have *finished* at or " + "before this moment (e.g. end of the day, or 03:00 tonight). A " + "date time without timezone is read as local time. See " + "'deadline_policy' for what happens when no start can meet it." + ), + "examples": [None, "2026-07-16T03:00:00+02:00"], + }, + ) + deadline_policy: ConsumerDeadlinePolicy = Field( + default=ConsumerDeadlinePolicy.BEST_EFFORT, + json_schema_extra={ + "x-scope": [str(ConfigScope.GENETIC)], + "description": ( + "What to do when 'deadline_datetime' cannot be met: BEST_EFFORT " + "runs as early as possible instead (warning logged), STRICT keeps " + "the deadline (a ONCE consumer then fails the optimization)." + ), + "examples": ["BEST_EFFORT", "STRICT"], + }, + ) + + @field_validator("earliest_start_datetime", "deadline_datetime", mode="before") + @classmethod + def transform_to_datetime(cls, value: Any) -> Optional[DateTime]: + """Accept the usual date time representations, naive input is local time.""" + if value is None: + return None + return to_datetime(value) + + @model_validator(mode="after") + def validate_load_definition(self) -> Self: + """Ensure exactly one complete, valid load definition is provided.""" + validate_home_appliance_load_definition( + load_profile_power_w=self.load_profile_power_w, + load_profile_interval_seconds=self.load_profile_interval_seconds, + consumption_wh=self.consumption_wh, + duration_h=self.duration_h, + ) + return self + + @model_validator(mode="after") + def validate_schedule_bounds(self) -> Self: + """Reject an empty scheduling interval.""" + if self.earliest_start_datetime is not None and self.deadline_datetime is not None: + if compare_datetimes(self.deadline_datetime, self.earliest_start_datetime).le: + raise ValueError( + f"deadline_datetime {self.deadline_datetime} must be after " + f"earliest_start_datetime {self.earliest_start_datetime}." + ) + return self + @model_validator(mode="after") def _validate_num_cycles_specified(self) -> "HomeApplianceCommonSettings": """Require num_cycles when windows are not provided.""" @@ -231,6 +368,13 @@ class HomeApplianceCommonSettings(DevicesBaseSettings): num_cycles=self.effective_num_cycles, min_cycle_gap_h=self.min_cycle_gap_h, time_windows=self.cycle_time_windows, + shared_time_windows=self.time_windows, + load_profile_power_w=self.load_profile_power_w, + load_profile_interval_seconds=self.load_profile_interval_seconds, + schedule_mode=self.schedule_mode, + earliest_start_datetime=self.earliest_start_datetime, + deadline_datetime=self.deadline_datetime, + deadline_policy=self.deadline_policy, ) # ------------------------------------------------------------------ @@ -243,6 +387,10 @@ class HomeApplianceCommonSettings(DevicesBaseSettings): Genetic0HomeApplianceParameters, ) + if self.load_profile_power_w is not None: + raise ValueError("Explicit appliance load profiles require the GENETIC optimizer.") + if self.consumption_wh is None or self.duration_h is None: + raise ValueError("GENETIC0 requires a flat appliance load definition.") return Genetic0HomeApplianceParameters( device_id=self.device_id, consumption_wh=self.consumption_wh, @@ -254,5 +402,4 @@ class HomeApplianceCommonSettings(DevicesBaseSettings): @property def measurement_keys(self) -> Optional[list[str]]: """Measurement keys for the home appliance stati that are measurements.""" - keys: list[str] = [] - return keys + return [self.cycles_completed_measurement_key or f"{self.device_id}.cycles_completed"] diff --git a/src/akkudoktoreos/optimization/genetic/configrequest.py b/src/akkudoktoreos/optimization/genetic/configrequest.py new file mode 100644 index 00000000..e72a5f41 --- /dev/null +++ b/src/akkudoktoreos/optimization/genetic/configrequest.py @@ -0,0 +1,241 @@ +"""Configuration-owned GENETIC requests with fresh runtime observations.""" + +import math +from typing import Annotated, Any, Optional + +from pydantic import AliasChoices, Field, field_validator + +from akkudoktoreos.core.coreabc import ( + ConfigMixin, + MeasurementMixin, + PredictionMixin, + get_ems, +) +from akkudoktoreos.devices.settings.batterysettings import BatteriesCommonSettings +from akkudoktoreos.optimization.genetic.forecast import bounded_forecast_array +from akkudoktoreos.optimization.genetic.geneticabc import GeneticParametersBaseModel +from akkudoktoreos.optimization.genetic.geneticparams import ( + GeneticEnergyManagementParameters, + GeneticOptimizationParameters, +) +from akkudoktoreos.utils.datetimeutil import DateTime, to_duration + + +class RuntimeForecasts(GeneticParametersBaseModel): + """External series from local midnight, in slot Wh and currency per Wh. + + The interval is configured in optimization.genetic.interval_sec. Omitted + series are read from the configured providers, preserving their raw coverage. + """ + + pv_forecast_wh: Optional[list[float]] = Field( + default=None, validation_alias=AliasChoices("pv_forecast_wh", "pv_prognose_wh") + ) + total_load: Optional[list[float]] = Field( + default=None, validation_alias=AliasChoices("total_load", "gesamtlast") + ) + electricity_price_per_wh: Optional[list[float]] = Field( + default=None, + validation_alias=AliasChoices("electricity_price_per_wh", "strompreis_euro_pro_wh"), + ) + feed_in_tariff_per_wh: Optional[list[float]] = Field( + default=None, + validation_alias=AliasChoices("feed_in_tariff_per_wh", "einspeiseverguetung_euro_pro_wh"), + ) + temperature_forecast: Optional[list[Optional[float]]] = None + + +class ConfigOptimizationRequest( + ConfigMixin, MeasurementMixin, PredictionMixin, GeneticParametersBaseModel +): + """Supply observations while hardware, tariffs and scheduling remain in config.""" + + soc: dict[str, Annotated[int, Field(ge=0, le=100)]] = Field(default_factory=dict) + forecasts: RuntimeForecasts = Field(default_factory=RuntimeForecasts) + start_solution: Optional[list[float]] = None + start_solution_datetime: Optional[DateTime] = None + + @field_validator("start_solution_datetime", mode="before") + @classmethod + def validate_start_solution_datetime(cls, value: Any) -> Optional[DateTime]: + """Parse a previous-plan timestamp while retaining its explicit timezone.""" + return GeneticOptimizationParameters.transform_start_solution_datetime(value) + + async def resolve(self) -> GeneticOptimizationParameters: + """Resolve inside the EMS lock after its slot start has been established.""" + config = self.config + settings = config.optimization.genetic + config.validate_optimization_horizons() + ems = get_ems() + start = ems.start_datetime + observation_time = ems.observation_datetime + devices = config.devices + groups: dict[str, list[Any]] = {} + for name in ("batteries", "electric_vehicles", "inverters", "home_appliances"): + entries = list((getattr(devices, name) or {}).values()) + maximum = getattr(devices, "max_" + name) + if maximum is not None and len(entries) > maximum: + raise ValueError(f"devices.{name} exceeds configured maximum {maximum}.") + if name != "home_appliances" and len(entries) > 1: + raise ValueError(f"GENETIC supports at most one device in devices.{name}.") + groups[name] = entries + ids = [device.device_id for entries in groups.values() for device in entries] + if len(ids) != len(set(ids)): + raise ValueError("Configured device IDs must be unique across device groups.") + storage = groups["batteries"] + groups["electric_vehicles"] + unknown = set(self.soc) - {device.device_id for device in storage} + if unknown: + raise ValueError(f"SoC supplied for unconfigured devices: {sorted(unknown)}.") + + async def state(device: BatteriesCommonSettings) -> int: + if device.device_id in self.soc: + return self.soc[device.device_id] + try: + dates, values = await self.measurement.key_to_lists( + key=device.measurement_key_soc_factor, + start_datetime=observation_time.subtract( + seconds=settings.measurement_max_age_seconds + ), + end_datetime=observation_time.add(seconds=1), + dropna=False, + ) + samples = [ + (date, value) + for date, value in zip(dates, values) + if date.timestamp() <= observation_time.timestamp() + ] + date, value = max(samples, key=lambda sample: sample[0].timestamp()) + if ( + observation_time.timestamp() - date.timestamp() + > settings.measurement_max_age_seconds + or value is None + or not math.isfinite(value) + or not 0 <= value <= 1 + ): + raise ValueError("Invalid or stale SoC factor") + return int(value * 100) + except (ValueError, KeyError) as exc: + raise ValueError( + f"Fresh SoC missing for {device.device_id}; supply soc or a measurement." + ) from exc + + battery = None + if groups["batteries"]: + device = groups["batteries"][0] + battery = device.to_genetic_pv_bat_param() + battery.initial_soc_percentage = await state(device) + ev = None + if groups["electric_vehicles"]: + device = groups["electric_vehicles"][0] + ev = device.to_genetic_ev_bat_param() + ev.initial_soc_percentage = await state(device) + inverter = None + if groups["inverters"]: + device = groups["inverters"][0] + if device.battery_id != (battery.device_id if battery else None): + raise ValueError("Inverter battery_id must match the configured battery.") + inverter = device.to_genetic_param() + if inverter is None: + raise ValueError("Configure an inverter to model PV and grid energy flows.") + appliances = [device.to_genetic_param() for device in groups["home_appliances"]] + for device, appliance in zip(groups["home_appliances"], appliances): + key = device.cycles_completed_measurement_key or f"{device.device_id}.cycles_completed" + try: + dates, counts = await self.measurement.key_to_lists( + key=key, + start_datetime=observation_time.start_of("day"), + end_datetime=observation_time.add(seconds=1), + dropna=False, + ) + except KeyError: + continue + samples = [ + (date, count) + for date, count in zip(dates, counts) + if date.timestamp() <= observation_time.timestamp() + ] + if not samples: + continue + count = max(samples, key=lambda item: item[0].timestamp())[1] + if ( + count is None + or not math.isfinite(count) + or int(count) != count + or not 0 <= count <= appliance.num_cycles + ): + raise ValueError(f"Invalid completed cycle count for {device.device_id}.") + appliance.completed_cycles = int(count) + + origin = start.start_of("day") + if config.prediction.hours is None or config.prediction.hours <= 0: + raise ValueError("Configure a positive prediction horizon.") + end = start.add(hours=config.prediction.hours) + interval = to_duration(settings.interval_sec) + updated = False + + async def forecast( + supplied: Optional[list[float]], key: str, energy: bool = False + ) -> list[float]: + nonlocal updated + if supplied is not None: + return list(supplied) + if not updated: + await self.prediction.update_data() + updated = True + values = await bounded_forecast_array( + self.prediction, key=key, start_datetime=origin, end_datetime=end, interval=interval + ) + if energy: + values = values * (settings.interval_sec / 3600) + return values.tolist() + + supplied = self.forecasts + pv = await forecast(supplied.pv_forecast_wh, "pvforecast_ac_power", energy=True) + load = await forecast(supplied.total_load, "loadforecast_power_w", energy=True) + prices = await forecast(supplied.electricity_price_per_wh, "elecprice_marketprice_wh") + tariffs = await forecast(supplied.feed_in_tariff_per_wh, "feed_in_tariff_wh") + length = min(len(pv), len(load), len(prices), len(tariffs)) + if not length: + raise ValueError("Forecast series must be nonempty.") + pv, load, prices, tariffs = (values[:length] for values in (pv, load, prices, tariffs)) + first = int((start - origin).total_seconds() / settings.interval_sec) + last = first + settings.horizon + if settings.horizon <= 0 or len(pv) < last: + raise ValueError("Forecast does not cover the positive GENETIC control horizon.") + for key, values in ( + ("PV", pv), + ("load", load), + ("prices", prices), + ("feed-in tariff", tariffs), + ): + if not all(math.isfinite(value) for value in values[first:last]): + raise ValueError(f"Missing or invalid {key} within the control horizon.") + if any(value < 0 for values in (pv, load) for value in values[first:last]): + raise ValueError("PV and load energy must be nonnegative.") + previous = ems.genetic_solution() + warm_start = self.start_solution + warm_start_time = self.start_solution_datetime + if warm_start is None and previous is not None: + warm_start = previous.start_solution + warm_start_time = previous.start_solution_datetime + return GeneticOptimizationParameters( + forecast_interval_seconds=settings.interval_sec, + ems=GeneticEnergyManagementParameters( + pv_forecast_wh=pv, + total_load=load, + electricity_price_per_wh=prices, + feed_in_tariff_per_wh=tariffs, + price_per_wh_battery=settings.terminal_value_euro_per_kwh / 1000, + ), + pv_battery=battery, + ev=ev, + inverter=inverter, + home_appliances=appliances, + temperature_forecast=( + (supplied.temperature_forecast[:length] + [None] * length)[:length] + if supplied.temperature_forecast is not None + else None + ), + start_solution=warm_start, + start_solution_datetime=warm_start_time, + ) diff --git a/src/akkudoktoreos/optimization/genetic/forecast.py b/src/akkudoktoreos/optimization/genetic/forecast.py new file mode 100644 index 00000000..58d31606 --- /dev/null +++ b/src/akkudoktoreos/optimization/genetic/forecast.py @@ -0,0 +1,61 @@ +"""Read forecast interval averages without filling gaps or extrapolating the tail.""" + +from typing import Any + +import numpy as np +import pandas as pd + +from akkudoktoreos.utils.datetimeutil import DateTime, Duration + + +async def bounded_forecast_array( + prediction: Any, + *, + key: str, + start_datetime: DateTime, + end_datetime: DateTime, + interval: Duration, +) -> np.ndarray: + """Integrate stored interval averages only across completely covered target slots. + + Source timestamps are interval starts. Infer their smallest positive cadence, + capped at one hour. A missing source interval, explicit NaN or the end of the + available forecast remains missing. UTC elapsed time handles DST transitions. + The returned values retain the input units; callers convert W to slot Wh once. + """ + seconds = int(interval.total_seconds()) + start = start_datetime.timestamp() + end = end_datetime.timestamp() + if seconds <= 0 or end <= start or (end - start) % seconds: + raise ValueError("Forecast bounds must contain a positive whole number of slots.") + result = np.full(int((end - start) / seconds), np.nan) + try: + series = await prediction.key_to_raw_series(key=key, dropna=False) + except KeyError: + return result + if series.empty: + return result + series = pd.to_numeric(series, errors="coerce").sort_index() + series = series[~series.index.duplicated(keep="last")] + stamps = pd.to_datetime(series.index, utc=True).as_unit("ns").asi8 / 1e9 + values = series.to_numpy(dtype=float) + differences = np.diff(stamps) + cadence = min(3600.0, float(np.min(differences))) if len(differences) else 3600.0 + for slot in range(len(result)): + left = start + slot * seconds + right = left + seconds + position = max(0, int(np.searchsorted(stamps, left, side="right")) - 1) + covered = 0.0 + weighted = 0.0 + while position < len(stamps) and stamps[position] < right: + source_end = stamps[position] + cadence + if position + 1 < len(stamps): + source_end = min(source_end, stamps[position + 1]) + overlap = max(0.0, min(right, source_end) - max(left, stamps[position])) + if overlap and np.isfinite(values[position]): + covered += overlap + weighted += (overlap / seconds) * values[position] + position += 1 + if abs(covered - seconds) < 1e-6: + result[slot] = weighted + return result diff --git a/src/akkudoktoreos/optimization/genetic/genetic.py b/src/akkudoktoreos/optimization/genetic/genetic.py index ee2d3b3f..398f0c3c 100644 --- a/src/akkudoktoreos/optimization/genetic/genetic.py +++ b/src/akkudoktoreos/optimization/genetic/genetic.py @@ -1,16 +1,21 @@ """Genetic algorithm.""" +import math import random import time +from collections import defaultdict +from dataclasses import dataclass, field +from functools import lru_cache from typing import Any, Optional import numpy as np -from deap import algorithms, base, creator, tools +from deap import base, creator, tools from loguru import logger from numpydantic import NDArray, Shape from pydantic import ConfigDict, Field from akkudoktoreos.core.pydantic import PydanticBaseModel +from akkudoktoreos.devices.devicesabc import ConsumerScheduleMode from akkudoktoreos.devices.genetic.battery import Battery from akkudoktoreos.devices.genetic.homeappliance import HomeAppliance from akkudoktoreos.devices.genetic.inverter import Inverter @@ -22,7 +27,97 @@ from akkudoktoreos.optimization.genetic.geneticsolution import ( GeneticSimulationResult, GeneticSolution, ) +from akkudoktoreos.optimization.genetic.tailvalue import ( + TailValueCurve, + build_tail_value_curve, +) +from akkudoktoreos.optimization.genetic.terminalvalue import ( + TailDiagnostics, + TerminalValueCurve, + TerminalValueResult, + build_terminal_value_curve, + trailing_window, +) from akkudoktoreos.optimization.optimizationabc import OptimizationBase +from akkudoktoreos.utils.datetimeutil import DateTime + + +@dataclass +class ApplianceGeneSlot: + """One appliance start gene in the genome. + + The gene value is an **index into ``allowed_start_slots``**, not an absolute + slot. This guarantees every gene value maps to a genuinely valid start and + keeps all allowed starts equally reachable by mutation/crossover. + """ + + gene_index: int + appliance_index: int + device_id: str + run_index: int + # Local calendar date of the run for DAILY appliances; None for ONCE. + run_date: Optional[Any] + allowed_start_slots: list[int] + cycle_index: int = 0 + deadline_relaxed: bool = False + + +@dataclass +class ApplianceGeneLayout: + """Ordered descriptor of the appliance part of the genome. + + Every genome-building step (create/split/merge/mutate/decode) consumes only + this descriptor, so the appliance gene block can vary in length with the + number of devices and DAILY run days without any hard-coded gene positions. + """ + + genes: list[ApplianceGeneSlot] = field(default_factory=list) + + @property + def n_genes(self) -> int: + """Number of appliance start genes.""" + return len(self.genes) + + def signature(self) -> tuple: + """Stable identity of the layout for start-solution compatibility. + + Two layouts with the same length can still describe different schedules; + the signature captures device, run date and the allowed-start list so a + cached start solution built for a different layout is not silently + reused. + """ + return tuple( + (gene.device_id, str(gene.run_date), gene.cycle_index, tuple(gene.allowed_start_slots)) + for gene in self.genes + ) + + +@dataclass(frozen=True) +class FitnessCacheEntry: + """One canonical, successful fitness evaluation within an optimization run.""" + + genome: tuple[int, ...] + fitness: tuple[float] + extra_data: tuple[float, float, float] + + +@dataclass(frozen=True) +class BatteryStateLayout: + """Indices of optional battery states appended to the legacy state ranges. + + With graded direct-marketing export there is one state per configured export + rate. ``grid_export_states`` holds them in the order of + ``bat_possible_grid_export_values`` (full power first), and + ``grid_export_state`` is that full-power state - the one every seeding + heuristic uses when it wants "export in this slot". + """ + + total_states: int + dc_not_allowed_state: Optional[int] = None + dc_allowed_state: Optional[int] = None + grid_export_state: Optional[int] = None + self_consumption_state: Optional[int] = None + grid_export_states: tuple[int, ...] = () class GeneticSimulation(PydanticBaseModel): @@ -67,20 +162,26 @@ class GeneticSimulation(PydanticBaseModel): elect_price_hourly: Optional[NDArray[Shape["*"], float]] = Field( default=None, json_schema_extra={ - "description": "An array of floats representing the electricity price per watt-hour for different time intervals." + "description": "An array of floats representing the electricity price in euros per watt-hour for different time intervals." }, ) elect_revenue_per_hour_arr: Optional[NDArray[Shape["*"], float]] = Field( default=None, json_schema_extra={ - "description": "An array of floats representing the feed-in compensation per watt-hour." + "description": "An array of floats representing the feed-in compensation in euros per watt-hour." + }, + ) + direct_marketing_enabled: bool = Field( + default=False, + json_schema_extra={ + "description": "Use direct marketing behavior for feed-in/export decisions." }, ) - battery: Optional[Battery] = Field(default=None, json_schema_extra={"description": "TBD."}) ev: Optional[Battery] = Field(default=None, json_schema_extra={"description": "TBD."}) - home_appliance: Optional[HomeAppliance] = Field( - default=None, json_schema_extra={"description": "TBD."} + home_appliances: list[HomeAppliance] = Field( + default_factory=list, + json_schema_extra={"description": "Flexible consumers scheduled by the optimizer."}, ) inverter: Optional[Inverter] = Field(default=None, json_schema_extra={"description": "TBD."}) @@ -93,16 +194,27 @@ class GeneticSimulation(PydanticBaseModel): bat_discharge_hours: Optional[NDArray[Shape["*"], float]] = Field( default=None, json_schema_extra={"description": "TBD"} ) + bat_grid_export_hours: Optional[NDArray[Shape["*"], float]] = Field( + default=None, + json_schema_extra={"description": "Hourly permission for battery discharge into the grid."}, + ) ev_charge_hours: Optional[NDArray[Shape["*"], float]] = Field( default=None, json_schema_extra={"description": "TBD"} ) ev_discharge_hours: Optional[NDArray[Shape["*"], float]] = Field( default=None, json_schema_extra={"description": "TBD"} ) - home_appliance_start_hour: Optional[int] = Field( - default=None, - json_schema_extra={"description": "Home appliance start hour - None denotes no start."}, - ) + + home_appliance_start_hour: Optional[int] = Field(default=None) + + @property + def home_appliance(self) -> Optional[HomeAppliance]: + """Deprecated singular device accessor.""" + return self.home_appliances[0] if self.home_appliances else None + + @home_appliance.setter + def home_appliance(self, appliance: Optional[HomeAppliance]) -> None: + self.home_appliances = [appliance] if appliance is not None else [] def prepare( self, @@ -112,20 +224,24 @@ class GeneticSimulation(PydanticBaseModel): ev: Optional[Battery] = None, home_appliance: Optional[HomeAppliance] = None, inverter: Optional[Inverter] = None, + direct_marketing_enabled: bool = False, + home_appliances: Optional[list[HomeAppliance]] = None, ) -> None: """Prepare simulation runs. Populate internal arrays and device references used during simulation. """ + self.home_appliance_start_hour = None self.optimization_hours = optimization_hours self.prediction_hours = prediction_hours + self.direct_marketing_enabled = direct_marketing_enabled # Load arrays from provided EMS parameters self.load_energy_array = np.array(parameters.total_load, float) self.pv_prediction_wh = np.array(parameters.pv_forecast_wh, float) self.elect_price_hourly = np.array(parameters.electricity_price_per_wh, float) self.elect_revenue_per_hour_arr = ( - np.asarray(parameters.feed_in_tariff_per_wh, dtype=float) + np.array(parameters.feed_in_tariff_per_wh, float) if isinstance(parameters.feed_in_tariff_per_wh, list) else np.full(len(self.load_energy_array), parameters.feed_in_tariff_per_wh, float) ) @@ -136,30 +252,39 @@ class GeneticSimulation(PydanticBaseModel): else: self.battery = None self.ev = ev - self.home_appliance = home_appliance + if home_appliance is not None and home_appliances is not None: + raise ValueError("Use home_appliance or home_appliances, not both.") + self.home_appliances = home_appliances or ( + [home_appliance] if home_appliance is not None else [] + ) self.inverter = inverter # Initialize per-hour action arrays for the prediction horizon self.ac_charge_hours = np.full(self.prediction_hours, 0.0) self.dc_charge_hours = np.full(self.prediction_hours, 0.0) self.bat_discharge_hours = np.full(self.prediction_hours, 0.0) + self.bat_grid_export_hours = np.full(self.prediction_hours, 0.0) self.ev_charge_hours = np.full(self.prediction_hours, 0.0) self.ev_discharge_hours = np.full(self.prediction_hours, 0.0) - self.home_appliance_start_hour = None def reset(self) -> None: + self.home_appliance_start_hour = None if self.ev: self.ev.reset() if self.battery: self.battery.reset() - self.home_appliance_start_hour = None def simulate(self, start_hour: int) -> dict[str, Any]: """Simulate energy usage and costs for the given start hour. - battery_soc_per_hour begin of the hour, initial hour state! - load_wh_per_hour integral of last hour (end state) + akku_soc_pro_stunde begin of the hour, initial hour state! + last_wh_pro_stunde integral of last hour (end state) """ + # Preserve the singular hourly simulator API. Native schedules are built beforehand. + if self.home_appliance is not None and self.home_appliance_start_hour is not None: + self.home_appliance_start_hour = self.home_appliance.set_starting_time( + self.home_appliance_start_hour, start_hour + ) # Remember start hour self.start_hour = start_hour @@ -170,13 +295,15 @@ class GeneticSimulation(PydanticBaseModel): ac_charge_hours_fast = self.ac_charge_hours dc_charge_hours_fast = self.dc_charge_hours bat_discharge_hours_fast = self.bat_discharge_hours + bat_grid_export_hours_fast = self.bat_grid_export_hours elect_price_hourly_fast = self.elect_price_hourly elect_revenue_per_hour_arr_fast = self.elect_revenue_per_hour_arr pv_prediction_wh_fast = self.pv_prediction_wh battery_fast = self.battery ev_fast = self.ev - home_appliance_fast = self.home_appliance + home_appliances_fast = self.home_appliances inverter_fast = self.inverter + direct_marketing_enabled_fast = self.direct_marketing_enabled # Check for simulation integrity (in a way that mypy understands) if ( @@ -188,6 +315,7 @@ class GeneticSimulation(PydanticBaseModel): or dc_charge_hours_fast is None or elect_revenue_per_hour_arr_fast is None or bat_discharge_hours_fast is None + or bat_grid_export_hours_fast is None or ev_discharge_hours_fast is None ): missing = [] @@ -207,6 +335,8 @@ class GeneticSimulation(PydanticBaseModel): missing.append("Electricity Revenue Per Hour") if bat_discharge_hours_fast is None: missing.append("Battery Discharge Hours") + if bat_grid_export_hours_fast is None: + missing.append("Battery Grid Export Hours") if ev_discharge_hours_fast is None: missing.append("EV Discharge Hours") msg = ", ".join(missing) @@ -222,7 +352,9 @@ class GeneticSimulation(PydanticBaseModel): logger.error(error_msg) raise ValueError(error_msg) - end_hour = len(load_energy_array_fast) + end_hour = min( + len(load_energy_array_fast), self.prediction_hours or len(load_energy_array_fast) + ) total_hours = end_hour - start_hour # Pre-allocate arrays for the results, optimized for speed @@ -233,6 +365,7 @@ class GeneticSimulation(PydanticBaseModel): revenue_per_hour = np.full((total_hours), np.nan) losses_wh_per_hour = np.full((total_hours), np.nan) electricity_price_per_hour = np.full((total_hours), np.nan) + feed_in_tariff_per_hour = np.full((total_hours), np.nan) # Set initial state if battery_fast: @@ -255,9 +388,13 @@ class GeneticSimulation(PydanticBaseModel): max_ac_charge_w_fast is None or max_ac_charge_w_fast > 0 ) - # If AC charging is disabled via inverter, zero out AC charge hours + # If AC charging is disabled via inverter, zero out AC charge hours. + # In place, not by rebinding: the reported plan is read back from + # this very array, so a rebind would leave AC charge values in the + # solution that the simulation never executed - and a controller + # acting on them would grid-charge the battery unplanned. if not ac_charging_possible: - ac_charge_hours_fast = np.zeros_like(ac_charge_hours_fast) + ac_charge_hours_fast[:] = 0.0 # Fill the charge array of the battery dc_charge_hours_fast[0:start_hour] = 0 @@ -270,7 +407,18 @@ class GeneticSimulation(PydanticBaseModel): # Fill the discharge array of the battery bat_discharge_hours_fast[0:start_hour] = 0 bat_discharge_hours_fast[end_hour:] = 0 - battery_fast.discharge_array = bat_discharge_hours_fast + bat_grid_export_hours_fast[0:start_hour] = 0 + bat_grid_export_hours_fast[end_hour:] = 0 + battery_fast.discharge_array = np.where( + (bat_discharge_hours_fast > 0) + | ( + direct_marketing_enabled_fast + & (bat_grid_export_hours_fast > 0) + & (elect_revenue_per_hour_arr_fast[: len(bat_grid_export_hours_fast)] > 0.0) + ), + 1, + 0, + ) else: # Default return if no battery is available soc_per_hour = np.full((total_hours), 0) @@ -296,14 +444,12 @@ class GeneticSimulation(PydanticBaseModel): # Default return if no electric vehicle is available soc_ev_per_hour = np.full((total_hours), 0) - if home_appliance_fast and self.home_appliance_start_hour is not None: + if home_appliances_fast: home_appliance_enabled = True - # Pre-allocate arrays for the results, optimized for speed + # Pre-allocate the aggregate appliance load array (sum over all + # devices). Each appliance already carries its own resampled load + # curve, built from the decoded start(s) before this call. home_appliance_wh_per_hour = np.full((total_hours), np.nan) - - self.home_appliance_start_hour = home_appliance_fast.set_starting_time( - self.home_appliance_start_hour, start_hour - ) else: home_appliance_enabled = False # Default return if no home appliance is available @@ -316,9 +462,11 @@ class GeneticSimulation(PydanticBaseModel): consumption = load_energy_array_fast[hour] losses_wh_per_hour[hour_idx] = 0.0 - # Home appliances + # Home appliances (sum the per-slot load of all flexible consumers) if home_appliance_enabled: - ha_load = home_appliance_fast.get_load_for_hour(hour) # type: ignore[union-attr] + ha_load = 0.0 + for appliance in home_appliances_fast: + ha_load += appliance.get_load_for_hour(hour) consumption += ha_load home_appliance_wh_per_hour[hour_idx] = ha_load @@ -326,11 +474,13 @@ class GeneticSimulation(PydanticBaseModel): if ev_fast: soc_ev_per_hour[hour_idx] = ev_fast.current_soc_percentage() # save begin state if ev_charge_hours_fast[hour] > 0: - loaded_energy_ev, ev_charge_losses = ev_fast.charge_energy( + stored_energy_ev, verluste_eauto = ev_fast.charge_energy( wh=None, hour=hour, charge_factor=ev_charge_hours_fast[hour] ) - consumption += loaded_energy_ev - losses_wh_per_hour[hour_idx] += ev_charge_losses + # The inverter/grid must supply the EV charger's raw input, + # not only the energy stored after charging losses. + consumption += stored_energy_ev + verluste_eauto + losses_wh_per_hour[hour_idx] += verluste_eauto # Save battery SOC before inverter processing = true begin-of-interval state. # Must be recorded here (before DC charge/discharge) so the displayed SOC at @@ -340,18 +490,36 @@ class GeneticSimulation(PydanticBaseModel): soc_per_hour[hour_idx] = battery_fast.current_soc_percentage() # Process inverter logic - energy_feedin_grid_actual = energy_consumption_grid_actual = losses = ( - self_consumption - ) = 0.0 + energy_feedin_grid_actual = energy_consumption_grid_actual = losses = eigenverbrauch = ( + 0.0 + ) if inverter_fast: energy_produced = pv_prediction_wh_fast[hour] + hourly_feed_in_tariff = elect_revenue_per_hour_arr_fast[hour] + # bat_grid_export_hours carries the export level per slot: + # 0.0 = no export, otherwise the factor of the rated discharge + # power the optimizer selected. + battery_grid_export_factor = float(bat_grid_export_hours_fast[hour]) + battery_grid_export_allowed = ( + direct_marketing_enabled_fast + and hourly_feed_in_tariff > 0.0 + and battery_grid_export_factor > 0.0 + ) ( energy_feedin_grid_actual, energy_consumption_grid_actual, losses, - self_consumption, - ) = inverter_fast.process_energy(energy_produced, consumption, hour) + eigenverbrauch, + ) = inverter_fast.process_energy( + energy_produced, + consumption, + hour, + allow_battery_grid_export=battery_grid_export_allowed, + battery_grid_export_factor=battery_grid_export_factor, + ) + else: + hourly_feed_in_tariff = elect_revenue_per_hour_arr_fast[hour] # AC PV Battery Charge if battery_fast: @@ -389,18 +557,36 @@ class GeneticSimulation(PydanticBaseModel): ) # Update hourly arrays + if ( + direct_marketing_enabled_fast + and hourly_feed_in_tariff < 0.0 + and energy_feedin_grid_actual > 0.0 + ): + losses_wh_per_hour[hour_idx] += energy_feedin_grid_actual + energy_feedin_grid_actual = 0.0 + feedin_energy_per_hour[hour_idx] = energy_feedin_grid_actual consumption_energy_per_hour[hour_idx] = energy_consumption_grid_actual losses_wh_per_hour[hour_idx] += losses loads_energy_per_hour[hour_idx] = consumption hourly_electricity_price = elect_price_hourly_fast[hour] electricity_price_per_hour[hour_idx] = hourly_electricity_price + feed_in_tariff_per_hour[hour_idx] = hourly_feed_in_tariff # Financial calculations - costs_per_hour[hour_idx] = energy_consumption_grid_actual * hourly_electricity_price - revenue_per_hour[hour_idx] = ( - energy_feedin_grid_actual * elect_revenue_per_hour_arr_fast[hour] - ) + grid_cost = energy_consumption_grid_actual * hourly_electricity_price + # LCOS is charged exactly once on battery-delivered DC energy. It is + # not charged on input energy, internal discharge losses, or the + # downstream DC-to-AC inverter loss. + battery_lcos_cost = 0.0 + if battery_fast: + battery_lcos_cost = ( + battery_fast.discharged_energy_wh(hour) + * battery_fast.levelized_cost_of_storage_kwh + / 1000.0 + ) + costs_per_hour[hour_idx] = grid_cost + battery_lcos_cost + revenue_per_hour[hour_idx] = energy_feedin_grid_actual * hourly_feed_in_tariff total_cost = np.nansum(costs_per_hour) total_losses = np.nansum(losses_wh_per_hour) @@ -422,30 +608,134 @@ class GeneticSimulation(PydanticBaseModel): "Gesamt_Verluste": total_losses, "Home_appliance_wh_per_hour": home_appliance_wh_per_hour, "Electricity_price": electricity_price_per_hour, + "Feed_in_tariff": feed_in_tariff_per_hour, } class GeneticOptimization(OptimizationBase): """GENETIC algorithm to solve energy optimization.""" + WARM_START_COPIES = 10 + WARM_START_MUTATIONS = 50 + EDUCATED_GUESS_TARGET = 100 + MIN_RANDOM_POPULATION_FRACTION = 0.25 + WARM_START_COPY_FRACTION = 0.10 + WARM_START_MUTATION_FRACTION = 0.20 + EDUCATED_GUESS_FRACTION = 0.40 + LOCAL_SEARCH_MAX_EVALUATIONS = 96 + LOCAL_SEARCH_MAX_PASSES = 4 + EDUCATED_GUESS_EXPORT_QUANTILES = (0.60, 0.75, 0.90) + CROSSOVER_PROBABILITY = 0.50 + MUTATION_PROBABILITY = 0.55 + STAGNATION_MUTATION_PROBABILITY = 0.80 + STAGNATION_GENERATIONS = 8 + SOFT_RESTART_GENERATIONS = 20 + # The selection keeps SELECTION_DIVERSITY_FLOOR of the population unique, so a + # boost threshold at or above that floor would fire in every converged + # generation and make the boost the normal operating state instead of an + # intervention. Keep it strictly below the floor. + SELECTION_DIVERSITY_FLOOR = 0.30 + DIVERSITY_BOOST_THRESHOLD = 0.25 + SOFT_RESTART_DIVERSITY_THRESHOLD = 0.10 + IMMIGRANT_FRACTION = 0.12 + # Fresh immigrants are the worst individuals in the pool, so a plain + # tournament removes them in the generation they are born and their genes + # never get a chance to recombine. Keep a bounded number of them for a few + # selections so a boost can actually explore. + IMMIGRANT_PROTECTION_GENERATIONS = 2 + IMMIGRANT_PROTECTION_FRACTION = 0.25 + SOFT_RESTART_SURVIVOR_FRACTION = 0.20 + POINT_MUTATION_EXPECTED_GENES = 3.0 + + # Independent forecast and control durations on the optimization grid. + @property + def slot_duration_h(self) -> float: + """Length of one optimization slot in hours (1.0 hourly, 0.25 at 15 min).""" + interval = self.config.optimization.genetic.interval_sec or 3600 + return interval / 3600 + + @property + def slots_per_hour(self) -> int: + """Number of optimization slots per hour (1 hourly, 4 at 15 min).""" + interval = self.config.optimization.genetic.interval_sec or 3600 + return 3600 // interval + + @property + def control_slots(self) -> int: + """Number of executable control intervals, measured from now.""" + return self.config.optimization.genetic.horizon_hours * self.slots_per_hour + + @property + def prediction_slots(self) -> int: + """Forecast duration, independent of the control genome.""" + return int(self.config.prediction.hours * self.slots_per_hour) + + @property + def tail_slots(self) -> int: + """Requested lookahead, bounded by the forecast the configuration budgets. + + A prediction horizon that does not cover control plus tail shortens the + tail rather than failing the run, so the shortfall is not reported as + missing provider data. + """ + requested = self.config.optimization.genetic.tail_horizon_hours * self.slots_per_hour + budget = max(0, self.prediction_slots - self.control_slots) + return min(requested, budget) + + @property + def control_end_slot(self) -> int: + """Exclusive control end in run-relative device arrays.""" + return self._control_start_slot() + self.control_slots + + def _control_start_slot(self) -> int: + """Genomes and device arrays start at now, independently of wall-clock hour.""" + return 0 + + def _start_day_slot(self) -> int: + """Offset used only to trim legacy midnight-indexed forecast inputs.""" + sd = self.ems.start_datetime + midnight = sd.set(hour=0, minute=0, second=0, microsecond=0) + return int((sd - midnight).total_seconds() // (self.slot_duration_h * 3600)) + def __init__( self, verbose: bool = False, fixed_seed: Optional[int] = None, ): """Initialize the optimization problem with the required parameters.""" + if self.config.optimization.genetic.interval_sec not in (900, 3600): + logger.warning( + "Genetic optimization interval {} seconds is unsupported; using 3600 seconds.", + self.config.optimization.genetic.interval_sec, + ) + self.config.optimization.genetic.interval_sec = 3600 self.opti_param: dict[str, Any] = {} - self.fixed_ev_hours = ( - self.config.prediction.hours - self.config.optimization.genetic.horizon_hours - ) + # EV genes cover precisely the control horizon; no fixed prediction tail. + self.fixed_eauto_hours = 0 self.ev_possible_charge_values: list[float] = [1.0] # Separate charge-level list for battery AC charging (independent of EV rates). # Populated from parameters.pv_battery.charge_rates in optimize_ems. self.bat_possible_charge_values: list[float] = [1.0] + # Battery-to-grid export levels (direct marketing), full power first. + # Populated from parameters.pv_battery.grid_export_rates in optimize_ems; + # the single full-power default keeps the all-or-nothing export. + self.bat_possible_grid_export_values: list[float] = [1.0] + # Slot by which the EV has to reach its target SoC. None means the SoC is + # only required at the end of the horizon (the behaviour without a deadline). + self._ev_soc_deadline_slot: Optional[int] = None + # Value of the energy left in the battery at the end of the + # horizon. None means the fixed scalar terminal value is used instead. + self._terminal_value_curve: Optional[TerminalValueCurve] = None + self._continuation_value_curve: Optional[TerminalValueCurve] = None + self._tail_diagnostics: Optional[TailDiagnostics] = None + # Why that is - reported with the solution, because a run that silently + # falls back to the scalar looks exactly like a run configured for it. + self._terminal_value_reason: str = "" self.verbose = verbose self.fix_seed = fixed_seed self.optimize_ev = True self.optimize_dc_charge = False + self.optimize_battery_grid_export = False self.fitness_history: dict[str, Any] = {} # Set a fixed seed for random operations if provided or in debug mode @@ -455,13 +745,844 @@ class GeneticOptimization(OptimizationBase): self.fix_seed = random.randint(1, 100000000000) # noqa: S311 random.seed(self.fix_seed) + # Per-run cache for the AC-charge break-even penalty (see evaluate()). + self._ac_break_even_best_prices: Optional[list[float]] = None + + # Fitness memoization is activated only around optimize(). The cache is + # never shared across runs because forecasts, prices and device state may + # have changed even when the genome is identical. + self._fitness_cache_enabled = False + self._fitness_cache: dict[tuple[int, ...], FitnessCacheEntry] = {} + self._fitness_cache_hits = 0 + self._fitness_cache_misses = 0 + + # Appliance genome layout, built once per optimization run in + # optimize_ems(). Empty by default so setup_deap_environment() can be + # exercised standalone (e.g. in tests) without appliances. + self.appliance_layout: ApplianceGeneLayout = ApplianceGeneLayout([]) + # Local datetime of slot index 0 (the run start), needed to + # turn decoded start slots into absolute local timestamps. + self._slot0_datetime: Optional[Any] = None + # Create Simulation self.simulation = GeneticSimulation() + def _direct_marketing_enabled(self) -> bool: + """Return whether direct marketing mode is enabled in configuration.""" + try: + return bool(self.config.feedintariff.direct_marketing_enabled) + except Exception: + return False + + def _battery_state_layout(self) -> BatteryStateLayout: + """Build optional state indices without renumbering legacy warm starts. + + The pre-existing order is retained exactly: base charge/discharge ranges, + two optional DC states, then optional grid export. SELF_CONSUMPTION is + appended last so an old export gene never changes its meaning. + """ + next_state = 3 * len(self.bat_possible_charge_values) + dc_not_allowed_state: Optional[int] = None + dc_allowed_state: Optional[int] = None + grid_export_state: Optional[int] = None + self_consumption_state: Optional[int] = None + + if self.optimize_dc_charge: + dc_not_allowed_state = next_state + dc_allowed_state = next_state + 1 + next_state += 2 + + grid_export_states: tuple[int, ...] = () + if self.optimize_battery_grid_export: + export_count = max(len(self.bat_possible_grid_export_values), 1) + grid_export_states = tuple(range(next_state, next_state + export_count)) + # The first export state stays the full-power one, so its index does + # not move when further rates are configured. + grid_export_state = grid_export_states[0] + next_state += export_count + + if self.optimize_dc_charge: + self_consumption_state = next_state + next_state += 1 + + return BatteryStateLayout( + total_states=next_state, + dc_not_allowed_state=dc_not_allowed_state, + dc_allowed_state=dc_allowed_state, + grid_export_state=grid_export_state, + self_consumption_state=self_consumption_state, + grid_export_states=grid_export_states, + ) + + def _appliance_horizon_end_slot(self) -> int: + """Exclusive upper slot bound for appliance runs (end of horizon). + + A run must complete within the optimization horizon. The horizon starts + at the current slot and lasts ``horizon_hours``; the bound is capped to + the total slot grid. + """ + start_slot = self._control_start_slot() + horizon_slots = self.config.optimization.genetic.horizon_hours * self.slots_per_hour + return min(self.control_end_slot, start_slot + horizon_slots) + + def _ev_deadline_slot(self, parameters: GeneticOptimizationParameters) -> Optional[int]: + """Slot index by which the EV has to reach ``min_soc_percentage``. + + The deadline may be given as an absolute datetime, as a maximum duration + from the start of the optimization, or both - then the earlier one wins. + The returned slot excludes charging intervals that finish after the + deadline; a departure inside a slot cannot credit that whole slot. + + Args: + parameters: Optimization parameters of this run. + + Returns: + Absolute slot index, or None when the target is only required at the + end of the horizon (no deadline, or one beyond the horizon). + """ + ev_parameters = parameters.ev + if ev_parameters is None: + return None + + start_slot = self._control_start_slot() + slot_seconds = self.slot_duration_h * 3600 + candidates: list[int] = [] + + deadline = ev_parameters.min_soc_deadline_datetime + if deadline is not None: + if self._slot0_datetime is None: + raise ValueError("EV deadline requires a run start timestamp.") + seconds = ( + deadline.in_timezone(self._slot0_datetime.timezone) - self._slot0_datetime + ).total_seconds() + candidates.append(math.floor(seconds / slot_seconds + 1e-9)) + + duration_h = ev_parameters.min_soc_max_duration_h + if duration_h is not None: + candidates.append(start_slot + math.floor(duration_h * 3600 / slot_seconds + 1e-9)) + + if not candidates: + return None + + deadline_slot = min(candidates) + if deadline_slot >= self.control_end_slot: + # Beyond the horizon: the end-of-horizon requirement already covers it. + return None + # A deadline in the past means the target is due right now. + return max(deadline_slot, start_slot) + + def _validate_forecast_availability(self) -> None: + """Use only the contiguous finite forecast prefix after now.""" + start = self._control_start_slot() + required = self.control_end_slot + requested = required + self.tail_slots + available = requested + limiting = [] + for name, values in ( + ("load", self.simulation.load_energy_array), + ("pv", self.simulation.pv_prediction_wh), + ("import price", self.simulation.elect_price_hourly), + ("feed-in tariff", self.simulation.elect_revenue_per_hour_arr), + ): + end = min(len(values), requested) if values is not None else 0 + if values is not None: + missing = np.flatnonzero(~np.isfinite(values[start:end])) + if missing.size: + end = start + int(missing[0]) + if end < required: + raise ValueError( + f"Incomplete control forecast: {name} ends at slot {end}; control requires slot {required}." + ) + if end < requested: + limiting.append(name) + available = min(available, end) + self._effective_tail_slots = max(0, available - required) + self._forecast_reason = "" + if available < requested: + self._forecast_reason = ( + f"Tail forecast shortened: requested {self.config.optimization.genetic.tail_horizon_hours} h, " + f"effective {self._effective_tail_slots * self.slot_duration_h:g} h; " + f"limited by {', '.join(limiting)}. Continuation starts at slot {available}." + ) + logger.warning(self._forecast_reason) + + def _build_terminal_value_curve( + self, + battery: Optional[Battery], + inverter: Optional[Inverter], + ) -> Optional[TerminalValueCurve]: + """Build continuation at the effective tail end, then solve the tail. + + Only built in AUTO mode and only with a battery: the curve describes + what the energy left in that battery is worth once the horizon ends. + + Args: + battery: The house battery of this run, if any. + inverter: The inverter, needed for the DC/AC conversion. + + Returns: + The curve, or None when the fixed scalar terminal value applies. + """ + if battery is None: + self._terminal_value_reason = "no battery in this optimization" + return None + try: + mode = self.config.optimization.genetic.terminal_value_mode + window_hours = self.config.optimization.genetic.terminal_value_window_hours + except Exception: + self._terminal_value_reason = "terminal value configuration unavailable" + return None + if str(mode) != "AUTO": + self._terminal_value_reason = "terminal_value_mode is FIXED" + return None + + dc_to_ac = inverter.dc_to_ac_efficiency if inverter else 1.0 + # A full battery, expressed in the same unit as the curve: AC energy + # that can actually leave the house. + max_energy_wh = ( + max(battery.max_soc_wh - battery.min_soc_wh, 0.0) + * battery.discharging_efficiency + * dc_to_ac + ) + window_slots = max(int(window_hours) * self.slots_per_hour, 1) + end_slot = self.control_end_slot + getattr(self, "_effective_tail_slots", 0) + + curve = build_terminal_value_curve( + prices_euro_per_wh=trailing_window( + self.simulation.elect_price_hourly, end_slot, window_slots + ), + load_wh=trailing_window(self.simulation.load_energy_array, end_slot, window_slots), + pv_wh=trailing_window(self.simulation.pv_prediction_wh, end_slot, window_slots), + feed_in_euro_per_wh=trailing_window( + self.simulation.elect_revenue_per_hour_arr, end_slot, window_slots + ), + max_energy_wh=max_energy_wh, + lcos_euro_per_kwh=getattr(battery, "levelized_cost_of_storage_kwh", 0.0), + dc_to_ac_efficiency=dc_to_ac, + grid_export_allowed=self.optimize_battery_grid_export, + ) + self._continuation_value_curve = curve + self._tail_diagnostics = None + if self.tail_slots and inverter is not None: + tail = slice(self.control_end_slot, end_slot) + prices = self.simulation.elect_price_hourly + loads = self.simulation.load_energy_array + pv = self.simulation.pv_prediction_wh + tariffs = self.simulation.elect_revenue_per_hour_arr + if prices is None or loads is None or pv is None or tariffs is None: + raise ValueError("Tail evaluation requires prepared forecasts") + tail_prices = prices[tail] + tail_tariffs = tariffs[tail] + self._tail_diagnostics = TailDiagnostics( + slots=len(tail_prices), + slot_hours=self.slot_duration_h, + soc_grid_points=101, + min_import_price_euro_per_kwh=( + float(np.min(tail_prices)) * 1000 if len(tail_prices) else 0.0 + ), + max_import_price_euro_per_kwh=( + float(np.max(tail_prices)) * 1000 if len(tail_prices) else 0.0 + ), + min_feed_in_tariff_euro_per_kwh=( + float(np.min(tail_tariffs)) * 1000 if len(tail_tariffs) else 0.0 + ), + max_feed_in_tariff_euro_per_kwh=( + float(np.max(tail_tariffs)) * 1000 if len(tail_tariffs) else 0.0 + ), + negative_import_price_slots=int(np.count_nonzero(tail_prices < 0.0)), + positive_battery_export_slots=( + int(np.count_nonzero(tail_tariffs > 0.0)) + if self.optimize_battery_grid_export + else 0 + ), + ) + return build_tail_value_curve( + battery=battery, + inverter=inverter, + prices_euro_per_wh=prices[tail], + load_wh=loads[tail], + pv_wh=pv[tail], + feed_in_euro_per_wh=tariffs[tail], + continuation=curve, + charge_rates=self.bat_possible_charge_values, + export_rates=self.bat_possible_grid_export_values, + direct_marketing=self.optimize_battery_grid_export, + ) + if curve.energy_wh: + self._terminal_value_reason = "" + logger.debug( + "Terminal value curve: {} segments, first {:.3f} EUR/kWh, last {:.3f} EUR/kWh, " + "knee at {:.0f} Wh.", + len(curve.marginal_euro_per_kwh), + curve.marginal_euro_per_kwh[0], + curve.marginal_euro_per_kwh[-1], + curve.energy_wh[-1], + ) + else: + # Almost always an input problem: an all-zero price forecast, or a + # window whose load is fully covered by PV. Falling back to the + # scalar is quiet, so say it out loud. + self._terminal_value_reason = ( + "AUTO could not derive a curve: the last " + f"{window_slots} slots of the horizon carry no priced residual load " + "(check the electricity price forecast) - falling back to the fixed value" + ) + logger.warning(self._terminal_value_reason) + return curve + + def _terminal_value( + self, + parameters: GeneticOptimizationParameters, + *, + include_tail_plan: bool = False, + ) -> tuple[float, TerminalValueResult]: + """Credit for the energy left in the battery, plus its report. + + Args: + parameters: Optimization parameters, holding the fixed scalar value. + + Returns: + The credit in EUR and the result object for the solution. + """ + diagnostics = dict( + control_horizon_hours=self.config.optimization.genetic.horizon_hours, + requested_tail_hours=self.config.optimization.genetic.tail_horizon_hours, + effective_tail_hours=0.0, + tail_end_hour=float(self.config.optimization.genetic.horizon_hours), + ) + battery = self.simulation.battery + if battery is None: + return 0.0, TerminalValueResult( + mode="FIXED", reason="no battery in this optimization", **diagnostics + ) + + # Usable DC energy, converted to the AC energy that can serve a load. + energy_wh = battery.current_energy_content() + if self.simulation.inverter: + energy_wh *= self.simulation.inverter.dc_to_ac_efficiency + + curve = getattr(self, "_terminal_value_curve", None) + if curve is not None and curve.energy_wh: + credit = curve.value(energy_wh) + if isinstance(curve, TailValueCurve): + tail_operating_euro, continuation_value_euro = curve.component_values(energy_wh) + tail_plan = ( + curve.diagnostic_plan( + energy_wh, float(self.config.optimization.genetic.horizon_hours) + ) + if include_tail_plan + else [] + ) + else: + tail_operating_euro, continuation_value_euro = 0.0, credit + tail_plan = [] + return credit, TerminalValueResult( + mode="TAIL" if isinstance(curve, TailValueCurve) else "AUTO", + control_horizon_hours=self.config.optimization.genetic.horizon_hours, + requested_tail_hours=self.config.optimization.genetic.tail_horizon_hours, + effective_tail_hours=getattr(self, "_effective_tail_slots", 0) + * self.slot_duration_h, + tail_end_hour=(self.control_end_slot + getattr(self, "_effective_tail_slots", 0)) + * self.slot_duration_h, + continuation_mode="AUTO", + reason=getattr(self, "_forecast_reason", ""), + battery_energy_wh=energy_wh, + credited_euro=credit, + tail_operating_euro=tail_operating_euro, + continuation_value_euro=continuation_value_euro, + curve=curve, + continuation_curve=getattr(self, "_continuation_value_curve", None), + tail_diagnostics=getattr(self, "_tail_diagnostics", None), + tail_plan=tail_plan, + ) + + credit = energy_wh * parameters.ems.price_per_wh_battery + return credit, TerminalValueResult( + mode="FIXED", + battery_energy_wh=energy_wh, + credited_euro=credit, + continuation_value_euro=credit, + reason=" ".join( + filter( + None, + [ + getattr(self, "_terminal_value_reason", "") + or "terminal_value_mode is FIXED", + getattr(self, "_forecast_reason", ""), + ], + ) + ), + **diagnostics, + ) + + def _build_appliance_layout( + self, appliances: list[HomeAppliance], slot0_datetime: Any + ) -> ApplianceGeneLayout: + """Keep each cycle's windows, completed cycles and local-day identity.""" + self._appliance_devices = appliances + self._appliance_order_cache: dict[int, list[int]] = {} + start_slot = self._control_start_slot() + horizon_end_slot = self._appliance_horizon_end_slot() + genes: list[ApplianceGeneSlot] = [] + first_date = slot0_datetime.date() + for appliance_index, appliance in enumerate(appliances): + cycles = range(appliance.num_cycles) + allowed_by_cycle: dict[int, list[int]] = {} + relaxed_by_cycle: dict[int, bool] = {} + for cycle in cycles: + allowed_by_cycle[cycle] = appliance.allowed_start_slots( + slot0_datetime=slot0_datetime, + earliest_slot=start_slot, + horizon_end_slot=horizon_end_slot, + cycle_index=cycle, + ) + relaxed_by_cycle[cycle] = appliance.deadline_relaxed + if appliance.schedule_mode == ConsumerScheduleMode.ONCE: + groups = [ + (None, cycle, allowed_by_cycle[cycle]) + for cycle in cycles + if cycle >= appliance.completed_cycles + ] + else: + by_date: dict[Any, dict[int, list[int]]] = {} + for cycle, allowed in allowed_by_cycle.items(): + for slot in allowed: + date = slot0_datetime.add( + seconds=slot * appliance.slot_interval_seconds + ).date() + if date == first_date and cycle < appliance.completed_cycles: + continue + by_date.setdefault(date, {}).setdefault(cycle, []).append(slot) + groups = [ + (date, cycle, allowed) + for date, cycle_slots in sorted(by_date.items()) + for cycle, allowed in sorted(cycle_slots.items()) + ] + # A partially elapsed first day may have no feasible remaining run; + # later complete planning days must retain all configured cycles. + for date, cycle_slots in by_date.items(): + expected = { + cycle + for cycle in cycles + if date != first_date or cycle >= appliance.completed_cycles + } + if set(cycle_slots) != expected: + raise ValueError( + f"Home appliance '{appliance.device_id}' has no complete cycle schedule on {date}." + ) + for run_index, (date, cycle, allowed) in enumerate(groups): + if not allowed: + raise ValueError( + f"Home appliance '{appliance.device_id}' cycle {cycle} has no valid start within its windows, deadline and control horizon." + ) + genes.append( + ApplianceGeneSlot( + gene_index=len(genes), + appliance_index=appliance_index, + device_id=appliance.device_id, + run_index=run_index, + run_date=date, + allowed_start_slots=allowed, + cycle_index=cycle, + deadline_relaxed=relaxed_by_cycle[cycle], + ) + ) + layout = ApplianceGeneLayout(genes) + # Detect impossible cross-cycle gaps before running a population search. + previous_layout = self.appliance_layout + self.appliance_layout = layout + try: + self._decode_appliance_starts([0] * layout.n_genes) + finally: + self.appliance_layout = previous_layout + return layout + + def _decode_appliance_starts(self, appliance_gene_values: list[int]) -> dict[int, list[int]]: + """Repair requested starts jointly, preserving cycle windows and minimum gaps. + + Backward latest-feasible bounds and a forward nearest-choice pass prevent + an unlucky late first gene from making otherwise feasible cycles invalid. + The canonical gene values are updated to describe the executed schedule. + """ + starts_per_appliance: dict[int, list[int]] = defaultdict(list) + grouped: dict[int, list[tuple[int, ApplianceGeneSlot]]] = defaultdict(list) + for position, gene in enumerate(self.appliance_layout.genes): + grouped[gene.appliance_index].append((position, gene)) + for appliance_index, entries in grouped.items(): + appliances = getattr(self, "_appliance_devices", self.simulation.home_appliances) + appliance = appliances[appliance_index] + separation = appliance.run_slots + math.ceil( + appliance.min_cycle_gap_h / self.slot_duration_h + ) + by_position = dict(entries) + + def requested_start(entry: tuple[int, ApplianceGeneSlot]) -> int: + position, gene = entry + index = min( + max(int(appliance_gene_values[position]), 0), len(gene.allowed_start_slots) - 1 + ) + return gene.allowed_start_slots[0 if gene.deadline_relaxed else index] + + def latest_bounds(order: list[tuple[int, ApplianceGeneSlot]]) -> Optional[list[int]]: + bounds: list[int] = [] + limit = self.control_end_slot + for _, gene in reversed(order): + candidates = [slot for slot in gene.allowed_start_slots if slot <= limit] + if not candidates: + return None + last = candidates[-1] + bounds.append(last) + limit = last - separation + return list(reversed(bounds)) + + # Cycle numbers identify masks; they do not impose temporal order. + # Keep every feasible order represented by a candidate genome. + entries.sort(key=lambda entry: (requested_start(entry), entry[0])) + latest = latest_bounds(entries) + if latest is None: + if appliance_index not in self._appliance_order_cache: + + @lru_cache(maxsize=None) + def feasible_order( + remaining: tuple[int, ...], earliest: int + ) -> Optional[tuple[int, ...]]: + if not remaining: + return () + choices = [] + for position in remaining: + allowed = [ + slot + for slot in by_position[position].allowed_start_slots + if slot >= earliest + ] + if not allowed: + return None + choices.append((allowed[0], allowed[-1], position)) + # For a fixed order the earliest start dominates all later + # starts for feasibility. Search permutations only once + # per run, memoizing impossible remaining-cycle states. + for first, _, position in sorted(choices): + rest = feasible_order( + tuple(index for index in remaining if index != position), + first + separation, + ) + if rest is not None: + return (position, *rest) + return None + + order = feasible_order(tuple(sorted(by_position)), self._control_start_slot()) + if order is None: + raise ValueError( + f"Home appliance '{appliance.device_id}' cycles cannot fit their windows and minimum gaps." + ) + self._appliance_order_cache[appliance_index] = list(order) + entries = [ + (position, by_position[position]) + for position in self._appliance_order_cache[appliance_index] + ] + latest = latest_bounds(entries) + if latest is None: + raise ValueError( + f"Home appliance '{appliance.device_id}' has no feasible cycle schedule." + ) + earliest = self._control_start_slot() + for (position, gene), last in zip(entries, latest): + allowed = gene.allowed_start_slots + requested_index = min( + max(int(appliance_gene_values[position]), 0), len(allowed) - 1 + ) + requested = allowed[requested_index] + candidates = [slot for slot in allowed if earliest <= slot <= last] + if not candidates: + raise ValueError( + f"Home appliance '{gene.device_id}' has no feasible cycle start." + ) + chosen = ( + candidates[0] + if gene.deadline_relaxed + else min(candidates, key=lambda slot: (abs(slot - requested), slot)) + ) + starts_per_appliance[appliance_index].append(chosen) + appliance_gene_values[position] = allowed.index(chosen) + earliest = chosen + separation + return starts_per_appliance + + def _apply_appliance_starts(self, appliance_gene_values: list[int]) -> None: + """Build every appliance's load curve from the decoded starts.""" + if not self.simulation.home_appliances: + return + starts_per_appliance = self._decode_appliance_starts(appliance_gene_values) + for appliance_index, appliance in enumerate(self.simulation.home_appliances): + appliance.build_load_curve(starts_per_appliance.get(appliance_index, [])) + + def _start_solution_matches_layout(self, start_solution: list[float]) -> bool: + """Check that a start solution's appliance tail fits the current layout. + + A length match alone is insufficient (two different layouts can share a + length), so every appliance gene value must be a valid index into its + gene's ``allowed_start_slots``. + """ + n_genes = self.appliance_layout.n_genes + if n_genes == 0: + return True + if len(start_solution) < n_genes: + return False + tail = start_solution[-n_genes:] + for value, gene in zip(tail, self.appliance_layout.genes): + if not gene.allowed_start_slots: + return False + if not (0 <= int(value) < len(gene.allowed_start_slots)): + return False + return True + + def _ac_break_even_prices( + self, + prices_arr: Any, + load_arr: Any, + free_ac_wh: float, + ) -> list[float]: + """Best still-uncovered future price per potential AC-charge slot. + + The AC-charge break-even penalty needs, for every potential charge slot, + the highest future price whose load is not already covered by the energy + that is in the battery at simulation start. Prices, loads and the free + battery energy are constant within one optimization run, so this table + is computed once per run and looked up in every fitness evaluation. + (Previously the future list was rebuilt and sorted per slot per + individual, which dominated the fitness runtime.) The loops replicate + the former inline computation exactly, keeping results bit-identical. + """ + n = min(len(prices_arr), self.control_end_slot) + best_prices = [0.0] * n + for hour in range(n): + # Build list of (price, load_wh) for all future hours in the horizon + future = [(float(prices_arr[h]), float(load_arr[h])) for h in range(hour + 1, n)] + # Sort descending by price so we "use" the most expensive hours first + future.sort(key=lambda x: -x[0]) + + # Consume free PV energy against the highest-price future hours. + # The first uncovered (partially or fully) hour defines the best + # price still available for the new AC charge. + remaining_free = free_ac_wh + best_uncovered_price = 0.0 + for fp, fl in future: + if remaining_free >= fl: + # Entire expensive hour is already covered by free PV energy + remaining_free -= fl + else: + # First hour not (fully) covered: this is where new charge goes + best_uncovered_price = fp + break + best_prices[hour] = best_uncovered_price + return best_prices + + def _parameters_for_config( + self, parameters: GeneticOptimizationParameters + ) -> GeneticOptimizationParameters: + """Keep supplied sale revenues authoritative, including constant imports.""" + return parameters + + def _parameters_for_slot_grid( + self, parameters: GeneticOptimizationParameters + ) -> GeneticOptimizationParameters: + """Normalize hourly or native-slot EMS input onto the optimization grid. + + API clients historically provide one value per prediction hour. At a + sub-hourly interval, energy quantities are distributed across the slots + while price quantities are held constant. Inputs already matching the + native slot grid are preserved exactly. Short native forecasts must + declare their interval; availability is validated separately. + """ + + def normalize( + values: list[float] | list[Optional[float]], name: str, *, energy: bool + ) -> list[float]: + data = np.asarray(values, dtype=float) + # API inputs default to hourly; native callers declare their interval. + native = ( + parameters.forecast_interval_seconds + == self.config.optimization.genetic.interval_sec + ) + if parameters.forecast_interval_seconds is None: + max_hourly = self.config.prediction.hours + math.ceil( + self._start_day_slot() / self.slots_per_hour + ) + if self.slots_per_hour > 1 and max_hourly < len(data) < self.prediction_slots: + raise ValueError( + f"{name}: ambiguous forecast interval; expected either {self.config.prediction.hours} hourly values or {self.prediction_slots} native values. Set forecast_interval_seconds for shortened native forecasts." + ) + native = self.slots_per_hour == 1 or len(data) >= self.prediction_slots + if parameters.forecast_interval_seconds == 900 and self.slots_per_hour == 1: + remainder = len(data) % 4 + if remainder: + data = np.pad(data, (0, 4 - remainder), constant_values=np.nan) + blocks = data.reshape(-1, 4) + data = blocks.sum(axis=1) if energy else blocks.mean(axis=1) + elif not native: + data = np.repeat(data, self.slots_per_hour) + if energy: + data /= self.slots_per_hour + return data[self._start_day_slot() :].tolist() + + ems = parameters.ems + feed_in_tariff = ems.feed_in_tariff_per_wh + if isinstance(feed_in_tariff, list): + normalized_feed_in_tariff: list[float] | float = normalize( + feed_in_tariff, + "feed_in_tariff_per_wh", + energy=False, + ) + else: + normalized_feed_in_tariff = [float(feed_in_tariff)] * ( + self._control_start_slot() + self.prediction_slots + ) + + normalized_ems = ems.model_copy( + update={ + "pv_forecast_wh": normalize(ems.pv_forecast_wh, "pv_forecast_wh", energy=True), + "total_load": normalize(ems.total_load, "total_load", energy=True), + "electricity_price_per_wh": normalize( + ems.electricity_price_per_wh, + "electricity_price_per_wh", + energy=False, + ), + "feed_in_tariff_per_wh": normalized_feed_in_tariff, + }, + deep=True, + ) + temperature_forecast = ( + normalize(parameters.temperature_forecast, "temperature_forecast", energy=False) + if parameters.temperature_forecast is not None + else None + ) + return parameters.model_copy( + update={ + "ems": normalized_ems, + "temperature_forecast": temperature_forecast, + "forecast_interval_seconds": self.config.optimization.genetic.interval_sec, + }, + deep=True, + ) + + def _start_solution_for_slot_grid(self, start_solution: list[float]) -> list[float]: + """Expand a legacy hourly genome to the configured slot grid when possible. + + Only the battery and EV parts are grid-expanded. The appliance start + genes are indices into interval-dependent allowed-start lists, so they + are copied verbatim and validated later against the current layout + (incompatible tails cause the whole start solution to be discarded). + """ + n_appliance_genes = self.appliance_layout.n_genes + expected_length = self.control_end_slot * (2 if self.optimize_ev else 1) + n_appliance_genes + hourly_length = ( + self.config.optimization.genetic.horizon_hours * (2 if self.optimize_ev else 1) + + n_appliance_genes + ) + + if len(start_solution) == expected_length or self.slots_per_hour == 1: + return list(start_solution) + if len(start_solution) != hourly_length: + return list(start_solution) + + battery_end = self.config.optimization.genetic.horizon_hours + migrated = np.repeat(start_solution[:battery_end], self.slots_per_hour).tolist() + if self.optimize_ev: + ev_end = battery_end + self.config.optimization.genetic.horizon_hours + migrated.extend( + np.repeat(start_solution[battery_end:ev_end], self.slots_per_hour).tolist() + ) + if n_appliance_genes > 0: + migrated.extend(list(start_solution[-n_appliance_genes:])) + logger.info( + "Expanded hourly start_solution from {} to {} slot values.", + hourly_length, + expected_length, + ) + return migrated + + def _resolve_start_solution_datetime( + self, parameters: GeneticOptimizationParameters + ) -> Optional[DateTime]: + """Start of the slot that gene 0 of the supplied warm start controls. + + An explicit ``start_solution_datetime`` wins. Clients that only echo + ``start_solution`` get the start of this server's last solution when the + genomes are identical; for any other genome the start is unknown. + """ + if parameters.start_solution_datetime is not None: + return parameters.start_solution_datetime + if parameters.start_solution is None: + return None + last_solution = self.ems.genetic_solution() + if ( + last_solution is not None + and last_solution.start_solution is not None + and list(last_solution.start_solution) == list(parameters.start_solution) + ): + return last_solution.start_solution_datetime + return None + + def _start_solution_for_run_start( + self, + start_solution: Optional[list[float]], + start_solution_datetime: Optional[DateTime], + ) -> Optional[list[float]]: + """Align a warm start from an earlier run with the slot this run starts in. + + Genomes are run-relative, so a solution returned one slot ago describes + every battery and EV decision one slot too late. Reused unchanged, a + search that keeps the seed postpones each planned action by one slot per + run. The battery and EV blocks therefore drop the elapsed slots and + repeat their last gene to refill the horizon. + + Appliance genes index into per-run lists of allowed start slots that + cannot be rebuilt for the earlier run; they are kept and validated + against the current layout as before. + """ + if ( + start_solution is None + or start_solution_datetime is None + or self._slot0_datetime is None + ): + return start_solution + start_solution = self._start_solution_for_slot_grid(start_solution) + blocks = 2 if self.optimize_ev else 1 + if len(start_solution) != self.control_end_slot * blocks + self.appliance_layout.n_genes: + # optimize() rejects the length and logs why. + return start_solution + + elapsed_s = (self._slot0_datetime - start_solution_datetime).total_seconds() + if elapsed_s < 0: + logger.warning( + "Ignoring start_solution from {}: it starts after this run ({}).", + start_solution_datetime, + self._slot0_datetime, + ) + return None + elapsed_slots = int(elapsed_s // (self.slot_duration_h * 3600)) + if elapsed_slots == 0: + return start_solution + if elapsed_slots >= self.control_slots: + logger.info( + "Ignoring start_solution from {}: all {} control slots have elapsed.", + start_solution_datetime, + self.control_slots, + ) + return None + + aligned = list(start_solution) + for block in range(blocks): + begin = self._control_start_slot() + block * self.control_end_slot + end = begin + self.control_slots + genes = aligned[begin:end] + aligned[begin:end] = genes[elapsed_slots:] + [genes[-1]] * elapsed_slots + logger.debug("Shifted start_solution by {} elapsed slots.", elapsed_slots) + return aligned + def decode_charge_discharge( self, discharge_hours_bin: np.ndarray - ) -> tuple[np.ndarray, np.ndarray, np.ndarray]: - """Decode the input array into ac_charge, dc_charge, and discharge arrays.""" + ) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]: + """Decode the input array into charge, self-consumption discharge and export arrays.""" discharge_hours_bin_np = np.array(discharge_hours_bin) # Battery AC charge uses its own charge-level list (bat_possible_charge_values). len_bat = len(self.bat_possible_charge_values) @@ -471,9 +1592,9 @@ class GeneticOptimization(OptimizationBase): # Discharge: len_bat .. 2*len_bat - 1 # AC Charge: 2*len_bat .. 3*len_bat - 1 (maps to bat_possible_charge_values) # DC optional: 3*len_bat (not allowed), 3*len_bat + 1 (allowed) - - # Idle states - idle_mask = (discharge_hours_bin_np >= 0) & (discharge_hours_bin_np < len_bat) + # Grid export: next state, if direct marketing/export optimization is enabled + # Self-consumption: final state, with DC charging and local discharge enabled + state_layout = self._battery_state_layout() # Discharge states discharge_mask = (discharge_hours_bin_np >= len_bat) & ( @@ -485,59 +1606,227 @@ class GeneticOptimization(OptimizationBase): ac_indices = (discharge_hours_bin_np[ac_mask] - 2 * len_bat).astype(int) # DC states (if enabled) - if self.optimize_dc_charge: - dc_not_allowed_state = 3 * len_bat - dc_allowed_state = 3 * len_bat + 1 - dc_charge = np.where(discharge_hours_bin_np == dc_allowed_state, 1, 0) + if state_layout.dc_allowed_state is not None: + dc_mask = discharge_hours_bin_np == state_layout.dc_allowed_state + if state_layout.self_consumption_state is not None: + dc_mask |= discharge_hours_bin_np == state_layout.self_consumption_state + dc_charge = np.where(dc_mask, 1, 0) else: dc_charge = np.ones_like(discharge_hours_bin_np, dtype=float) # Generate the result arrays discharge = np.zeros_like(discharge_hours_bin_np, dtype=int) discharge[discharge_mask] = 1 # Set Discharge states to 1 + if state_layout.self_consumption_state is not None: + discharge[discharge_hours_bin_np == state_layout.self_consumption_state] = 1 ac_charge = np.zeros_like(discharge_hours_bin_np, dtype=float) ac_charge[ac_mask] = [self.bat_possible_charge_values[i] for i in ac_indices] - # Idle is just 0, already default. - - return ac_charge, dc_charge, discharge - - def mutate(self, individual: list[int]) -> tuple[list[int]]: - """Custom mutation function for the individual.""" - # Calculate the number of states using battery charge levels - len_bat = len(self.bat_possible_charge_values) - if self.optimize_dc_charge: - total_states = 3 * len_bat + 2 - else: - total_states = 3 * len_bat - - # 1. Mutating the charge_discharge part - charge_discharge_part = individual[: self.config.prediction.hours] - (charge_discharge_mutated,) = self.toolbox.mutate_charge_discharge(charge_discharge_part) - - # Instead of a fixed clamping to 0..8 or 0..6 dynamically: - charge_discharge_mutated = np.clip(charge_discharge_mutated, 0, total_states - 1) - individual[: self.config.prediction.hours] = charge_discharge_mutated - - # 2. Mutating the EV charge part, if active - if self.optimize_ev: - ev_charge_part = individual[ - self.config.prediction.hours : self.config.prediction.hours * 2 - ] - (ev_charge_part_mutated,) = self.toolbox.mutate_ev_charge_index(ev_charge_part) - ev_charge_part_mutated[self.config.prediction.hours - self.fixed_ev_hours :] = [ - 0 - ] * self.fixed_ev_hours - individual[self.config.prediction.hours : self.config.prediction.hours * 2] = ( - ev_charge_part_mutated + # Export rate per slot: 0.0 = no export, otherwise the factor of the + # rated discharge power the optimizer picked for that slot. + battery_grid_export = np.zeros_like(discharge_hours_bin_np, dtype=float) + for index, export_state in enumerate(state_layout.grid_export_states): + rate = ( + self.bat_possible_grid_export_values[index] + if index < len(self.bat_possible_grid_export_values) + else 1.0 + ) + battery_grid_export = np.where( + discharge_hours_bin_np == export_state, rate, battery_grid_export ) - # 3. Mutating the appliance start time, if applicable - if self.opti_param["home_appliance"] > 0: - appliance_part = [individual[-1]] - (appliance_part_mutated,) = self.toolbox.mutate_hour(appliance_part) - individual[-1] = appliance_part_mutated[0] + # Idle is just 0, already default. + + return ac_charge, dc_charge, discharge, battery_grid_export + + def _mutate_battery_block(self, individual: list[int]) -> None: + """Mutate a short future block to one coherent operating policy.""" + start_slot = self._control_start_slot() + if start_slot >= self.control_end_slot: + return + + state_layout = self._battery_state_layout() + len_bat = len(self.bat_possible_charge_values) + policy_states = [0, len_bat] + if state_layout.self_consumption_state is not None: + policy_states.append(state_layout.self_consumption_state) + if state_layout.dc_allowed_state is not None: + policy_states.append(state_layout.dc_allowed_state) + # Every export level is a coherent policy for a whole block. + policy_states.extend(state_layout.grid_export_states) + + block_start = random.randint(start_slot, self.control_end_slot - 1) # noqa: S311 + max_length = min(12, self.control_end_slot - block_start) + block_length = random.randint(2, max(2, max_length)) if max_length > 1 else 1 # noqa: S311 + state = random.choice(policy_states) # noqa: S311 + individual[block_start : block_start + block_length] = [state] * block_length + + def _energy_shift_target_slots( + self, + individual: list[int], + source_slot: int, + ) -> list[int]: + """Return later idle slots where retained battery energy avoids costly import.""" + try: + prices = np.asarray(self.simulation.elect_price_hourly, dtype=float) + feed_in = np.asarray(self.simulation.elect_revenue_per_hour_arr, dtype=float) + pv = np.asarray(self.simulation.pv_prediction_wh, dtype=float) + load = np.asarray(self.simulation.load_energy_array, dtype=float) + except Exception: + return [] + if any(values.size < self.control_end_slot for values in (prices, feed_in, pv, load)): + return [] + + len_bat = len(self.bat_possible_charge_values) + source_tariff = float(feed_in[source_slot]) + candidates = [ + slot + for slot in range(source_slot + 1, self.control_end_slot) + if 0 <= int(individual[slot]) < len_bat + and load[slot] > pv[slot] + and prices[slot] > source_tariff + ] + return sorted( + candidates, + key=lambda slot: (float(prices[slot]), float(load[slot] - pv[slot])), + reverse=True, + ) + + def _mutate_energy_shift(self, individual: list[int]) -> bool: + """Move battery energy from a weak export into later expensive self-consumption.""" + state_layout = self._battery_state_layout() + export_states = set(state_layout.grid_export_states) + self_state = state_layout.self_consumption_state + if not export_states or self_state is None: + return False + + start_slot = self._control_start_slot() + viable: list[tuple[int, list[int]]] = [] + for source_slot in range(start_slot, self.control_end_slot): + if int(individual[source_slot]) not in export_states: + continue + targets = self._energy_shift_target_slots(individual, source_slot) + if targets: + viable.append((source_slot, targets)) + if not viable: + return False + + # Prefer later/lower-value exports, but retain random diversity among + # the viable tail instead of always producing one identical neighbour. + try: + feed_in = np.asarray(self.simulation.elect_revenue_per_hour_arr, dtype=float) + viable.sort(key=lambda item: (float(feed_in[item[0]]), -item[0])) + except Exception: + viable.sort(key=lambda item: -item[0]) + source_slot, targets = random.choice(viable[: min(6, len(viable))]) # noqa: S311 + + individual[source_slot] = self_state + target_count = min(len(targets), random.randint(4, 10)) # noqa: S311 + len_bat = len(self.bat_possible_charge_values) + pv = np.asarray(self.simulation.pv_prediction_wh, dtype=float) + for target_slot in targets[:target_count]: + individual[target_slot] = self_state if pv[target_slot] > 0.0 else len_bat + return True + + @staticmethod + def _force_segment_change(values: list[int], low: int, up: int) -> bool: + """Change one value when probabilistic mutation produced no effective change.""" + if not values or up <= low: + return False + position = random.randrange(len(values)) # noqa: S311 + old_value = int(values[position]) + replacement = random.randint(low, up - 1) # noqa: S311 + if replacement >= old_value: + replacement += 1 + values[position] = replacement + return True + + def _mutate_point_controls(self, individual: list[int]) -> bool: + """Apply a small point mutation only to controls that can still affect fitness.""" + changed = False + start_slot = self._control_start_slot() + total_states = self._battery_state_layout().total_states + battery_part = list(individual[start_slot : self.control_end_slot]) + battery_before = list(battery_part) + (battery_part,) = self.toolbox.mutate_charge_discharge(battery_part) + if battery_part == battery_before: + self._force_segment_change(battery_part, 0, total_states - 1) + if battery_part != battery_before: + individual[start_slot : self.control_end_slot] = battery_part + changed = True + + if self.optimize_ev and random.random() < 0.40: # noqa: S311 + ev_start = self.control_end_slot + start_slot + ev_end = self.control_end_slot * 2 - self.fixed_eauto_hours + ev_part = list(individual[ev_start:ev_end]) + ev_before = list(ev_part) + (ev_part,) = self.toolbox.mutate_ev_charge_index(ev_part) + if ev_part == ev_before: + self._force_segment_change(ev_part, 0, len(self.ev_possible_charge_values) - 1) + if ev_part != ev_before: + individual[ev_start:ev_end] = ev_part + changed = True + + return changed + + def _mutate_flexible_controls(self, individual: list[int]) -> bool: + """Mutate EV or appliance controls without disturbing a good battery schedule.""" + changed = False + if self.optimize_ev: + ev_start = self.control_end_slot + self._control_start_slot() + ev_end = self.control_end_slot * 2 - self.fixed_eauto_hours + ev_part = list(individual[ev_start:ev_end]) + ev_before = list(ev_part) + (ev_part,) = self.toolbox.mutate_ev_charge_index(ev_part) + if ev_part == ev_before: + self._force_segment_change(ev_part, 0, len(self.ev_possible_charge_values) - 1) + if ev_part != ev_before: + individual[ev_start:ev_end] = ev_part + changed = True + + n_appliance_genes = self.appliance_layout.n_genes + if n_appliance_genes > 0: + base = len(individual) - n_appliance_genes + mutable_positions = [ + (base + position, len(gene.allowed_start_slots) - 1) + for position, gene in enumerate(self.appliance_layout.genes) + if len(gene.allowed_start_slots) > 1 + ] + if mutable_positions: + position, upper = random.choice(mutable_positions) # noqa: S311 + old_value = int(individual[position]) + replacement = random.randint(0, upper - 1) # noqa: S311 + if replacement >= old_value: + replacement += 1 + individual[position] = replacement + changed = True + return changed + + def mutate(self, individual: list[int]) -> tuple[list[int]]: + """Apply one coherent mutation family instead of stacking destructive changes.""" + operation = random.random() # noqa: S311 + changed = False + if operation < 0.50: + changed = self._mutate_point_controls(individual) + elif operation < 0.70: + before = list(individual) + self._mutate_battery_block(individual) + changed = individual != before + elif operation < 0.90: + changed = self._mutate_energy_shift(individual) + else: + changed = self._mutate_flexible_controls(individual) + + # Some specialized moves are unavailable without EV, appliances or a + # viable grid-export opportunity. Always return a genuinely changed + # future control so an offspring budget is not silently wasted. + if not changed: + self._mutate_point_controls(individual) + + if self.optimize_ev and self.fixed_eauto_hours > 0: + ev_end = self.control_end_slot * 2 + individual[ev_end - self.fixed_eauto_hours : ev_end] = [0] * self.fixed_eauto_hours return (individual,) @@ -545,33 +1834,39 @@ class GeneticOptimization(OptimizationBase): def create_individual(self) -> list[int]: # Start with discharge states for the individual individual_components = [ - self.toolbox.attr_discharge_state() for _ in range(self.config.prediction.hours) + self.toolbox.attr_discharge_state() for _ in range(self.control_end_slot) ] # Add EV charge index values if optimize_ev is True if self.optimize_ev: - individual_components += [ - self.toolbox.attr_ev_charge_index() for _ in range(self.config.prediction.hours) + ev_controls = [ + self.toolbox.attr_ev_charge_index() for _ in range(self.control_end_slot) ] + if self.fixed_eauto_hours > 0: + ev_controls[-self.fixed_eauto_hours :] = [0] * self.fixed_eauto_hours + individual_components += ev_controls - # Add the start time of the household appliance if it's being optimized - if self.opti_param["home_appliance"] > 0: - individual_components += [self.toolbox.attr_int()] + # Add one appliance start gene per scheduled run (index into that run's + # allowed_start_slots). No draws happen when there are no appliances, so + # the battery/EV-only genome is unchanged. + for gene in self.appliance_layout.genes: + individual_components.append(random.randint(0, len(gene.allowed_start_slots) - 1)) # noqa: S311 return creator.Individual(individual_components) def merge_individual( self, discharge_hours_bin: np.ndarray, - ev_charge_hours_index: Optional[np.ndarray], - washingstart_int: Optional[int], + eautocharge_hours_index: Optional[np.ndarray], + appliance_gene_values: Optional[list[int]], ) -> list[int]: """Merge the individual components back into a single solution list. Parameters: discharge_hours_bin (np.ndarray): Binary discharge hours. - ev_charge_hours_index (Optional[np.ndarray]): EV charge hours as integers, or None. - washingstart_int (Optional[int]): Dishwasher start time as integer, or None. + eautocharge_hours_index (Optional[np.ndarray]): EV charge hours as integers, or None. + appliance_gene_values (Optional[list[int]]): One index per appliance + start gene (into the gene's allowed_start_slots), or None. Returns: list[int]: The merged individual solution as a list of integers. @@ -580,53 +1875,828 @@ class GeneticOptimization(OptimizationBase): individual = discharge_hours_bin.tolist() # Add EV charge hours if applicable - if self.optimize_ev and ev_charge_hours_index is not None: - individual.extend(ev_charge_hours_index.tolist()) + if self.optimize_ev and eautocharge_hours_index is not None: + individual.extend(eautocharge_hours_index.tolist()) elif self.optimize_ev: - # If optimize_ev is active but no EV data is available, append zeros - individual.extend([0] * self.config.prediction.hours) + # optimize_ev active but no EV data present: pad with zeros + individual.extend([0] * self.control_end_slot) - # Add dishwasher start time if applicable - if self.opti_param.get("home_appliance", 0) > 0 and washingstart_int is not None: - individual.append(washingstart_int) - elif self.opti_param.get("home_appliance", 0) > 0: - # If a home appliance is optimized but no start time is available - individual.append(0) + # Add appliance start genes (one index per scheduled run). + n_appliance_genes = self.appliance_layout.n_genes + if n_appliance_genes > 0: + if appliance_gene_values is not None: + individual.extend(int(value) for value in appliance_gene_values) + else: + individual.extend([0] * n_appliance_genes) return individual def split_individual( self, individual: list[int] - ) -> tuple[np.ndarray, Optional[np.ndarray], Optional[int]]: + ) -> tuple[np.ndarray, Optional[np.ndarray], list[int]]: """Split the individual solution into its components. Components: 1. Discharge hours (binary as int NumPy array), 2. Electric vehicle charge hours (float as int NumPy array, if applicable), - 3. Dishwasher start time (integer if applicable). + 3. Appliance start genes (list of indices, one per scheduled run). """ # Discharge hours as a NumPy array of ints - discharge_hours_bin = np.array(individual[: self.config.prediction.hours], dtype=int) + discharge_hours_bin = np.array(individual[: self.control_end_slot], dtype=int) # EV charge hours as a NumPy array of ints (if optimize_ev is True) - ev_charge_hours_index = ( + eautocharge_hours_index = ( # append ev charging states to individual np.array( - individual[self.config.prediction.hours : self.config.prediction.hours * 2], + individual[self.control_end_slot : self.control_end_slot * 2], dtype=int, ) if self.optimize_ev else None ) - # Washing machine start time as an integer (if applicable) - washingstart_int = ( - int(individual[-1]) - if self.opti_param and self.opti_param.get("home_appliance", 0) > 0 - else None + # Appliance start genes are the trailing entries of the genome. + n_appliance_genes = self.appliance_layout.n_genes + if n_appliance_genes > 0: + appliance_gene_values = [int(value) for value in individual[-n_appliance_genes:]] + else: + appliance_gene_values = [] + + return discharge_hours_bin, eautocharge_hours_index, appliance_gene_values + + def _repair_ev_charge_at_full_soc( + self, + individual: list[int], + simulation_result: dict[str, Any], + ) -> bool: + """Remove EV charging genes in slots that begin at full SoC. + + The repair is deliberately separated from fitness calculation. Callers + must re-simulate after a change so the individual's genome, simulation + state and assigned fitness always describe the same schedule. + """ + ev_possible_charge_values = getattr(self, "ev_possible_charge_values", None) + if not self.optimize_ev or not ev_possible_charge_values: + return False + + zero_charge_index = min( + range(len(ev_possible_charge_values)), + key=lambda index: abs(ev_possible_charge_values[index]), + ) + if abs(ev_possible_charge_values[zero_charge_index]) > 1e-12: + return False + + _, ev_charge_indices, _ = self.split_individual(individual) + if ev_charge_indices is None: + return False + + ev_soc = np.asarray(simulation_result.get("EAuto_SoC_pro_Stunde", []), dtype=float) + start_slot = self._control_start_slot() + result_slots = min(ev_soc.size, self.control_end_slot - start_slot) + if result_slots <= 0: + return False + + changed = False + for offset in range(result_slots): + slot = start_slot + offset + charge_index = int(ev_charge_indices[slot]) + if ev_soc[offset] >= 100.0 - 1e-9 and ev_possible_charge_values[charge_index] > 0.0: + ev_charge_indices[slot] = zero_charge_index + changed = True + + if changed: + battery_genes, _, appliance_genes = self.split_individual(individual) + individual[:] = self.merge_individual( + battery_genes, + ev_charge_indices, + appliance_genes, + ) + return changed + + def _heuristic_ev_schedule(self, *, prefer_pv: bool) -> list[int]: + """Build a low-cost EV schedule that reaches the configured minimum SoC.""" + if not self.optimize_ev or not self.ev_possible_charge_values: + return [] + + zero_index = min( + range(len(self.ev_possible_charge_values)), + key=lambda index: abs(self.ev_possible_charge_values[index]), + ) + schedule = [zero_index] * self.control_end_slot + ev = self.simulation.ev + if ev is None: + return schedule + + required_stored_wh = max( + ev.min_soc_wh - ev.capacity_wh * ev.initial_soc_percentage / 100.0, + 0.0, + ) + if required_stored_wh <= 0.0: + return schedule + + start_slot = self._control_start_slot() + end_slot = max(start_slot, self.control_end_slot - self.fixed_eauto_hours) + deadline_slot = self._ev_soc_deadline_slot + if deadline_slot is not None: + # Charging after the deadline does not help to reach the target. + end_slot = max(start_slot, min(end_slot, deadline_slot)) + prices = np.asarray(self.simulation.elect_price_hourly, dtype=float) + feed_in = np.asarray(self.simulation.elect_revenue_per_hour_arr, dtype=float) + pv = np.asarray(self.simulation.pv_prediction_wh, dtype=float) + load = np.asarray(self.simulation.load_energy_array, dtype=float) + + def marginal_cost(slot: int) -> tuple[float, float]: + surplus = pv[slot] - load[slot] + if prefer_pv and surplus > 0.0: + return (float(feed_in[slot]), -float(surplus)) + return (float(prices[slot]), -float(surplus)) + + candidates = sorted(range(start_slot, end_slot), key=marginal_cost) + positive_rates = sorted( + ( + (rate, index) + for index, rate in enumerate(self.ev_possible_charge_values) + if rate > 0.0 + ), + key=lambda item: item[0], + ) + if not positive_rates: + return schedule + + max_stored_wh = ev.max_charge_power_w * self.slot_duration_h * ev.charging_efficiency + remaining_wh = required_stored_wh + for slot in candidates: + required_rate = remaining_wh / max(max_stored_wh, 1e-9) + rate, rate_index = next( + (item for item in positive_rates if item[0] >= required_rate), + positive_rates[-1], + ) + schedule[slot] = rate_index + remaining_wh -= max_stored_wh * rate + if remaining_wh <= 1e-9: + break + return schedule + + def _heuristic_appliance_genes(self) -> list[int]: + """Choose low-opportunity-cost starts for flexible appliances.""" + if self.appliance_layout.n_genes == 0: + return [] + prices = np.asarray(self.simulation.elect_price_hourly, dtype=float) + feed_in = np.asarray(self.simulation.elect_revenue_per_hour_arr, dtype=float) + pv = np.asarray(self.simulation.pv_prediction_wh, dtype=float) + load = np.asarray(self.simulation.load_energy_array, dtype=float) + genes: list[int] = [] + for gene in self.appliance_layout.genes: + + def opportunity_cost(position: int) -> float: + slot = gene.allowed_start_slots[position] + return float(feed_in[slot] if pv[slot] > load[slot] else prices[slot]) + + genes.append(min(range(len(gene.allowed_start_slots)), key=opportunity_cost)) + return genes + + def _educated_guess_individuals( + self, + target_count: int = EDUCATED_GUESS_TARGET, + ) -> list[list[int]]: + """Create a randomized family of domain-informed initial candidates.""" + if target_count <= 0: + return [] + + slots = self.control_end_slot + start_slot = self._control_start_slot() + len_bat = len(self.bat_possible_charge_values) + state_layout = self._battery_state_layout() + idle_state = 0 + discharge_state = len_bat + ac_charge_state = 3 * len_bat - 1 + dc_allowed_state = state_layout.dc_allowed_state + export_state = state_layout.grid_export_state + self_consumption_state = state_layout.self_consumption_state + + prices = np.asarray(self.simulation.elect_price_hourly, dtype=float) + feed_in = np.asarray(self.simulation.elect_revenue_per_hour_arr, dtype=float) + pv = np.asarray(self.simulation.pv_prediction_wh, dtype=float) + load = np.asarray(self.simulation.load_energy_array, dtype=float) + future = slice(start_slot, slots) + future_prices = prices[future] + future_feed_in = feed_in[future] + high_import_price = float(np.quantile(future_prices, 0.70)) + low_import_price = float(np.quantile(future_prices, 0.25)) + feed_spread = float(np.ptp(future_feed_in)) if future_feed_in.size else 0.0 + + ev_price = self._heuristic_ev_schedule(prefer_pv=False) + ev_pv = self._heuristic_ev_schedule(prefer_pv=True) + appliance_genes = self._heuristic_appliance_genes() + + def compose(battery_genes: list[int], ev_genes: list[int]) -> list[int]: + individual = list(battery_genes) + if self.optimize_ev: + individual.extend(ev_genes) + individual.extend(appliance_genes) + return individual + + unique: dict[tuple[int, ...], list[int]] = {} + + def add_guess(battery_genes: list[int], ev_genes: list[int]) -> None: + guess = compose(battery_genes, ev_genes) + unique.setdefault(tuple(guess), guess) + + def policy_guess( + *, + import_quantile: float, + export_quantile: Optional[float], + pv_surplus_ratio: float, + allow_ac_arbitrage: bool, + ) -> list[int]: + import_threshold = float(np.quantile(future_prices, import_quantile)) + export_threshold = ( + float(np.quantile(future_feed_in, export_quantile)) + if export_quantile is not None and future_feed_in.size + else float("inf") + ) + low_price_threshold = float( + np.quantile(future_prices, max(0.05, 1.0 - import_quantile)) + ) + battery_genes = [idle_state] * slots + for slot in range(start_slot, slots): + high_feed_in = ( + export_quantile is not None + and export_state is not None + and feed_spread > 1e-12 + and feed_in[slot] > 0.0 + and feed_in[slot] >= export_threshold + ) + pv_surplus = pv[slot] > load[slot] * pv_surplus_ratio + if high_feed_in and export_state is not None: + battery_genes[slot] = export_state + elif self_consumption_state is not None and pv[slot] > 0.0 and load[slot] > 0.0: + # The probabilistic inverter model can see a residual load + # and a PV surplus within the same coarse slot. Normal + # self-consumption must therefore allow both directions. + battery_genes[slot] = self_consumption_state + elif dc_allowed_state is not None and pv_surplus: + battery_genes[slot] = dc_allowed_state + elif allow_ac_arbitrage and prices[slot] <= low_price_threshold: + battery_genes[slot] = ac_charge_state + elif prices[slot] >= import_threshold and load[slot] > pv[slot]: + battery_genes[slot] = discharge_state + return battery_genes + + # Baseline and self-consumption candidates are useful even without + # direct marketing and anchor the population with feasible schedules. + add_guess([idle_state] * slots, ev_price) + add_guess( + policy_guess( + import_quantile=0.70, + export_quantile=None, + pv_surplus_ratio=1.0, + allow_ac_arbitrage=False, + ), + ev_pv, ) - return discharge_hours_bin, ev_charge_hours_index, washingstart_int + # Direct marketing candidates export only in the relatively expensive + # feed-in slots. At low tariffs PV is preferentially stored instead. + if self.optimize_battery_grid_export and future_feed_in.size: + for quantile in self.EDUCATED_GUESS_EXPORT_QUANTILES: + export_guess = policy_guess( + import_quantile=0.70, + export_quantile=quantile, + pv_surplus_ratio=1.0, + allow_ac_arbitrage=False, + ) + add_guess( + export_guess, + ev_pv, + ) + # Seed coordinated alternatives that retain a weak export and + # spend the energy in later expensive import slots. + for shifted in self._grid_export_shift_candidates( + export_guess, + max_sources=2, + )[:6]: + add_guess(shifted, ev_pv) + + inverter = self.simulation.inverter + ac_arbitrage_possible = inverter is not None and ( + inverter.max_ac_charge_power_w is None or inverter.max_ac_charge_power_w > 0 + ) + if ac_arbitrage_possible: + price_arbitrage = [idle_state] * slots + for slot in range(start_slot, slots): + if prices[slot] <= low_import_price: + price_arbitrage[slot] = ac_charge_state + elif prices[slot] >= high_import_price: + price_arbitrage[slot] = discharge_state + add_guess(price_arbitrage, ev_price) + + # Randomize policy thresholds rather than merely cloning a handful of + # templates. Every candidate remains policy-safe: a flat/low-information + # feed-in series never acquires export actions through blind mutation. + attempts = max(target_count * 20, 100) + for _ in range(attempts): + export_quantile = ( + random.uniform(0.50, 0.98) # noqa: S311 + if self.optimize_battery_grid_export and feed_spread > 1e-12 + else None + ) + randomized = policy_guess( + import_quantile=random.uniform(0.55, 0.95), # noqa: S311 + export_quantile=export_quantile, + pv_surplus_ratio=random.uniform(0.80, 1.20), # noqa: S311 + allow_ac_arbitrage=ac_arbitrage_possible and random.random() < 0.35, # noqa: S311 + ) + + # Add small policy-safe local variations. These provide diversity + # even when price quantiles collapse to only a few distinct slot + # masks. Export is only ever removed here, never introduced into a + # slot that the tariff policy did not mark as attractive. + future_slots = list(range(start_slot, slots)) + perturbations = random.randint(1, max(2, len(future_slots) // 12)) # noqa: S311 + for slot in random.sample(future_slots, min(perturbations, len(future_slots))): # noqa: S311 + if randomized[slot] != idle_state: + randomized[slot] = idle_state + elif self_consumption_state is not None and pv[slot] > 0.0 and load[slot] > 0.0: + randomized[slot] = self_consumption_state + elif dc_allowed_state is not None and pv[slot] > load[slot]: + randomized[slot] = dc_allowed_state + elif load[slot] > pv[slot] and prices[slot] >= high_import_price: + randomized[slot] = discharge_state + + if random.random() < 0.5: # noqa: S311 + self._mutate_energy_shift(randomized) + + add_guess( + randomized, + ev_pv if random.random() < 0.5 else ev_price, # noqa: S311 + ) + if len(unique) >= target_count: + break + + return list(unique.values())[:target_count] + + def _mutated_warm_start_neighbors( + self, + start_solution: list[float], + count: int, + ) -> list[list[int]]: + """Create unique local variants while preserving already elapsed slots.""" + original = [int(value) for value in start_solution] + start_slot = self._control_start_slot() + seen = {tuple(original)} + neighbors: list[list[int]] = [] + for _ in range(max(count * 10, 1)): + neighbor = creator.Individual(original) + self.mutate(neighbor) + neighbor[:start_slot] = original[:start_slot] + if self.optimize_ev: + ev_start = self.control_end_slot + neighbor[ev_start : ev_start + start_slot] = original[ + ev_start : ev_start + start_slot + ] + key = tuple(int(value) for value in neighbor) + if key in seen: + continue + seen.add(key) + neighbors.append(list(key)) + if len(neighbors) >= count: + break + return neighbors + + def _grid_export_shift_candidates( + self, + individual: list[int], + *, + max_sources: int = 6, + ) -> list[list[int]]: + """Build deterministic export-to-self-consumption neighbourhood candidates.""" + state_layout = self._battery_state_layout() + export_states = set(state_layout.grid_export_states) + self_state = state_layout.self_consumption_state + if not export_states or self_state is None: + return [] + + start_slot = self._control_start_slot() + try: + feed_in = np.asarray(self.simulation.elect_revenue_per_hour_arr, dtype=float) + pv = np.asarray(self.simulation.pv_prediction_wh, dtype=float) + except Exception: + return [] + if feed_in.size < self.control_end_slot or pv.size < self.control_end_slot: + return [] + + sources = [ + slot + for slot in range(start_slot, self.control_end_slot) + if int(individual[slot]) in export_states + ] + # Search weak and late export decisions first. They are the most likely + # to compete with later, more valuable avoided grid imports. + sources.sort(key=lambda slot: (float(feed_in[slot]), -slot)) + + len_bat = len(self.bat_possible_charge_values) + candidates: list[list[int]] = [] + seen: set[tuple[int, ...]] = set() + viable_sources = 0 + for source_slot in sources: + targets = self._energy_shift_target_slots(individual, source_slot) + if not targets: + continue + viable_sources += 1 + counts = sorted({min(len(targets), count) for count in (2, 4, 6, 8, 10, 12)}) + for count in counts: + candidate = list(individual) + candidate[source_slot] = self_state + for target_slot in targets[:count]: + candidate[target_slot] = self_state if pv[target_slot] > 0.0 else len_bat + key = tuple(int(value) for value in candidate) + if key in seen: + continue + seen.add(key) + candidates.append(candidate) + if viable_sources >= max_sources: + break + return candidates + + def _locally_improve_grid_export( + self, + individual: list[int], + *, + max_evaluations: int, + ) -> tuple[Any, int, int, float, float]: + """Improve the incumbent through bounded, fitness-checked energy shifts.""" + best = creator.Individual(individual) + original_fitness = getattr(individual, "fitness", None) + if original_fitness is not None and original_fitness.valid: + best.fitness.values = original_fitness.values + if hasattr(individual, "extra_data"): + best.extra_data = individual.extra_data + + if not hasattr(self.toolbox, "evaluate"): + value = float(best.fitness.values[0]) if best.fitness.valid else float("inf") + return best, 0, 0, value, value + if not best.fitness.valid: + best.fitness.values = self.toolbox.evaluate(best) + + initial_value = float(best.fitness.values[0]) + evaluations = 0 + improvements = 0 + for _ in range(self.LOCAL_SEARCH_MAX_PASSES): + pass_best = best + for genome in self._grid_export_shift_candidates(best): + if evaluations >= max_evaluations: + break + candidate = creator.Individual(genome) + candidate.fitness.values = self.toolbox.evaluate(candidate) + evaluations += 1 + if candidate.fitness.values[0] < pass_best.fitness.values[0] - 1e-9: + pass_best = candidate + if pass_best is best: + break + best = pass_best + improvements += 1 + if evaluations >= max_evaluations: + break + + final_value = float(best.fitness.values[0]) + return best, evaluations, improvements, initial_value, final_value + + def _population_diversity(self, population: list[Any]) -> float: + """Return the fraction of fitness-relevant unique genomes.""" + if not population: + return 0.0 + return len({self._fitness_key(individual) for individual in population}) / len(population) + + def _invalidate_individual(self, individual: Any) -> None: + """Invalidate inherited fitness and auxiliary simulation values.""" + if individual.fitness.valid: + del individual.fitness.values + if hasattr(individual, "extra_data"): + del individual.extra_data + # A child of a protected immigrant is an ordinary offspring. + if hasattr(individual, "immigrant_protection"): + del individual.immigrant_protection + + def _evaluate_invalid(self, population: list[Any]) -> int: + """Evaluate invalid individuals and return the number of cache lookups.""" + invalid = [individual for individual in population if not individual.fitness.valid] + fitnesses = self.toolbox.map(self.toolbox.evaluate, invalid) + for individual, fitness in zip(invalid, fitnesses): + individual.fitness.values = fitness + return len(invalid) + + def _fresh_population(self, count: int, *, educated_fraction: float) -> list[Any]: + """Create a mixed set of current educated guesses and random immigrants.""" + if count <= 0: + return [] + educated_target = min(count, int(count * educated_fraction + 0.5)) + educated = self._educated_guess_individuals(educated_target) + fresh = [creator.Individual(genome) for genome in educated[:count]] + fresh.extend(self.toolbox.population(n=count - len(fresh))) + return fresh + + def _best_unique(self, population: list[Any], count: int) -> list[Any]: + """Return the best fitness-relevant unique candidates.""" + selected: list[Any] = [] + seen: set[tuple[int, ...]] = set() + for candidate in tools.selBest(population, len(population)): + key = self._fitness_key(candidate) + if key in seen: + continue + seen.add(key) + selected.append(candidate) + if len(selected) >= count: + break + return selected + + def _reserve_immigrant_slots( + self, + candidates: list[Any], + selected: list[Any], + selected_keys: list[tuple[int, ...]], + best_key: tuple[int, ...], + ) -> bool: + """Carry still-protected immigrants into ``selected`` in place. + + The tournament judges immigrants on the fitness they have before any + recombination, which they lose. Reserving a bounded share of the seats + gives their genes the generations they need to be crossed into the + incumbents. + + Returns whether any seat was reassigned. + """ + protected = [ + candidate + for candidate in candidates + if getattr(candidate, "immigrant_protection", 0) > 0 + ] + if not protected: + return False + + limit = max(1, int(len(selected) * self.IMMIGRANT_PROTECTION_FRACTION)) + chosen = {id(candidate) for candidate in selected} + seated = sum(1 for candidate in protected if id(candidate) in chosen) + missing = [candidate for candidate in protected if id(candidate) not in chosen] + if seated >= limit or not missing: + return False + + # Evict the weakest seats that carry neither the incumbent genome nor a + # protection of their own, worst first. + evictable = sorted( + ( + index + for index, candidate in enumerate(selected) + if selected_keys[index] != best_key + and getattr(candidate, "immigrant_protection", 0) <= 0 + ), + key=lambda index: selected[index].fitness.values[0], + reverse=True, + ) + reassigned = False + for immigrant, index in zip(missing[: limit - seated], evictable): + selected[index] = immigrant + reassigned = True + return reassigned + + def _age_immigrant_protection(self, population: list[Any]) -> None: + """Spend one generation of the surviving immigrants' protection.""" + for individual in population: + remaining = getattr(individual, "immigrant_protection", 0) + if remaining > 0: + individual.immigrant_protection = remaining - 1 + + def _select_diverse(self, candidates: list[Any], count: int) -> list[Any]: + """Tournament-select while repairing only severe duplicate takeover.""" + if not candidates or count <= 0: + return [] + + selected = tools.selTournament(candidates, count, tournsize=3) + best = tools.selBest(candidates, 1)[0] + best_key = self._fitness_key(best) + selected_keys = [self._fitness_key(candidate) for candidate in selected] + if best_key not in selected_keys: + worst_index = max( + range(len(selected)), + key=lambda index: selected[index].fitness.values[0], + ) + selected[worst_index] = best + selected_keys[worst_index] = best_key + + if self._reserve_immigrant_slots(candidates, selected, selected_keys, best_key): + selected_keys = [self._fitness_key(candidate) for candidate in selected] + + # Duplicates are useful for exploitation and cache hits. Replace only + # enough duplicate selections to keep a minimum search breadth. + target_unique = min( + count, + max(1, int(count * self.SELECTION_DIVERSITY_FLOOR + 0.999999)), + ) + key_counts: dict[tuple[int, ...], int] = defaultdict(int) + for key in selected_keys: + key_counts[key] += 1 + if len(key_counts) >= target_unique: + return selected + + for candidate in tools.selBest(candidates, len(candidates)): + candidate_key = self._fitness_key(candidate) + if candidate_key in key_counts: + continue + replaceable = [index for index, key in enumerate(selected_keys) if key_counts[key] > 1] + if not replaceable: + break + replace_index = max( + replaceable, + key=lambda index: selected[index].fitness.values[0], + ) + replaced_key = selected_keys[replace_index] + key_counts[replaced_key] -= 1 + selected[replace_index] = candidate + selected_keys[replace_index] = candidate_key + key_counts[candidate_key] = 1 + if len(key_counts) >= target_unique: + break + return selected + + def _make_offspring( + self, + population: list[Any], + count: int, + *, + mutation_probability: float, + ) -> list[Any]: + """Create offspring where crossover and mutation can both be applied.""" + offspring: list[Any] = [] + for _ in range(count): + child = self.toolbox.clone(random.choice(population)) # noqa: S311 + crossed = False + if ( + len(child) > 1 + and len(population) > 1 + and random.random() < self.CROSSOVER_PROBABILITY # noqa: S311 + ): + partner = self.toolbox.clone(random.choice(population)) # noqa: S311 + child, _ = self.toolbox.mate(child, partner) + crossed = True + + # Non-crossover offspring are always mutated. Crossover children are + # independently mutated, preventing identical parents from turning + # most of the generation into unchanged copies. + if not crossed or random.random() < mutation_probability: # noqa: S311 + (child,) = self.toolbox.mutate(child) + self._invalidate_individual(child) + offspring.append(child) + return offspring + + def _evolve_population_adaptive( + self, + population: list[Any], + *, + mu: int, + lambda_: int, + ngen: int, + stats: Any, + halloffame: Any, + ) -> tuple[list[Any], Any]: + """Evolve with diversity boosts and incumbent-preserving soft restarts.""" + logbook = tools.Logbook() + logbook.header = [ + "gen", + "nevals", + *stats.fields, + "diversity", + "stagnation", + "immigrants", + "restart", + ] + + nevals = self._evaluate_invalid(population) + halloffame.update(population) + best_fitness = float(halloffame[0].fitness.values[0]) + stagnation = 0 + diversity = self._population_diversity(population) + record = stats.compile(population) + logbook.record( + gen=0, + nevals=nevals, + diversity=diversity, + stagnation=stagnation, + immigrants=0, + restart=0, + **record, + ) + if self.verbose: + print(logbook.stream) + + diversity_boost_active = False + soft_restarts = 0 + total_immigrants = 0 + minimum_diversity = diversity + for generation in range(1, ngen + 1): + diversity = self._population_diversity(population) + soft_restart = ( + stagnation >= self.SOFT_RESTART_GENERATIONS + or diversity < self.SOFT_RESTART_DIVERSITY_THRESHOLD + ) + immigrants = 0 + + if soft_restart: + survivor_count = max(1, int(mu * self.SOFT_RESTART_SURVIVOR_FRACTION)) + survivors = self._best_unique(population, survivor_count) + immigrants = mu - len(survivors) + population = survivors + self._fresh_population( + immigrants, + educated_fraction=0.40, + ) + nevals = self._evaluate_invalid(population) + halloffame.update(population) + soft_restarts += 1 + total_immigrants += immigrants + stagnation = 0 + diversity_boost_active = False + self._age_immigrant_protection(population) + logger.info( + "Genetic soft restart at generation {}: kept {} unique survivors, " + "injected {} immigrants (diversity {:.1%}).", + generation, + len(survivors), + immigrants, + diversity, + ) + else: + diversity_boost = ( + stagnation >= self.STAGNATION_GENERATIONS + or diversity < self.DIVERSITY_BOOST_THRESHOLD + ) + if diversity_boost and not diversity_boost_active: + logger.info( + "Genetic diversity boost at generation {}: stagnation {}, " + "diversity {:.1%}.", + generation, + stagnation, + diversity, + ) + elif diversity_boost_active and not diversity_boost: + logger.info( + "Genetic diversity boost ended at generation {}: stagnation {}, " + "diversity {:.1%}.", + generation, + stagnation, + diversity, + ) + diversity_boost_active = diversity_boost + mutation_probability = ( + self.STAGNATION_MUTATION_PROBABILITY + if diversity_boost + else self.MUTATION_PROBABILITY + ) + if diversity_boost: + immigrants = max(1, int(lambda_ * self.IMMIGRANT_FRACTION + 0.5)) + offspring = self._make_offspring( + population, + lambda_ - immigrants, + mutation_probability=mutation_probability, + ) + fresh = self._fresh_population(immigrants, educated_fraction=0.50) + for immigrant in fresh: + immigrant.immigrant_protection = self.IMMIGRANT_PROTECTION_GENERATIONS + offspring.extend(fresh) + nevals = self._evaluate_invalid(offspring) + halloffame.update(offspring) + population = self._select_diverse(population + offspring, mu) + self._age_immigrant_protection(population) + total_immigrants += immigrants + + current_best = float(halloffame[0].fitness.values[0]) + if current_best < best_fitness - 1e-9: + best_fitness = current_best + stagnation = 0 + elif not soft_restart: + stagnation += 1 + + diversity = self._population_diversity(population) + minimum_diversity = min(minimum_diversity, diversity) + record = stats.compile(population) + logbook.record( + gen=generation, + nevals=nevals, + diversity=diversity, + stagnation=stagnation, + immigrants=immigrants, + restart=int(soft_restart), + **record, + ) + if self.verbose: + print(logbook.stream) + + self._adaptive_evolution_metrics = { + "soft_restarts": soft_restarts, + "immigrants": total_immigrants, + "minimum_diversity": minimum_diversity, + "final_diversity": self._population_diversity(population), + "final_stagnation": stagnation, + } + return population, logbook def setup_deap_environment(self, opti_param: dict[str, Any], start_hour: int) -> None: """Set up the DEAP environment with fitness and individual creation rules.""" @@ -650,10 +2720,9 @@ class GeneticOptimization(OptimizationBase): # Discharge: len_bat states # AC-Charge: len_bat states (maps to bat_possible_charge_values) # With DC: + 2 additional states - if self.optimize_dc_charge: - total_states = 3 * len_bat + 2 - else: - total_states = 3 * len_bat + # With battery grid export: + 1 additional state + # With DC: + 1 final SELF_CONSUMPTION state + total_states = self._battery_state_layout().total_states # State space: 0 .. (total_states - 1) self.toolbox.register("attr_discharge_state", random.randint, 0, total_states - 1) @@ -667,16 +2736,25 @@ class GeneticOptimization(OptimizationBase): len_ev - 1, ) - # Household appliance start time - self.toolbox.register("attr_int", random.randint, start_hour, 23) - self.toolbox.register("individual", self.create_individual) self.toolbox.register("population", tools.initRepeat, list, self.toolbox.individual) self.toolbox.register("mate", tools.cxTwoPoint) - # Mutation operator for battery charge/discharge states + # Keep point mutations local enough to refine a mature schedule. The + # expected number of changed controls remains close to three regardless + # of interval and elapsed slots; coherent block/energy moves are handled + # by separate mutation families. + active_slots = max(self.control_end_slot - self._control_start_slot(), 1) + mutation_probability = min( + 0.10, + self.POINT_MUTATION_EXPECTED_GENES / active_slots, + ) self.toolbox.register( - "mutate_charge_discharge", tools.mutUniformInt, low=0, up=total_states - 1, indpb=0.2 + "mutate_charge_discharge", + tools.mutUniformInt, + low=0, + up=total_states - 1, + indpb=mutation_probability, ) # Mutation operator for EV states (separate index space) @@ -685,12 +2763,9 @@ class GeneticOptimization(OptimizationBase): tools.mutUniformInt, low=0, up=len_ev - 1, - indpb=0.2, + indpb=mutation_probability, ) - # Mutation for household appliance - self.toolbox.register("mutate_hour", tools.mutUniformInt, low=start_hour, up=23, indpb=0.2) - # Custom mutate function remains unchanged self.toolbox.register("mutate", self.mutate) self.toolbox.register("select", tools.selTournament, tournsize=3) @@ -701,39 +2776,44 @@ class GeneticOptimization(OptimizationBase): This is an internal function. """ self.simulation.reset() - discharge_hours_bin, ev_charge_hours_index, washingstart_int = self.split_individual( + discharge_hours_bin, eautocharge_hours_index, appliance_gene_values = self.split_individual( individual ) - if self.opti_param.get("home_appliance", 0) > 0 and washingstart_int: - # Set start hour for appliance - self.simulation.home_appliance_start_hour = washingstart_int + # Decode the appliance start genes and (re)build each appliance's load + # curve for this candidate solution. + self._apply_appliance_starts(appliance_gene_values) + if appliance_gene_values: + individual[-len(appliance_gene_values) :] = appliance_gene_values - ac_charge_hours, dc_charge_hours, discharge = self.decode_charge_discharge( - discharge_hours_bin + ac_charge_hours, dc_charge_hours, discharge, battery_grid_export = ( + self.decode_charge_discharge(discharge_hours_bin) ) self.simulation.bat_discharge_hours = discharge + self.simulation.bat_grid_export_hours = battery_grid_export # Set DC charge hours only if DC optimization is enabled if self.optimize_dc_charge: self.simulation.dc_charge_hours = dc_charge_hours else: - self.simulation.dc_charge_hours = np.full(self.config.prediction.hours, 1) + self.simulation.dc_charge_hours = np.full(self.control_end_slot, 1) self.simulation.ac_charge_hours = ac_charge_hours - if ev_charge_hours_index is not None: - ev_charge_hours_float = np.array( - [self.ev_possible_charge_values[i] for i in ev_charge_hours_index], + if eautocharge_hours_index is not None: + eautocharge_hours_float = np.array( + [self.ev_possible_charge_values[i] for i in eautocharge_hours_index], float, ) # discharge is set to 0 by default - self.simulation.ev_charge_hours = ev_charge_hours_float + self.simulation.ev_charge_hours = eautocharge_hours_float else: # discharge is set to 0 by default - self.simulation.ev_charge_hours = np.full(self.config.prediction.hours, 0) + self.simulation.ev_charge_hours = np.full(self.control_end_slot, 0) - # Do the simulation and return result. - return self.simulation.simulate(self.ems.start_datetime.hour) + # Do the simulation and return result. simulate()'s argument is a slot + # index into the prediction/charge arrays, not an hour-of-day, so pass + # the start_day_slot to keep sub-hourly runs aligned. + return self.simulation.simulate(self._control_start_slot()) def evaluate( self, @@ -741,6 +2821,93 @@ class GeneticOptimization(OptimizationBase): parameters: GeneticOptimizationParameters, start_hour: int, worst_case: bool, + ) -> tuple[float]: + """Evaluate an individual, using run-local canonical memoization when active.""" + # Some lightweight callers construct the optimizer without __init__ + # (for example isolated penalty evaluations). Memoization is opt-in, so + # a missing flag must behave exactly like a disabled cache. + if not getattr(self, "_fitness_cache_enabled", False): + return self._evaluate_uncached(individual, parameters, start_hour, worst_case) + + original_key = self._fitness_key(individual) + cached = self._fitness_cache.get(original_key) + if cached is not None: + individual[:] = cached.genome + individual.extra_data = cached.extra_data # type: ignore[attr-defined] + self._fitness_cache_hits += 1 + return cached.fitness + + self._fitness_cache_misses += 1 + fitness = self._evaluate_uncached(individual, parameters, start_hour, worst_case) + extra_data = getattr(individual, "extra_data", None) + if extra_data is None: + # Failed evaluations use the sentinel fitness and are intentionally + # not cached: an unexpected transient failure must never become a + # persistent result for the remainder of the run. + return fitness + + canonical_key = self._fitness_key(individual) + extra_value1, extra_value2, extra_value3 = extra_data + entry = FitnessCacheEntry( + genome=tuple(int(value) for value in individual), + fitness=fitness, + extra_data=( + float(extra_value1), + float(extra_value2), + float(extra_value3), + ), + ) + self._fitness_cache[original_key] = entry + self._fitness_cache[canonical_key] = entry + return fitness + + def _fitness_key(self, individual: list[int]) -> tuple[int, ...]: + """Return the fitness-relevant genome, excluding elapsed control slots.""" + start_slot = self._control_start_slot() + relevant = list(individual[start_slot : self.control_end_slot]) + if self.optimize_ev: + ev_start = self.control_end_slot + start_slot + relevant.extend(individual[ev_start : self.control_end_slot * 2]) + n_appliance_genes = self.appliance_layout.n_genes + if n_appliance_genes > 0: + relevant.extend(individual[-n_appliance_genes:]) + return tuple(int(value) for value in relevant) + + def _ev_soc_at_deadline(self, simulation_result: dict[str, Any], start_slot: int) -> float: + """EV state of charge the target is checked against [%]. + + Without a deadline this is the SoC after the last slot, which is what the + penalty always used. With a deadline it is the SoC at the beginning of + the deadline slot, i.e. after every charge that completes in time. + + Args: + simulation_result: Result of the simulation run for this individual. + start_slot: Slot index the result arrays start at. + + Returns: + State of charge in percent. + """ + ev = self.simulation.ev + if ev is None: + return 0.0 + # Lightweight callers construct the optimizer without __init__ (see + # evaluate()); a missing deadline must behave like no deadline. + deadline_slot = getattr(self, "_ev_soc_deadline_slot", None) + if deadline_slot is None: + return ev.current_soc_percentage() + + soc_per_slot = simulation_result.get("EAuto_SoC_pro_Stunde") + index = deadline_slot - start_slot + if soc_per_slot is None or index >= len(soc_per_slot): + return ev.current_soc_percentage() + return float(soc_per_slot[max(index, 0)]) + + def _evaluate_uncached( + self, + individual: list[int], + parameters: GeneticOptimizationParameters, + start_hour: int, + worst_case: bool, ) -> tuple[float]: """Evaluate the fitness score of a single individual in the DEAP genetic algorithm. @@ -778,43 +2945,15 @@ class GeneticOptimization(OptimizationBase): """ try: simulation_result = self.evaluate_inner(individual) - except Exception as e: + if self._repair_ev_charge_at_full_soc(individual, simulation_result): + simulation_result = self.evaluate_inner(individual) + except Exception: # Return bad fitness score ("FitnessMin") in case of an exception + if hasattr(individual, "extra_data"): + del individual.extra_data return (100000.0,) - total_balance = simulation_result["Gesamtbilanz_Euro"] * (-1.0 if worst_case else 1.0) - - # EV 100% & charge not allowed - if self.optimize_ev: - discharge_hours_bin, ev_charge_hours_index, washingstart_int = self.split_individual( - individual - ) - - ev_soc_per_hour = np.array( - simulation_result.get("EAuto_SoC_pro_Stunde", []) - ) # Beispielkey - - if ev_soc_per_hour is None or ev_charge_hours_index is None: - raise ValueError("ev_soc_per_hour or ev_charge_hours_index is None") - min_length = min(ev_soc_per_hour.size, ev_charge_hours_index.size) - ev_soc_per_hour_tail = ev_soc_per_hour[-min_length:] - ev_charge_hours_index_tail = ev_charge_hours_index[-min_length:] - - # Mask - invalid_charge_mask = (ev_soc_per_hour_tail == 100) & (ev_charge_hours_index_tail > 0) - - if np.any(invalid_charge_mask): - invalid_indices = np.where(invalid_charge_mask)[0] - if len(invalid_indices) > 1: - ev_charge_hours_index_tail[invalid_indices] = 0 - - ev_charge_hours_index[-min_length:] = ev_charge_hours_index_tail.tolist() - - adjusted_individual = self.merge_individual( - discharge_hours_bin, ev_charge_hours_index, washingstart_int - ) - - individual[:] = adjusted_individual + gesamtbilanz = simulation_result["Gesamtbilanz_Euro"] * (-1.0 if worst_case else 1.0) # New check: Activate discharge when battery SoC is 0 # battery_soc_per_hour = np.array( @@ -842,7 +2981,7 @@ class GeneticOptimization(OptimizationBase): # # discharge_hours_bin_tail[zero_soc_mask] = ( # # len_ac + 2 # # ) # Activate discharge for these hours - # set_to_len_ac_plus_2 = np.random.rand() < 0.5 # True with 50% probability + # set_to_len_ac_plus_2 = np.random.rand() < 0.5 # True mit 50% Wahrscheinlichkeit # # Werte setzen basierend auf der zufälligen Entscheidung # value_to_set = len_ac + 2 if set_to_len_ac_plus_2 else 0 @@ -850,7 +2989,7 @@ class GeneticOptimization(OptimizationBase): # # Merge the updated discharge_hours_bin back into the individual # adjusted_individual = self.merge_individual( - # discharge_hours_bin, ev_charge_hours_index, washingstart_int + # discharge_hours_bin, eautocharge_hours_index, washingstart_int # ) # individual[:] = adjusted_individual @@ -863,15 +3002,13 @@ class GeneticOptimization(OptimizationBase): else 0, ) - # Adjust total balance with battery value and penalties for unmet SOC + # Adjust total balance with battery value and penalties for unmet SOC. + # The terminal value is concave in AUTO mode: the first stored kWh + # replaces the most expensive hour after the horizon, the last one + # replaces nothing. A scalar cannot express that (see terminalvalue.py). if self.simulation.battery: - battery_energy_content = self.simulation.battery.current_energy_content() - # Apply DC→AC inverter efficiency to residual battery value - # (stored DC energy must pass through inverter to be usable as AC) - if self.simulation.inverter: - battery_energy_content *= self.simulation.inverter.dc_to_ac_efficiency - battery_residual_value = battery_energy_content * parameters.ems.price_per_wh_battery - total_balance += -battery_residual_value + restwert_akku, _ = self._terminal_value(parameters) + gesamtbilanz += -restwert_akku # --- AC charging break-even penalty --- # Penalise AC charging decisions that cannot be economically justified given the @@ -889,6 +3026,7 @@ class GeneticOptimization(OptimizationBase): if ( self.simulation.battery and self.simulation.inverter + and not isinstance(getattr(self, "_terminal_value_curve", None), TailValueCurve) and self.simulation.ac_charge_hours is not None and self.simulation.elect_price_hourly is not None and self.simulation.load_energy_array is not None @@ -904,11 +3042,21 @@ class GeneticOptimization(OptimizationBase): * inv.dc_to_ac_efficiency ) - if round_trip_eff > 0: + # Configurable penalty multiplier (default 1 = economic loss in €) + try: + ac_penalty_factor = float( + self.config.optimization.genetic.penalties["ac_charge_break_even"] + ) + except Exception: + ac_penalty_factor = 1.0 + + # A factor of 0 multiplies every penalty term to zero - skip the + # whole computation in that case. + if round_trip_eff > 0 and ac_penalty_factor != 0.0: ac_charge_arr = self.simulation.ac_charge_hours prices_arr = self.simulation.elect_price_hourly load_arr = self.simulation.load_energy_array - n = len(prices_arr) + n = min(len(prices_arr), self.control_end_slot) # Usable AC energy already in battery from prior PV charging (zero grid cost). # This covers the most expensive future hours first, pushing AC charging demand @@ -920,13 +3068,13 @@ class GeneticOptimization(OptimizationBase): * inv.dc_to_ac_efficiency ) - # Configurable penalty multiplier (default 1 = economic loss in currency units) - try: - ac_penalty_factor = float( - self.config.optimization.genetic.penalties["ac_charge_break_even"] - ) - except Exception: - ac_penalty_factor = 1.0 + # Prices/loads/free energy are constant within one optimization + # run - compute the break-even lookup once, reuse it for every + # individual (cache is reset per run in optimize_ems()). + best_prices = getattr(self, "_ac_break_even_best_prices", None) + if best_prices is None: + best_prices = self._ac_break_even_prices(prices_arr, load_arr, free_ac_wh) + self._ac_break_even_best_prices = best_prices for hour in range(start_hour, min(len(ac_charge_arr), n)): ac_factor = ac_charge_arr[hour] @@ -937,75 +3085,68 @@ class GeneticOptimization(OptimizationBase): if charge_price <= 0: continue - # Price that a future discharge hour must reach to break even - break_even_price = charge_price / round_trip_eff + # Price that a future AC discharge hour must reach to break + # even. LCOS is defined per DC Wh delivered by the battery; + # dividing it by DC-to-AC efficiency converts it to the + # corresponding cost per useful/exported AC Wh. + lcos_per_wh_dc = getattr(bat, "levelized_cost_of_storage_kwh", 0.0) / 1000.0 + break_even_price = ( + charge_price / round_trip_eff + lcos_per_wh_dc / inv.dc_to_ac_efficiency + ) - # Build list of (price, load_wh) for all future hours in the horizon - future = [ - (float(prices_arr[h]), float(load_arr[h])) for h in range(hour + 1, n) - ] - # Sort descending by price so we "use" the most expensive hours first - future.sort(key=lambda x: -x[0]) - - # Consume free PV energy against the highest-price future hours. - # The first uncovered (partially or fully) hour defines the best - # price still available for the new AC charge. - remaining_free = free_ac_wh - best_uncovered_price = 0.0 - for fp, fl in future: - if remaining_free >= fl: - # Entire expensive hour is already covered by free PV energy - remaining_free -= fl - else: - # First hour not (fully) covered: this is where new charge goes - best_uncovered_price = fp - break + best_uncovered_price = best_prices[hour] if best_uncovered_price < break_even_price: # AC charging at this hour is economically unjustified. - # Penalty = excess cost per Wh × DC energy requested this hour. - dc_wh = bat.max_charge_power_w * ac_factor + # Penalty = excess cost per Wh × DC energy requested this slot. + # max_charge_power_w is a power [W]; the energy movable in + # one slot is power × slot_duration_h (¼ at 15 min). + dc_wh = bat.max_charge_power_w * self.slot_duration_h * ac_factor ac_wh = dc_wh / max(inv.ac_to_dc_efficiency, 1e-9) excess_cost_per_wh = break_even_price - best_uncovered_price - total_balance += ac_wh * excess_cost_per_wh * ac_penalty_factor + gesamtbilanz += ac_wh * excess_cost_per_wh * ac_penalty_factor if self.optimize_ev and parameters.ev and self.simulation.ev: try: penalty = self.config.optimization.genetic.penalties["ev_soc_miss"] - except Exception: + except: # Use default penalty = 10 logger.error( "Penalty function parameter `ev_soc_miss` not configured, using {}.", penalty ) - ev_soc_percentage = self.simulation.ev.current_soc_percentage() - if ( - ev_soc_percentage < parameters.ev.min_soc_percentage - or ev_soc_percentage > parameters.ev.max_soc_percentage - ): - total_balance += abs(parameters.ev.min_soc_percentage - ev_soc_percentage) * penalty + ev_soc_percentage = self._ev_soc_at_deadline(simulation_result, start_hour) + if ev_soc_percentage < parameters.ev.min_soc_percentage: + gesamtbilanz += abs(parameters.ev.min_soc_percentage - ev_soc_percentage) * penalty - return (total_balance,) + return (gesamtbilanz,) def optimize( self, start_solution: Optional[list[float]] = None, ngen: int = 200, + individuals: Optional[int] = None, ) -> tuple[Any, dict[str, list[Any]]]: """Run the optimization process using a genetic algorithm. @TODO: optimize() ngen default (200) is different from optimize_ems() ngen default (400). """ + # Re-seed at the actual optimization boundary. Setup and validation may + # consume random values elsewhere in a long-running process; a fixed seed + # must nevertheless produce the same population and result. + if self.fix_seed is not None: + random.seed(self.fix_seed) + # Set the number of inviduals in a generation - try: - individuals = self.config.optimization.genetic.individuals - if individuals is None: - raise - except Exception: - individuals = 300 - logger.error("Individuals not configured. Using {}.", individuals) + if individuals is None: + try: + individuals = self.config.optimization.genetic.individuals + if individuals is None: + raise ValueError("individuals is not configured") + except Exception: + individuals = 300 + logger.error("Individuals not configured. Using {}.", individuals) - population = self.toolbox.population(n=individuals) hof = tools.HallOfFame(1) stats = tools.Statistics(lambda ind: ind.fitness.values) stats.register("min", np.min) @@ -1014,23 +3155,152 @@ class GeneticOptimization(OptimizationBase): logger.debug("Start optimize: {}", start_solution) - # Insert the start solution into the population if provided + # Validate the warm start before assigning the fixed population budget. + valid_start_solution: Optional[list[float]] = None if start_solution is not None: - for _ in range(10): - population.insert(0, creator.Individual(start_solution)) + n_appliance_genes = self.appliance_layout.n_genes + expected_length = ( + self.control_end_slot * (2 if self.optimize_ev else 1) + n_appliance_genes + ) + start_solution = self._start_solution_for_slot_grid(start_solution) - # Run the evolutionary algorithm - pop, log = algorithms.eaMuPlusLambda( - population, - self.toolbox, - mu=100, - lambda_=150, - cxpb=0.6, - mutpb=0.4, - ngen=ngen, - stats=stats, - halloffame=hof, - verbose=self.verbose, + if len(start_solution) != expected_length: + logger.warning( + "Ignoring start_solution with incompatible length {} (expected {}).", + len(start_solution), + expected_length, + ) + elif not self._start_solution_matches_layout(start_solution): + logger.warning( + "Ignoring start_solution: appliance genes do not match the current " + "appliance layout." + ) + else: + valid_start_solution = start_solution + + # Scale the seed families with small populations without changing the + # established 300-individual defaults. This prevents a 100-member run + # from spending 60% of its budget on the warm-start neighbourhood. + exact_warm_target = min( + self.WARM_START_COPIES, + max(1, int(individuals * self.WARM_START_COPY_FRACTION + 0.999999)), + ) + warm_mutation_target = min( + self.WARM_START_MUTATIONS, + max(1, int(individuals * self.WARM_START_MUTATION_FRACTION + 0.999999)), + ) + educated_guess_target = min( + self.EDUCATED_GUESS_TARGET, + max(1, int(individuals * self.EDUCATED_GUESS_FRACTION + 0.999999)), + ) + minimum_random = max( + int(individuals * self.MIN_RANDOM_POPULATION_FRACTION + 0.999999), + individuals - (exact_warm_target + warm_mutation_target + educated_guess_target), + ) + seed_budget = max(individuals - minimum_random, 0) + seeded: list[list[Any]] = [] + + exact_warm_count = 0 + warm_neighbors: list[list[int]] = [] + if valid_start_solution is not None and seed_budget > 0: + exact_warm_count = min(exact_warm_target, seed_budget) + seeded.extend([valid_start_solution] * exact_warm_count) + remaining_seed_budget = seed_budget - len(seeded) + warm_neighbors = self._mutated_warm_start_neighbors( + valid_start_solution, + min(warm_mutation_target, remaining_seed_budget), + ) + seeded.extend(warm_neighbors) + + remaining_seed_budget = seed_budget - len(seeded) + educated_guesses = self._educated_guess_individuals( + min(educated_guess_target, remaining_seed_budget) + ) + seeded.extend(educated_guesses) + + random_count = max(individuals - len(seeded), 0) + population = [creator.Individual(seed) for seed in seeded] + population.extend(self.toolbox.population(n=random_count)) + logger.info( + "Genetic settings: {} individuals, {} generations, {} survivors, " + "{} offspring per generation, adaptive mutation {:.0%}/{:.0%}.", + individuals, + ngen, + individuals, + individuals, + self.MUTATION_PROBABILITY, + self.STAGNATION_MUTATION_PROBABILITY, + ) + logger.info( + "Initial population {}: {} exact warm starts, {} warm mutations, " + "{} educated guesses, {} random candidates.", + len(population), + exact_warm_count, + len(warm_neighbors), + len(educated_guesses), + random_count, + ) + + # The memoization scope is exactly one optimizer invocation. Always turn + # it off again, including when DEAP raises, so no later caller can reuse + # results under changed forecasts or device state. + self._fitness_cache.clear() + self._fitness_cache_hits = 0 + self._fitness_cache_misses = 0 + self._fitness_cache_enabled = True + local_evaluations = 0 + local_improvements = 0 + local_initial_fitness = float("nan") + local_final_fitness = float("nan") + self._adaptive_evolution_metrics = {} + try: + pop, log = self._evolve_population_adaptive( + population, + mu=individuals, + lambda_=individuals, + ngen=ngen, + stats=stats, + halloffame=hof, + ) + population = pop + ( + best_solution, + local_evaluations, + local_improvements, + local_initial_fitness, + local_final_fitness, + ) = self._locally_improve_grid_export( + hof[0], + max_evaluations=min( + self.LOCAL_SEARCH_MAX_EVALUATIONS, + max(individuals, 1), + ), + ) + except Exception: + self._fitness_cache.clear() + raise + finally: + self._fitness_cache_enabled = False + + if local_improvements: + logger.info( + "Grid-export local search: {} improvements in {} evaluations, " + "fitness {:.6f} -> {:.6f}.", + local_improvements, + local_evaluations, + local_initial_fitness, + local_final_fitness, + ) + + cache_lookups = self._fitness_cache_hits + self._fitness_cache_misses + cache_hit_rate = self._fitness_cache_hits / cache_lookups if cache_lookups > 0 else 0.0 + cache_keys = len(self._fitness_cache) + logger.info( + "Fitness cache: {} hits, {} misses, {:.1%} hit rate, {} keys.", + self._fitness_cache_hits, + self._fitness_cache_misses, + cache_hit_rate, + cache_keys, ) # Store fitness history @@ -1039,17 +3309,37 @@ class GeneticOptimization(OptimizationBase): "avg": log.select("avg"), # Average fitness for each generation (Y-axis) "max": log.select("max"), # Maximum fitness for each generation (Y-axis) "min": log.select("min"), # Minimum fitness for each generation (Y-axis) + "diversity": log.select("diversity"), + "stagnation": log.select("stagnation"), + "immigrants": log.select("immigrants"), + "restart": log.select("restart"), + "fitness_cache": { + "hits": self._fitness_cache_hits, + "misses": self._fitness_cache_misses, + "hit_rate": cache_hit_rate, + "keys": cache_keys, + }, + "adaptive_evolution": self._adaptive_evolution_metrics, + "local_search": { + "evaluations": local_evaluations, + "improvements": local_improvements, + "initial_fitness": local_initial_fitness, + "final_fitness": local_final_fitness, + }, } - member: dict[str, list[float]] = {"balance": [], "losses": [], "constraints": []} + member: dict[str, list[float]] = {"bilanz": [], "verluste": [], "nebenbedingung": []} for ind in population: if hasattr(ind, "extra_data"): extra_value1, extra_value2, extra_value3 = ind.extra_data - member["balance"].append(extra_value1) - member["losses"].append(extra_value2) - member["constraints"].append(extra_value3) + member["bilanz"].append(extra_value1) + member["verluste"].append(extra_value2) + member["nebenbedingung"].append(extra_value3) - return hof[0], member + # Avoid retaining large genome tuples in a long-lived API process until + # cyclic garbage collection happens. Cache statistics above are scalar. + self._fitness_cache.clear() + return best_solution, member def optimize_ems( self, @@ -1057,8 +3347,19 @@ class GeneticOptimization(OptimizationBase): start_hour: Optional[int] = None, worst_case: bool = False, ngen: Optional[int] = None, + individuals: Optional[int] = None, ) -> GeneticSolution: """Perform EMS (Energy Management System) optimization and visualize results.""" + self.config.validate_optimization_horizons() + direct_marketing_enabled = self._direct_marketing_enabled() + parameters = self._parameters_for_config(parameters) + parameters = self._parameters_for_slot_grid(parameters) + # Home-appliance scheduling now supports sub-hourly intervals via the + # energy-preserving per-slot run profile. + home_appliance_params = parameters.resolved_home_appliances() + self.optimize_dc_charge = direct_marketing_enabled + self.optimize_battery_grid_export = direct_marketing_enabled + if start_hour is None: start_hour = self.ems.start_datetime.hour # Start hour has to be in sync with energy management @@ -1066,48 +3367,57 @@ class GeneticOptimization(OptimizationBase): raise ValueError( f"Start hour not synced. EMS {self.ems.start_datetime.hour} vs. GENETIC {start_hour}." ) + # Forecasts are trimmed to now; all genome/device indices are run-relative. + start_slot = self._control_start_slot() # Set the number of generations generations = ngen if generations is None: try: generations = self.config.optimization.genetic.generations - except Exception: + except: generations = 400 logger.error("Generations not configured. Using {}.", generations) self.simulation.reset() + # Prices/loads/initial SoC may differ from the previous run - the + # break-even lookup must be rebuilt lazily on first evaluation. + self._ac_break_even_best_prices = None - # Initialize PV and EV batteries - battery: Optional[Battery] = None + # Initialize PV and EV batteries. slot_duration_h lets the Battery scale + # its power caps (max_charge_power_w) to a per-slot energy cap. + akku: Optional[Battery] = None if parameters.pv_battery: - battery = Battery( + akku = Battery( parameters.pv_battery, - prediction_hours=self.config.prediction.hours, + prediction_hours=self.control_end_slot, + slot_duration_h=self.slot_duration_h, ) - battery.set_charge_per_hour(np.full(self.config.prediction.hours, 0)) + akku.set_charge_per_hour(np.full(self.control_end_slot, 0)) - ev: Optional[Battery] = None + eauto: Optional[Battery] = None if parameters.ev: - ev = Battery( + eauto = Battery( parameters.ev, - prediction_hours=self.config.prediction.hours, + prediction_hours=self.control_end_slot, + slot_duration_h=self.slot_duration_h, ) - ev.set_charge_per_hour(np.full(self.config.prediction.hours, 1)) + eauto.set_charge_per_hour(np.full(self.control_end_slot, 1)) self.optimize_ev = ( - parameters.ev.min_soc_percentage - parameters.ev.initial_soc_percentage >= 0 + parameters.ev.min_soc_percentage > parameters.ev.initial_soc_percentage ) # electrical vehicle charge rates if parameters.ev.charge_rates is not None: self.ev_possible_charge_values = parameters.ev.charge_rates elif ( self.config.devices.electric_vehicles - and len(self.config.devices.electric_vehicles) > 0 - and list(self.config.devices.electric_vehicles.values())[0].charge_rates is not None + and next(iter(self.config.devices.electric_vehicles.values())) + and next(iter(self.config.devices.electric_vehicles.values())).charge_rates + is not None ): - self.ev_possible_charge_values = list( - self.config.devices.electric_vehicles.values() - )[0].charge_rates + self.ev_possible_charge_values = next( + iter(self.config.devices.electric_vehicles.values()) + ).charge_rates else: warning_msg = "No charge rates provided for electric vehicle - using default." logger.warning(warning_msg) @@ -1136,105 +3446,220 @@ class GeneticOptimization(OptimizationBase): ] or [1.0] elif ( self.config.devices.batteries - and len(self.config.devices.batteries) > 0 - and list(self.config.devices.batteries.values())[0].charge_rates + and next(iter(self.config.devices.batteries.values())) + and next(iter(self.config.devices.batteries.values())).charge_rates ): self.bat_possible_charge_values = [ - r for r in list(self.config.devices.batteries.values())[0].charge_rates if r > 0.0 + r + for r in next(iter(self.config.devices.batteries.values())).charge_rates + if r > 0.0 ] or [1.0] else: self.bat_possible_charge_values = [1.0] logger.debug("Battery AC charge levels: {}", self.bat_possible_charge_values) - # Initialize household appliance if applicable - dishwasher = ( - HomeAppliance( - parameters=parameters.dishwasher, - optimization_hours=self.config.optimization.genetic.horizon_hours, - prediction_hours=self.config.prediction.hours, + # Battery-to-grid export levels (direct marketing only). Same resolution + # order as the charge rates: request parameters win over the configured + # battery, and the fallback is the previous all-or-nothing export. + export_rates: Optional[list[float]] = None + if parameters.pv_battery and parameters.pv_battery.grid_export_rates: + export_rates = list(parameters.pv_battery.grid_export_rates) + elif ( + self.config.devices.batteries + and next(iter(self.config.devices.batteries.values())) + and next(iter(self.config.devices.batteries.values())).grid_export_rates is not None + ): + export_rates = list( + next(iter(self.config.devices.batteries.values())).grid_export_rates ) - if parameters.dishwasher is not None - else None - ) + # Highest rate first so the full-power state keeps the lowest index and + # every heuristic that seeds "export here" keeps seeding full power. + self.bat_possible_grid_export_values = sorted( + (rate for rate in (export_rates or [1.0]) if rate > 0.0), reverse=True + ) or [1.0] + if self.optimize_battery_grid_export: + logger.debug("Battery grid export levels: {}", self.bat_possible_grid_export_values) - # Initialize the inverter and energy management system + # Initialize the flexible consumers (home appliances) and their genome + # layout. slot0_datetime (the run start) turns decoded start + # slots into absolute local timestamps and drives DAILY day grouping. + self._slot0_datetime = self.ems.start_datetime + home_appliances = [ + HomeAppliance( + parameters=appliance_params, + optimization_hours=self.config.optimization.genetic.horizon_hours, + prediction_hours=self.control_end_slot, + slot_duration_h=self.slot_duration_h, + ) + for appliance_params in home_appliance_params + ] + self.appliance_layout = self._build_appliance_layout(home_appliances, self._slot0_datetime) + + # EV charging deadline (departure). Resolved once per run; the seeding + # heuristic and the SoC penalty both read it. + self._ev_soc_deadline_slot = self._ev_deadline_slot(parameters) + if self._ev_soc_deadline_slot is not None: + logger.debug( + "EV target SoC required by slot {} ({}).", + self._ev_soc_deadline_slot, + self._slot0_datetime.add( + seconds=self._ev_soc_deadline_slot * self.slot_duration_h * 3600 + ), + ) + + # Initialize the inverter and energy management system. slot_duration_h + # lets the Inverter scale max_power_wh to a per-slot energy cap. inverter: Optional[Inverter] = None if parameters.inverter: inverter = Inverter( parameters.inverter, - battery=battery, + battery=akku, + slot_duration_h=self.slot_duration_h, ) # Prepare device simulation self.simulation.prepare( parameters=parameters.ems, optimization_hours=self.config.optimization.genetic.horizon_hours, - prediction_hours=self.config.prediction.hours, + prediction_hours=self.control_end_slot, inverter=inverter, # battery is part of inverter - ev=ev, - home_appliance=dishwasher, + ev=eauto, + home_appliances=home_appliances, + direct_marketing_enabled=direct_marketing_enabled, ) - # Setup the DEAP environment and optimization process - self.setup_deap_environment({"home_appliance": 1 if dishwasher else 0}, start_hour) + self._validate_forecast_availability() + + # Terminal value of the energy left in the battery. Built once per run - + # it needs the prepared price/load/PV series - so every fitness + # evaluation only interpolates on it. + self._terminal_value_curve = self._build_terminal_value_curve(akku, inverter) + + # The curve owns all lookahead information from here on. Simulation + # and control heuristics receive only equally sized control forecasts, + # even when providers supplied different amounts of tail data. + for name in ( + "load_energy_array", + "pv_prediction_wh", + "elect_price_hourly", + "elect_revenue_per_hour_arr", + ): + setattr(self.simulation, name, getattr(self.simulation, name)[: self.control_slots]) + + # Setup the DEAP environment and optimization process. The appliance + # genome layout (built above) drives the appliance gene block; evaluate + # gets the slot index (its break-even loop walks the slot arrays from "now"). + self.setup_deap_environment({"home_appliance": self.appliance_layout.n_genes}, start_hour) self.toolbox.register( "evaluate", - lambda ind: self.evaluate(ind, parameters, start_hour, worst_case), + lambda ind: self.evaluate(ind, parameters, start_slot, worst_case), ) start_time = time.time() - start_solution, extra_data = self.optimize(parameters.start_solution, ngen=generations) + start_solution_datetime = self._resolve_start_solution_datetime(parameters) + start_solution, extra_data = self.optimize( + self._start_solution_for_run_start(parameters.start_solution, start_solution_datetime), + ngen=generations, + individuals=individuals, + ) elapsed_time = time.time() - start_time logger.debug(f"Time evaluate inner: {elapsed_time:.4f} sec.") # Perform final evaluation on the best solution simulation_result = self.evaluate_inner(start_solution) + # Read the terminal value off the final battery state, for the solution. + _, terminal_value_result = self._terminal_value(parameters, include_tail_plan=True) # Prepare results - discharge_hours_bin, ev_charge_hours_index, washingstart_int = self.split_individual( + discharge_hours_bin, eautocharge_hours_index, appliance_gene_values = self.split_individual( start_solution ) - # home appliance may have choosen a different appliance start hour - if self.simulation.home_appliance: - washingstart_int = self.simulation.home_appliance_start_hour - ev_charge_hours_float = ( - [self.ev_possible_charge_values[i] for i in ev_charge_hours_index] - if ev_charge_hours_index is not None - else None - ) + # Materialize the per-device appliance results only for the final best + # solution. Each appliance's load curve (already built by the final + # evaluate_inner above) starts at slot 0; slice it to the simulation + # window so it aligns with the other per-slot result arrays. + starts_per_appliance = self._decode_appliance_starts(appliance_gene_values) + home_appliance_energy_wh: dict[str, list[float]] = {} + home_appliance_running: dict[str, list[bool]] = {} + appliance_starts: dict[str, list[Any]] = {} + appliance_deadline_missed: dict[str, bool] = {} + timezone = self.config.general.timezone + for appliance_index, appliance in enumerate(self.simulation.home_appliances): + device_id = appliance.device_id + home_appliance_energy_wh[device_id] = appliance.get_load_curve()[start_slot:].tolist() + starts = sorted(starts_per_appliance.get(appliance_index, [])) + home_appliance_running[device_id] = [ + any(run_start <= slot < run_start + appliance.run_slots for run_start in starts) + for slot in range(self.control_slots) + ] + appliance_starts[device_id] = [ + self._slot0_datetime.add( + seconds=start * appliance.slot_interval_seconds + ).in_timezone(timezone) + for start in starts + ] + # Report a deadline that could not be kept (no run scheduled at all, + # or a run that ends late because a BEST_EFFORT deadline was dropped) + # so the caller can warn instead of silently trusting the schedule. + if appliance.deadline_datetime is not None: + appliance_deadline_missed[device_id] = appliance.deadline_missed( + starts, self._slot0_datetime + ) + simulation_result["home_appliance_energy_wh"] = home_appliance_energy_wh + simulation_result["home_appliance_running"] = home_appliance_running - # Simulation may have changed something, use simulation values - ac_charge_hours = ( - self.simulation.ac_charge_hours.tolist() - if self.simulation.ac_charge_hours is not None - else [] - ) - dc_charge_hours = ( - self.simulation.dc_charge_hours.tolist() - if self.simulation.dc_charge_hours is not None - else [] - ) - discharge = ( - self.simulation.bat_discharge_hours.tolist() - if self.simulation.bat_discharge_hours is not None + # Deprecated single-device hourly start (kept for backward compatibility). + # Only meaningful for the legacy case: exactly one appliance on the hourly + # grid. Otherwise None; use appliance_starts instead. + washingstart_int: Optional[int] = None + if self.slots_per_hour == 1 and len(self.simulation.home_appliances) == 1: + single_starts = starts_per_appliance.get(0, []) + if single_starts: + washingstart_int = self._start_day_slot() + int(min(single_starts)) + + eautocharge_hours_float = None + if eautocharge_hours_index is not None and self.simulation.ev is not None: + eautocharge_hours_float = self.simulation.ev.charge_array.tolist() + + # Report executed controls, already indexed from the run start. + def control_values(values: Optional[np.ndarray]) -> list[float]: + return values.tolist() if values is not None else [] + + ac_charge_hours = control_values(self.simulation.ac_charge_hours) + dc_charge_hours = control_values(self.simulation.dc_charge_hours) + discharge = control_values(self.simulation.bat_discharge_hours) + battery_grid_export_factor = ( + control_values(self.simulation.bat_grid_export_hours) + if direct_marketing_enabled else [] ) + battery_grid_export = [1 if value > 0 else 0 for value in battery_grid_export_factor] return GeneticSolution( **{ "parameters": parameters, - "ac_charge": ac_charge_hours, - "dc_charge": dc_charge_hours, - "discharge_allowed": discharge, - "ev_charge_hours_float": ev_charge_hours_float, - "result": GeneticSimulationResult(**simulation_result), - "ev_obj": self.simulation.ev, + "interval_seconds": self.config.optimization.genetic.interval_sec, "start_hour": start_hour, - "start_solution": start_solution, - "washingstart": washingstart_int, "extra_data": extra_data, "fitness_history": self.fitness_history, "fixed_seed": self.fix_seed, + "controls_start_at_now": True, + "ac_charge": ac_charge_hours, + "dc_charge": dc_charge_hours, + "discharge_allowed": discharge, + "battery_grid_export_allowed": battery_grid_export, + "battery_grid_export_factor": battery_grid_export_factor, + "terminal_value": terminal_value_result, + "ev_charge_hours_float": eautocharge_hours_float[start_slot:] + if eautocharge_hours_float is not None + else None, + "result": GeneticSimulationResult(**simulation_result), + "ev_obj": self.simulation.ev, + "start_solution": start_solution, + "start_solution_datetime": self._slot0_datetime, + "washingstart": washingstart_int, + "appliance_starts": appliance_starts, + "appliance_deadline_missed": appliance_deadline_missed, } ) diff --git a/src/akkudoktoreos/optimization/genetic/geneticparams.py b/src/akkudoktoreos/optimization/genetic/geneticparams.py index a6384634..550c1c77 100644 --- a/src/akkudoktoreos/optimization/genetic/geneticparams.py +++ b/src/akkudoktoreos/optimization/genetic/geneticparams.py @@ -8,9 +8,9 @@ It also provides a method to assemble these parameters from predictions, forecasts, and fallback defaults, preparing them for optimization runs. """ -from typing import Optional, Union +from datetime import datetime +from typing import Any, Optional, Union -import numpy as np from loguru import logger from pydantic import ( AliasChoices, @@ -26,7 +26,6 @@ from akkudoktoreos.core.coreabc import ( ConfigMixin, MeasurementMixin, PredictionMixin, - get_ems, ) from akkudoktoreos.devices.genetic.battery import ( ElectricVehicleParameters, @@ -35,7 +34,7 @@ from akkudoktoreos.devices.genetic.battery import ( from akkudoktoreos.devices.genetic.homeappliance import HomeApplianceParameters from akkudoktoreos.devices.genetic.inverter import InverterParameters from akkudoktoreos.optimization.genetic.geneticabc import GeneticParametersBaseModel -from akkudoktoreos.utils.datetimeutil import to_duration +from akkudoktoreos.utils.datetimeutil import DateTime, to_datetime # Do not import directly from akkudoktoreos.core.coreabc # EnergyManagementSystemMixin - Creates circular dependency with ems.py @@ -50,7 +49,7 @@ class GeneticEnergyManagementParameters(GeneticParametersBaseModel): pv_forecast_wh: list[float] = Field( validation_alias=AliasChoices("pv_forecast_wh", "pv_prognose_wh"), json_schema_extra={ - "description": "An array of floats representing the forecasted photovoltaic output in watts for different time intervals." + "description": "An array of floats representing the forecasted photovoltaic energy in watt-hours per slot for different time intervals." }, ) electricity_price_per_wh: list[float] = Field( @@ -74,7 +73,7 @@ class GeneticEnergyManagementParameters(GeneticParametersBaseModel): total_load: list[float] = Field( validation_alias=AliasChoices("total_load", "gesamtlast"), json_schema_extra={ - "description": "An array of floats representing the total load (consumption) in watts for different time intervals." + "description": "An array of floats representing the total load (consumption) in watt-hours per slot for different time intervals." }, ) @@ -105,22 +104,13 @@ class GeneticEnergyManagementParameters(GeneticParametersBaseModel): return self.total_load @model_validator(mode="after") - def validate_list_length(self) -> Self: - """Validate that all input lists are of the same length. - - Raises: - ValueError: If input list lengths differ. - """ - pv_forecast_length = len(self.pv_forecast_wh) - if ( - pv_forecast_length != len(self.electricity_price_per_wh) - or pv_forecast_length != len(self.total_load) - or ( - isinstance(self.feed_in_tariff_per_wh, list) - and pv_forecast_length != len(self.feed_in_tariff_per_wh) - ) - ): - raise ValueError("Input lists have different lengths") + def validate_forecast_arrays(self) -> Self: + """Require forecasts; the optimizer validates control coverage and clips the tail.""" + arrays = [self.pv_forecast_wh, self.electricity_price_per_wh, self.total_load] + if isinstance(self.feed_in_tariff_per_wh, list): + arrays.append(self.feed_in_tariff_per_wh) + if any(not values for values in arrays): + raise ValueError("Forecast arrays must not be empty.") return self @@ -164,6 +154,91 @@ class GeneticOptimizationParameters( }, ) + forecast_interval_seconds: Optional[int] = Field( + default=None, + description="Input interval: 3600 for hourly, or optimization interval for native slots.", + ) + + home_appliances: Optional[list[HomeApplianceParameters]] = Field( + default=None, + json_schema_extra={ + "description": "List of flexible consumers (home appliances) to schedule." + }, + ) + + start_solution_datetime: Optional[DateTime] = Field( + default=None, + json_schema_extra={ + "description": ( + "Start of the slot that gene 0 of 'start_solution' controls, as " + "returned with the previous solution. The warm start is shifted by " + "the slots that have elapsed until this run. Without it, a " + "'start_solution' identical to the last solution of this server " + "uses that solution's start; any other one is used unshifted." + ), + "examples": [None, "2026-09-14T07:45:00+02:00"], + }, + ) + + @field_validator("forecast_interval_seconds") + @classmethod + def validate_forecast_interval(cls, value: Optional[int]) -> Optional[int]: + if value not in (None, 900, 3600): + raise ValueError("forecast_interval_seconds must be 900 or 3600") + return value + + @field_validator("start_solution_datetime", mode="before") + @classmethod + def transform_start_solution_datetime(cls, value: Any) -> Optional[DateTime]: + """Accept the usual date time representations, naive input is local time.""" + if value is None: + return None + # A stored run must retain its timezone/fold even when the server host + # uses another local timezone. ISO offsets survive JSON round trips. + aware = value + if isinstance(value, str): + try: + aware = datetime.fromisoformat(value) + except ValueError: + pass + if isinstance(aware, datetime) and aware.tzinfo is not None: + return DateTime.instance(aware) + return to_datetime(value) + + @model_validator(mode="after") + def validate_home_appliances(self) -> Self: + """Reject conflicting home appliance definitions. + + The deprecated ``dishwasher`` field and the new ``home_appliances`` list + must not be set at the same time; nothing is silently overwritten. + Device ids within ``home_appliances`` must be unique. + """ + # Read the deprecated field via __dict__ to avoid emitting a deprecation + # warning on every internal validation. + dishwasher = self.__dict__.get("dishwasher") + if dishwasher is not None and self.home_appliances is not None: + raise ValueError( + "Provide either 'home_appliances' or the deprecated 'dishwasher', not both." + ) + appliances = self.home_appliances or [] + device_ids = [appliance.device_id for appliance in appliances] + if len(device_ids) != len(set(device_ids)): + raise ValueError("home_appliances device_id values must be unique.") + return self + + def resolved_home_appliances(self) -> list[HomeApplianceParameters]: + """Return the effective home appliance list. + + Maps the deprecated single ``dishwasher`` onto a one-element list so the + optimizer only ever deals with the list form. + """ + if self.home_appliances is not None: + return list(self.home_appliances) + dishwasher = self.__dict__.get("dishwasher") + if dishwasher is not None: + return [dishwasher] + return [] + # Computed fields for backward compatibility (deprecated German names) @computed_field(json_schema_extra={"deprecated": True}) def pv_akku(self) -> Optional[SolarPanelBatteryParameters]: @@ -208,620 +283,21 @@ class GeneticOptimizationParameters( @classmethod async def prepare(cls) -> "Optional[GeneticOptimizationParameters]": - """Prepare optimization parameters from config, forecast and measurement data. + """Resolve configured devices and fresh forecasts for automatic optimization. - Fills in values needed for optimization from available configuration, predictions and - measurements. If some data is missing, default or demo values are used. - - Parameters start by definition of the genetic algorithm at hour 0 of the actual date - (not at start datetime of energy management run) - - Returns: - GeneticOptimizationParameters: The fully prepared optimization parameters. - - Raises: - ValueError: If required configuration values like start time are missing. + Missing inputs cancel the run without changing providers or inventing devices. + The same resolver serves the configuration-owned HTTP request. """ - ems = get_ems() + from akkudoktoreos.optimization.genetic.configrequest import ( + ConfigOptimizationRequest, + ) - # The optimization paramters - oparams: "Optional[GeneticOptimizationParameters]" = None - - # Check for run definitions - if ems.start_datetime is None: - error_msg = "Start datetime unknown." - logger.error(error_msg) - raise ValueError(error_msg) - # Check for general predictions conditions - if cls.config.general.latitude is None: - default_latitude = 52.52 - logger.info(f"Latitude unknown - defaulting to {default_latitude}.") - cls.config.general.latitude = default_latitude - if cls.config.general.longitude is None: - default_longitude = 13.405 - logger.info(f"Longitude unknown - defaulting to {default_longitude}.") - cls.config.general.longitude = default_longitude - if cls.config.prediction.hours is None: - logger.info("Prediction hours unknown - defaulting to 48 hours.") - cls.config.prediction.hours = 48 - if cls.config.prediction.historic_hours is None: - logger.info("Prediction historic hours unknown - defaulting to 24 hours.") - cls.config.prediction.historic_hours = 24 - # Check optimization definitions - if cls.config.optimization.genetic.horizon_hours is None: - logger.info("Optimization horizon unknown - defaulting to 24 hours.") - cls.config.optimization.genetic.horizon_hours = 24 - if cls.config.optimization.genetic.interval_sec is None: - logger.info("Optimization interval unknown - defaulting to 3600 seconds.") - cls.config.optimization.genetic.interval_sec = 3600 - if cls.config.optimization.genetic.interval_sec != 3600: - logger.info( - f"Optimization interval '{cls.config.optimization.genetic.interval_sec}' seconds " - "not supported - forced to 3600 seconds." + try: + return await ConfigOptimizationRequest().resolve() + except Exception as exc: + logger.error( + "Cannot prepare GENETIC parameters; canceling optimization with provider {}: {}", + cls.config.feedintariff.provider, + exc, ) - cls.config.optimization.genetic.interval_sec = 3600 - # Check genetic algorithm definitions - if cls.config.optimization.genetic.individuals is None: - logger.info("Genetic individuals unknown - defaulting to 300.") - cls.config.optimization.genetic.individuals = 300 - if cls.config.optimization.genetic.generations is None: - logger.info("Genetic generations unknown - defaulting to 400.") - cls.config.optimization.genetic.generations = 400 - if "ev_soc_miss" not in cls.config.optimization.genetic.penalties: - logger.info("Genetic penalties unknown - defaulting to ev_soc_miss = 10.") - cls.config.optimization.genetic.penalties["ev_soc_miss"] = 10 - # Setup some basic providers if not set - if not cls.config.weather.provider: - cls.config.weather.provider = "OpenMeteo" - if not cls.config.load.provider: - cls.config.load.provider = "LoadAkkudoktor" - - # Get start solution from last run - start_solution = None - last_solution = ems.genetic_solution() - if last_solution and last_solution.start_solution: - start_solution = last_solution.start_solution - - # Add forecast and device data - interval = to_duration(cls.config.optimization.genetic.interval_sec) - power_to_energy_per_interval_factor = cls.config.optimization.genetic.interval_sec / 3600 - parameter_start_datetime = ems.start_datetime.set(hour=0, second=0, microsecond=0) - parameter_end_datetime = parameter_start_datetime.add(hours=cls.config.prediction.hours) - max_retries = 10 - - for attempt in range(1, max_retries + 1): - # Collect all the data for optimisation, but do not exceed max retries - if attempt > max_retries: - error_msg = f"Maximum retries {max_retries} for parameter collection exceeded. Parameter preparation attempt {attempt}." - logger.error(error_msg) - raise ValueError(error_msg) - - # Assure predictions are uptodate - await cls.prediction.update_data() - - try: # Try weather first - predition is also needed by the default PV forecast - array = await cls.prediction.key_to_array( - key="weather_temp_air", - start_datetime=parameter_start_datetime, - end_datetime=parameter_end_datetime, - interval=interval, - fill_method="ffill", - ) - weather_temp_air = array.tolist() - except Exception as e: - logger.info( - "No weather forecast data available - defaulting to demo data. Parameter preparation attempt {}: {}", - attempt, - e, - ) - cls.config.weather.provider = "OpenMeteo" - # Retry - continue - # Try electricity fees next - predition is also needed by the default electricity price - # If no provider is set the fees default to 0 anyway - if cls.config.elecfee.provider: - try: - array = await cls.prediction.key_to_array( - key="elecfee_consumption_amt_kwh", - start_datetime=parameter_start_datetime, - end_datetime=parameter_end_datetime, - interval=interval, - fill_method="ffill", - ) - except Exception as e: - logger.info( - "No electricity fee data available - defaulting to demo data. Parameter preparation attempt {}: {}", - attempt, - e, - ) - cls.config.merge_settings_from_dict( - { - "elecfee": { - "provider": "ElecFeeFixed", - "elecfeefixed": { - "consumption_amt_kwh": { - "windows": [ - { - "start_time": "00:00", - "duration": "24 hours", - "value": 0.21, - }, - ] - }, - "consumption_percent_amt": { - "windows": [ - { - "start_time": "00:00", - "duration": "24 hours", - "value": 19.0, - }, - ] - }, - "feedin_amt_kwh": { - "windows": [ - { - "start_time": "00:00", - "duration": "24 hours", - "value": 0.0, - }, - ] - }, - "feedin_percent_amt": { - "windows": [ - { - "start_time": "00:00", - "duration": "24 hours", - "value": 0.0, - }, - ] - }, - }, - }, - } - ) - # Retry - continue - try: - array = await cls.prediction.key_to_array( - key="pvforecast_ac_power", - start_datetime=parameter_start_datetime, - end_datetime=parameter_end_datetime, - interval=interval, - fill_method="linear", - ) - pvforecast_ac_power = (array * power_to_energy_per_interval_factor).tolist() - except Exception as e: - logger.info( - "No PV forecast data available - defaulting to demo data. Parameter preparation attempt {}: {}", - attempt, - e, - ) - cls.config.merge_settings_from_dict( - { - "pvforecast": { - "provider": "PVForecastPVLib", - "max_planes": 4, - "planes": [ - { - "surface_tilt": 7, - "surface_azimuth": 170, - "userhorizon": [20, 27, 22, 20], - "peakpower": 5.0, - "module_model": "AXITEC_AC_410MH_144S", - "inverter_model": "Sungrow__SH25T", - "inverter_paco": 10000, - "modules_per_string": 12, - "strings_per_inverter": 1, - }, - { - "surface_tilt": 7, - "surface_azimuth": 90, - "userhorizon": [30, 30, 30, 50], - "peakpower": 4.8, - "module_model": "AXITEC_AC_410MH_144S", - "inverter_model": "Sungrow__SH25T", - "inverter_paco": 10000, - "modules_per_string": 12, - "strings_per_inverter": 1, - }, - { - "surface_tilt": 60, - "surface_azimuth": 140, - "userhorizon": [60, 30, 0, 30], - "peakpower": 1.4, - "module_model": "AXITEC_AC_410MH_144S", - "inverter_model": "Sungrow__SH25T", - "inverter_paco": 2000, - "modules_per_string": 5, - "strings_per_inverter": 1, - }, - { - "surface_tilt": 45, - "surface_azimuth": 185, - "userhorizon": [45, 25, 30, 60], - "peakpower": 1.6, - "module_model": "AXITEC_AC_410MH_144S", - "inverter_model": "Sungrow__SH25T", - "inverter_paco": 1400, - "modules_per_string": 4, - "strings_per_inverter": 1, - }, - ], - }, - } - ) - # Retry - continue - try: - array = await cls.prediction.key_to_array( - key="elecprice_marketprice_wh", - start_datetime=parameter_start_datetime, - end_datetime=parameter_end_datetime, - interval=interval, - fill_method="ffill", - ) - elecprice_marketprice_wh = array.tolist() - except Exception as e: - logger.info( - "No Electricity Marketprice forecast data available - defaulting to demo data. Parameter preparation attempt {}: {}", - attempt, - e, - ) - cls.config.merge_settings_from_dict( - { - "elecprice": { - "elecpricefixed": { - "elecprice_marketprice_amt_kwh": { - "windows": [ - { - "duration": "1 day", - "start_time": "00:00:00.000000", - "value": 0.288, - } - ] - } - }, - "provider": "ElecPriceFixed", - }, - }, - ) - # Retry - continue - try: - array = await cls.prediction.key_to_array( - key="loadforecast_power_w", - start_datetime=parameter_start_datetime, - end_datetime=parameter_end_datetime, - interval=interval, - fill_method="ffill", - ) - loadforecast_power_w = array.tolist() - except Exception as e: - logger.info( - "No Load forecast data available - defaulting to demo data. Parameter preparation attempt {}: {}", - attempt, - e, - ) - cls.config.merge_settings_from_dict( - { - "load": { - "provider": "LoadAkkudoktor", - "loadakkudoktor": { - "loadakkudoktor_year_energy_kwh": "3000", - }, - }, - } - ) - # Retry - continue - try: - array = await cls.prediction.key_to_array( - key="feed_in_tariff_wh", - start_datetime=parameter_start_datetime, - end_datetime=parameter_end_datetime, - interval=interval, - fill_method="ffill", - ) - # Prediction records already contain amount/Wh; only fixed-provider - # configuration is expressed in amount/kWh. Preserve signs and units. - if cls.config.feedintariff.provider == "FeedInTariffImport": - if array.ndim != 1 or len(array) != len(pvforecast_ac_power): - raise ValueError( - "Imported feed-in tariff length does not match the forecast horizon" - ) - if not np.isfinite(array).all(): - raise ValueError( - "Imported feed-in tariff contains missing or non-finite values" - ) - feed_in_tariff_wh = array.tolist() - except Exception as e: - if cls.config.feedintariff.provider == "FeedInTariffImport": - # An external EMS supplies the resolved sale revenue. Replacing - # missing imports with demo or purchase prices changes the economics. - logger.error( - "Cannot prepare GENETIC parameters: FeedInTariffImport revenue is " - "unavailable or invalid; keeping the configured provider and " - "canceling optimization: {}", - e, - ) - return None - logger.info( - "No feed in tariff forecast data available - defaulting to demo data. Parameter preparation attempt {}: {}", - attempt, - e, - ) - cls.config.merge_settings_from_dict( - { - "feedintariff": { - "provider": "FeedInTariffFixed", - "feedintarifffixed": { - "feed_in_tariff_amt_kwh": { - "windows": [ - { - "start_time": "00:00", - "duration": "24 hours", - "value": 0.078, - }, - ], - }, - }, - }, - } - ) - # Retry - continue - - # Add device data - - # Batteries - # --------- - if cls.config.devices.max_batteries is None: - logger.info("Number of battery devices not configured - defaulting to 1.") - cls.config.devices.max_batteries = 1 - if cls.config.devices.max_batteries == 0: - battery_params = None - battery_lcos_kwh = 0 - else: - if cls.config.devices.batteries is None: - logger.info("No battery device data available - defaulting to demo data.") - cls.config.devices.batteries = { - "battery1": { - "device_id": "battery1", - "capacity_wh": 8000, - }, - } - try: - # Take first battery - battery_config = list(cls.config.devices.batteries.values())[0] - battery_params = battery_config.to_genetic_pv_bat_param() - except Exception as e: - logger.info( - "No battery device data available - defaulting to demo data. Parameter preparation attempt {}: {}", - attempt, - e, - ) - cls.config.devices.batteries = { - "battery1": { - "device_id": "battery1", - "capacity_wh": 8000, - }, - } - # Retry - continue - # Levelized cost of ownership - if battery_config.levelized_cost_of_storage_amt_kwh is None: - logger.info( - "No battery device LCOS data available - defaulting to 0 [amount/kWh]. Parameter preparation attempt {}.", - attempt, - ) - battery_config.levelized_cost_of_storage_amt_kwh = 0 - battery_lcos_kwh = battery_config.levelized_cost_of_storage_amt_kwh - # Initial SOC - try: - initial_soc_factor = await cls.measurement.key_to_value( - key=battery_config.measurement_key_soc_factor, - target_datetime=ems.start_datetime, - time_window=to_duration(to_duration("48 hours")), - ) - if initial_soc_factor > 1.0 or initial_soc_factor < 0.0: - logger.error( - f"Invalid battery initial SoC factor {initial_soc_factor} - defaulting to 0.0." - ) - initial_soc_factor = 0.0 - # genetic parameter is 0..100 as int - initial_soc_percentage = int(initial_soc_factor * 100) - except Exception: - initial_soc_percentage = None - if initial_soc_percentage is None: - logger.info( - f"No battery device SoC data (measurement key = '{battery_config.measurement_key_soc_factor}') available - defaulting to 0." - ) - initial_soc_percentage = 0 - battery_params.initial_soc_percentage = initial_soc_percentage - - # Electric Vehicles - # ----------------- - if cls.config.devices.max_electric_vehicles is None: - logger.info("Number of electric_vehicle devices not configured - defaulting to 1.") - cls.config.devices.max_electric_vehicles = 1 - if cls.config.devices.max_electric_vehicles == 0: - electric_vehicle_params = None - else: - if cls.config.devices.electric_vehicles is None: - logger.info( - "No electric vehicle device data available - defaulting to demo data." - ) - cls.config.devices.max_electric_vehicles = 1 - cls.config.devices.electric_vehicles = { - "ev1": { - "device_id": "ev1", - "capacity_wh": 50000, - "charge_rates": [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], - "min_soc_percentage": 70, - }, - } - try: - # Take first electric_vehicle - electric_vehicle_config = list(cls.config.devices.electric_vehicles.values())[0] - electric_vehicle_params = electric_vehicle_config.to_genetic_ev_bat_param() - except Exception as e: - logger.info( - "No electric_vehicle device data available - defaulting to demo data. Parameter preparation attempt {}: {}", - attempt, - e, - ) - cls.config.devices.max_electric_vehicles = 1 - cls.config.devices.electric_vehicles = { - "ev1": { - "device_id": "ev1", - "capacity_wh": 50000, - "charge_rates": [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], - "min_soc_percentage": 70, - }, - } - # Retry - continue - # Initial SOC - try: - initial_soc_factor = await cls.measurement.key_to_value( - key=electric_vehicle_config.measurement_key_soc_factor, - target_datetime=ems.start_datetime, - time_window=to_duration(to_duration("48 hours")), - ) - if initial_soc_factor > 1.0 or initial_soc_factor < 0.0: - logger.error( - f"Invalid electric vehicle initial SoC factor {initial_soc_factor} - defaulting to 0.0." - ) - initial_soc_factor = 0.0 - # genetic parameter is 0..100 as int - initial_soc_percentage = int(initial_soc_factor * 100) - except Exception: - initial_soc_percentage = None - if initial_soc_percentage is None: - logger.info( - f"No electric vehicle device SoC data (measurement key = '{electric_vehicle_config.measurement_key_soc_factor}') available - defaulting to 0." - ) - initial_soc_percentage = 0 - electric_vehicle_params.initial_soc_percentage = initial_soc_percentage - - # Inverters - # --------- - if cls.config.devices.max_inverters is None: - logger.info("Number of inverter devices not configured - defaulting to 1.") - cls.config.devices.max_inverters = 1 - if cls.config.devices.max_inverters == 0: - inverter_params = None - else: - if cls.config.devices.inverters is None: - logger.info("No inverter device data available - defaulting to demo data.") - cls.config.devices.inverters = { - "inverter1": { - "device_id": "inverter1", - "max_power_w": 10000, - "battery_id": battery_config.device_id, - }, - } - try: - # Take first inverter - inverter_config = list(cls.config.devices.inverters.values())[0] - inverter_params = inverter_config.to_genetic_param() - except Exception as e: - logger.info( - "No inverter device data available - defaulting to demo data. Parameter preparation attempt {}: {}", - attempt, - e, - ) - cls.config.devices.inverters = { - "inverter1": { - "device_id": "inverter1", - "max_power_w": 10000, - "battery_id": battery_config.device_id, - }, - } - # Retry - continue - - # Home Appliances - # --------------- - if cls.config.devices.max_home_appliances is None: - logger.info("Number of home appliance devices not configured - defaulting to 1.") - cls.config.devices.max_home_appliances = 1 - if cls.config.devices.max_home_appliances == 0: - home_appliance_params = None - else: - home_appliance_params = None - if cls.config.devices.home_appliances is None: - logger.info( - "No home appliance device data available - defaulting to demo data." - ) - cls.config.devices.home_appliances = { - "dishwasher1": { - "device_id": "dishwasher1", - "consumption_wh": 2000, - "duration_h": 3.0, - "cycle_time_windows": { - "windows": [ - { - "start_time": "08:00", - "duration": "5 hours", - }, - { - "start_time": "15:00", - "duration": "3 hours", - }, - ], - }, - }, - } - try: - # Take first appliance - home_appliance_config = list(cls.config.devices.home_appliances.values())[0] - home_appliance_params = home_appliance_config.to_genetic_param() - except Exception as e: - logger.info( - "No home appliance device data available - defaulting to demo data. Parameter preparation attempt {}: {}", - attempt, - e, - ) - cls.config.devices.home_appliances = { - "dishwasher1": { - "device_id": "dishwasher1", - "consumption_wh": 2000, - "duration_h": 3.0, - "cycle_time_windows": None, - }, - } - # Retry - continue - - # We got all parameter data - try: - oparams = GeneticOptimizationParameters( - ems=GeneticEnergyManagementParameters( - pv_forecast_wh=pvforecast_ac_power, - electricity_price_per_wh=elecprice_marketprice_wh, - feed_in_tariff_per_wh=feed_in_tariff_wh, - total_load=loadforecast_power_w, - price_per_wh_battery=battery_lcos_kwh / 1000, - ), - temperature_forecast=weather_temp_air, - pv_battery=battery_params, - ev=electric_vehicle_params, - inverter=inverter_params, - dishwasher=home_appliance_params, - start_solution=start_solution, - ) - except Exception as e: - logger.info( - "Can not prepare optimization parameters - will retry. Parameter preparation attempt {}: {}", - attempt, - e, - ) - oparams = None - # Retry - continue - - # Parameters prepared - break - - return oparams + return None diff --git a/src/akkudoktoreos/optimization/genetic/geneticsettings.py b/src/akkudoktoreos/optimization/genetic/geneticsettings.py index 8c184346..9e9e600e 100644 --- a/src/akkudoktoreos/optimization/genetic/geneticsettings.py +++ b/src/akkudoktoreos/optimization/genetic/geneticsettings.py @@ -3,20 +3,69 @@ Kept in an extra module to avoid cyclic dependencies on package import. """ -from typing import Optional, Union +from enum import StrEnum +from typing import Any, Literal, Optional, Union from pydantic import Field, computed_field from akkudoktoreos.config.configabc import SettingsBaseModel +def normalize_genetic_settings(value: Any) -> Any: + """Preserve old feature settings in the current algorithm-specific namespace. + + Explicit nested values win. The classic flat configuration without GENETIC + markers remains available to the existing GENETIC0 migration. + """ + if not isinstance(value, dict): + return value + feature_keys = ( + "tail_horizon_hours", + "terminal_value_mode", + "terminal_value_euro_per_kwh", + "terminal_value_window_hours", + "measurement_max_age_seconds", + ) + is_genetic = value.get("algorithm") == "GENETIC" or any(key in value for key in feature_keys) + if not is_genetic or not any( + key in value for key in (*feature_keys, "interval", "horizon_hours") + ): + return value + result = dict(value) + nested = dict(result.get("genetic") or {}) + for key in (*feature_keys, "interval", "horizon_hours"): + if key in result: + nested.setdefault("interval_sec" if key == "interval" else key, result.pop(key)) + result["genetic"] = nested + return result + + +class TerminalValueMode(StrEnum): + """How the energy left in the battery at the end of the horizon is valued. + + Modes + ----- + - AUTO: + Solve the deterministic forecast tail and apply a conservative + continuation proxy at its end. Tail values may decrease with SOC when + empty capacity is valuable. With a zero tail, use the proxy directly. + + - FIXED: + Credit every stored kWh with the configured + ``terminal_value_euro_per_kwh`` (or ``preis_euro_pro_wh_akku`` of the + request). The historical behaviour; a value of 0 makes the optimizer + empty the battery towards the end of the horizon. + """ + + AUTO = "AUTO" + FIXED = "FIXED" + + class GeneticCommonSettings(SettingsBaseModel): """GENETIC Optimization Algorithm Configuration.""" - interval_sec: int = Field( + interval_sec: Literal[900, 3600] = Field( default=3600, - ge=15 * 60, - le=60 * 60, json_schema_extra={ "description": "The optimization interval [sec]. Defaults to 3600 seconds (1 hour)", "examples": [60 * 60, 15 * 60], @@ -59,6 +108,59 @@ class GeneticCommonSettings(SettingsBaseModel): }, ) + measurement_max_age_seconds: int = Field( + default=300, + gt=0, + description="Maximum age of SoC measurements for configuration-based optimization [s].", + ) + + tail_horizon_hours: int = Field( + default=48, + ge=0, + json_schema_extra={ + "description": "Forecast lookahead after the control horizon [h]. No tail commands are issued. Set 0 to disable." + }, + ) + + terminal_value_mode: TerminalValueMode = Field( + default=TerminalValueMode.AUTO, + json_schema_extra={ + "description": ( + "How to value the energy left in the battery at the end of the " + "control horizon. AUTO solves the forecast tail with an AUTO " + "continuation proxy at its end (or only the proxy if tail is zero); FIXED " + "uses 'terminal_value_euro_per_kwh'. Defaults to AUTO." + ), + "examples": ["AUTO", "FIXED"], + }, + ) + + terminal_value_euro_per_kwh: float = Field( + default=0.0, + json_schema_extra={ + "description": ( + "Value assigned to usable battery energy remaining at the end of the " + "optimization horizon [EUR/kWh]. This terminal value is independent " + "of the battery LCOS. Only used with terminal_value_mode = FIXED. " + "Defaults to 0 EUR/kWh." + ), + "examples": [0.0, 0.20], + }, + ) + + terminal_value_window_hours: int = Field( + default=24, + ge=1, + json_schema_extra={ + "description": ( + "Length of the trailing window at the effective tail end the AUTO continuation " + "curve is derived from [h]. One day covers a full load and PV " + "cycle. Defaults to 24 hours." + ), + "examples": [24], + }, + ) + # --- Penalties (existing) ------------------------------------------------- penalties: dict[str, Union[float, int, str]] = Field( diff --git a/src/akkudoktoreos/optimization/genetic/geneticsolution.py b/src/akkudoktoreos/optimization/genetic/geneticsolution.py index 0d79974d..450431f4 100644 --- a/src/akkudoktoreos/optimization/genetic/geneticsolution.py +++ b/src/akkudoktoreos/optimization/genetic/geneticsolution.py @@ -1,6 +1,6 @@ """Genetic algorithm optimisation solution.""" -from typing import Any, Optional, Union +from typing import Any, Optional, Union, cast import numpy as np import pandas as pd @@ -30,8 +30,9 @@ from akkudoktoreos.optimization.genetic.geneticdevices import GeneticParametersB from akkudoktoreos.optimization.genetic.geneticparams import ( GeneticOptimizationParameters, ) +from akkudoktoreos.optimization.genetic.terminalvalue import TerminalValueResult from akkudoktoreos.optimization.optimization import OptimizationSolution -from akkudoktoreos.utils.datetimeutil import to_datetime, to_duration +from akkudoktoreos.utils.datetimeutil import DateTime, to_datetime, to_duration from akkudoktoreos.utils.utils import NumpyEncoder @@ -239,11 +240,48 @@ class GeneticSimulationResult(GeneticParametersBaseModel): "losses_per_hour", "home_appliance_wh_per_hour", "electricity_price", + "feed_in_tariff", mode="before", ) def convert_numpy(cls, field: Any) -> Any: return NumpyEncoder.convert_numpy(field)[0] + home_appliance_energy_wh: dict[str, list[float]] = Field( + default_factory=dict, + json_schema_extra={ + "description": ( + "Per-device appliance energy in watt-hours per optimization slot, " + "keyed by device_id." + ) + }, + ) + + feed_in_tariff: list[float] = Field( + validation_alias=AliasChoices("feed_in_tariff", "Feed_in_tariff"), + default_factory=list, + json_schema_extra={ + "description": "Used feed-in tariff in €/Wh per hour, including predictions" + }, + ) + + home_appliance_running: dict[str, list[bool]] = Field( + default_factory=dict, description="Active run occupancy, including zero-power phases." + ) + + @field_validator("home_appliance_energy_wh", mode="before") + def convert_numpy_appliance_energy(cls, field: Any) -> Any: + if isinstance(field, dict): + return { + device_id: NumpyEncoder.convert_numpy(values)[0] + for device_id, values in field.items() + } + return field + + @computed_field(json_schema_extra={"deprecated": True}) + def Feed_in_tariff(self) -> Optional[list[float]]: + """Deprecated: use feed_in_tariff.""" + return self.feed_in_tariff + class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): """**Note**: The first value of "load_wh_per_hour", "grid_feed_in_wh_per_hour", and "grid_consumption_wh_per_hour", will be set to null in the JSON output and represented as NaN or None in the corresponding classes' data returns. This approach is adopted to ensure that the current hour's processing remains unchanged.""" @@ -253,55 +291,66 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): parameters: GeneticOptimizationParameters = Field( json_schema_extra={"description": "Optimization parameters used to generate solution."} ) + ac_charge: list[float] = Field( json_schema_extra={ "description": "Array with AC charging values as relative power (0.0-1.0), other values set to 0." } ) + dc_charge: list[float] = Field( json_schema_extra={ "description": "Array with DC charging values as relative power (0-1), other values set to 0." } ) + discharge_allowed: list[int] = Field( json_schema_extra={ "description": "Array with discharge values (1 for discharge, 0 otherwise)." } ) + ev_charge_hours_float: Optional[list[float]] = Field( validation_alias=AliasChoices("ev_charge_hours_float", "eautocharge_hours_float"), json_schema_extra={ "description": "Array with EV charging values as relative power (0.0-1.0), or `null` if no EV is optimized." }, ) + result: GeneticSimulationResult + ev_obj: Optional[ElectricVehicleResult] = Field( validation_alias=AliasChoices("ev_obj", "eauto_obj"), json_schema_extra={"description": "Electric vehicle state after optimization."}, ) + start_hour: int = Field( default=0, json_schema_extra={"description": "Start hour."}, ) + start_solution: Optional[list[float]] = Field( default=None, json_schema_extra={ "description": "An array of binary values (0 or 1) representing a possible starting solution for the simulation." }, ) + washingstart: Optional[int] = Field( default=None, json_schema_extra={ "description": "Can be `null` or contain an object representing the start of washing (if applicable)." }, ) + extra_data: Optional[dict[str, Union[list[int], list[float]]]] = Field( default=None, json_schema_extra={ "description": ("Dictionary of balance: TBD, losses: TBD, constraints: TBD.") }, ) - fitness_history: Optional[dict[str, Union[list[int], list[float]]]] = Field( + + fitness_history: Optional[dict[str, Any]] = Field( default=None, json_schema_extra={ "description": ( @@ -313,12 +362,12 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): ) }, ) + fixed_seed: Optional[int] = Field( default=None, json_schema_extra={"description": "Fixed seed."}, ) - # Computed fields for backward compatibility (deprecated German names) @computed_field(json_schema_extra={"deprecated": True}) def eautocharge_hours_float(self) -> Optional[list[float]]: """Deprecated: Use ev_charge_hours_float instead.""" @@ -333,6 +382,8 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): "ac_charge", "dc_charge", "discharge_allowed", + "battery_grid_export_allowed", + "battery_grid_export_factor", mode="before", ) def convert_numpy(cls, field: Any) -> Any: @@ -347,39 +398,131 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): return ElectricVehicleResult(**field.to_dict()) return field + controls_start_at_now: bool = Field( + default=False, description="Control arrays start at the run timestamp instead of midnight." + ) + + battery_grid_export_allowed: list[int] = Field( + default_factory=list, + json_schema_extra={ + "description": "Array with battery-to-grid export values (1 for export discharge, 0 otherwise)." + }, + ) + + terminal_value: Optional[TerminalValueResult] = Field( + default=None, + json_schema_extra={ + "description": ( + "The terminal value applied to the energy left in the battery at " + "the end of the horizon, including the curve it was read from. " + "None when no battery is part of the optimization." + ) + }, + ) + + battery_grid_export_factor: list[float] = Field( + default_factory=list, + json_schema_extra={ + "description": ( + "Array with the battery-to-grid export level per slot as factor " + "of the rated discharge power (0.0 for no export). Empty when " + "direct marketing is disabled; a solution without this array " + "exports at full power wherever " + "'battery_grid_export_allowed' is 1." + ) + }, + ) + + start_solution_datetime: Optional[DateTime] = Field( + default=None, + json_schema_extra={ + "description": ( + "Start of the slot that gene 0 of 'start_solution' controls. Send it " + "back together with 'start_solution' so the next run can shift the " + "warm start by the slots that have elapsed since." + ), + "examples": [None, "2026-09-14T07:45:00+02:00"], + }, + ) + + appliance_starts: dict[str, list[DateTime]] = Field( + default_factory=dict, + json_schema_extra={ + "description": ( + "Scheduled run start times per appliance device_id as absolute local datetimes." + ) + }, + ) + + appliance_deadline_missed: dict[str, bool] = Field( + default_factory=dict, + json_schema_extra={ + "description": ( + "Per appliance device_id with a 'deadline_datetime': whether the " + "scheduled run misses that deadline (or was not scheduled at " + "all). Appliances without a deadline are not listed." + ) + }, + ) + + interval_seconds: int = Field( + default=3600, description="Duration of one result/control slot in seconds." + ) + + @field_validator("start_solution_datetime", mode="before") + @classmethod + def transform_start_solution_datetime(cls, value: Any) -> Optional[DateTime]: + """Accept the usual date time representations, naive input is local time.""" + return GeneticOptimizationParameters.transform_start_solution_datetime(value) + def _battery_device_id(self) -> str: """Get battery device id.""" + parameters = getattr(self, "parameters", None) + if parameters is not None and parameters.pv_battery is not None: + return parameters.pv_battery.device_id try: - return list(self.config.devices.batteries.values())[0].device_id + return next(iter(self.config.devices.batteries.values())).device_id except Exception: return "battery1" def _ev_device_id(self) -> str: """Get electric vehicle device id.""" + parameters = getattr(self, "parameters", None) + if parameters is not None and parameters.ev is not None: + return parameters.ev.device_id try: - return self.config.devices.electric_vehicles.values()[0].device_id + return next(iter(self.config.devices.electric_vehicles.values())).device_id except Exception: return "ev1" def _homeappliance_device_id(self) -> str: """Get home appliance device id.""" + parameters = getattr(self, "parameters", None) + if parameters is not None and parameters.resolved_home_appliances(): + return parameters.resolved_home_appliances()[0].device_id try: - return self.config.devices.home_appliances.values()[0].device_id + return next(iter(self.config.devices.home_appliances.values())).device_id except Exception: return "homeappliance1" + @staticmethod def _battery_operation_from_solution( - self, ac_charge: float, dc_charge: float, discharge_allowed: bool, + battery_grid_export_allowed: bool = False, + battery_grid_export_factor: float = 1.0, ) -> tuple[BatteryOperationMode, float]: """Maps low-level solution to a representative operation mode and factor. Args: ac_charge (float): Allowed AC-side charging power (relative units). dc_charge (float): Allowed DC-side charging power (relative units). - discharge_allowed (bool): Whether discharging is permitted. + discharge_allowed (bool): Whether discharging to local load is permitted. + battery_grid_export_allowed (bool): Whether discharge into the grid is permitted. + battery_grid_export_factor (float): Export level as factor of the rated + discharge power ]0.0 ... 1.0]. Becomes the operation factor of + GRID_SUPPORT_EXPORT. Returns: tuple[BatteryOperationMode, float]: A tuple containing @@ -387,15 +530,30 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): - `float`: the operation factor corresponding to the active signal. Notes: - - The mapping prioritizes AC charge > DC charge > discharge. + - Explicit grid export is separate from local-load discharge. + - The mapping prioritizes export > AC charge > DC charge > discharge. - Multiple strategies can produce the same low-level signals; this function returns a representative mode based on a defined priority order. """ # (0,0,0) → Nothing allowed - if ac_charge <= 0.0 and dc_charge <= 0.0 and not discharge_allowed: + if ( + ac_charge <= 0.0 + and dc_charge <= 0.0 + and not discharge_allowed + and not battery_grid_export_allowed + ): return BatteryOperationMode.IDLE, 1.0 - # (0,0,1) → Discharge only + if battery_grid_export_allowed: + if ac_charge > 0.0 or dc_charge > 0.0: + raise ValueError( + "Illegal state: battery_grid_export_allowed cannot be combined with charging" + ) + return BatteryOperationMode.GRID_SUPPORT_EXPORT, min( + max(float(battery_grid_export_factor), 0.0), 1.0 + ) + + # (0,0,1) -> Discharge for local load only if ac_charge <= 0.0 and dc_charge <= 0.0 and discharge_allowed: return BatteryOperationMode.PEAK_SHAVING, 1.0 @@ -436,7 +594,8 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): dc_charge: float, discharge_allowed: bool, soc_pct: float, - ) -> tuple[float, float, bool]: + battery_grid_export_allowed: bool = False, + ) -> tuple[float, float, bool, bool]: """Clamp raw genetic gene values by the battery's actual SOC at that hour. The raw gene values represent the optimizer's *intent* and are stored @@ -448,16 +607,24 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): Clamping rules: - AC charge factor: scaled down proportionally when the battery headroom (max_soc − current_soc) is smaller than what the - commanded factor would store in one hour. Set to 0 when full. + commanded factor would store in one optimization slot. Set to 0 when full. - DC charge factor (PV): zeroed when battery is at or above max SOC (the inverter curtails automatically, but this makes intent clear). - Discharge: blocked when SOC is at or below min SOC. + - Battery grid export: blocked when SOC is at or below min SOC. """ - bat_dict = self.config.devices.batteries - if bat_dict is None or len(bat_dict) <= 0: - return ac_charge, dc_charge, discharge_allowed + parameters = getattr(self, "parameters", None) + bat_list = ( + [parameters.pv_battery] + if parameters is not None and parameters.pv_battery is not None + else list((self.config.devices.batteries or {}).values()) + if parameters is None + else [] + ) + if not bat_list: + return ac_charge, dc_charge, discharge_allowed, battery_grid_export_allowed - bat = list(bat_dict.values())[0] + bat = bat_list[0] min_soc = float(bat.min_soc_percentage) max_soc = float(bat.max_soc_percentage) capacity_wh = float(bat.capacity_wh) @@ -470,16 +637,27 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): if headroom_wh <= 0.0: effective_ac = 0.0 else: - inv_list = self.config.devices.inverters + inv_list = ( + [parameters.inverter] + if parameters is not None and parameters.inverter is not None + else list((self.config.devices.inverters or {}).values()) + if parameters is None + else [] + ) ac_to_dc_eff = float(inv_list[0].ac_to_dc_efficiency) if inv_list else 1.0 max_ac_cp_w = ( float(inv_list[0].max_ac_charge_power_w) if inv_list and inv_list[0].max_ac_charge_power_w is not None else float(bat.max_charge_power_w) ) - max_dc_per_h_wh = effective_ac * max_ac_cp_w * ac_to_dc_eff * ch_eff - if max_dc_per_h_wh > headroom_wh: - effective_ac = effective_ac * (headroom_wh / max_dc_per_h_wh) + # Energy storable in one optimization slot, not per hour: scale the + # power [W] by the slot duration (1.0 hourly, 0.25 at 15 min). + slot_duration_h = float(self.interval_seconds) / 3600.0 + max_dc_per_slot_wh = ( + effective_ac * max_ac_cp_w * slot_duration_h * ac_to_dc_eff * ch_eff + ) + if max_dc_per_slot_wh > headroom_wh: + effective_ac = effective_ac * (headroom_wh / max_dc_per_slot_wh) # --- DC charge (PV): zero when battery is full --- effective_dc = dc_charge @@ -488,36 +666,49 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): # --- Discharge: block at min SOC --- effective_dis = discharge_allowed and (soc_pct > min_soc) + effective_grid_export = battery_grid_export_allowed and (soc_pct > min_soc) - return effective_ac, effective_dc, effective_dis + return effective_ac, effective_dc, effective_dis, effective_grid_export async def optimization_solution(self) -> OptimizationSolution: """Provide the genetic solution as a general optimization solution. The battery modes are controlled by the grid control triggers: - ac_charge: charge from grid - - discharge_allowed: discharge to grid + - discharge_allowed: discharge to local load + - battery_grid_export_allowed: discharge to grid The following battery modes are supported: - - SELF_CONSUMPTION: ac_charge == 0 and discharge_allowed == 0 - - GRID_SUPPORT_EXPORT: ac_charge == 0 and discharge_allowed == 1 + - SELF_CONSUMPTION: dc_charge > 0 and discharge_allowed == 1 + - PEAK_SHAVING: ac_charge == 0 and discharge_allowed == 1 + - GRID_SUPPORT_EXPORT: battery_grid_export_allowed == 1 - GRID_SUPPORT_IMPORT: ac_charge > 0 and discharge_allowed == 0 or 1 """ - start_datetime = get_ems().start_datetime - start_day_hour = start_datetime.in_timezone(self.config.general.timezone).hour - interval_hours = 1 - power_to_energy_per_interval_factor = 1.0 + start_datetime = self.start_solution_datetime or get_ems().start_datetime + # New controls use the run-relative control horizon; old payloads retain a midnight prefix. + # entries indexed by slot (slot 0 == 00:00 local). Index this serializer + # by slot too. At the default interval of 3600 s slots_per_hour == 1 and + # this is the established hourly behaviour. + interval_s = self.interval_seconds + start_local = start_datetime.in_timezone(self.config.general.timezone) + start_day_slot = ( + 0 + if self.controls_start_at_now + else int((start_local - start_local.start_of("day")).total_seconds() // interval_s) + ) + # power [W] -> energy per slot [Wh]: multiply by the slot duration in hours. + power_to_energy_per_interval_factor = interval_s / 3600.0 # --- Create index based on list length and interval --- # Ensure we only use the minimum of results and commands if differing - periods = min(len(self.result.costs_per_hour), len(self.ac_charge) - start_day_hour) + periods = min(len(self.result.costs_per_hour), len(self.ac_charge) - start_day_slot) time_index = pd.date_range( start=start_datetime, periods=periods, - freq=f"{interval_hours}h", + freq=f"{interval_s}s", ) n_points = len(time_index) - end_datetime = start_datetime.add(hours=n_points) + end_datetime = start_datetime.add(seconds=interval_s * n_points) # Fill solution into dataframe with correct column names # - load_energy_wh: Load of all energy consumers in wh" @@ -533,7 +724,7 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): solution = pd.DataFrame( { "date_time": time_index, - # result starts at start_day_hour + # result starts at start_day_slot "load_energy_wh": self.result.load_wh_per_hour[:n_points], "grid_feedin_energy_wh": self.result.grid_feed_in_wh_per_hour[:n_points], "grid_consumption_energy_wh": self.result.grid_consumption_wh_per_hour[:n_points], @@ -544,68 +735,91 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): index=time_index, ) - # Add battery data - battery_device_id = self._battery_device_id() - solution[f"{battery_device_id}_soc_factor"] = [ - v / 100 - for v in self.result.battery_soc_per_hour[:n_points] # result starts at start_day_hour - ] - operation: dict[str, list[float]] = { - "genetic_ac_charge_factor": [], - "genetic_dc_charge_factor": [], - "genetic_discharge_allowed_factor": [], - } - # ac_charge, dc_charge, discharge_allowed start at hour 0 of start day - for hour_idx, rate in enumerate(self.ac_charge): - if hour_idx < start_day_hour: - continue - if hour_idx >= start_day_hour + n_points: - break - ac_charge_hour = self.ac_charge[hour_idx] - dc_charge_hour = self.dc_charge[hour_idx] - discharge_allowed_hour = bool(self.discharge_allowed[hour_idx]) + if self.parameters.pv_battery is not None: + # Add battery data + battery_device_id = self._battery_device_id() + solution[f"{battery_device_id}_soc_factor"] = [ + v / 100 + for v in self.result.battery_soc_per_hour[ + :n_points + ] # result starts at start_day_slot + ] + operation: dict[str, list[float]] = { + "genetic_ac_charge_factor": [], + "genetic_dc_charge_factor": [], + "genetic_discharge_allowed_factor": [], + "genetic_battery_grid_export_allowed_factor": [], + } + # ac_charge, dc_charge, discharge_allowed start at hour 0 of start day + for hour_idx, rate in enumerate(self.ac_charge): + if hour_idx < start_day_slot: + continue + if hour_idx >= start_day_slot + n_points: + break + ac_charge_hour = self.ac_charge[hour_idx] + dc_charge_hour = self.dc_charge[hour_idx] + discharge_allowed_hour = bool(self.discharge_allowed[hour_idx]) + battery_grid_export_allowed_hour = ( + bool(self.battery_grid_export_allowed[hour_idx]) + if hour_idx < len(self.battery_grid_export_allowed) + else False + ) + # Solutions written before graded export carry no factor array; they + # exported at full power wherever the signal was set. + battery_grid_export_factor_hour = ( + float(self.battery_grid_export_factor[hour_idx]) + if hour_idx < len(self.battery_grid_export_factor) + else (1.0 if battery_grid_export_allowed_hour else 0.0) + ) - # Raw genetic gene values — optimizer intent, stored verbatim - operation["genetic_ac_charge_factor"].append(ac_charge_hour) - operation["genetic_dc_charge_factor"].append(dc_charge_hour) - operation["genetic_discharge_allowed_factor"].append(float(discharge_allowed_hour)) + # Raw genetic gene values — optimizer intent, stored verbatim + operation["genetic_ac_charge_factor"].append(ac_charge_hour) + operation["genetic_dc_charge_factor"].append(dc_charge_hour) + operation["genetic_discharge_allowed_factor"].append(float(discharge_allowed_hour)) + operation["genetic_battery_grid_export_allowed_factor"].append( + battery_grid_export_factor_hour + ) - # SOC-clamped effective values — what can physically be executed at - # this hour given the expected battery state of charge. - result_idx = hour_idx - start_day_hour - soc_h_pct = ( - self.result.battery_soc_per_hour[result_idx] - if result_idx < len(self.result.battery_soc_per_hour) - else 0.0 - ) - eff_ac, eff_dc, eff_dis = self._soc_clamped_operation_factors( - ac_charge_hour, dc_charge_hour, discharge_allowed_hour, soc_h_pct - ) - operation_mode, operation_mode_factor = self._battery_operation_from_solution( - eff_ac, eff_dc, eff_dis - ) - for mode in BatteryOperationMode: - mode_key = f"{battery_device_id}_{mode.lower()}_op_mode" - factor_key = f"{battery_device_id}_{mode.lower()}_op_factor" - if mode_key not in operation.keys(): - operation[mode_key] = [] - operation[factor_key] = [] - if mode == operation_mode: - operation[mode_key].append(1.0) - operation[factor_key].append(operation_mode_factor) - else: - operation[mode_key].append(0.0) - operation[factor_key].append(0.0) - for key in operation.keys(): - if len(operation[key]) != n_points: - error_msg = f"instruction {key} has invalid length {len(operation[key])} - expected {n_points}" - logger.error(error_msg) - raise ValueError(error_msg) - solution[key] = operation[key] + # SOC-clamped effective values — what can physically be executed at + # this hour given the expected battery state of charge. + result_idx = hour_idx - start_day_slot + soc_h_pct = ( + self.result.battery_soc_per_hour[result_idx] + if result_idx < len(self.result.battery_soc_per_hour) + else 0.0 + ) + eff_ac, eff_dc, eff_dis, eff_grid_export = self._soc_clamped_operation_factors( + ac_charge_hour, + dc_charge_hour, + discharge_allowed_hour, + soc_h_pct, + battery_grid_export_allowed_hour, + ) + operation_mode, operation_mode_factor = self._battery_operation_from_solution( + eff_ac, eff_dc, eff_dis, eff_grid_export, battery_grid_export_factor_hour + ) + for mode in BatteryOperationMode: + mode_key = f"{battery_device_id}_{mode.lower()}_op_mode" + factor_key = f"{battery_device_id}_{mode.lower()}_op_factor" + if mode_key not in operation.keys(): + operation[mode_key] = [] + operation[factor_key] = [] + if mode == operation_mode: + operation[mode_key].append(1.0) + operation[factor_key].append(operation_mode_factor) + else: + operation[mode_key].append(0.0) + operation[factor_key].append(0.0) + for key in operation.keys(): + if len(operation[key]) != n_points: + error_msg = f"instruction {key} has invalid length {len(operation[key])} - expected {n_points}" + logger.error(error_msg) + raise ValueError(error_msg) + solution[key] = operation[key] # Add EV battery solution - # ev_charge_hours_float start at hour 0 of start day - # result.ev_soc_per_hour start at start_datetime.hour + # eautocharge_hours_float start at hour 0 of start day + # result.EAuto_SoC_pro_Stunde start at start_datetime.hour if self.ev_obj: ev_device_id = self._ev_device_id() if self.ev_charge_hours_float is None: @@ -633,9 +847,9 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): "genetic_ev_charge_factor": [], } for hour_idx, rate in enumerate(self.ev_charge_hours_float): - if hour_idx < start_day_hour: + if hour_idx < start_day_slot: continue - if hour_idx >= start_day_hour + n_points: + if hour_idx >= start_day_slot + n_points: break operation["genetic_ev_charge_factor"].append(rate) operation_mode, operation_mode_factor = self._battery_operation_from_solution( @@ -660,32 +874,33 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): raise ValueError(error_msg) solution[key] = operation[key] - # Add home appliance data - if self.config.devices.max_home_appliances and self.config.devices.max_home_appliances > 0: - # Use config and not self.washingstart as washingstart may be None (no start) - # even if configured to be started. - homeappliance_device_id = self._homeappliance_device_id() - # result starts at start_day_hour - solution[f"{homeappliance_device_id}_energy_wh"] = ( - self.result.home_appliance_wh_per_hour[:n_points] - ) + # Add home appliance data, one block of columns per device. Per-device + # energy arrays start at start_day_slot, like the other result arrays. + for device_id, energy_wh in self.result.home_appliance_energy_wh.items(): + solution[f"{device_id}_energy_wh"] = energy_wh[:n_points] operation = { - f"{homeappliance_device_id}_run_op_mode": [], - f"{homeappliance_device_id}_run_op_factor": [], - f"{homeappliance_device_id}_off_op_mode": [], - f"{homeappliance_device_id}_off_op_factor": [], + f"{device_id}_run_op_mode": [], + f"{device_id}_run_op_factor": [], + f"{device_id}_off_op_mode": [], + f"{device_id}_off_op_factor": [], } - for hour_idx, energy in enumerate(solution[f"{homeappliance_device_id}_energy_wh"]): - if energy > 0.0: - operation[f"{homeappliance_device_id}_run_op_mode"].append(1.0) - operation[f"{homeappliance_device_id}_run_op_factor"].append(1.0) - operation[f"{homeappliance_device_id}_off_op_mode"].append(0.0) - operation[f"{homeappliance_device_id}_off_op_factor"].append(0.0) + running_slots = self.result.home_appliance_running.get(device_id, []) + for hour_idx, energy in enumerate(solution[f"{device_id}_energy_wh"]): + running = ( + running_slots[hour_idx] + if hour_idx < len(running_slots) + else bool(energy and energy > 0.0) + ) + if running: + operation[f"{device_id}_run_op_mode"].append(1.0) + operation[f"{device_id}_run_op_factor"].append(1.0) + operation[f"{device_id}_off_op_mode"].append(0.0) + operation[f"{device_id}_off_op_factor"].append(0.0) else: - operation[f"{homeappliance_device_id}_run_op_mode"].append(0.0) - operation[f"{homeappliance_device_id}_run_op_factor"].append(0.0) - operation[f"{homeappliance_device_id}_off_op_mode"].append(1.0) - operation[f"{homeappliance_device_id}_off_op_factor"].append(1.0) + operation[f"{device_id}_run_op_mode"].append(0.0) + operation[f"{device_id}_run_op_factor"].append(0.0) + operation[f"{device_id}_off_op_mode"].append(1.0) + operation[f"{device_id}_off_op_factor"].append(1.0) for key in operation.keys(): if len(operation[key]) != n_points: error_msg = f"instruction {key} has invalid length {len(operation[key])} - expected {n_points}" @@ -706,18 +921,33 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): }, index=time_index, ) - pred = get_prediction() + if self.controls_start_at_now: + snapshot = self.parameters.ems + prediction["pvforecast_ac_energy_wh"] = snapshot.pv_forecast_wh[:n_points] + prediction["loadforecast_energy_wh"] = snapshot.total_load[:n_points] + prediction["elec_price_amt_kwh"] = ( + np.asarray(snapshot.electricity_price_per_wh[:n_points]) * 1000 + ) + tariffs = snapshot.feed_in_tariff_per_wh + prediction["feed_in_tariff_amt_kwh"] = ( + np.asarray(tariffs[:n_points]) * 1000 + if isinstance(tariffs, list) + else tariffs * 1000 + ) + if self.controls_start_at_now and self.parameters.temperature_forecast is not None: + prediction["weather_air_temp_celcius"] = self.parameters.temperature_forecast[:n_points] + pred = None if self.controls_start_at_now else get_prediction() - prediction_specs: list[tuple[str, FillMethod, str, float]] = [ + for pred_key, pred_fill_method, pred_solution_key, pred_solution_factor in [ ( "pvforecast_ac_power", - "linear", + "ffill", "pvforecast_ac_energy_wh", power_to_energy_per_interval_factor, ), ( "pvforecast_dc_power", - "linear", + "ffill", "pvforecast_dc_energy_wh", power_to_energy_per_interval_factor, ), @@ -741,32 +971,32 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): ), ( "loadforecast_power_w", - "linear", + "ffill", "loadforecast_energy_wh", power_to_energy_per_interval_factor, ), ( "loadakkudoktor_std_power_w", - "linear", + "ffill", "loadakkudoktor_std_energy_wh", power_to_energy_per_interval_factor, ), ( "loadakkudoktor_mean_power_w", - "linear", + "ffill", "loadakkudoktor_mean_energy_wh", power_to_energy_per_interval_factor, ), - ] - - for pred_key, pred_fill_method, pred_solution_key, pred_solution_factor in prediction_specs: - if pred_key in pred.record_keys: + ]: + if pred_solution_key in prediction.columns: + continue + if pred is not None and pred_key in pred.record_keys: array = await pred.key_to_array( key=pred_key, start_datetime=start_datetime, end_datetime=end_datetime, - interval=to_duration(f"{interval_hours} hours"), - fill_method=pred_fill_method, + interval=to_duration(f"{interval_s} seconds"), + fill_method=cast(FillMethod, pred_fill_method), ) # 'key_to_array()' creates None values array if no data records are available. if array is not None and array.size > 0 and not np.any(pd.isna(array)): @@ -777,7 +1007,7 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): generated_at=to_datetime(), comment="Optimization solution derived from GeneticSolution.", valid_from=start_datetime, - valid_until=start_datetime.add(hours=self.config.optimization.genetic.horizon_hours), + valid_until=end_datetime, total_losses_energy_wh=self.result.total_losses, total_revenues_amt=self.result.total_revenue, total_costs_amt=self.result.total_costs, @@ -792,8 +1022,16 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): def energy_management_plan(self) -> EnergyManagementPlan: """Provide the genetic solution as an energy management plan.""" - start_datetime = get_ems().start_datetime - start_day_hour = start_datetime.in_timezone(self.config.general.timezone).hour + start_datetime = self.start_solution_datetime or get_ems().start_datetime + # Index by slot, not hour (mirrors optimization_solution). At the default + # interval of 3600 s this reduces to the start hour-of-day. + interval_s = self.interval_seconds + start_local = start_datetime.in_timezone(self.config.general.timezone) + start_day_slot = ( + 0 + if self.controls_start_at_now + else int((start_local - start_local.start_of("day")).total_seconds() // interval_s) + ) plan = EnergyManagementPlan( id=f"plan-genetic@{to_datetime(as_string=True)}", generated_at=to_datetime(), @@ -801,55 +1039,70 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): comment="Energy management plan derived from GeneticSolution.", ) - # Add battery instructions (fill rate based control) - last_operation_mode: Optional[str] = None - last_operation_mode_factor: Optional[float] = None - resource_id = self._battery_device_id() - # ac_charge, dc_charge, discharge_allowed start at hour 0 of start day - logger.debug("BAT: {} - {}", resource_id, self.ac_charge[start_day_hour:]) - for hour_idx, rate in enumerate(self.ac_charge): - if hour_idx < start_day_hour: - continue - # Derive SOC-clamped effective factors so that FRBCInstruction - # operation_mode_factor reflects what can physically be executed, - # while the raw genetic gene values are preserved in the solution - # dataframe (genetic_*_factor columns). - result_idx = hour_idx - start_day_hour - soc_h_pct = ( - self.result.battery_soc_per_hour[result_idx] - if result_idx < len(self.result.battery_soc_per_hour) - else 0.0 - ) - eff_ac, eff_dc, eff_dis = self._soc_clamped_operation_factors( - self.ac_charge[hour_idx], - self.dc_charge[hour_idx], - bool(self.discharge_allowed[hour_idx]), - soc_h_pct, - ) - operation_mode, operation_mode_factor = self._battery_operation_from_solution( - eff_ac, eff_dc, eff_dis - ) - if ( - operation_mode == last_operation_mode - and operation_mode_factor == last_operation_mode_factor - ): - # Skip, we already added the instruction - continue - last_operation_mode = operation_mode - last_operation_mode_factor = operation_mode_factor - execution_time = start_datetime.add(hours=hour_idx - start_day_hour) - plan.add_instruction( - FRBCInstruction( - resource_id=resource_id, - execution_time=execution_time, - actuator_id=resource_id, - operation_mode_id=operation_mode, - operation_mode_factor=operation_mode_factor, + if self.parameters.pv_battery is not None: + # Add battery instructions (fill rate based control) + last_operation_mode: Optional[str] = None + last_operation_mode_factor: Optional[float] = None + resource_id = self._battery_device_id() + # ac_charge, dc_charge, discharge_allowed start at hour 0 of start day + logger.debug("BAT: {} - {}", resource_id, self.ac_charge[start_day_slot:]) + for hour_idx, rate in enumerate(self.ac_charge): + if hour_idx < start_day_slot: + continue + # Derive SOC-clamped effective factors so that FRBCInstruction + # operation_mode_factor reflects what can physically be executed, + # while the raw genetic gene values are preserved in the solution + # dataframe (genetic_*_factor columns). + result_idx = hour_idx - start_day_slot + soc_h_pct = ( + self.result.battery_soc_per_hour[result_idx] + if result_idx < len(self.result.battery_soc_per_hour) + else 0.0 + ) + battery_grid_export_allowed_hour = ( + bool(self.battery_grid_export_allowed[hour_idx]) + if hour_idx < len(self.battery_grid_export_allowed) + else False + ) + eff_ac, eff_dc, eff_dis, eff_grid_export = self._soc_clamped_operation_factors( + self.ac_charge[hour_idx], + self.dc_charge[hour_idx], + bool(self.discharge_allowed[hour_idx]), + soc_h_pct, + battery_grid_export_allowed_hour, + ) + operation_mode, operation_mode_factor = self._battery_operation_from_solution( + eff_ac, + eff_dc, + eff_dis, + eff_grid_export, + self.battery_grid_export_factor[hour_idx] + if hour_idx < len(self.battery_grid_export_factor) + else 1.0, + ) + if ( + operation_mode == last_operation_mode + and operation_mode_factor == last_operation_mode_factor + ): + # Skip, we already added the instruction + continue + last_operation_mode = operation_mode + last_operation_mode_factor = operation_mode_factor + execution_time = start_datetime.add( + seconds=interval_s * (hour_idx - start_day_slot) + ) + plan.add_instruction( + FRBCInstruction( + resource_id=resource_id, + execution_time=execution_time, + actuator_id=resource_id, + operation_mode_id=operation_mode, + operation_mode_factor=operation_mode_factor, + ) ) - ) # Add EV battery instructions (fill rate based control) - # ev_charge_hours_float start at hour 0 of start day + # eautocharge_hours_float start at hour 0 of start day if self.ev_obj: resource_id = self._ev_device_id() if self.ev_charge_hours_float is None: @@ -868,10 +1121,10 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): last_operation_mode = None last_operation_mode_factor = None logger.debug( - "EV: {} - {}", resource_id, self.ev_charge_hours_float[start_day_hour:] + "EV: {} - {}", resource_id, self.ev_charge_hours_float[start_day_slot:] ) for hour_idx, rate in enumerate(self.ev_charge_hours_float): - if hour_idx < start_day_hour: + if hour_idx < start_day_slot: continue operation_mode, operation_mode_factor = self._battery_operation_from_solution( rate, 0.0, False @@ -884,7 +1137,9 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): continue last_operation_mode = operation_mode last_operation_mode_factor = operation_mode_factor - execution_time = start_datetime.add(hours=hour_idx - start_day_hour) + execution_time = start_datetime.add( + seconds=interval_s * (hour_idx - start_day_slot) + ) plan.add_instruction( FRBCInstruction( resource_id=resource_id, @@ -895,34 +1150,36 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel): ) ) - # Add home appliance instructions (demand driven based control) - if self.config.devices.max_home_appliances and self.config.devices.max_home_appliances > 0: - # Use config and not self.washingstart as washingstart may be None (no start) - # even if configured to be started. - resource_id = self._homeappliance_device_id() - last_energy: Optional[float] = None - for hours, energy in enumerate(self.result.home_appliance_wh_per_hour): + # Add home appliance instructions (demand driven based control), one + # stream of instructions per device. A new instruction is only emitted on + # a transition between OFF (energy == 0) and RUN (energy > 0); a mere + # power change within a running profile does not add an instruction. + for resource_id, energy_wh in self.result.home_appliance_energy_wh.items(): + last_state: Optional[bool] = None + for hours, energy in enumerate(energy_wh): # hours starts at start_datetime with 0 if energy is None: raise ValueError( - f"Unexpected value {energy} in {self.result.home_appliance_wh_per_hour}" + f"Unexpected value {energy} in home_appliance_energy_wh[{resource_id}]" ) - if last_energy is None or energy != last_energy: - if energy > 0.0: - operation_mode = ApplianceOperationMode.RUN # type: ignore[assignment] - else: - operation_mode = ApplianceOperationMode.OFF # type: ignore[assignment] - operation_mode_factor = 1.0 - execution_time = start_datetime.add(hours=hours) + running_slots = self.result.home_appliance_running.get(resource_id, []) + running = running_slots[hours] if hours < len(running_slots) else energy > 0.0 + if last_state is None or running != last_state: + appliance_mode = ( + ApplianceOperationMode.RUN if running else ApplianceOperationMode.OFF + ) + execution_time = start_datetime.add(seconds=interval_s * hours) plan.add_instruction( DDBCInstruction( resource_id=resource_id, execution_time=execution_time, actuator_id=resource_id, - operation_mode_id=operation_mode, - operation_mode_factor=operation_mode_factor, + operation_mode_id=appliance_mode, + operation_mode_factor=1.0, ) ) - last_energy = energy + last_state = running + if not plan.instructions: + plan.valid_from = start_datetime return plan diff --git a/src/akkudoktoreos/optimization/genetic/geneticvisualize.py b/src/akkudoktoreos/optimization/genetic/geneticvisualize.py index c03f8702..80d6402b 100644 --- a/src/akkudoktoreos/optimization/genetic/geneticvisualize.py +++ b/src/akkudoktoreos/optimization/genetic/geneticvisualize.py @@ -27,7 +27,11 @@ class GeneticVisualizationReport(ConfigMixin): def __init__( self, version: str = "0.0.1", + interval_seconds: int = 3600, ) -> None: + if interval_seconds <= 0: + raise ValueError("Report interval must be positive") + self.interval_seconds = interval_seconds # Initialize the report with empty groups self.groups: list[list[Callable[[], None]]] = [] # Store groups of charts self.current_group: list[ @@ -129,12 +133,16 @@ class GeneticVisualizationReport(ConfigMixin): line_styles: Optional[list[str]] = None, x2label: Optional[Union[str, None]] = "Hours Since Start", ) -> None: - """Create a line chart and add it to the current group.""" + """Plot interval values at their start timestamps without energy rescaling.""" + if not y_list or not len(y_list[0]): + return + if any(len(values) != len(y_list[0]) for values in y_list): + raise ValueError("Date chart series must cover the same intervals") def chart() -> None: timestamps = [ - start_date.add(hours=i) for i in range(len(y_list[0])) - ] # 840 timestamps at 1-hour intervals + start_date.add(seconds=i * self.interval_seconds) for i in range(len(y_list[0])) + ] for idx, y_data in enumerate(y_list): label = labels[idx] if labels else None # Chart label @@ -150,18 +158,14 @@ class GeneticVisualizationReport(ConfigMixin): # Format the time axis plt.gca().xaxis.set_major_formatter( - mdates.DateFormatter("%Y-%m-%d", tz=self.config.general.timezone) + mdates.DateFormatter("%Y-%m-%d", tz=start_date.tzinfo) ) # Show date and time plt.gca().xaxis.set_major_locator( - mdates.DayLocator(interval=1, tz=self.config.general.timezone) + mdates.DayLocator(interval=1, tz=start_date.tzinfo) ) # Major ticks every day - plt.gca().xaxis.set_minor_locator( - mdates.HourLocator(interval=2, tz=self.config.general.timezone) - ) + plt.gca().xaxis.set_minor_locator(mdates.HourLocator(interval=2, tz=start_date.tzinfo)) # Minor ticks every 6 hours - plt.gca().xaxis.set_minor_formatter( - mdates.DateFormatter("%H", tz=self.config.general.timezone) - ) + plt.gca().xaxis.set_minor_formatter(mdates.DateFormatter("%H", tz=start_date.tzinfo)) # plt.gcf().autofmt_xdate(rotation=45, which="major") # Auto-format the x-axis for readability @@ -203,10 +207,12 @@ class GeneticVisualizationReport(ConfigMixin): # ax2.set_xticklabels([f"{int(h)}" for h in hours_since_start[::48]]) # ax2.set_xticks(timestamps[:: len(timestamps) // 24]) # Select 10 evenly spaced ticks ax2.set_xticks( - mdates.date2num(timestamps[:: len(timestamps) // 12]) + mdates.date2num(timestamps[:: max(1, len(timestamps) // 12)]) ) # Select 10 evenly spaced ticks # ax2.set_xticklabels([f"{int(h)}" for h in hours_since_start[:: len(timestamps) // 24]]) - ax2.set_xticklabels([f"{int(h)}" for h in hours_since_start[:: len(timestamps) // 12]]) + ax2.set_xticklabels( + [f"{h:g}" for h in hours_since_start[:: max(1, len(timestamps) // 12)]] + ) if x2label: ax2.set_xlabel(x2label) @@ -455,14 +461,18 @@ def genetic_prepare_visualize( ) -> bytes: global debug_visualize - report = GeneticVisualizationReport() - next_full_hour_date = get_ems().start_datetime - start_hour = solution.start_hour + report = GeneticVisualizationReport(interval_seconds=solution.interval_seconds) + # New solutions own their timeline. The fallback supports historical hourly solutions. + start_datetime = solution.start_solution_datetime or get_ems().start_datetime + start_offset = 0 if solution.controls_start_at_now else solution.start_hour + control_slots = len(solution.result.load_wh_per_hour) + end_offset = start_offset + control_slots + # Forecasts can include a diagnostic tail; report executable intervals here. # Group 1: report.create_line_chart_date( - next_full_hour_date, + start_datetime, [ - solution.parameters.ems.total_load[start_hour:], + solution.parameters.ems.total_load[start_offset:end_offset], ], title="Load Profile", # xlabel="Hours", # not enough space @@ -470,9 +480,9 @@ def genetic_prepare_visualize( labels=["Total Load (Wh)"], ) report.create_line_chart_date( - next_full_hour_date, + start_datetime, [ - solution.parameters.ems.pv_forecast_wh[start_hour:], + solution.parameters.ems.pv_forecast_wh[start_offset:end_offset], ], title="PV Forecast", # xlabel="Hours", # not enough space @@ -480,14 +490,11 @@ def genetic_prepare_visualize( ) report.create_line_chart_date( - next_full_hour_date, + start_datetime, [ - np.full( - len(solution.parameters.ems.total_load) - start_hour, - solution.parameters.ems.feed_in_tariff_per_wh[start_hour:] - if isinstance(solution.parameters.ems.feed_in_tariff_per_wh, list) - else solution.parameters.ems.feed_in_tariff_per_wh, - ) + np.asarray(solution.parameters.ems.feed_in_tariff_per_wh[start_offset:end_offset]) + if isinstance(solution.parameters.ems.feed_in_tariff_per_wh, list) + else np.full(control_slots, solution.parameters.ems.feed_in_tariff_per_wh) ], title="Remuneration", # xlabel="Hours", # not enough space @@ -496,9 +503,9 @@ def genetic_prepare_visualize( ) if solution.parameters.temperature_forecast: report.create_line_chart_date( - next_full_hour_date, + start_datetime, [ - solution.parameters.temperature_forecast[start_hour:], + solution.parameters.temperature_forecast[start_offset:end_offset], ], title="Temperature Forecast", # xlabel="Hours", # not enough space @@ -509,7 +516,7 @@ def genetic_prepare_visualize( # Group 2: report.create_line_chart_date( - start_date=next_full_hour_date, # start_date + start_date=start_datetime, # start_date y_list=[ solution.result.load_wh_per_hour, solution.result.home_appliance_wh_per_hour, @@ -517,7 +524,7 @@ def genetic_prepare_visualize( solution.result.grid_consumption_wh_per_hour, solution.result.losses_per_hour, ], - title="Energy Flow per Hour", + title="Energy Flow per Interval", # xlabel="Date", # not enough space ylabel="Energy (Wh)", labels=[ @@ -534,9 +541,9 @@ def genetic_prepare_visualize( # Group 3: report.create_line_chart_date( - start_date=next_full_hour_date, + start_date=start_datetime, y_list=[solution.result.battery_soc_per_hour, solution.result.ev_soc_per_hour], - title="Battery SOC", + title="Battery SOC at Interval Start", # xlabel="Date", # not enough space ylabel="%", labels=[ @@ -546,54 +553,60 @@ def genetic_prepare_visualize( markers=["o", "x"], ) report.create_line_chart_date( - start_date=next_full_hour_date, # start_date - y_list=[solution.parameters.ems.electricity_price_per_wh[start_hour:]], + start_date=start_datetime, # start_date + y_list=[solution.parameters.ems.electricity_price_per_wh[start_offset:end_offset]], # title="Electricity Price", # not enough space # xlabel="Date", # not enough space ylabel="Electricity Price (amount/Wh)", x2label=None, # not enough space ) - labels = list( - item - for sublist in zip( - list(str(i) for i in range(0, 23, 2)), list(str(" ") for i in range(0, 23, 2)) - ) - for item in sublist - ) - labels = labels[start_hour:] + labels - - report.create_bar_chart( - labels=labels, - values_list=[ - solution.ac_charge[start_hour:], - solution.dc_charge[start_hour:], - solution.discharge_allowed[start_hour:], - ], - title="AC/DC Charging and Discharge Overview", - ylabel="Relative Power (0-1) / Discharge (0 or 1)", - label_names=["AC Charging (relative)", "DC Charging (relative)", "Discharge Allowed"], - colors=["blue", "green", "red"], - bottom=3, - xlabels=labels, + controls: list[Union[np.ndarray, list[Optional[float]], list[float]]] = [ + solution.ac_charge[start_offset:end_offset], + solution.dc_charge[start_offset:end_offset], + np.asarray(solution.discharge_allowed[start_offset:end_offset]), + ] + control_labels = ["AC Charging (relative)", "DC Charging (relative)", "Discharge Allowed"] + if solution.battery_grid_export_allowed: + controls.append(np.asarray(solution.battery_grid_export_allowed[start_offset:end_offset])) + control_labels.append("Battery Grid Export Allowed") + if solution.battery_grid_export_factor: + controls.append(solution.battery_grid_export_factor[start_offset:end_offset]) + control_labels.append("Battery Grid Export (relative)") + report.create_line_chart_date( + start_datetime, + controls, + title="Executable Battery Controls", + ylabel="Relative Power / Allowed (0-1)", + labels=control_labels, ) report.finalize_group() # Group 4: report.create_line_chart_date( - next_full_hour_date, # start_date + start_datetime, # start_date [ solution.result.costs_per_hour, solution.result.revenue_per_hour, ], - title="Financial Balance per Hour", + title="Financial Balance per Interval", # xlabel="Date", # not enough space ylabel="Amount", labels=["Costs", "Revenue"], ) extra_data = solution.extra_data + if extra_data: + # Historical native runs used German diagnostic keys. Prefer current public keys. + extra_data = { + key: extra_data.get(key, extra_data.get(alias, [])) + for key, alias in ( + ("losses", "verluste"), + ("balance", "bilanz"), + ("constraints", "nebenbedingung"), + ) + } if extra_data: report.create_scatter_plot( x=np.array(extra_data["losses"]), @@ -683,6 +696,8 @@ def genetic_prepare_visualize( ) report.finalize_group() + _add_solution_diagnostics(report, solution, start_datetime) + # Generate the PDF report pdf = report.generate_pdf() @@ -690,6 +705,119 @@ def genetic_prepare_visualize( return pdf +def _add_solution_diagnostics( + report: GeneticVisualizationReport, solution: GeneticSolution, start: DateTime +) -> None: + """Render the retained run diagnostics; tail actions are never executable controls.""" + report.add_text_page( + f"Run start: {start.isoformat()}. Interval: {report.interval_seconds} seconds. " + "Energy values are Wh per interval; prices are amount/Wh. " + "Timeline positions mark interval starts. Battery SOC shows the beginning of each interval.", + title="GENETIC Run", + ) + report.finalize_group() + device_ids = sorted( + solution.result.home_appliance_energy_wh.keys() + | solution.appliance_starts.keys() + | solution.appliance_deadline_missed.keys() + ) + for device_id in device_ids: + energy = solution.result.home_appliance_energy_wh.get(device_id, []) + report.create_line_chart_date( + start, + [energy], + title=f"Flexible Consumer: {device_id}", + ylabel="Energy (Wh per interval)", + ) + if energy: + report.finalize_group() + starts = ", ".join( + value.isoformat() for value in solution.appliance_starts.get(device_id, []) + ) + report.add_text_page( + f"Scheduled starts: {starts or 'none'}. " + f"Deadline missed: {solution.appliance_deadline_missed.get(device_id, False)}.", + title=f"Flexible Consumer Schedule: {device_id}", + ) + report.finalize_group() + terminal = solution.terminal_value + if terminal is None: + return + report.add_text_page( + f"Mode: {terminal.mode}. Reason: {terminal.reason or 'none'}. " + f"Control horizon: {terminal.control_horizon_hours:g} h. " + f"Requested tail: {terminal.requested_tail_hours:g} h; " + f"effective tail: {terminal.effective_tail_hours:g} h; " + f"tail end: {terminal.tail_end_hour:g} h after run start. " + f"Battery usable AC energy at control end: {terminal.battery_energy_wh:g} Wh. " + f"Credited terminal value: {terminal.credited_euro:g} EUR. " + f"Tail operating value: {terminal.tail_operating_euro:g} EUR. " + f"Continuation value: {terminal.continuation_value_euro:g} EUR " + f"({terminal.continuation_mode}). " + "Tail results are diagnostic lookahead only, not executable controls. " + "The terminal credit is a modeled residual value, not realized revenue.", + title="Terminal Value and Tail Diagnostics", + ) + report.finalize_group() + if terminal.tail_diagnostics is not None: + report.add_json_page( + terminal.tail_diagnostics.model_dump(), title="Tail Forecast Summary", fontsize=10 + ) + report.finalize_group() + curve = terminal.curve + if curve is not None and curve.energy_wh: + + def value_chart() -> None: + plt.plot(curve.energy_wh, curve.value_euro, label="Terminal credit") + plt.scatter( + [terminal.battery_energy_wh], [terminal.credited_euro], label="Selected state" + ) + plt.xlabel("Usable AC Battery Energy (Wh)") + plt.ylabel("Terminal Value (EUR)") + plt.title("Residual Battery Value Curve") + plt.legend() + plt.grid(True) + + report.add_chart_to_group(value_chart, "Residual Battery Value Curve") + report.finalize_group() + if terminal.tail_plan: + # Use explicit hour offsets, which also preserve a clipped tail's true start. + plan = terminal.tail_plan + + def tail_chart() -> None: + dates = [start.add(seconds=slot.hour_from_start * 3600) for slot in plan] + for field, label in ( + ("grid_import_wh", "Grid import"), + ("grid_export_wh", "Grid export"), + ("battery_charge_wh", "Battery charge"), + ("battery_discharge_wh", "Battery discharge"), + ): + plt.plot( + mdates.date2num(dates), + [getattr(slot, field) for slot in plan], + label=label, + marker="o" if len(plan) == 1 else None, + ) + if len(plan) == 1: + # A lone sample otherwise disappears and datetime autoscaling spans years. + half_interval = report.interval_seconds / 2 + plt.xlim( + mdates.date2num(dates[0].subtract(seconds=half_interval)), + mdates.date2num(dates[0].add(seconds=half_interval)), + ) + plt.gca().xaxis.set_major_formatter( + mdates.DateFormatter("%m-%d %H:%M", tz=start.tzinfo) + ) + plt.xlabel("Interval Start (diagnostic only)") + plt.ylabel("Energy (Wh per interval)") + plt.title("Tail Lookahead: Not Executable Controls") + plt.legend() + plt.grid(True) + + report.add_chart_to_group(tail_chart, "Tail Lookahead") + report.finalize_group() + + def genetic_generate_example_report(filename: str = "example_report.pdf") -> None: """Generate example visualization report.""" global debug_visualize diff --git a/src/akkudoktoreos/optimization/genetic/tailvalue.py b/src/akkudoktoreos/optimization/genetic/tailvalue.py new file mode 100644 index 00000000..c28c98b6 --- /dev/null +++ b/src/akkudoktoreos/optimization/genetic/tailvalue.py @@ -0,0 +1,305 @@ +"""Chronological, run-local battery lookahead using the production device physics. + +The Bellman recursion interpolates continuation values on a stored-energy grid. +Tail actions never enter the executable control arrays. The selected path may +be returned separately as diagnostics so users can inspect the lookahead. +""" + +import numpy as np +from pydantic import PrivateAttr + +from akkudoktoreos.devices.genetic.battery import Battery +from akkudoktoreos.devices.genetic.inverter import Inverter +from akkudoktoreos.optimization.genetic.terminalvalue import ( + TailPlanSlot, + TerminalValueCurve, +) + +TailAction = tuple[int, int, float, float] + + +def _action_name(action: TailAction) -> str: + dc, discharge, ac_rate, export_rate = action + if export_rate > 0: + return "BATTERY_EXPORT" + if ac_rate > 0: + return "GRID_CHARGE" + if dc and discharge: + return "SELF_CONSUMPTION" + if dc: + return "PV_CHARGE_ONLY" + if discharge: + return "DISCHARGE_ONLY" + return "HOLD" + + +def _simulate_action( + *, + bat: Battery, + inv: Inverter, + energy_wh: float, + action: TailAction, + price: float, + load: float, + pv: float, + tariff: float, + direct_marketing: bool, +) -> dict[str, float]: + """Apply one tail action from one stored-energy state.""" + dc, discharge, ac_rate, export = action + bat.soc_wh = float(energy_wh) + bat._charged_raw_wh_per_slot.fill(0) + bat._discharged_raw_wh_per_slot.fill(0) + ac_enabled = inv.ac_to_dc_efficiency > 0 and ( + inv.max_ac_charge_power_w is None or inv.max_ac_charge_power_w > 0 + ) + bat.charge_array[0] = ac_rate if ac_rate > 0 and ac_enabled else dc + bat.discharge_array[0] = discharge if export == 0 or tariff > 0 else 0 + sold, bought, losses, _ = inv.process_energy( + pv, + load, + 0, + allow_battery_grid_export=direct_marketing and export > 0 and tariff > 0, + battery_grid_export_factor=export, + ) + ac_grid_charge_wh = 0.0 + if ac_rate > 0 and inv.ac_to_dc_efficiency > 0: + rate = ac_rate + if inv.max_ac_charge_power_w is not None and bat.max_charge_power_w > 0: + rate = min( + rate, + inv.max_ac_charge_power_w * inv.ac_to_dc_efficiency / bat.max_charge_power_w, + ) + bat.charge_array[0] = rate + if rate > 0: + stored, loss = bat.charge_energy(None, 0, charge_factor=rate) + ac_grid_charge_wh = (stored + loss) / inv.ac_to_dc_efficiency + bought += ac_grid_charge_wh + losses += loss + max(ac_grid_charge_wh - stored - loss, 0.0) + if direct_marketing and tariff < 0: + sold = 0.0 + discharged_wh = bat.discharged_energy_wh(0) + reward = ( + sold * tariff - bought * price - (discharged_wh * bat.levelized_cost_of_storage_kwh / 1000) + ) + return { + "next_state_wh": bat.soc_wh, + "reward_euro": reward, + "grid_export_wh": sold, + "grid_import_wh": bought, + "battery_charge_wh": (bat._charged_raw_wh_per_slot[0] * bat.charging_efficiency), + "battery_discharge_wh": discharged_wh, + "losses_wh": losses, + "ac_grid_charge_wh": ac_grid_charge_wh, + } + + +class TailValueCurve(TerminalValueCurve): + """Value of usable AC battery energy, including the value of empty capacity. + + Neither values nor marginal values are constrained to be monotone. The empty + state can earn money by charging at negative prices and selling later. + """ + + _trace_context: dict = PrivateAttr(default_factory=dict) + + def value(self, energy_wh: float) -> float: + return ( + float(np.interp(energy_wh, self.energy_wh, self.value_euro)) if self.energy_wh else 0.0 + ) + + def component_values(self, energy_wh: float) -> tuple[float, float]: + """Return tail operating cash flow and continuation credit separately.""" + if not self.energy_wh: + return 0.0, 0.0 + operating = float(np.interp(energy_wh, self.energy_wh, self.operating_value_euro)) + continuation = float(np.interp(energy_wh, self.energy_wh, self.continuation_value_euro)) + return operating, continuation + + def diagnostic_plan(self, energy_wh: float, control_horizon_hours: float) -> list[TailPlanSlot]: + """Replay the optimal tail path for one control-end battery state.""" + context = self._trace_context + if not context: + return [] + bat = Battery( + context["battery_parameters"], + prediction_hours=1, + slot_duration_h=context["slot_duration_h"], + ) + inv = Inverter( + context["inverter_parameters"], + battery=bat, + slot_duration_h=context["slot_duration_h"], + ) + conversion = bat.discharging_efficiency * inv.dc_to_ac_efficiency + state_wh = bat.min_soc_wh + (energy_wh / conversion if conversion > 0 else 0.0) + state_wh = float(np.clip(state_wh, bat.min_soc_wh, bat.max_soc_wh)) + plan: list[TailPlanSlot] = [] + arrays = zip( + context["prices"], + context["load"], + context["pv"], + context["tariffs"], + ) + for slot, (price, load, pv, tariff) in enumerate(arrays): + candidates: list[tuple[float, TailAction, dict[str, float], float]] = [] + for action in context["actions"]: + result = _simulate_action( + bat=bat, + inv=inv, + energy_wh=state_wh, + action=action, + price=price, + load=load, + pv=pv, + tariff=tariff, + direct_marketing=context["direct_marketing"], + ) + remaining = float( + np.interp( + result["next_state_wh"], + context["states"], + context["future_values"][slot + 1], + ) + ) + candidates.append((result["reward_euro"] + remaining, action, result, remaining)) + candidates.sort(key=lambda candidate: candidate[0], reverse=True) + chosen_value, chosen_action, chosen_result, remaining = candidates[0] + alternative = next( + ( + candidate + for candidate in candidates[1:] + if _action_name(candidate[1]) != _action_name(chosen_action) + ), + candidates[1] if len(candidates) > 1 else candidates[0], + ) + dc, discharge, ac_rate, export_rate = chosen_action + start_soc = state_wh / bat.capacity_wh * 100 + state_wh = chosen_result["next_state_wh"] + plan.append( + TailPlanSlot( + slot=slot, + hour_from_start=control_horizon_hours + slot * context["slot_duration_h"], + action=_action_name(chosen_action), + alternative_action=_action_name(alternative[1]), + decision_margin_euro=max(chosen_value - alternative[0], 0.0), + soc_start_percentage=start_soc, + soc_end_percentage=state_wh / bat.capacity_wh * 100, + pv_wh=pv, + load_wh=load, + grid_import_wh=chosen_result["grid_import_wh"], + grid_export_wh=chosen_result["grid_export_wh"], + battery_charge_wh=chosen_result["battery_charge_wh"], + battery_discharge_wh=chosen_result["battery_discharge_wh"], + import_price_euro_per_kwh=price * 1000, + feed_in_tariff_euro_per_kwh=tariff * 1000, + slot_value_euro=chosen_result["reward_euro"], + remaining_value_euro=remaining, + ac_charge_factor=ac_rate, + dc_charge_allowed=dc, + discharge_allowed=discharge, + battery_grid_export_factor=export_rate, + ) + ) + return plan + + +def build_tail_value_curve( + *, + battery: Battery, + inverter: Inverter, + prices_euro_per_wh: np.ndarray, + load_wh: np.ndarray, + pv_wh: np.ndarray, + feed_in_euro_per_wh: np.ndarray, + continuation: TerminalValueCurve, + charge_rates: list[float], + export_rates: list[float], + direct_marketing: bool, + grid_points: int = 101, +) -> TailValueCurve: + """Solve the finite tail once, backwards in time, without mutating run devices. + + LCOS uses delivered DC energy, exactly as in GeneticSimulation. State grid + endpoints include battery minimum and maximum SOC. Continuous next states are + interpolated rather than rounded (which would invent or destroy energy). + """ + arrays = [ + np.asarray(a, dtype=float) + for a in (prices_euro_per_wh, load_wh, pv_wh, feed_in_euro_per_wh) + ] + if len({len(a) for a in arrays}) != 1 or any(not np.isfinite(a).all() for a in arrays): + raise ValueError("Tail forecasts must have equal lengths and contain only finite values") + bat = Battery(battery.parameters, prediction_hours=1, slot_duration_h=battery.slot_duration_h) + bat.charge_array = np.zeros(1, dtype=float) + inv = Inverter(inverter.parameters, battery=bat, slot_duration_h=battery.slot_duration_h) + states = np.linspace(bat.min_soc_wh, bat.max_soc_wh, grid_points) + usable = (states - bat.min_soc_wh) * bat.discharging_efficiency * inv.dc_to_ac_efficiency + continuation_values = np.array([continuation.value(e) for e in usable]) + operating_values = np.zeros(len(states)) + values = continuation_values.copy() + # (DC charge, local discharge, AC rate, export rate). Preserve production + # modes; direct marketing permits disabling DC charge to make headroom. + actions: list[TailAction] = [(1, 0, 0.0, 0.0), (1, 1, 0.0, 0.0)] + actions += [(1, 0, rate, 0.0) for rate in charge_rates if rate > 0] + if direct_marketing: + actions += [(0, 0, 0.0, 0.0), (0, 1, 0.0, 0.0)] + actions += [(0, 0, rate, 0.0) for rate in charge_rates if rate > 0] + actions += [(dc, 1, 0.0, rate) for dc in (0, 1) for rate in export_rates if rate > 0] + future_values: list[np.ndarray] = [np.empty(0)] * (len(arrays[0]) + 1) + future_values[-1] = continuation_values.copy() + for slot in reversed(range(len(arrays[0]))): + price, load, pv, tariff = (array[slot] for array in arrays) + best = np.full(len(states), -np.inf) + best_operating = np.zeros(len(states)) + best_continuation = np.zeros(len(states)) + for action in actions: + next_states = np.empty(len(states)) + rewards = np.empty(len(states)) + for i, energy in enumerate(states): + action_result = _simulate_action( + bat=bat, + inv=inv, + energy_wh=energy, + action=action, + price=price, + load=load, + pv=pv, + tariff=tariff, + direct_marketing=direct_marketing, + ) + rewards[i] = action_result["reward_euro"] + next_states[i] = action_result["next_state_wh"] + candidate_operating = rewards + np.interp(next_states, states, operating_values) + candidate_continuation = np.interp(next_states, states, continuation_values) + candidate = candidate_operating + candidate_continuation + better = candidate > best + best[better] = candidate[better] + best_operating[better] = candidate_operating[better] + best_continuation[better] = candidate_continuation[better] + values = best + operating_values = best_operating + continuation_values = best_continuation + future_values[slot] = best.copy() + result = TailValueCurve( + energy_wh=usable.tolist(), + value_euro=values.tolist(), + operating_value_euro=operating_values.tolist(), + continuation_value_euro=continuation_values.tolist(), + marginal_euro_per_kwh=(np.diff(values) / np.maximum(np.diff(usable), 1e-9) * 1000).tolist(), + window_slots=len(arrays[0]), + ) + result._trace_context = { + "battery_parameters": battery.parameters, + "inverter_parameters": inverter.parameters, + "slot_duration_h": battery.slot_duration_h, + "states": states, + "future_values": future_values, + "actions": actions, + "prices": arrays[0], + "load": arrays[1], + "pv": arrays[2], + "tariffs": arrays[3], + "direct_marketing": direct_marketing, + } + return result diff --git a/src/akkudoktoreos/optimization/genetic/terminalvalue.py b/src/akkudoktoreos/optimization/genetic/terminalvalue.py new file mode 100644 index 00000000..f8c45990 --- /dev/null +++ b/src/akkudoktoreos/optimization/genetic/terminalvalue.py @@ -0,0 +1,359 @@ +"""Terminal value of the energy left in the battery at the end of the horizon. + +The optimizer stops at the horizon, but the energy still stored in the battery +keeps its worth: it replaces grid imports that would otherwise be paid for +afterwards. Crediting that worth with a single price per kWh - the historical +``preis_euro_pro_wh_akku`` - cannot describe it, because the value of stored +energy is **not linear in the amount stored**: + +- The first kWh replaces the most expensive hour after the horizon. +- The next one replaces the second most expensive hour, and so on. +- Once every hour that PV cannot cover is served, further energy replaces + nothing; it is worth an export at best, and nothing at worst. + +The resulting value function is monotone and concave. A scalar has to pick one +slope: high enough for the first kWh means hoarding a full battery, low enough +for the last kWh means running it empty by midnight. This module builds the +curve instead. + +There is no forecast beyond the horizon, so the trailing window of the horizon +itself stands in for the day that follows: same season, same household rhythm, +same tariff structure. That approximation is the reason the curve is a planning +aid, not a prediction - which is also why the marginal values are deliberately +conservative wherever a choice exists. +""" + +from typing import Optional + +import numpy as np +from loguru import logger +from pydantic import Field + +from akkudoktoreos.core.pydantic import PydanticBaseModel + + +class TerminalValueCurve(PydanticBaseModel): + """Piecewise linear, concave value of battery energy left at the horizon. + + ``energy_wh`` and ``value_euro`` are the breakpoints of the cumulative + value, ``marginal_euro_per_kwh`` the slope of each segment. Both arrays + start at the origin; the curve is flat beyond its last breakpoint. + """ + + energy_wh: list[float] = Field( + default_factory=list, + json_schema_extra={ + "description": "Breakpoints of usable AC energy left in the battery [Wh]." + }, + ) + value_euro: list[float] = Field( + default_factory=list, + json_schema_extra={"description": "Cumulative credit at each breakpoint [EUR]."}, + ) + operating_value_euro: list[float] = Field( + default_factory=list, + json_schema_extra={ + "description": "Tail operating component at each breakpoint [EUR]; empty for a proxy curve." + }, + ) + continuation_value_euro: list[float] = Field( + default_factory=list, + json_schema_extra={ + "description": "Continuation component at each breakpoint [EUR]; empty for a proxy curve." + }, + ) + marginal_euro_per_kwh: list[float] = Field( + default_factory=list, + json_schema_extra={ + "description": ( + "Marginal value of the segment that starts at each breakpoint " + "[EUR/kWh]. May be negative or non-monotone in TAIL mode." + ) + }, + ) + residual_energy_wh: float = Field( + default=0.0, + json_schema_extra={ + "description": ( + "Energy up to which the curve is backed by residual load - the " + "knee. Everything beyond it is only worth an export." + ) + }, + ) + window_slots: int = Field( + default=0, + json_schema_extra={ + "description": ( + "Number of trailing horizon slots the curve was derived from. " + "Fewer slots than a full day mean a shorter proxy period." + ) + }, + ) + + def value(self, energy_wh: float) -> float: + """Return the credit for ``energy_wh`` of usable AC energy [EUR]. + + Args: + energy_wh: Usable AC energy left in the battery. + + Returns: + Interpolated value of the curve; 0.0 for an empty curve. + """ + if not self.energy_wh or energy_wh <= 0.0: + return 0.0 + return float(np.interp(energy_wh, self.energy_wh, self.value_euro)) + + +class TailDiagnostics(PydanticBaseModel): + """Forecast summary used by the deterministic tail optimization.""" + + slots: int = 0 + slot_hours: float = 0.0 + soc_grid_points: int = 0 + min_import_price_euro_per_kwh: float = 0.0 + max_import_price_euro_per_kwh: float = 0.0 + min_feed_in_tariff_euro_per_kwh: float = 0.0 + max_feed_in_tariff_euro_per_kwh: float = 0.0 + negative_import_price_slots: int = 0 + positive_battery_export_slots: int = 0 + + +class TailPlanSlot(PydanticBaseModel): + """One diagnostic slot of the optimal tail path. + + These values explain the lookahead used for fitness. They are diagnostics + only and are never copied into the executable control arrays. + """ + + slot: int + hour_from_start: float + action: str + alternative_action: str = "" + decision_margin_euro: float = 0.0 + soc_start_percentage: float + soc_end_percentage: float + pv_wh: float + load_wh: float + grid_import_wh: float + grid_export_wh: float + battery_charge_wh: float + battery_discharge_wh: float + import_price_euro_per_kwh: float + feed_in_tariff_euro_per_kwh: float + slot_value_euro: float + remaining_value_euro: float + ac_charge_factor: float + dc_charge_allowed: int + discharge_allowed: int + battery_grid_export_factor: float + + +class TerminalValueResult(PydanticBaseModel): + """What the optimizer credited for the energy left in the battery.""" + + control_horizon_hours: float = 0 + requested_tail_hours: float = 0 + effective_tail_hours: float = 0 + tail_end_hour: float = 0 + continuation_mode: str = "FIXED" + + mode: str = Field( + json_schema_extra={ + "description": "Terminal value mode the run used: TAIL, AUTO or FIXED.", + "examples": ["TAIL", "AUTO", "FIXED"], + } + ) + battery_energy_wh: float = Field( + default=0.0, + json_schema_extra={ + "description": "Usable AC energy left in the battery at the end of the horizon [Wh]." + }, + ) + credited_euro: float = Field( + default=0.0, + json_schema_extra={"description": "Credit applied to the total balance [EUR]."}, + ) + tail_operating_euro: float = Field( + default=0.0, + json_schema_extra={ + "description": ( + "Optimal net cash flow within the effective tail for the selected " + "control-end battery state [EUR]." + ) + }, + ) + continuation_value_euro: float = Field( + default=0.0, + json_schema_extra={ + "description": ( + "Continuation credit remaining at the end of the optimal tail path [EUR]." + ) + }, + ) + curve: Optional[TerminalValueCurve] = Field( + default=None, + json_schema_extra={ + "description": ( + "Combined tail value curve (tail operation plus continuation) read by fitness; " + "None in FIXED mode." + ) + }, + ) + continuation_curve: Optional[TerminalValueCurve] = Field( + default=None, + json_schema_extra={ + "description": "Conservative AUTO proxy constructed at the effective tail end." + }, + ) + tail_diagnostics: Optional[TailDiagnostics] = None + tail_plan: list[TailPlanSlot] = Field( + default_factory=list, + json_schema_extra={ + "description": ( + "Diagnostic optimal battery path inside the tail. It explains the " + "lookahead but is never an executable control plan." + ) + }, + ) + reason: str = Field( + default="", + json_schema_extra={ + "description": ( + "Why this mode applied. Empty in AUTO mode; in FIXED mode it " + "says whether FIXED was configured or whether AUTO fell back " + "because no curve could be derived." + ), + "examples": ["", "terminal_value_mode is FIXED"], + }, + ) + + +def build_terminal_value_curve( + *, + prices_euro_per_wh: np.ndarray, + load_wh: np.ndarray, + pv_wh: np.ndarray, + feed_in_euro_per_wh: np.ndarray, + max_energy_wh: float, + lcos_euro_per_kwh: float = 0.0, + dc_to_ac_efficiency: float = 1.0, + grid_export_allowed: bool = False, +) -> TerminalValueCurve: + """Build the terminal value curve from the trailing horizon window. + + Every slot of the window contributes its residual load - the part of the + load that PV does not cover - at its import price. Sorting those slots by + price and accumulating them yields the marginal value of the first, second, + ... kWh in the battery. Energy beyond the residual load can only be + exported, and only when direct marketing allows it. + + Args: + prices_euro_per_wh: Import prices of the window [EUR/Wh]. + load_wh: Load per slot of the window [Wh]. + pv_wh: PV generation per slot of the window [Wh]. + feed_in_euro_per_wh: Feed-in tariff of the window [EUR/Wh]. + max_energy_wh: Usable AC energy of a full battery [Wh]; the curve ends here. + lcos_euro_per_kwh: Levelized cost of storage, already charged per + delivered DC energy in the simulation and therefore subtracted here + so stored energy is not credited twice. + dc_to_ac_efficiency: Inverter efficiency, used to convert the LCOS from + delivered DC energy to the AC energy of the curve. + grid_export_allowed: Whether the battery may feed the grid (direct + marketing). Without it, energy beyond the residual load gets no + credit: it can neither be exported nor is its use covered by the + proxy window. + + Returns: + The curve; empty when the window carries no usable information. + """ + window = min(len(prices_euro_per_wh), len(load_wh), len(pv_wh)) + if window <= 0 or max_energy_wh <= 0.0: + return TerminalValueCurve() + + residual = np.maximum(load_wh[:window] - pv_wh[:window], 0.0) + prices = np.asarray(prices_euro_per_wh[:window], dtype=float) + + # LCOS is charged on delivered DC energy; the curve is in AC energy. + lcos_per_wh_ac = (lcos_euro_per_kwh / 1000.0) / max(dc_to_ac_efficiency, 1e-9) + + order = np.argsort(-prices) + energy_points: list[float] = [0.0] + value_points: list[float] = [0.0] + marginals: list[float] = [] + + cumulative_energy = 0.0 + cumulative_value = 0.0 + for index in order: + slot_energy = float(residual[index]) + if slot_energy <= 0.0: + continue + # Negative or very cheap hours are not worth storing energy for. + marginal = max(float(prices[index]) - lcos_per_wh_ac, 0.0) + if marginal <= 0.0: + continue + slot_energy = min(slot_energy, max_energy_wh - cumulative_energy) + if slot_energy <= 0.0: + break + cumulative_energy += slot_energy + cumulative_value += slot_energy * marginal + energy_points.append(cumulative_energy) + value_points.append(cumulative_value) + marginals.append(marginal * 1000.0) + + # Everything beyond the residual load can only be sold. A median feed-in + # tariff rather than the best one: exporting all of it in the single best + # slot is not something the horizon can promise. + residual_energy_wh = cumulative_energy + if grid_export_allowed and cumulative_energy < max_energy_wh: + positive_feed_in = [ + float(value) for value in feed_in_euro_per_wh[:window] if float(value) > 0.0 + ] + export_marginal = max( + (float(np.median(positive_feed_in)) if positive_feed_in else 0.0) - lcos_per_wh_ac, + 0.0, + ) + if export_marginal > 0.0: + remaining = max_energy_wh - cumulative_energy + cumulative_energy += remaining + cumulative_value += remaining * export_marginal + energy_points.append(cumulative_energy) + value_points.append(cumulative_value) + marginals.append(export_marginal * 1000.0) + + if len(energy_points) <= 1: + logger.debug("Terminal value curve is empty - no priced residual load in the window.") + return TerminalValueCurve(window_slots=window) + + # The segment slopes are decreasing by construction (prices were sorted), + # so the curve is concave; the export tail is the flattest segment. + return TerminalValueCurve( + energy_wh=energy_points, + value_euro=value_points, + marginal_euro_per_kwh=marginals, + residual_energy_wh=residual_energy_wh, + window_slots=window, + ) + + +def trailing_window( + values: Optional[np.ndarray], + end_slot: int, + window_slots: int, +) -> np.ndarray: + """Return the ``window_slots`` values in front of ``end_slot``. + + Args: + values: Full slot array, or None. + end_slot: Exclusive end of the window (end of the optimization horizon). + window_slots: Desired window length; a shorter horizon yields less. + + Returns: + The window as a float array, empty when no data is available. + """ + if values is None: + return np.zeros(0, dtype=float) + end = min(int(end_slot), len(values)) + start = max(end - int(window_slots), 0) + if end <= start: + return np.zeros(0, dtype=float) + return np.asarray(values[start:end], dtype=float) diff --git a/src/akkudoktoreos/optimization/optimization.py b/src/akkudoktoreos/optimization/optimization.py index 51770b86..48cb09e7 100644 --- a/src/akkudoktoreos/optimization/optimization.py +++ b/src/akkudoktoreos/optimization/optimization.py @@ -1,7 +1,7 @@ from enum import StrEnum -from typing import Optional +from typing import Any, Optional -from pydantic import Field, computed_field +from pydantic import Field, computed_field, model_validator from akkudoktoreos.config.configabc import SettingsBaseModel from akkudoktoreos.core.coreabc import get_ems @@ -10,7 +10,10 @@ from akkudoktoreos.core.pydantic import ( PydanticDateTimeDataFrame, ) from akkudoktoreos.optimization.genetic0.genetic0settings import Genetic0CommonSettings -from akkudoktoreos.optimization.genetic.geneticsettings import GeneticCommonSettings +from akkudoktoreos.optimization.genetic.geneticsettings import ( + GeneticCommonSettings, + normalize_genetic_settings, +) from akkudoktoreos.utils.datetimeutil import DateTime @@ -29,6 +32,12 @@ def optimization_default_algorithm() -> OptimizationAlgorithm: class OptimizationCommonSettings(SettingsBaseModel): """General Optimization Configuration.""" + @model_validator(mode="before") + @classmethod + def normalize_old_genetic_settings(cls, value: Any) -> Any: + """Accept the feature branch's flat GENETIC settings without losing values.""" + return normalize_genetic_settings(value) + algorithm: OptimizationAlgorithm = Field( default_factory=optimization_default_algorithm, json_schema_extra={ diff --git a/src/akkudoktoreos/prediction/feedintariff.py b/src/akkudoktoreos/prediction/feedintariff.py index 81975a24..6f9cd5e2 100644 --- a/src/akkudoktoreos/prediction/feedintariff.py +++ b/src/akkudoktoreos/prediction/feedintariff.py @@ -42,6 +42,14 @@ def feedintariff_provider_ids() -> list[str]: class FeedInTariffCommonSettings(SettingsBaseModel): """Feed In Tariff Prediction Configuration.""" + direct_marketing_enabled: bool = Field( + default=False, + description=( + "Enable export-aware GENETIC optimization. Sale revenues remain those of the " + "configured feed-in provider or explicit forecast; purchase prices never replace them." + ), + ) + provider: Optional[str] = Field( default=None, json_schema_extra={ diff --git a/src/akkudoktoreos/server/eos.py b/src/akkudoktoreos/server/eos.py index 184d08f3..8bf834fd 100755 --- a/src/akkudoktoreos/server/eos.py +++ b/src/akkudoktoreos/server/eos.py @@ -64,7 +64,11 @@ from akkudoktoreos.optimization.genetic0.genetic0solution import ( from akkudoktoreos.optimization.genetic0.genetic0visualize import ( genetic0_prepare_visualize, ) +from akkudoktoreos.optimization.genetic.configrequest import ConfigOptimizationRequest from akkudoktoreos.optimization.genetic.geneticsolution import GeneticSolution +from akkudoktoreos.optimization.genetic.geneticvisualize import ( + genetic_prepare_visualize, +) from akkudoktoreos.optimization.optimization import ( OptimizationAlgorithm, OptimizationSolution, @@ -1867,6 +1871,35 @@ async def fastapi_energy_management_optimization_solution_algorithm_get( return solution +@app.get( + "/v1/energy-management/optimization/solution/GENETIC/pdf", + tags=["energy-management"], + response_class=Response, + responses={200: {"content": {"application/pdf": {}}}}, +) +async def fastapi_energy_management_optimization_solution_genetic_pdf_get() -> Response: + """Render the retained GENETIC result without rerunning optimization. + + Rendering runs outside the event loop. Copy the result before offloading; + its recorded timestamp, interval and inputs own the report's time grid. + The legacy /visualization_results.pdf route continues to serve GENETIC0. + """ + retained = get_ems().genetic_solution() + if retained is None: + raise EOSProblem( + status=404, + title="Optimization solution report retrieval failed", + detail="Can not get the 'GENETIC' optimization solution.", + ) + snapshot = retained.model_copy(deep=True) + pdf = await asyncio.to_thread(genetic_prepare_visualize, solution=snapshot) + return Response( + content=pdf, + media_type="application/pdf", + headers={"Content-Disposition": 'inline; filename="optimization-genetic.pdf"'}, + ) + + @app.get("/v1/energy-management/plan", tags=["energy-management"]) def fastapi_energy_management_plan_get() -> EnergyManagementPlan: """Get the latest energy management plan.""" @@ -2174,6 +2207,44 @@ async def fastapi_pvforecast() -> ForecastResponse: return ForecastResponse(temperature=temp_air_list, pvpower=ac_power_list) +@app.post("/v1/optimize", tags=["optimize"]) +async def fastapi_optimize_config( + request: Request, + parameters: ConfigOptimizationRequest = Body(default_factory=ConfigOptimizationRequest), +) -> GeneticSolution: + """Optimize GENETIC using configured devices and optional fresh runtime inputs. + + Static settings belong in configuration; query overrides are rejected. + Forecast arrays start at local + midnight and contain Wh per configured GENETIC slot; prices are currency/Wh. + An empty body uses configured providers and fresh measured states of charge. + The deprecated /optimize endpoint continues to run hourly GENETIC0. + """ + if request.query_params: + raise HTTPException( + status_code=422, + detail="Configure optimization settings; query overrides are not supported.", + ) + solution = await get_ems().run( + mode=EnergyManagementMode.OPTIMIZATION, + algorithm=OptimizationAlgorithm.GENETIC, + genetic_parameters=parameters, + ) + if solution is None: + raise EOSProblem( + status=503, + title="GENETIC optimization failed", + detail="No new solution was produced. Check configured devices, fresh SoC and forecast coverage.", + ) + if not isinstance(solution, GeneticSolution): + raise EOSProblem( + status=500, + title="Unexpected optimization algorithm result", + detail="GENETIC did not return its native solution type.", + ) + return solution + + @app.post("/optimize", tags=["optimize"]) async def fastapi_optimize( parameters: Genetic0OptimizationParameters, diff --git a/tests/test_config_optimization_request.py b/tests/test_config_optimization_request.py new file mode 100644 index 00000000..477e971d --- /dev/null +++ b/tests/test_config_optimization_request.py @@ -0,0 +1,398 @@ +"""Configuration-owned requests and the real automatic GENETIC path.""" + +from types import SimpleNamespace +from unittest.mock import AsyncMock, Mock + +import httpx +import numpy as np +import pandas as pd +import pytest +from fastapi.testclient import TestClient +from pydantic import ValidationError + +from akkudoktoreos.config.configmigrate import migrate_config_data +from akkudoktoreos.core.ems import EnergyManagement, EnergyManagementStage +from akkudoktoreos.core.emsettings import EnergyManagementMode +from akkudoktoreos.optimization.genetic import configrequest +from akkudoktoreos.optimization.genetic.configrequest import ConfigOptimizationRequest +from akkudoktoreos.optimization.optimization import ( + OptimizationAlgorithm, + OptimizationCommonSettings, +) +from akkudoktoreos.utils.datetimeutil import to_datetime + + +@pytest.fixture +def configured_request(config_eos, monkeypatch): + config_eos.merge_settings_from_dict( + { + "prediction": {"hours": 24}, + "optimization": { + "algorithm": "GENETIC", + "genetic": { + "interval_sec": 900, + "horizon_hours": 1, + "tail_horizon_hours": 0, + "terminal_value_mode": "FIXED", + "terminal_value_euro_per_kwh": 0.23, + "individuals": 10, + "generations": 10, + "seed": 42, + }, + }, + "devices": { + "max_batteries": 1, + "max_electric_vehicles": 0, + "max_inverters": 1, + "max_home_appliances": 2, + "batteries": { + "storage": { + "capacity_wh": 2000, + "max_charge_power_w": 1000, + "levelized_cost_of_storage_amt_kwh": 0.01, + } + }, + "electric_vehicles": {}, + "inverters": {"inverter": {"battery_id": "storage", "max_power_w": 1000}}, + "home_appliances": {}, + }, + } + ) + start = to_datetime("2026-09-12T10:15:00Z", in_timezone="UTC") + ems = SimpleNamespace( + start_datetime=start, observation_datetime=start, genetic_solution=lambda: None + ) + monkeypatch.setattr(configrequest, "get_ems", lambda: ems) + measurement = Mock(key_to_lists=AsyncMock(return_value=([], []))) + monkeypatch.setattr(ConfigOptimizationRequest, "measurement", measurement) + data = { + "soc": {"storage": 42}, + "forecasts": { + "pv_forecast_wh": [100.0] * 96, + "total_load": [200.0] * 96, + "electricity_price_per_wh": [0.0003] * 96, + "feed_in_tariff_per_wh": [0.00008] * 96, + }, + } + return config_eos, ems, measurement, data + + +@pytest.mark.asyncio +async def test_hardware_costs_and_state_are_resolved_without_config_changes(configured_request): + config, _, _, data = configured_request + before = config.model_dump_json() + parameters = await ConfigOptimizationRequest.model_validate(data).resolve() + assert parameters.pv_battery is not None + assert parameters.pv_battery.device_id == "storage" + assert parameters.pv_battery.capacity_wh == 2000 + assert parameters.pv_battery.initial_soc_percentage == 42 + assert parameters.pv_battery.levelized_cost_of_storage_kwh == 0.01 + assert parameters.ems.price_per_wh_battery == pytest.approx(0.00023) + assert parameters.forecast_interval_seconds == 900 + assert config.model_dump_json() == before + + +@pytest.mark.parametrize("field", ["devices", "optimization", "pv_battery", "inverter", "ems"]) +def test_static_http_overrides_are_rejected(field): + with pytest.raises(ValidationError): + ConfigOptimizationRequest.model_validate({field: {}}) + + +@pytest.mark.parametrize("host_timezone", ["UTC", "Europe/Berlin"]) +def test_warmstart_timestamp_retains_explicit_zone_and_json_instant( + set_other_timezone, host_timezone +): + set_other_timezone(host_timezone) + previous = to_datetime("2026-10-25T02:30:00+01:00", in_timezone="Europe/Berlin") + request = ConfigOptimizationRequest(start_solution_datetime=previous) + assert request.start_solution_datetime is not None + assert request.start_solution_datetime.timezone_name == "Europe/Berlin" + assert request.start_solution_datetime.timestamp() == previous.timestamp() + restored = ConfigOptimizationRequest.model_validate_json(request.model_dump_json()) + assert restored.start_solution_datetime is not None + assert restored.start_solution_datetime.timestamp() == previous.timestamp() + assert restored.start_solution_datetime.utcoffset() == previous.utcoffset() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("age,value", [(301, 0.45), (-1, 0.45), (1, None), (1, np.nan), (1, 1.1)]) +async def test_missing_stale_future_or_invalid_soc_never_becomes_zero( + configured_request, age, value +): + _, ems, measurement, data = configured_request + data["soc"] = {} + measurement.key_to_lists.return_value = ( + [ems.observation_datetime.subtract(seconds=age)], + [value], + ) + with pytest.raises(ValueError, match="Fresh SoC missing"): + await ConfigOptimizationRequest.model_validate(data).resolve() + + +@pytest.mark.asyncio +async def test_soc_freshness_uses_actual_run_time_inside_quarter_hour(configured_request): + _, ems, measurement, data = configured_request + data["soc"] = {} + ems.observation_datetime = ems.start_datetime.add(minutes=7) + measurement.key_to_lists.return_value = ( + [ems.observation_datetime.subtract(seconds=30)], + [0.456], + ) + parameters = await ConfigOptimizationRequest.model_validate(data).resolve() + assert parameters.pv_battery is not None + assert parameters.pv_battery.initial_soc_percentage == 45 + assert measurement.key_to_lists.call_args.kwargs[ + "end_datetime" + ] == ems.observation_datetime.add(seconds=1) + + +@pytest.mark.asyncio +async def test_future_soc_in_repeated_hour_is_rejected(configured_request): + _, ems, measurement, data = configured_request + data["soc"] = {} + ems.observation_datetime = to_datetime("2026-10-25T02:45:00+02:00", in_timezone="Europe/Berlin") + measurement.key_to_lists.return_value = ( + [to_datetime("2026-10-25T02:15:00+01:00", in_timezone="Europe/Berlin")], + [0.8], + ) + with pytest.raises(ValueError, match="Fresh SoC missing"): + await ConfigOptimizationRequest.model_validate(data).resolve() + + +@pytest.mark.asyncio +async def test_unknown_soc_and_mismatched_device_link_are_rejected(configured_request): + config, _, _, data = configured_request + request = ConfigOptimizationRequest.model_validate(data) + request.soc["unknown"] = 20 + with pytest.raises(ValueError, match="unconfigured"): + await request.resolve() + assert config.devices.inverters is not None + config.devices.inverters["inverter"].battery_id = "missing" + with pytest.raises(ValueError, match="battery_id"): + await ConfigOptimizationRequest.model_validate(data).resolve() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("tariff", [0.00008, 0.0, -0.00005]) +async def test_direct_marketing_keeps_explicit_imported_sale_prices(configured_request, tariff): + config, _, _, data = configured_request + config.feedintariff.direct_marketing_enabled = True + config.feedintariff.provider = "FeedInTariffImport" + data["forecasts"]["feed_in_tariff_per_wh"] = [tariff] * 96 + parameters = await ConfigOptimizationRequest.model_validate(data).resolve() + assert parameters.ems.feed_in_tariff_per_wh == [tariff] * 96 + + +@pytest.mark.asyncio +async def test_control_gaps_fail_but_shorter_tail_is_accepted(configured_request): + _, _, _, data = configured_request + data["forecasts"]["electricity_price_per_wh"] = [0.0003] * 50 + parameters = await ConfigOptimizationRequest.model_validate(data).resolve() + assert len(parameters.ems.total_load) == 50 + data["forecasts"]["electricity_price_per_wh"][42] = np.nan + with pytest.raises(ValueError, match="control horizon"): + await ConfigOptimizationRequest.model_validate(data).resolve() + + +@pytest.mark.asyncio +async def test_provider_power_is_integrated_once_and_update_is_awaited( + configured_request, monkeypatch +): + _, _, _, data = configured_request + data["forecasts"] = {} + series = { + "pvforecast_ac_power": 1000.0, + "loadforecast_power_w": 2000.0, + "elecprice_marketprice_wh": 0.0003, + "feed_in_tariff_wh": -0.00005, + } + + async def read(key, **kwargs): + return pd.Series( + [series[key]] * 48, index=pd.date_range("2026-09-12T00:00:00Z", periods=48, freq="h") + ) + + prediction = Mock(update_data=AsyncMock(), key_to_raw_series=AsyncMock(side_effect=read)) + monkeypatch.setattr(ConfigOptimizationRequest, "prediction", prediction) + parameters = await ConfigOptimizationRequest.model_validate(data).resolve() + assert parameters.ems.pv_forecast_wh == [250.0] * len(parameters.ems.pv_forecast_wh) + assert parameters.ems.total_load == [500.0] * len(parameters.ems.total_load) + assert parameters.ems.feed_in_tariff_per_wh == [-0.00005] * len(parameters.ems.total_load) + prediction.update_data.assert_awaited_once() + + +@pytest.mark.parametrize( + "stamp,interval,expected", + [ + ("2026-03-29T03:17:00+02:00", 900, "2026-03-29T03:15:00+02:00"), + ("2026-10-25T02:47:00+01:00", 900, "2026-10-25T02:45:00+01:00"), + ("2026-10-25T02:47:00+01:00", 3600, "2026-10-25T02:00:00+01:00"), + ], +) +def test_slot_alignment_preserves_dst_fold(config_eos, stamp, interval, expected): + time = to_datetime(stamp, in_timezone="Europe/Berlin") + aligned = EnergyManagement.set_start_datetime(time, interval_seconds=interval) + assert aligned == to_datetime(expected, in_timezone="Europe/Berlin") + assert aligned.utcoffset() == to_datetime(expected, in_timezone="Europe/Berlin").utcoffset() + + +@pytest.mark.parametrize("timezone", ["Asia/Kolkata", "Asia/Kathmandu"]) +@pytest.mark.parametrize("interval,minute", [(900, 30), (3600, 0)]) +def test_slot_alignment_uses_local_midnight(config_eos, timezone, interval, minute): + time = to_datetime("2026-09-12T10:40:00", in_timezone=timezone) + aligned = EnergyManagement.set_start_datetime(time, interval_seconds=interval) + assert aligned.hour == 10 + assert aligned.minute == minute + assert aligned.timezone_name == timezone + assert EnergyManagement().observation_datetime == time + + +def test_old_feature_config_migrates_without_changing_explicit_nested_values(config_eos): + raw = { + "optimization": { + "algorithm": "GENETIC", + "interval": 900, + "horizon_hours": 12, + "terminal_value_euro_per_kwh": 0.23, + "genetic": {"horizon_hours": 8}, + } + } + migrated = migrate_config_data(raw) + assert migrated.optimization.genetic.interval_sec == 900 + assert migrated.optimization.genetic.horizon_hours == 8 + assert migrated.optimization.genetic.terminal_value_euro_per_kwh == 0.23 + settings = OptimizationCommonSettings.model_validate(raw["optimization"]) + settings.algorithm = OptimizationAlgorithm.GENETIC0 + assert settings.genetic.interval_sec == 900 + assert settings.genetic.generations == 400 + + +def test_http_empty_body_is_configuration_request_and_failure_cannot_return_cache(monkeypatch): + from akkudoktoreos.server import eos + + run = AsyncMock(return_value=None) + monkeypatch.setattr(eos, "get_ems", lambda: SimpleNamespace(run=run)) + client = TestClient(eos.app) + response = client.post("/v1/optimize") + assert response.status_code == 503 + assert isinstance(run.call_args.kwargs["genetic_parameters"], ConfigOptimizationRequest) + assert run.call_args.kwargs["algorithm"] == OptimizationAlgorithm.GENETIC + response = client.post("/v1/optimize", json={"devices": {}}) + assert response.status_code == 422 + assert run.await_count == 1 + + +@pytest.mark.parametrize("query", ["start_hour=4", "ngen=1", "interval=3600", "unknown="]) +def test_http_configuration_request_rejects_query_overrides(monkeypatch, query): + from akkudoktoreos.server import eos + + run = AsyncMock() + monkeypatch.setattr(eos, "get_ems", lambda: SimpleNamespace(run=run)) + response = TestClient(eos.app).post(f"/v1/optimize?{query}", json={}) + assert response.status_code == 422 + assert "query overrides are not supported" in response.text + run.assert_not_awaited() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("automatic", [False, True]) +async def test_real_ems_returns_coherent_quarter_hour_solution_and_plan( + configured_request, monkeypatch, automatic +): + _, fake, _, data = configured_request + ems = EnergyManagement() + monkeypatch.setattr(configrequest, "get_ems", lambda: ems) + monkeypatch.setattr(EnergyManagement, "prediction", Mock(update_data=AsyncMock())) + monkeypatch.setattr(EnergyManagement, "adapter", Mock(update_data=AsyncMock())) + request = ConfigOptimizationRequest.model_validate(data) + if automatic: + original = ConfigOptimizationRequest.resolve + + async def resolve_default(self): + return await original(request) + + monkeypatch.setattr(ConfigOptimizationRequest, "resolve", resolve_default) + solution = await ems.run( + start_datetime=fake.start_datetime.add(minutes=2), + mode=EnergyManagementMode.OPTIMIZATION, + genetic_parameters=None if automatic else request, + genetic_generations=10, + genetic_seed=42, + ) + assert solution is not None + assert solution is ems.genetic_solution() + assert solution.start_solution_datetime == fake.start_datetime + assert len(solution.ac_charge) == 4 + assert ems.optimization_solution() is not None + assert ems.plan() is not None + assert ems.stage() == EnergyManagementStage.IDLE + + +@pytest.mark.asyncio +async def test_automatic_run_reads_real_resolver_provider_units_and_measured_soc( + configured_request, monkeypatch +): + _, fake, measurement, _ = configured_request + ems = EnergyManagement() + monkeypatch.setattr(configrequest, "get_ems", lambda: ems) + monkeypatch.setattr(EnergyManagement, "_genetic_solution", None) + measurement.key_to_lists.return_value = ([fake.start_datetime], [0.42]) + values = { + "pvforecast_ac_power": 1000.0, + "loadforecast_power_w": 2000.0, + "elecprice_marketprice_wh": 0.0003, + "feed_in_tariff_wh": 0.00008, + } + + async def read(key, **kwargs): + return pd.Series( + [values[key]] * 48, index=pd.date_range("2026-09-12T00:00:00Z", periods=48, freq="h") + ) + + prediction = Mock(update_data=AsyncMock(), key_to_raw_series=AsyncMock(side_effect=read)) + monkeypatch.setattr(EnergyManagement, "prediction", prediction) + monkeypatch.setattr(ConfigOptimizationRequest, "prediction", prediction) + monkeypatch.setattr(EnergyManagement, "adapter", Mock(update_data=AsyncMock())) + solution = await ems.run( + start_datetime=fake.start_datetime, + mode=EnergyManagementMode.OPTIMIZATION, + genetic_generations=10, + genetic_seed=42, + ) + assert solution is not None + assert solution.parameters is not None + assert solution.parameters.pv_battery is not None + assert solution.parameters.pv_battery.initial_soc_percentage == 42 + assert solution.parameters.ems.pv_forecast_wh[:4] == [250.0] * 4 + assert solution.parameters.ems.total_load[:4] == [500.0] * 4 + assert len(solution.ac_charge) == 4 + prediction.key_to_raw_series.assert_awaited() + + +@pytest.mark.asyncio +async def test_http_real_genetic_result_serializes_native_quarter_hour_contract( + configured_request, monkeypatch +): + from akkudoktoreos.server import eos + + _, fake, _, data = configured_request + ems = EnergyManagement() + monkeypatch.setattr(configrequest, "get_ems", lambda: ems) + monkeypatch.setattr(EnergyManagement, "_genetic_solution", None) + monkeypatch.setattr(EnergyManagement, "prediction", Mock(update_data=AsyncMock())) + monkeypatch.setattr(EnergyManagement, "adapter", Mock(update_data=AsyncMock())) + + async def run(**kwargs): + return await ems.run(start_datetime=fake.start_datetime, **kwargs) + + monkeypatch.setattr(eos, "get_ems", lambda: SimpleNamespace(run=run)) + async with httpx.AsyncClient( + transport=httpx.ASGITransport(app=eos.app), base_url="http://test" + ) as client: + response = await client.post("/v1/optimize", json=data) + assert response.status_code == 200, response.text + result = response.json() + assert result["interval_seconds"] == 900 + assert result["controls_start_at_now"] is True + assert len(result["ac_charge"]) == 4 + assert result["parameters"]["pv_battery"]["device_id"] == "storage" diff --git a/tests/test_consolidation_file_restore.py b/tests/test_consolidation_file_restore.py new file mode 100644 index 00000000..d4876a44 --- /dev/null +++ b/tests/test_consolidation_file_restore.py @@ -0,0 +1,29 @@ +"""Verify fallback JSON loading does not lose records through the singleton.""" + +from unittest.mock import AsyncMock + +import pytest + +from akkudoktoreos.core.coreabc import get_measurement +from akkudoktoreos.core.dataabc import DataSequence +from akkudoktoreos.utils.datetimeutil import to_datetime + + +@pytest.mark.asyncio +async def test_measurement_json_roundtrip(config_eos, tmp_path, monkeypatch): + m = get_measurement() + config_eos.measurement.load_emr_keys = ["meter"] + config_eos.general.data_folder_path = tmp_path + config_eos.database.provider = None + m._db_reset_state() + try: + await m.update_value(to_datetime("2026-09-16T08:00:00Z"), "meter", 123.5) + monkeypatch.setattr(DataSequence, "save", AsyncMock(return_value=False)) + monkeypatch.setattr(DataSequence, "load", AsyncMock(return_value=False)) + assert await m.save() + m._db_reset_state() + assert await m.load() + assert len(m.records) == 1 + assert m.records[0]["meter"] == 123.5 + finally: + m._db_reset_state() diff --git a/tests/test_dataabcsequence.py b/tests/test_dataabcsequence.py index b392c49b..4a3b83c2 100644 --- a/tests/test_dataabcsequence.py +++ b/tests/test_dataabcsequence.py @@ -994,10 +994,11 @@ class TestDataSequence: data_dict_keep = await sequence.key_to_dict("data_value", dropna=False) assert pd.isna(data_dict_keep[to_datetime(datetime(2023, 11, 6), as_string=True)]) - async def test_key_to_lists_dropna_removes_nan(self, sequence): + @pytest.mark.parametrize("missing", [None, float("nan")]) + async def test_key_to_lists_dropna_removes_nan(self, sequence, missing): """`dropna=True` (default) must drop records whose value is NaN, not just None.""" record1 = self.create_test_record(datetime(2023, 11, 5), 0.8) - record2 = self.create_test_record(datetime(2023, 11, 6), float("nan")) + record2 = self.create_test_record(datetime(2023, 11, 6), missing) record3 = self.create_test_record(datetime(2023, 11, 7), 0.9) await sequence.insert_by_datetime(record1) await sequence.insert_by_datetime(record2) @@ -1013,6 +1014,27 @@ class TestDataSequence: assert len(values_keep) == 3 assert pd.isna(values_keep[1]) + async def test_raw_none_record_keeps_forecast_gap_and_true_interval_length(self, sequence): + from akkudoktoreos.optimization.genetic.forecast import bounded_forecast_array + + start = to_datetime("2026-09-16T10:00:00Z", in_timezone="UTC") + for minute, value in [(0, 100.0), (15, None), (30, 100.0)]: + await sequence.insert_by_datetime( + self.create_test_record(start.add(minutes=minute), value) + ) + raw = await sequence.key_to_raw_series("data_value", dropna=False) + assert len(raw) == 3 + assert raw.index[1].timestamp() == start.add(minutes=15).timestamp() + assert pd.isna(raw.iloc[1]) + filtered = await sequence.key_to_raw_series("data_value") + assert filtered.tolist() == [100.0, 100.0] + values = await bounded_forecast_array( + sequence, key="data_value", start_datetime=start, + end_datetime=start.add(hours=1), interval=to_duration(900), + ) + assert values[[0, 2]].tolist() == [100.0, 100.0] + assert np.isnan(values[[1, 3]]).all() + async def test_to_dataframe_full_data(self, sequence): """Test conversion of all records to a DataFrame without filtering.""" record1 = self.create_test_record("2024-01-01T12:00:00Z", 10) diff --git a/tests/test_emplan_dst_instants.py b/tests/test_emplan_dst_instants.py new file mode 100644 index 00000000..db2e803d --- /dev/null +++ b/tests/test_emplan_dst_instants.py @@ -0,0 +1,38 @@ +"""Repeated local clock times must dispatch by their real UTC instant.""" + +from akkudoktoreos.core.emplan import EnergyManagementPlan, FRBCInstruction +from akkudoktoreos.utils.datetimeutil import to_datetime + + +def test_repeated_hour_keeps_instruction_order_and_dispatch(): + first = to_datetime("2026-10-25T02:45:00+02:00").in_timezone("Europe/Berlin") + second = to_datetime("2026-10-25T02:15:00+01:00").in_timezone("Europe/Berlin") + plan = EnergyManagementPlan(id="fold", generated_at=first, instructions=[]) + later = FRBCInstruction( + resource_id="battery1", + actuator_id="battery1", + execution_time=second, + operation_mode_id="IDLE", + operation_mode_factor=1.0, + ) + earlier = FRBCInstruction( + resource_id="battery1", + actuator_id="battery1", + execution_time=first, + operation_mode_id="GRID_SUPPORT_IMPORT", + operation_mode_factor=0.5, + ) + plan.add_instruction(later) + plan.add_instruction(earlier) + assert [item.execution_time.timestamp() for item in plan.instructions] == [ + first.timestamp(), + second.timestamp(), + ] + assert plan.valid_from is not None + assert plan.valid_from.timestamp() == first.timestamp() + assert plan.valid_until is None + between = to_datetime("2026-10-25T02:00:00+01:00").in_timezone("Europe/Berlin") + assert plan.get_active_instructions(between) == [earlier] + assert plan.get_next_instruction(between) == later + assert plan.get_active_instructions(second) == [later] + assert plan.get_next_instruction(second) is None diff --git a/tests/test_genetic_complete_devices.py b/tests/test_genetic_complete_devices.py new file mode 100644 index 00000000..2a146bf6 --- /dev/null +++ b/tests/test_genetic_complete_devices.py @@ -0,0 +1,293 @@ +"""Profiles, cycle windows and deadlines on the GENETIC slot grid.""" + +from typing import Any + +import numpy as np +import pytest +from pydantic import ValidationError + +from akkudoktoreos.devices.devicesabc import ( + ConsumerDeadlinePolicy, + ConsumerScheduleMode, +) +from akkudoktoreos.devices.genetic.battery import Battery, ElectricVehicleParameters +from akkudoktoreos.devices.genetic.homeappliance import ( + HomeAppliance, + HomeApplianceParameters, + resample_power_to_slot_energy, +) +from akkudoktoreos.devices.settings.batterysettings import BatteriesCommonSettings +from akkudoktoreos.devices.settings.homeappliancesettings import ( + HomeApplianceCommonSettings, +) +from akkudoktoreos.utils.datetimeutil import to_datetime + + +def appliance(*, slots: int = 192, slot_h: float = 0.25, **kwargs: Any) -> HomeAppliance: + return HomeAppliance( + HomeApplianceParameters.model_validate( + { + "device_id": "washer", + "load_profile_power_w": [1200.0, 600.0, 300.0], + "load_profile_interval_seconds": 600, + **kwargs, + } + ), + optimization_hours=48, + prediction_hours=slots, + slot_duration_h=slot_h, + ) + + +def test_noninteger_resampling_conserves_real_energy() -> None: + # 10-minute 1.2kW/0.6kW/0.3kW phases: 350Wh, delivered in two 15min slots. + device = appliance() + np.testing.assert_allclose(device.run_energy_wh, [250.0, 100.0]) + device.build_load_curve([3, 12]) + assert device.get_load_curve().sum() == pytest.approx(700) + np.testing.assert_allclose(device.get_load_curve()[3:5], [250.0, 100.0]) + assert device.run_slots == 2 + + +@pytest.mark.parametrize("input_s,slot_s", [(600, 900), (1200, 900), (900, 3600), (3600, 900)]) +def test_profile_energy_independent_of_slot_grid(input_s: int, slot_s: int) -> None: + result = resample_power_to_slot_energy([400.0, 1600.0, 200.0], input_s, slot_s) + assert result.sum() == pytest.approx(2200 * input_s / 3600) + + +@pytest.mark.parametrize("profile", [[], [-1], [float("nan")], [float("inf")]]) +def test_invalid_profiles_rejected_in_parameters_and_settings(profile: list[float]) -> None: + for model in (HomeApplianceParameters, HomeApplianceCommonSettings): + with pytest.raises(ValidationError): + model.model_validate({"device_id": "bad", "load_profile_power_w": profile}) + + +@pytest.mark.parametrize( + "fields", [{"duration_h": 2}, {"consumption_wh": 500}, {"duration_h": 2, "consumption_wh": 500}] +) +def test_profile_cannot_silently_override_explicit_flat_definition(fields: dict[str, int]) -> None: + for model in (HomeApplianceParameters, HomeApplianceCommonSettings): + with pytest.raises(ValidationError, match="Conflicting"): + model.model_validate({"device_id": "bad", "load_profile_power_w": [100.0], **fields}) + + +def test_settings_profile_roundtrip_and_legacy_defaults() -> None: + defaults = HomeApplianceCommonSettings(device_id="legacy") + assert defaults.to_genetic_param().consumption_wh == 3000 + assert defaults.to_genetic0_param().duration_h == 3 + settings = HomeApplianceCommonSettings.model_validate( + { + "device_id": "washer", + "load_profile_power_w": [1200.0, 600.0], + "load_profile_interval_seconds": 600, + "schedule_mode": "DAILY", + "num_cycles": 2, + "min_cycle_gap_h": 1, + "time_windows": {"windows": [{"start_time": "07:00", "duration": "12 hours"}]}, + } + ) + restored = HomeApplianceCommonSettings.model_validate_json(settings.model_dump_json()) + params = restored.to_genetic_param() + assert params.load_profile_power_w == [1200.0, 600.0] + assert params.consumption_wh is None and params.duration_h is None + assert params.num_cycles == 2 and params.min_cycle_gap_h == 1 + assert params.schedule_mode == ConsumerScheduleMode.DAILY + assert params.shared_time_windows is not None + with pytest.raises(ValueError, match="GENETIC"): + restored.to_genetic0_param() + + +def test_cycle_and_shared_windows_intersect_after_completed_cycle() -> None: + settings = HomeApplianceCommonSettings.model_validate( + { + "device_id": "washer", + "load_profile_power_w": [1200.0], + "load_profile_interval_seconds": 1800, + "schedule_mode": "DAILY", + "cycle_time_windows": { + "windows": [ + {"start_time": "06:00", "duration": "2 hours", "value": 0}, + {"start_time": "18:00", "duration": "2 hours", "value": 1}, + ] + }, + "time_windows": {"windows": [{"start_time": "18:30", "duration": "1 hour"}]}, + } + ) + device = HomeAppliance(settings.to_genetic_param(), 48, 192, 0.25) + device.set_completed_cycles(1) + zero = to_datetime("2026-09-16T00:00:00+02:00", in_timezone="Europe/Berlin") + assert device.remaining_cycle_indices == [1] + assert device.allowed_start_slots( + slot0_datetime=zero, earliest_slot=0, horizon_end_slot=96, cycle_index=1 + ) == [74, 75, 76] + assert ( + device.allowed_start_slots( + slot0_datetime=zero, earliest_slot=0, horizon_end_slot=96, cycle_index=0 + ) + == [] + ) + # Next-day DAILY lookup remains possible using absolute configured cycle IDs. + assert device.allowed_start_slots( + slot0_datetime=zero, earliest_slot=96, horizon_end_slot=192, cycle_index=1 + ) == [170, 171, 172] + + +def test_legacy_multiple_cycles_use_slots_and_physical_idle_gap() -> None: + device = appliance(slots=96, num_cycles=2, min_cycle_gap_h=1) + assert device.set_starting_times([4, 4]) == [4, 10] + assert device.get_load_curve().sum() == pytest.approx(700) + + +def test_touching_cycle_windows_preserve_union() -> None: + device = appliance( + time_windows={ + "windows": [ + {"start_time": "08:00", "duration": "15 minutes", "value": 0}, + {"start_time": "08:15", "duration": "15 minutes", "value": 0}, + ] + } + ) + assert device.allowed_start_slots( + slot0_datetime=to_datetime("2026-09-16T00:00:00+02:00", in_timezone="Europe/Berlin"), + earliest_slot=0, + horizon_end_slot=96, + cycle_index=0, + ) == [32] + + +def test_absolute_bounds_round_inward_on_quarter_hour_grid() -> None: + device = appliance( + earliest_start_datetime="2026-09-16T08:01:00+02:00", + deadline_datetime="2026-09-16T09:01:00+02:00", + deadline_policy="STRICT", + ) + zero = to_datetime("2026-09-16T00:00:00+02:00", in_timezone="Europe/Berlin") + assert device.allowed_start_slots( + slot0_datetime=zero, earliest_slot=0, horizon_end_slot=96 + ) == [33, 34] + assert not device.deadline_missed([34], zero) + assert device.deadline_missed([35], zero) + + +def test_best_effort_relaxes_only_deadline_not_window_or_earliest() -> None: + device = appliance( + deadline_datetime="2026-09-16T08:00:00+02:00", + shared_time_windows={"windows": [{"start_time": "09:00", "duration": "1 hour"}]}, + ) + zero = to_datetime("2026-09-16T00:00:00+02:00", in_timezone="Europe/Berlin") + assert device.allowed_start_slots( + slot0_datetime=zero, earliest_slot=0, horizon_end_slot=96 + ) == [36] + assert device.deadline_relaxed and device.deadline_missed([36], zero) + device.deadline_policy = ConsumerDeadlinePolicy.STRICT + assert ( + device.allowed_start_slots(slot0_datetime=zero, earliest_slot=0, horizon_end_slot=96) == [] + ) + assert not device.deadline_relaxed + + +def test_overnight_window_uses_opening_date_and_weekday() -> None: + device = appliance( + shared_time_windows={ + "windows": [ + { + "start_time": "23:00", + "duration": "3 hours", + "date": "2026-09-15", + "day_of_week": 1, + } + ] + } + ) + zero = to_datetime("2026-09-16T00:00:00+02:00", in_timezone="Europe/Berlin") + assert device.allowed_start_slots( + slot0_datetime=zero, earliest_slot=0, horizon_end_slot=96 + ) == list(range(7)) + + +@pytest.mark.parametrize("date,expected", [("2026-03-29", 8), ("2026-10-25", 16)]) +def test_dst_elapsed_slots_and_local_window(date: str, expected: int) -> None: + zero = to_datetime(f"{date}T00:00:00", in_timezone="Europe/Berlin") + device = appliance( + shared_time_windows={"windows": [{"start_time": "03:00", "duration": "30 minutes"}]} + ) + assert device.allowed_start_slots( + slot0_datetime=zero, earliest_slot=0, horizon_end_slot=100 + ) == [expected] + assert device.run_end_datetime(expected, zero).hour == 3 + + +@pytest.mark.parametrize("starts", [[-1], [191], [192]]) +def test_complete_curve_does_not_silently_truncate_runs(starts: list[int]) -> None: + with pytest.raises(ValueError, match="complete"): + appliance().build_load_curve(starts) + + +def test_ev_deadlines_roundtrip_without_changing_slot_physics() -> None: + settings = BatteriesCommonSettings.model_validate( + { + "device_id": "car", + "capacity_wh": 10000, + "charging_efficiency": 0.8, + "max_charge_power_w": 4000, + "min_soc_deadline_datetime": "2026-09-16T07:30:00+02:00", + "min_soc_max_duration_h": 2.5, + } + ) + params = settings.to_genetic_ev_bat_param() + assert params.min_soc_max_duration_h == 2.5 + assert params.min_soc_deadline_datetime is not None + assert params.min_soc_deadline_datetime.in_timezone("Europe/Berlin").hour == 7 + battery = Battery(params, prediction_hours=16, slot_duration_h=0.25) + battery.charge_array[0] = 1 + charged, losses = battery.charge_energy(2000, hour=0) + assert charged == pytest.approx(800) + assert losses == pytest.approx(200) + assert "min_soc_deadline_datetime" not in settings.to_genetic0_ev_bat_param().model_dump() + + +@pytest.mark.parametrize("duration", [0.0, -1.0, float("inf"), float("nan")]) +def test_ev_invalid_departure_durations_rejected(duration: float) -> None: + for model in (ElectricVehicleParameters, BatteriesCommonSettings): + with pytest.raises(ValidationError): + model.model_validate( + {"device_id": "car", "capacity_wh": 10000, "min_soc_max_duration_h": duration} + ) + + +def test_completed_cycles_parameter_initializes_remaining_global_ids() -> None: + device = appliance(num_cycles=3, completed_cycles=2) + assert device.completed_cycles == 2 + assert device.remaining_cycle_indices == [2] + assert device.num_remaining_cycles == 1 + done = appliance(num_cycles=3, completed_cycles=3) + assert done.remaining_cycle_indices == [] + with pytest.raises(ValidationError, match="completed_cycles"): + appliance(num_cycles=2, completed_cycles=3) + + +def test_best_effort_multiple_cycles_retain_room_for_joint_gap_repair() -> None: + device = appliance( + num_cycles=2, + min_cycle_gap_h=1, + deadline_datetime="2026-09-16T08:00:00+02:00", + shared_time_windows={"windows": [{"start_time": "09:00", "duration": "3 hours"}]}, + ) + zero = to_datetime("2026-09-16T00:00:00+02:00", in_timezone="Europe/Berlin") + assert device.allowed_start_slots( + slot0_datetime=zero, earliest_slot=0, horizon_end_slot=96, cycle_index=0 + ) == list(range(36, 47)) + assert device.deadline_relaxed + device.build_load_curve([36, 42]) + assert device.get_load_curve().sum() == pytest.approx(700) + + +@pytest.mark.parametrize("deadline", ["2026-09-16T07:00:00Z", "2026-09-16T09:00:00+02:00"]) +def test_absolute_deadline_offsets_describe_same_instant(deadline: str) -> None: + device = appliance(deadline_datetime=deadline, deadline_policy="STRICT") + zero = to_datetime("2026-09-16T08:00:00+02:00", in_timezone="Europe/Berlin") + assert device.allowed_start_slots( + slot0_datetime=zero, earliest_slot=0, horizon_end_slot=16 + ) == [0, 1, 2] + assert not device.deadline_missed([2], zero) + assert device.deadline_missed([3], zero) diff --git a/tests/test_genetic_complete_optimization.py b/tests/test_genetic_complete_optimization.py new file mode 100644 index 00000000..5d867d64 --- /dev/null +++ b/tests/test_genetic_complete_optimization.py @@ -0,0 +1,385 @@ +from pathlib import Path +from typing import Optional +from unittest.mock import MagicMock + +import pytest + +from akkudoktoreos.config.config import ConfigEOS +from akkudoktoreos.core.cache import CacheEnergyManagementStore +from akkudoktoreos.core.coreabc import get_ems +from akkudoktoreos.devices.genetic.battery import Battery +from akkudoktoreos.optimization.genetic.genetic import GeneticOptimization +from akkudoktoreos.optimization.genetic.geneticparams import ( + GeneticOptimizationParameters, +) +from akkudoktoreos.optimization.genetic.geneticsolution import GeneticSolution +from akkudoktoreos.utils.datetimeutil import to_datetime + +ems_eos = get_ems(init=True) # init once + +DIR_TESTDATA = Path(__file__).parent / "testdata" + + +def test_direct_marketing_preserves_constant_supplied_feed_in_tariff(config_eos: ConfigEOS): + config_eos.merge_settings_from_dict({"feedintariff": {"direct_marketing_enabled": True}}) + parameters = GeneticOptimizationParameters.model_validate( + dict( + ems={ + "pv_prognose_wh": [0.0, 0.0], + "strompreis_euro_pro_wh": [0.0002, -0.0001], + "einspeiseverguetung_euro_pro_wh": [0.00007, 0.00007], + "preis_euro_pro_wh_akku": 0.0, + "gesamtlast": [0.0, 0.0], + }, + pv_battery=None, + # Without an inverter the simulation books no grid energy at all, so the + # price signal would never reach the fitness. + inverter={"device_id": "inverter1", "max_power_wh": 20000}, + ev=None, + ) + ) + + adjusted = GeneticOptimization()._parameters_for_config(parameters) + + assert adjusted.ems.einspeiseverguetung_euro_pro_wh == [0.00007, 0.00007] + assert parameters.ems.einspeiseverguetung_euro_pro_wh == [0.00007, 0.00007] + + +def test_direct_marketing_keeps_variable_feed_in_tariff(config_eos: ConfigEOS): + config_eos.merge_settings_from_dict({"feedintariff": {"direct_marketing_enabled": True}}) + parameters = GeneticOptimizationParameters.model_validate( + dict( + ems={ + "pv_prognose_wh": [0.0, 0.0], + "strompreis_euro_pro_wh": [0.0002, 0.0003], + "einspeiseverguetung_euro_pro_wh": [0.0001, -0.00005], + "preis_euro_pro_wh_akku": 0.0, + "gesamtlast": [0.0, 0.0], + }, + pv_battery=None, + inverter=None, + ev=None, + ) + ) + + adjusted = GeneticOptimization()._parameters_for_config(parameters) + + assert adjusted.ems.einspeiseverguetung_euro_pro_wh == [0.0001, -0.00005] + + +def test_grid_export_rates_reach_the_solution(config_eos: ConfigEOS): + """Configured export rates end up as per-slot export levels in the solution.""" + config_eos.merge_settings_from_dict( + { + "prediction": {"hours": 24}, + "optimization": { + "genetic": { + "individuals": 40, + "generations": 10, + "tail_horizon_hours": 0, + "horizon_hours": 24, + "interval_sec": 3600, + } + }, + "feedintariff": {"direct_marketing_enabled": True}, + "devices": { + "max_batteries": 1, + "batteries": { + "battery1": {"device_id": "battery1", "grid_export_rates": [0.5, 1.0]} + }, + }, + } + ) + ems_eos.set_start_datetime(to_datetime().set(hour=0, minute=0)) + CacheEnergyManagementStore().clear() + + hours = 24 + parameters = GeneticOptimizationParameters.model_validate( + dict( + ems={ + "pv_prognose_wh": [0.0] * hours, + "strompreis_euro_pro_wh": [0.0003] * hours, + # A pronounced tariff peak makes exporting worthwhile at all. + "einspeiseverguetung_euro_pro_wh": [0.0001] * 12 + [0.0009] * 12, + "preis_euro_pro_wh_akku": 0.0, + "gesamtlast": [200.0] * hours, + }, + pv_battery={ + "device_id": "battery1", + "capacity_wh": 10000, + "initial_soc_percentage": 100, + "min_soc_percentage": 0, + "max_charge_power_w": 5000, + }, + inverter={ + "device_id": "inverter1", + "max_power_wh": 10000, + "battery_id": "battery1", + }, + ev=None, + ) + ) + + optimization = GeneticOptimization(fixed_seed=42) + solution = optimization.optimize_ems(parameters=parameters, start_hour=0, ngen=3) + + # Full power first, so the full-power state keeps the lowest export index. + assert optimization.bat_possible_grid_export_values == [1.0, 0.5] + assert len(solution.battery_grid_export_factor) == len(solution.battery_grid_export_allowed) + assert set(solution.battery_grid_export_factor) <= {0.0, 0.5, 1.0} + assert [ + 1 if factor > 0.0 else 0 for factor in solution.battery_grid_export_factor + ] == solution.battery_grid_export_allowed + + # @TODO + + +def _ev_deadline_parameters(hours: int, **ev_extra) -> GeneticOptimizationParameters: + """Optimization parameters with an EV that has to be charged.""" + return GeneticOptimizationParameters.model_validate( + dict( + ems={ + "pv_prognose_wh": [0.0] * hours, + # Expensive for the first six hours, dirt cheap afterwards: without a + # deadline the optimizer would always wait for the cheap slots. + "strompreis_euro_pro_wh": [0.0009] * 6 + [0.00001] * (hours - 6), + "einspeiseverguetung_euro_pro_wh": [0.00007] * hours, + "preis_euro_pro_wh_akku": 0.0, + "gesamtlast": [300.0] * hours, + }, + pv_battery=None, + inverter=None, + ev={ + "device_id": "ev1", + "capacity_wh": 60000, + "charging_efficiency": 0.95, + "max_charge_power_w": 11040, + "initial_soc_percentage": 20, + "min_soc_percentage": 60, + **ev_extra, + }, + ) + ) + + +def test_ev_deadline_slot_resolution(config_eos: ConfigEOS): + """Datetime and maximum duration resolve to a slot; the earlier one wins.""" + config_eos.merge_settings_from_dict( + { + "prediction": {"hours": 48}, + "optimization": { + "genetic": {"tail_horizon_hours": 0, "horizon_hours": 48, "interval_sec": 3600} + }, + } + ) + ems_eos.set_start_datetime(to_datetime().set(hour=10, minute=0)) + optimization = GeneticOptimization(fixed_seed=1) + optimization._slot0_datetime = optimization.ems.start_datetime + slot0 = optimization._slot0_datetime + + # Duration only: 6 h after the start hour 10. + parameters = _ev_deadline_parameters(48, min_soc_max_duration_h=6) + assert optimization._ev_deadline_slot(parameters) == 6 + + # Datetime only. + parameters = _ev_deadline_parameters(48, min_soc_deadline_datetime=slot0.add(hours=14)) + assert optimization._ev_deadline_slot(parameters) == 14 + + # Both: the earlier one wins. + parameters = _ev_deadline_parameters( + 48, min_soc_deadline_datetime=slot0.add(hours=20), min_soc_max_duration_h=6 + ) + assert optimization._ev_deadline_slot(parameters) == 6 + + # Beyond the horizon: no deadline, the end-of-horizon target already covers it. + parameters = _ev_deadline_parameters(48, min_soc_deadline_datetime=slot0.add(hours=100)) + assert optimization._ev_deadline_slot(parameters) is None + + # In the past: due right now. + parameters = _ev_deadline_parameters(48, min_soc_deadline_datetime=slot0.subtract(hours=2)) + assert optimization._ev_deadline_slot(parameters) == 0 + + # No deadline at all. + assert optimization._ev_deadline_slot(_ev_deadline_parameters(48)) is None + + +def test_ev_soc_penalty_reads_the_deadline_slot(config_eos: ConfigEOS): + """With a deadline the penalty checks the SoC at that slot, not at the end.""" + config_eos.merge_settings_from_dict( + { + "prediction": {"hours": 48}, + "optimization": { + "genetic": {"tail_horizon_hours": 0, "horizon_hours": 48, "interval_sec": 3600} + }, + } + ) + ems_eos.set_start_datetime(to_datetime().set(hour=10, minute=0)) + optimization = GeneticOptimization(fixed_seed=1) + simulation_result = {"EAuto_SoC_pro_Stunde": [20.0, 35.0, 50.0, 80.0]} + + optimization.simulation.ev = MagicMock(spec=Battery) + optimization.simulation.ev.current_soc_percentage.return_value = 80.0 + + # Without a deadline the final SoC counts. + optimization._ev_soc_deadline_slot = None + assert optimization._ev_soc_at_deadline(simulation_result, 10) == 80.0 + + # With one, the SoC at the beginning of the deadline slot counts. + optimization._ev_soc_deadline_slot = 12 + assert optimization._ev_soc_at_deadline(simulation_result, 10) == 50.0 + + # A deadline beyond the reported slots falls back to the final SoC. + optimization._ev_soc_deadline_slot = 99 + assert optimization._ev_soc_at_deadline(simulation_result, 10) == 80.0 + + +def test_ev_deadline_charges_before_departure(config_eos: ConfigEOS): + """The EV reaches its target before the deadline even when energy is cheaper later.""" + hours = 24 + config_eos.merge_settings_from_dict( + { + "prediction": {"hours": hours}, + "optimization": { + "genetic": { + "individuals": 100, + "generations": 40, + "tail_horizon_hours": 0, + "horizon_hours": hours, + "interval_sec": 3600, + } + }, + } + ) + ems_eos.set_start_datetime(to_datetime().set(hour=0, minute=0)) + CacheEnergyManagementStore().clear() + + parameters = _ev_deadline_parameters(hours, min_soc_max_duration_h=6) + solution = GeneticOptimization(fixed_seed=42).optimize_ems( + parameters=parameters, start_hour=0, ngen=40 + ) + + soc_per_hour = solution.result.ev_soc_per_hour + # Slot 6 is the first slot at or after the deadline, so its start-of-slot SoC + # is what the target is checked against. + assert soc_per_hour[6] >= 60.0 + + +def _terminal_value_run( + config_eos: ConfigEOS, mode: str, prices: Optional[list[float]] = None +) -> GeneticSolution: + """48 h with expensive energy and two dirt-cheap slots at the very end. + + Charging in those last slots only pays off when the stored energy keeps a + value beyond the horizon. + """ + hours = 48 + config_eos.merge_settings_from_dict( + { + "prediction": {"hours": hours}, + "optimization": { + "genetic": { + "individuals": 80, + "generations": 20, + "tail_horizon_hours": 0, + "horizon_hours": hours, + "interval_sec": 3600, + "terminal_value_mode": mode, + "terminal_value_euro_per_kwh": 0.0, + } + }, + } + ) + ems_eos.set_start_datetime(to_datetime().set(hour=0, minute=0)) + CacheEnergyManagementStore().clear() + + if prices is None: + prices = [0.0004] * (hours - 2) + [0.00002] * 2 + parameters = GeneticOptimizationParameters.model_validate( + dict( + ems={ + "pv_prognose_wh": [0.0] * hours, + "strompreis_euro_pro_wh": prices, + "einspeiseverguetung_euro_pro_wh": [0.00007] * hours, + "preis_euro_pro_wh_akku": 0.0, + "gesamtlast": [200.0] * hours, + }, + pv_battery={ + "device_id": "battery1", + "capacity_wh": 10000, + "initial_soc_percentage": 20, + "min_soc_percentage": 0, + "max_soc_percentage": 100, + "charging_efficiency": 1.0, + "discharging_efficiency": 1.0, + "max_charge_power_w": 5000, + }, + inverter={ + "device_id": "inverter1", + "max_power_wh": 10000, + "battery_id": "battery1", + "ac_to_dc_efficiency": 1.0, + "dc_to_ac_efficiency": 1.0, + "max_ac_charge_power_w": 5000, + }, + ev=None, + ) + ) + return GeneticOptimization(fixed_seed=7).optimize_ems( + parameters=parameters, start_hour=0, ngen=20 + ) + + +def test_terminal_value_auto_keeps_energy_that_fixed_zero_throws_away(config_eos: ConfigEOS): + """AUTO values the energy left in the battery, a fixed zero does not.""" + auto = _terminal_value_run(config_eos, "AUTO") + fixed = _terminal_value_run(config_eos, "FIXED") + + assert auto.terminal_value is not None + assert auto.terminal_value.mode == "AUTO" + assert auto.terminal_value.curve is not None + assert auto.terminal_value.credited_euro > 0.0 + + assert fixed.terminal_value is not None + assert fixed.terminal_value.mode == "FIXED" + assert fixed.terminal_value.credited_euro == 0.0 + + # The cheap slots at the end are only worth using with a terminal value. + assert auto.result.battery_soc_per_hour[-1] > fixed.result.battery_soc_per_hour[-1] + + +def test_terminal_value_curve_is_concave_and_reported(config_eos: ConfigEOS): + """The reported curve is what the credit was read from.""" + solution = _terminal_value_run(config_eos, "AUTO") + assert solution.terminal_value is not None + curve = solution.terminal_value.curve + assert curve is not None + + assert curve.window_slots == 24 + assert len(curve.energy_wh) == len(curve.value_euro) + assert len(curve.marginal_euro_per_kwh) == len(curve.energy_wh) - 1 + marginals = curve.marginal_euro_per_kwh + assert all(a >= b for a, b in zip(marginals, marginals[1:])) + + # The credit is the curve evaluated at the energy left in the battery. + expected = curve.value(solution.terminal_value.battery_energy_wh) + assert solution.terminal_value.credited_euro == pytest.approx(expected) + + +def test_terminal_value_reports_why_it_fell_back_to_fixed(config_eos: ConfigEOS): + """AUTO without any prices cannot build a curve - and has to say so. + + A request whose price forecast is all zeros used to be indistinguishable + from a run configured for FIXED. + """ + hours = 48 + solution = _terminal_value_run(config_eos, "AUTO", prices=[0.0] * hours) + + assert solution.terminal_value is not None + assert solution.terminal_value.mode == "FIXED" + assert solution.terminal_value.curve is None + assert solution.terminal_value.reason is not None + assert "no priced residual load" in solution.terminal_value.reason + + configured = _terminal_value_run(config_eos, "FIXED") + assert configured.terminal_value is not None + assert configured.terminal_value.reason == "terminal_value_mode is FIXED" diff --git a/tests/test_genetic_complete_simulation.py b/tests/test_genetic_complete_simulation.py new file mode 100644 index 00000000..cf07e96f --- /dev/null +++ b/tests/test_genetic_complete_simulation.py @@ -0,0 +1,528 @@ +from unittest.mock import Mock + +import numpy as np +import pytest + +from akkudoktoreos.devices.genetic.battery import ( + Battery, + ElectricVehicleParameters, + SolarPanelBatteryParameters, +) +from akkudoktoreos.devices.genetic.homeappliance import ( + HomeAppliance, + HomeApplianceParameters, +) +from akkudoktoreos.devices.genetic.inverter import Inverter, InverterParameters +from akkudoktoreos.optimization.genetic.genetic import GeneticSimulation +from akkudoktoreos.optimization.genetic.geneticparams import ( + GeneticEnergyManagementParameters, +) + +start_hour = 1 + + +# Example initialization of necessary components +@pytest.fixture +def genetic_simulation(config_eos) -> GeneticSimulation: + """Fixture to create an EnergyManagement instance with given test parameters.""" + # Assure configuration holds the correct values + config_eos.merge_settings_from_dict( + { + "prediction": {"hours": 48}, + "optimization": {"hours": 24, "genetic": {"tail_horizon_hours": 0}}, + } + ) + assert config_eos.prediction.hours == 48 + assert config_eos.optimization.genetic.horizon_hours == 24 + + # Initialize the battery and the inverter + akku = Battery( + SolarPanelBatteryParameters( + device_id="battery1", + capacity_wh=5000, + initial_soc_percentage=80, + min_soc_percentage=10, + ), + prediction_hours=config_eos.prediction.hours, + ) + akku.reset() + + inverter = Inverter( + InverterParameters( + device_id="inverter1", max_power_wh=10000, battery_id=akku.parameters.device_id + ), + battery=akku, + ) + + # Flexible consumer (fixed start at slot 2 for this deterministic test) + home_appliance = HomeAppliance( + HomeApplianceParameters( + device_id="dishwasher1", + consumption_wh=2000, + duration_h=2, + time_windows=None, + ), + optimization_hours=config_eos.optimization.genetic.horizon_hours, + prediction_hours=config_eos.prediction.hours, + ) + home_appliance.build_load_curve([2]) + + # Example initialization of electric car battery + eauto = Battery( + ElectricVehicleParameters( + device_id="ev1", capacity_wh=26400, initial_soc_percentage=10, min_soc_percentage=10 + ), + prediction_hours=config_eos.prediction.hours, + ) + eauto.set_charge_per_hour(np.full(config_eos.prediction.hours, 1)) + + # Parameters based on previous example data + pv_prognose_wh = [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 8.05, + 352.91, + 728.51, + 930.28, + 1043.25, + 1106.74, + 1161.69, + 6018.82, + 5519.07, + 3969.88, + 3017.96, + 1943.07, + 1007.17, + 319.67, + 7.88, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 5.04, + 335.59, + 705.32, + 1121.12, + 1604.79, + 2157.38, + 1433.25, + 5718.49, + 4553.96, + 3027.55, + 2574.46, + 1720.4, + 963.4, + 383.3, + 0, + 0, + 0, + ] + + strompreis_euro_pro_wh = [ + 0.0003384, + 0.0003318, + 0.0003284, + 0.0003283, + 0.0003289, + 0.0003334, + 0.0003290, + 0.0003302, + 0.0003042, + 0.0002430, + 0.0002280, + 0.0002212, + 0.0002093, + 0.0001879, + 0.0001838, + 0.0002004, + 0.0002198, + 0.0002270, + 0.0002997, + 0.0003195, + 0.0003081, + 0.0002969, + 0.0002921, + 0.0002780, + 0.0003384, + 0.0003318, + 0.0003284, + 0.0003283, + 0.0003289, + 0.0003334, + 0.0003290, + 0.0003302, + 0.0003042, + 0.0002430, + 0.0002280, + 0.0002212, + 0.0002093, + 0.0001879, + 0.0001838, + 0.0002004, + 0.0002198, + 0.0002270, + 0.0002997, + 0.0003195, + 0.0003081, + 0.0002969, + 0.0002921, + 0.0002780, + ] + + einspeiseverguetung_euro_pro_wh = 0.00007 + preis_euro_pro_wh_akku = 0.0001 + + gesamtlast = [ + 676.71, + 876.19, + 527.13, + 468.88, + 531.38, + 517.95, + 483.15, + 472.28, + 1011.68, + 995.00, + 1053.07, + 1063.91, + 1320.56, + 1132.03, + 1163.67, + 1176.82, + 1216.22, + 1103.78, + 1129.12, + 1178.71, + 1050.98, + 988.56, + 912.38, + 704.61, + 516.37, + 868.05, + 694.34, + 608.79, + 556.31, + 488.89, + 506.91, + 804.89, + 1141.98, + 1056.97, + 992.46, + 1155.99, + 827.01, + 1257.98, + 1232.67, + 871.26, + 860.88, + 1158.03, + 1222.72, + 1221.04, + 949.99, + 987.01, + 733.99, + 592.97, + ] + + # Initialize the energy management system with the respective parameters + simulation = GeneticSimulation() + simulation.prepare( + GeneticEnergyManagementParameters.model_validate( + dict( + pv_prognose_wh=pv_prognose_wh, + strompreis_euro_pro_wh=strompreis_euro_pro_wh, + einspeiseverguetung_euro_pro_wh=einspeiseverguetung_euro_pro_wh, + preis_euro_pro_wh_akku=preis_euro_pro_wh_akku, + gesamtlast=gesamtlast, + ) + ), + optimization_hours=config_eos.optimization.genetic.horizon_hours, + prediction_hours=config_eos.prediction.hours, + inverter=inverter, + ev=eauto, + home_appliances=[home_appliance], + ) + + # Init for test + assert simulation.ac_charge_hours is not None + assert simulation.dc_charge_hours is not None + assert simulation.bat_discharge_hours is not None + assert simulation.bat_grid_export_hours is not None + assert simulation.ev_charge_hours is not None + simulation.ac_charge_hours[start_hour] = 1.0 + simulation.dc_charge_hours[start_hour] = 1.0 + simulation.bat_discharge_hours[start_hour] = 1.0 + simulation.ev_charge_hours[start_hour] = 1.0 + + return simulation + + +def test_ev_charging_uses_raw_input_energy_for_load_and_grid(config_eos): + config_eos.merge_settings_from_dict( + { + "prediction": {"hours": 1}, + "optimization": {"genetic": {"tail_horizon_hours": 0, "horizon_hours": 1}}, + } + ) + ev = Battery( + ElectricVehicleParameters( + device_id="ev1", + capacity_wh=1000, + charging_efficiency=0.8, + max_charge_power_w=100, + initial_soc_percentage=0, + min_soc_percentage=0, + ), + prediction_hours=1, + ) + inverter = Inverter(InverterParameters(device_id="inverter1", max_power_wh=1000.0)) + simulation = GeneticSimulation() + simulation.prepare( + GeneticEnergyManagementParameters.model_validate( + dict( + pv_prognose_wh=[0.0], + strompreis_euro_pro_wh=[0.001], + einspeiseverguetung_euro_pro_wh=[0.0], + preis_euro_pro_wh_akku=0.0, + gesamtlast=[0.0], + ) + ), + optimization_hours=1, + prediction_hours=1, + inverter=inverter, + ev=ev, + ) + simulation.ev_charge_hours = np.array([1.0]) + + result = simulation.simulate(start_hour=0) + + assert result["Last_Wh_pro_Stunde"][0] == pytest.approx(100.0) + assert result["Netzbezug_Wh_pro_Stunde"][0] == pytest.approx(100.0) + assert result["Kosten_Euro_pro_Stunde"][0] == pytest.approx(0.1) + assert result["Verluste_Pro_Stunde"][0] == pytest.approx(20.0) + assert ev.current_soc_percentage() == pytest.approx(8.0) + + +def test_direct_marketing_curtails_negative_feed_in(config_eos, monkeypatch): + config_eos.merge_settings_from_dict( + { + "prediction": {"hours": 2}, + "optimization": {"genetic": {"tail_horizon_hours": 0, "horizon_hours": 2}}, + } + ) + + inverter = Inverter(InverterParameters(device_id="inverter1", max_power_wh=1000.0)) + monkeypatch.setattr( + inverter.self_consumption_predictor, + "calculate_expected_direct_consumption", + Mock(side_effect=min), + ) + + simulation = GeneticSimulation() + simulation.prepare( + GeneticEnergyManagementParameters.model_validate( + dict( + pv_prognose_wh=[500.0, 500.0], + strompreis_euro_pro_wh=[-0.0001, -0.0001], + einspeiseverguetung_euro_pro_wh=[-0.0001, -0.0001], + preis_euro_pro_wh_akku=0.0, + gesamtlast=[0.0, 0.0], + ) + ), + optimization_hours=config_eos.optimization.genetic.horizon_hours, + prediction_hours=config_eos.prediction.hours, + inverter=inverter, + direct_marketing_enabled=True, + ) + + result = simulation.simulate(start_hour=0) + + assert result["Netzeinspeisung_Wh_pro_Stunde"][0] == 0.0 + assert result["Einnahmen_Euro_pro_Stunde"][0] == 0.0 + assert result["Verluste_Pro_Stunde"][0] == pytest.approx(500.0) + + +def _direct_marketing_battery_export_simulation( + config_eos, + levelized_cost_of_storage_kwh: float = 0.0, + dc_to_ac_efficiency: float = 1.0, +) -> GeneticSimulation: + config_eos.merge_settings_from_dict( + { + "prediction": {"hours": 2}, + "optimization": {"genetic": {"tail_horizon_hours": 0, "horizon_hours": 2}}, + } + ) + + battery = Battery( + SolarPanelBatteryParameters( + device_id="battery1", + capacity_wh=1000, + initial_soc_percentage=100, + min_soc_percentage=0, + charging_efficiency=1.0, + discharging_efficiency=1.0, + levelized_cost_of_storage_kwh=levelized_cost_of_storage_kwh, + max_charge_power_w=500, + ), + prediction_hours=config_eos.prediction.hours, + ) + inverter = Inverter( + InverterParameters( + device_id="inverter1", + max_power_wh=500.0, + battery_id=battery.parameters.device_id, + dc_to_ac_efficiency=dc_to_ac_efficiency, + ), + battery=battery, + ) + + simulation = GeneticSimulation() + simulation.prepare( + GeneticEnergyManagementParameters.model_validate( + dict( + pv_prognose_wh=[0.0, 0.0], + strompreis_euro_pro_wh=[0.0, 0.0], + einspeiseverguetung_euro_pro_wh=[0.0002, 0.0002], + preis_euro_pro_wh_akku=0.0, + gesamtlast=[0.0, 0.0], + ) + ), + optimization_hours=config_eos.optimization.genetic.horizon_hours, + prediction_hours=config_eos.prediction.hours, + inverter=inverter, + direct_marketing_enabled=True, + ) + return simulation + + +def test_direct_marketing_discharge_allowed_does_not_export_battery(config_eos): + simulation = _direct_marketing_battery_export_simulation(config_eos) + assert simulation.bat_discharge_hours is not None + simulation.bat_discharge_hours[0] = 1 + + result = simulation.simulate(start_hour=0) + + assert result["Netzeinspeisung_Wh_pro_Stunde"][0] == 0.0 + assert simulation.battery is not None + assert simulation.battery.current_soc_percentage() == 100.0 + + +def test_direct_marketing_battery_grid_export_uses_separate_signal(config_eos): + simulation = _direct_marketing_battery_export_simulation(config_eos) + assert simulation.bat_grid_export_hours is not None + simulation.bat_grid_export_hours[0] = 1 + + result = simulation.simulate(start_hour=0) + + assert result["Netzeinspeisung_Wh_pro_Stunde"][0] == pytest.approx(500.0) + assert result["Einnahmen_Euro_pro_Stunde"][0] == pytest.approx(0.1) + assert simulation.battery is not None + assert simulation.battery.current_soc_percentage() == 50.0 + + +def test_direct_marketing_grid_export_rate_limits_exported_energy(config_eos): + """A partial export level exports that share of the rated discharge power.""" + simulation = _direct_marketing_battery_export_simulation(config_eos) + assert simulation.bat_grid_export_hours is not None + # 500 W rated discharge power over a one hour slot -> 500 Wh at rate 1.0. + simulation.bat_grid_export_hours[0] = 0.5 + + result = simulation.simulate(start_hour=0) + + assert result["Netzeinspeisung_Wh_pro_Stunde"][0] == pytest.approx(250.0) + assert simulation.battery is not None + assert simulation.battery.current_soc_percentage() == 75.0 + + +def test_battery_lcos_is_charged_once_on_delivered_energy(config_eos): + simulation = _direct_marketing_battery_export_simulation( + config_eos, + levelized_cost_of_storage_kwh=0.12, + dc_to_ac_efficiency=0.8, + ) + assert simulation.bat_grid_export_hours is not None + simulation.bat_grid_export_hours[0] = 1 + + result = simulation.simulate(start_hour=0) + + # The battery delivers 500 Wh DC, so LCOS is 0.5 kWh * 0.12 EUR/kWh + # = 0.06 EUR exactly once. After the 80% inverter, 400 Wh AC reaches + # the grid and earns 400 Wh * 0.0002 EUR/Wh = 0.08 EUR. + assert result["Kosten_Euro_pro_Stunde"][0] == pytest.approx(0.06) + assert result["Gesamtkosten_Euro"] == pytest.approx(0.06) + assert result["Einnahmen_Euro_pro_Stunde"][0] == pytest.approx(0.08) + assert result["Gesamtbilanz_Euro"] == pytest.approx(-0.02) + + +def test_disabled_ac_charging_clears_the_reported_plan(config_eos): + """With AC charging off the reported plan must not keep charge commands. + + The simulation ignores the AC charge genes when the inverter forbids grid + charging. The solution is read back from the same array, so a controller + acting on it would grid-charge the battery although no such charge was ever + simulated or paid for. + """ + config_eos.merge_settings_from_dict( + { + "prediction": {"hours": 2}, + "optimization": {"genetic": {"tail_horizon_hours": 0, "horizon_hours": 2}}, + } + ) + + battery = Battery( + SolarPanelBatteryParameters( + device_id="battery1", + capacity_wh=10000, + initial_soc_percentage=50, + min_soc_percentage=0, + charging_efficiency=1.0, + discharging_efficiency=1.0, + max_charge_power_w=5000, + ), + prediction_hours=config_eos.prediction.hours, + ) + inverter = Inverter( + InverterParameters( + device_id="inverter1", + max_power_wh=5000.0, + battery_id=battery.parameters.device_id, + max_ac_charge_power_w=0, # Netzladen deaktiviert + ), + battery=battery, + ) + + simulation = GeneticSimulation() + simulation.prepare( + GeneticEnergyManagementParameters.model_validate( + dict( + pv_prognose_wh=[0.0, 0.0], + strompreis_euro_pro_wh=[0.0003, 0.0003], + einspeiseverguetung_euro_pro_wh=[0.0001, 0.0001], + preis_euro_pro_wh_akku=0.0, + gesamtlast=[0.0, 0.0], + ) + ), + optimization_hours=config_eos.optimization.genetic.horizon_hours, + prediction_hours=config_eos.prediction.hours, + inverter=inverter, + ) + simulation.ac_charge_hours = np.array([0.8, 0.0]) + + soc_before = battery.current_soc_percentage() + simulation.simulate(start_hour=0) + + # Nothing was charged ... + assert battery.current_soc_percentage() == pytest.approx(soc_before) + # ... and the plan says so. + assert simulation.ac_charge_hours is not None + assert list(simulation.ac_charge_hours) == [0.0, 0.0] diff --git a/tests/test_genetic_complete_solution.py b/tests/test_genetic_complete_solution.py new file mode 100644 index 00000000..11ccc664 --- /dev/null +++ b/tests/test_genetic_complete_solution.py @@ -0,0 +1,188 @@ +# ruff: noqa: S101 + +import numpy as np + +from akkudoktoreos.devices.devicesabc import BatteryOperationMode +from akkudoktoreos.optimization.genetic.genetic import GeneticOptimization +from akkudoktoreos.optimization.genetic.geneticsolution import GeneticSolution + + +def test_battery_discharge_allowed_remains_local_load_mode(config_eos): + config_eos.merge_settings_from_dict({"feedintariff": {"direct_marketing_enabled": True}}) + solution = GeneticSolution + + operation_mode, operation_mode_factor = solution._battery_operation_from_solution( + ac_charge=0.0, + dc_charge=0.0, + discharge_allowed=True, + ) + + assert operation_mode == BatteryOperationMode.PEAK_SHAVING + assert operation_mode_factor == 1.0 + + +def test_battery_grid_export_signal_maps_to_grid_support_export(config_eos): + config_eos.merge_settings_from_dict({"feedintariff": {"direct_marketing_enabled": True}}) + solution = GeneticSolution + + operation_mode, operation_mode_factor = solution._battery_operation_from_solution( + ac_charge=0.0, + dc_charge=0.0, + discharge_allowed=False, + battery_grid_export_allowed=True, + ) + + assert operation_mode == BatteryOperationMode.GRID_SUPPORT_EXPORT + assert operation_mode_factor == 1.0 + + +def test_decode_charge_discharge_has_separate_battery_grid_export_state(): + optimization = GeneticOptimization() + optimization.bat_possible_charge_values = [1.0] + optimization.optimize_dc_charge = True + optimization.optimize_battery_grid_export = True + + ac_charge, dc_charge, discharge, battery_grid_export = optimization.decode_charge_discharge( + np.array([5]) + ) + + assert ac_charge.tolist() == [0.0] + assert dc_charge.tolist() == [0] + assert discharge.tolist() == [0] + assert battery_grid_export.tolist() == [1] + + +def test_decode_charge_discharge_has_self_consumption_state_after_legacy_export(): + optimization = GeneticOptimization() + optimization.bat_possible_charge_values = [1.0] + optimization.optimize_dc_charge = True + optimization.optimize_battery_grid_export = True + + layout = optimization._battery_state_layout() + ac_charge, dc_charge, discharge, battery_grid_export = optimization.decode_charge_discharge( + np.array([6]) + ) + + assert layout.total_states == 7 + assert layout.grid_export_state == 5 + assert layout.self_consumption_state == 6 + assert ac_charge.tolist() == [0.0] + assert dc_charge.tolist() == [1] + assert discharge.tolist() == [1] + assert battery_grid_export.tolist() == [0] + + +def test_graded_grid_export_states_decode_to_rates(): + """Each configured export rate gets its own state; state 5 stays full power.""" + optimization = GeneticOptimization() + optimization.bat_possible_charge_values = [1.0] + optimization.bat_possible_grid_export_values = [1.0, 0.5, 0.25] + optimization.optimize_dc_charge = True + optimization.optimize_battery_grid_export = True + + layout = optimization._battery_state_layout() + + assert layout.grid_export_states == (5, 6, 7) + # The full-power state keeps its index, so existing seeds stay valid. + assert layout.grid_export_state == 5 + assert layout.self_consumption_state == 8 + assert layout.total_states == 9 + + _, _, _, battery_grid_export = optimization.decode_charge_discharge(np.array([0, 5, 6, 7])) + assert battery_grid_export.tolist() == [0.0, 1.0, 0.5, 0.25] + + +def test_single_export_rate_keeps_all_or_nothing_layout(): + """Without configured rates the state space is the one from before grading.""" + optimization = GeneticOptimization() + optimization.bat_possible_charge_values = [1.0] + optimization.optimize_dc_charge = True + optimization.optimize_battery_grid_export = True + + layout = optimization._battery_state_layout() + + assert layout.grid_export_states == (5,) + assert layout.total_states == 7 + + +def test_battery_grid_export_factor_becomes_operation_factor(config_eos): + """A partial export level is reported as the GRID_SUPPORT_EXPORT factor.""" + config_eos.merge_settings_from_dict({"feedintariff": {"direct_marketing_enabled": True}}) + solution = GeneticSolution + + operation_mode, operation_mode_factor = solution._battery_operation_from_solution( + ac_charge=0.0, + dc_charge=0.0, + discharge_allowed=False, + battery_grid_export_allowed=True, + battery_grid_export_factor=0.25, + ) + + assert operation_mode == BatteryOperationMode.GRID_SUPPORT_EXPORT + assert operation_mode_factor == 0.25 + + +def test_disjoint_cycle_masks_keep_feasible_non_deadline_order(config_eos): + from akkudoktoreos.core.coreabc import get_ems + from akkudoktoreos.optimization.genetic.geneticparams import ( + GeneticOptimizationParameters, + ) + from akkudoktoreos.utils.datetimeutil import to_datetime + + config_eos.merge_settings_from_dict( + { + "optimization": { + "genetic": { + "interval_sec": 900, + "horizon_hours": 2, + "tail_horizon_hours": 0, + "terminal_value_mode": "FIXED", + } + } + } + ) + get_ems(init=True).set_start_datetime( + to_datetime("2026-09-16T00:00:00+02:00", in_timezone="Europe/Berlin") + ) + params = GeneticOptimizationParameters.model_validate( + { + "ems": { + "pv_forecast_wh": [0.0] * 8, + "total_load": [0.0] * 8, + "electricity_price_per_wh": [0.0003] * 8, + "feed_in_tariff_per_wh": 0.0, + "price_per_wh_battery": 0.0, + }, + "forecast_interval_seconds": 900, + "pv_battery": None, + "ev": None, + "inverter": {"device_id": "inv", "max_power_wh": 1000}, + "home_appliances": [ + { + "device_id": "washer", + "num_cycles": 2, + "load_profile_power_w": [1000.0, 1000.0], + "load_profile_interval_seconds": 900, + "time_windows": { + "windows": [ + {"start_time": "01:00", "duration": "30 minutes", "value": 0}, + {"start_time": "00:00", "duration": "30 minutes", "value": 1}, + {"start_time": "01:15", "duration": "30 minutes", "value": 1}, + ] + }, + } + ], + } + ) + optimizer = GeneticOptimization(fixed_seed=42) + solution = optimizer.optimize_ems(params, ngen=1, individuals=6) + assert [(item.hour, item.minute) for item in solution.appliance_starts["washer"]] == [ + (0, 0), + (1, 0), + ] + assert sum(solution.result.home_appliance_energy_wh["washer"]) == 1000.0 + # Even a candidate choosing the second disconnected window repairs to the + # feasible order, without dropping a configured run or crossing its mask. + genes = [0, 1] + assert optimizer._decode_appliance_starts(genes) == {0: [0, 4]} + assert genes == [0, 0] diff --git a/tests/test_genetic_complete_tail.py b/tests/test_genetic_complete_tail.py new file mode 100644 index 00000000..1927446b --- /dev/null +++ b/tests/test_genetic_complete_tail.py @@ -0,0 +1,438 @@ +"""Economic tail scenarios and hard control/forecast boundaries.""" + +from unittest.mock import patch + +import numpy as np +import pandas as pd +import pytest + +from akkudoktoreos.config.config import SettingsEOSDefaults +from akkudoktoreos.core.coreabc import get_ems +from akkudoktoreos.devices.genetic.battery import Battery, SolarPanelBatteryParameters +from akkudoktoreos.devices.genetic.inverter import Inverter, InverterParameters +from akkudoktoreos.optimization.genetic.forecast import bounded_forecast_array +from akkudoktoreos.optimization.genetic.genetic import GeneticOptimization +from akkudoktoreos.optimization.genetic.geneticparams import ( + GeneticOptimizationParameters, +) +from akkudoktoreos.optimization.genetic.tailvalue import build_tail_value_curve +from akkudoktoreos.optimization.genetic.terminalvalue import TerminalValueCurve +from akkudoktoreos.utils.datetimeutil import to_datetime, to_duration + + +def devices(power=1000, efficiency=1.0, lcos=0, ac_limit=None, export_power=5000): + bat = Battery( + SolarPanelBatteryParameters( + device_id="battery1", + capacity_wh=1000, + max_charge_power_w=power, + charging_efficiency=efficiency, + discharging_efficiency=efficiency, + initial_soc_percentage=50, + levelized_cost_of_storage_kwh=lcos, + charge_rates=[0, 0.5, 1], + ), + prediction_hours=1, + ) + inv = Inverter( + InverterParameters( + device_id="inverter1", + battery_id="battery1", + max_power_wh=export_power, + dc_to_ac_efficiency=1, + ac_to_dc_efficiency=1, + max_ac_charge_power_w=ac_limit, + ), + battery=bat, + ) + return bat, inv + + +def curve( + prices=(-0.1, 0.3), + tariffs=(0, 0.3), + direct=True, + continuation=None, + load=None, + pv=None, + **kwargs, +): + bat, inv = devices(**kwargs) + return build_tail_value_curve( + battery=bat, + inverter=inv, + prices_euro_per_wh=np.array(prices) / 1000, + feed_in_euro_per_wh=np.array(tariffs) / 1000, + load_wh=np.zeros(len(prices)) if load is None else np.array(load), + pv_wh=np.zeros(len(prices)) if pv is None else np.array(pv), + continuation=continuation or TerminalValueCurve(), + charge_rates=[0.5, 1], + export_rates=[1], + direct_marketing=direct, + ) + + +def test_headroom_has_value_and_empty_state_can_earn(): + c = curve() + assert c.value(0) == pytest.approx(0.4) + tail, continuation = c.component_values(0) + assert tail == pytest.approx(0.4) + assert continuation == pytest.approx(0.0) + assert c.value(0) == pytest.approx(tail + continuation) + assert c.value(500) > c.value(1000) + assert any(v < 0 for v in c.marginal_euro_per_kwh) + + +def test_chronology_changes_arbitrage(): + forward = curve() + reverse = curve(prices=(0.3, -0.1), tariffs=(0.3, 0)) + assert forward.value(0) > reverse.value(0) + + +def test_discharge_and_ac_power_limits(): + limited = curve(prices=(1,), tariffs=(1,), power=100) + assert limited.value(1000) == pytest.approx(0.1) + limited_ac = curve(ac_limit=100) + assert limited_ac.value(0) == pytest.approx(0.04) + limited_inverter = curve(prices=(1,), tariffs=(1,), export_power=50) + assert limited_inverter.value(1000) == pytest.approx(0.05) + + +def test_losses_and_lcos_reduce_arbitrage(): + ideal = curve(prices=(0.1, 0.3)) + lossy = curve(prices=(0.1, 0.3), efficiency=0.8) + assert 0 < lossy.value(0) < ideal.value(0) + assert curve(prices=(0.1, 0.3), lcos=0.25).value(0) == pytest.approx(0) + + +def test_no_battery_export_without_permission(): + assert curve(prices=(0.1, 0.3), direct=False).value(0) == pytest.approx(0) + + +def test_pv_surplus_can_be_stored_for_local_load(): + c = curve(prices=(0.2, 0.3), tariffs=(0, 0), pv=[1000, 0], load=[0, 1000], direct=False) + assert c.value(0) == pytest.approx(0) + # Without PV the same empty battery must buy energy to serve the load. + assert curve(prices=(0.2, 0.3), tariffs=(0, 0), load=[0, 1000], direct=False).value( + 0 + ) < c.value(0) + + +def test_continuation_survives_tail_end(): + continuation = TerminalValueCurve(energy_wh=[0, 1000], value_euro=[0, 0.2]) + c = curve(prices=(0.5,), tariffs=(0,), continuation=continuation) + assert c.value(1000) == pytest.approx(0.2) + tail, continuation_credit = c.component_values(1000) + assert tail == pytest.approx(0.0) + assert continuation_credit == pytest.approx(0.2) + + +def test_tail_diagnostic_plan_explains_the_selected_path(): + c = curve() + plan = c.diagnostic_plan(0, control_horizon_hours=24) + assert len(plan) == 2 + assert plan[0].hour_from_start == 24 + assert plan[0].action == "GRID_CHARGE" + assert plan[0].soc_end_percentage > plan[0].soc_start_percentage + assert plan[0].grid_import_wh > 0 + assert plan[1].action == "BATTERY_EXPORT" + assert plan[1].soc_end_percentage < plan[1].soc_start_percentage + assert plan[1].grid_export_wh > 0 + assert sum(slot.slot_value_euro for slot in plan) == pytest.approx(c.value(0)) + + +def test_central_config_invariant(): + # Only the control horizon is mandatory. A prediction horizon that cannot + # cover the requested tail shortens the tail instead of failing the run, + # so existing configurations keep starting after an upgrade. + short = SettingsEOSDefaults.model_validate( + dict( + prediction={"hours": 48}, + optimization={"genetic": {"horizon_hours": 24, "tail_horizon_hours": 48}}, + ) + ) + assert short.prediction.hours == 48 + assert short.optimization.genetic.tail_horizon_hours == 48 + + # A control horizon the forecast cannot serve is not rejected here either - + # prediction.hours also serves callers that never optimize. The optimizer + # rejects the run itself, naming the series that ran out. + undersized = SettingsEOSDefaults.model_validate( + dict( + prediction={"hours": 48}, + optimization={"genetic": {"horizon_hours": 72, "tail_horizon_hours": 0}}, + ) + ) + assert undersized.optimization.genetic.horizon_hours == 72 + + settings = SettingsEOSDefaults() + assert settings.prediction.hours == 48 + assert settings.optimization.genetic.tail_horizon_hours == 48 + + +def setup_run(config, interval=3600, start_hour=0, hours=72, prediction_hours=72): + config.merge_settings_from_dict( + { + "prediction": {"hours": prediction_hours}, + "optimization": { + "genetic": {"horizon_hours": 24, "tail_horizon_hours": 48, "interval_sec": interval} + }, + "feedintariff": {"direct_marketing_enabled": True}, + } + ) + ems = get_ems(init=True) + ems.set_start_datetime(to_datetime("2026-09-05T00:00:00").set(hour=start_hour)) + bat, inv = devices() + params = GeneticOptimizationParameters.model_validate( + dict( + ems={ + "pv_prognose_wh": [0.0] * hours, + "gesamtlast": [0.0] * hours, + "strompreis_euro_pro_wh": [0.0002] * hours, + "einspeiseverguetung_euro_pro_wh": [0.0001] * hours, + "preis_euro_pro_wh_akku": 0, + }, + pv_battery=bat.parameters, + inverter=inv.parameters, + ev=None, + ) + ) + return GeneticOptimization(fixed_seed=42), params + + +@pytest.mark.parametrize("interval", [3600, 900]) +@pytest.mark.parametrize("start_hour", [0, 10]) +@pytest.mark.asyncio +async def test_genome_output_and_final_control_state(config_eos, interval, start_hour): + opt, params = setup_run(config_eos, interval, start_hour, hours=72 + start_hour) + + def choose(*args, **kwargs): + # Discharge only in the last control slot. Its POST-slot SOC is credited. + genome = opt.create_individual() + genome[:] = [0] * opt.control_end_slot + genome[-1] = opt._battery_state_layout().grid_export_state + assert len(genome) == 24 * (3600 // interval) + return genome, {} + + with ( + patch.object(opt, "optimize", side_effect=choose), + patch( + "akkudoktoreos.optimization.genetic.genetic.build_tail_value_curve", + wraps=build_tail_value_curve, + ) as builder, + ): + result = opt.optimize_ems(params) + assert builder.call_count == 1 + assert len(result.ac_charge) == opt.control_slots + assert len(result.dc_charge) == opt.control_slots + assert len(result.discharge_allowed) == opt.control_slots + assert len(result.battery_grid_export_factor) == opt.control_slots + assert len(result.result.Kosten_Euro_pro_Stunde) == opt.control_slots + assert result.terminal_value.mode == "TAIL" + assert result.terminal_value.battery_energy_wh == pytest.approx( + max(500 - 1000 * opt.slot_duration_h, 0) + ) + assert result.terminal_value.effective_tail_hours == 48 + assert result.terminal_value.credited_euro == pytest.approx( + result.terminal_value.tail_operating_euro + result.terminal_value.continuation_value_euro + ) + assert result.terminal_value.continuation_curve is not None + assert result.terminal_value.tail_diagnostics is not None + assert result.terminal_value.tail_diagnostics.slots == 48 * (3600 // interval) + assert result.terminal_value.tail_diagnostics.soc_grid_points == 101 + assert len(result.terminal_value.tail_plan) == result.terminal_value.tail_diagnostics.slots + assert result.terminal_value.tail_plan[0].hour_from_start == 24 + assert len(result.terminal_value.curve.operating_value_euro) == 101 + assert len(result.terminal_value.curve.continuation_value_euro) == 101 + assert len((await result.optimization_solution()).solution.to_dataframe()) == opt.control_slots + + +def test_short_tail_is_reported(config_eos, caplog): + opt, params = setup_run(config_eos, hours=52) + with patch.object( + opt, "optimize", side_effect=lambda *a, **k: ([0] * opt.control_end_slot, {}) + ): + result = opt.optimize_ems(params) + assert result.terminal_value.effective_tail_hours == 28 + assert "Tail forecast shortened" in caplog.text + assert result.terminal_value.reason + + +def test_missing_control_is_rejected(config_eos): + opt, params = setup_run(config_eos, hours=23) + with pytest.raises(ValueError, match="Incomplete control forecast"): + opt.optimize_ems(params) + + +@pytest.mark.asyncio +async def test_provider_values_are_not_extrapolated(): + from types import SimpleNamespace + + start = to_datetime("2026-09-05T00:00:00Z") + series = pd.Series([1.0, 2.0], index=pd.date_range(start=start, periods=2, freq="h")) + from unittest.mock import AsyncMock + + provider = SimpleNamespace(key_to_raw_series=AsyncMock(return_value=series)) + result = await bounded_forecast_array( + provider, + key="price", + start_datetime=start, + end_datetime=start.add(hours=3), + interval=to_duration("15 minutes"), + ) + assert result[:8].tolist() == [1.0] * 4 + [2.0] * 4 + assert np.isnan(result[8:]).all() + + +def test_future_opportunity_changes_optimal_control_soc(config_eos): + final_energy = [] + for negative_price in (0.5, -1.0): + opt, params = setup_run(config_eos, hours=3) + config_eos.merge_settings_from_dict( + {"optimization": {"genetic": {"horizon_hours": 1, "tail_horizon_hours": 2}}} + ) + params.ems.electricity_price_per_wh = [0.0005, negative_price / 1000, 0.0003] + params.ems.feed_in_tariff_per_wh = [0.00002, 0, 0.0003] + result = opt.optimize_ems(params, ngen=3, individuals=20) + final_energy.append(result.terminal_value.battery_energy_wh) + assert final_energy[0] > final_energy[1] + + +@pytest.mark.parametrize("control", [24, 48]) +def test_continuation_prevents_emptying_at_moved_boundary(config_eos, control): + opt, params = setup_run(config_eos, hours=96) + config_eos.merge_settings_from_dict( + {"prediction": {"hours": 96}, "optimization": {"genetic": {"horizon_hours": control}}} + ) + # Zero control load, with the same future local demand visible to both tails. + params.ems.gesamtlast[60] = 1000 + params.ems.feed_in_tariff_per_wh = [0.0] * 96 + config_eos.feedintariff.direct_marketing_enabled = False + + def choose(*a, **kw): + from deap import creator + + idle = creator.Individual([0] * control) + discharge = creator.Individual([len(opt.bat_possible_charge_values)] * control) + assert opt.toolbox.evaluate(idle)[0] <= opt.toolbox.evaluate(discharge)[0] + return idle, {} + + with patch.object(opt, "optimize", side_effect=choose): + result = opt.optimize_ems(params) + assert result.terminal_value.battery_energy_wh == pytest.approx(500) + assert result.terminal_value.continuation_mode == "AUTO" + + +def test_missing_price_inside_tail_stops_at_first_gap(config_eos): + opt, params = setup_run(config_eos) + params.ems.electricity_price_per_wh[30] = float("nan") + with patch.object(opt, "optimize", side_effect=lambda *a, **k: ([0] * opt.control_slots, {})): + result = opt.optimize_ems(params) + assert result.terminal_value.effective_tail_hours == 6 + + +def test_ev_genome_and_output_are_control_only(config_eos): + from akkudoktoreos.devices.genetic.battery import ElectricVehicleParameters + + opt, params = setup_run(config_eos, interval=900, start_hour=10, hours=82) + params.ev = ElectricVehicleParameters( + device_id="ev1", + capacity_wh=5000, + initial_soc_percentage=0, + min_soc_percentage=50, + charge_rates=[0, 0.5, 1], + ) + + def choose(*a, **kw): + genome = opt.create_individual() + assert len(genome) == 2 * 96 + return genome, {} + + with patch.object(opt, "optimize", side_effect=choose): + result = opt.optimize_ems(params) + assert len(result.ev_charge_hours_float) == 96 + + +def test_rejected_config_update_is_atomic(config_eos): + # The candidate is validated before the singleton is reinitialized, so a + # rejected update must leave the running configuration untouched rather + # than half-applied. `hours` is constrained to be non-negative. + before = config_eos.prediction.hours + with pytest.raises(ValueError): + config_eos.merge_settings_from_dict({"prediction": {"hours": -1}}) + assert config_eos.prediction.hours == before + + +def test_disabled_ac_conversion_cannot_earn_negative_price_revenue(): + bat, inv = devices() + inv.parameters.ac_to_dc_efficiency = 0 + c = build_tail_value_curve( + battery=bat, + inverter=inv, + prices_euro_per_wh=np.array([-0.001, 0.001]), + feed_in_euro_per_wh=np.array([0.0, 0.001]), + load_wh=np.zeros(2), + pv_wh=np.zeros(2), + continuation=TerminalValueCurve(), + charge_rates=[1], + export_rates=[1], + direct_marketing=True, + ) + assert c.value(0) == pytest.approx(0) + assert bat.soc_wh == 500 # Building the tail never mutates the real battery. + + +def test_short_native_forecast_declares_its_resolution(config_eos): + opt, params = setup_run(config_eos, interval=900, hours=52 * 4) + params.forecast_interval_seconds = 900 + with patch.object(opt, "optimize", side_effect=lambda *a, **k: ([0] * opt.control_slots, {})): + result = opt.optimize_ems(params) + assert result.terminal_value.effective_tail_hours == 28 + + +@pytest.mark.parametrize( + "field,reason", + [ + ("electricity_price_per_wh", "import price"), + ("feed_in_tariff_per_wh", "feed-in tariff"), + ], +) +def test_differing_provider_lengths_use_the_common_tail(config_eos, field, reason): + opt, params = setup_run(config_eos) + setattr(params.ems, field, getattr(params.ems, field)[:52]) + with patch.object(opt, "optimize", side_effect=lambda *a, **k: ([0] * opt.control_slots, {})): + result = opt.optimize_ems(params) + assert result.terminal_value.effective_tail_hours == 28 + assert reason in result.terminal_value.reason + + +@pytest.mark.asyncio +async def test_missing_provider_key_stays_missing(): + from types import SimpleNamespace + + async def unavailable(*a, **kw): + raise KeyError("price unavailable") + + start = to_datetime("2026-09-05T00:00:00Z") + result = await bounded_forecast_array( + SimpleNamespace(key_to_raw_series=unavailable), + key="price", + start_datetime=start, + end_datetime=start.add(hours=2), + interval=to_duration("1 hour"), + ) + assert np.isnan(result).all() + assert len(result) == 2 + + +def test_short_prediction_horizon_shortens_the_tail(config_eos): + # The forecast budget cannot serve the full 48 h tail. The run keeps going + # with the 12 h that are left after the control horizon instead of failing. + opt, params = setup_run(config_eos, hours=36, prediction_hours=36) + assert opt.control_slots == 24 + assert opt.tail_slots == 12 + + result = opt.optimize_ems(params, ngen=2) + assert result.terminal_value.mode == "TAIL" + assert result.terminal_value.requested_tail_hours == 48 + assert result.terminal_value.effective_tail_hours == 12 diff --git a/tests/test_genetic_complete_timegrid.py b/tests/test_genetic_complete_timegrid.py new file mode 100644 index 00000000..6711718a --- /dev/null +++ b/tests/test_genetic_complete_timegrid.py @@ -0,0 +1,165 @@ +"""Native GENETIC results retain the elapsed-time grid and immutable run inputs.""" + +from unittest.mock import patch + +import numpy as np +import pytest + +from akkudoktoreos.core.coreabc import get_ems +from akkudoktoreos.optimization.genetic.genetic import GeneticOptimization +from akkudoktoreos.optimization.genetic.geneticparams import ( + GeneticOptimizationParameters, +) +from akkudoktoreos.utils.datetimeutil import to_datetime + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "timestamp", + ["2026-03-29T03:15:00+02:00", "2026-10-25T02:30:00+02:00", "2026-10-25T02:30:00+01:00"], +) +async def test_native_result_retains_dst_grid_and_owned_inputs(config_eos, timestamp): + config_eos.merge_settings_from_dict( + { + "general": {"latitude": 52.52, "longitude": 13.405}, + "prediction": {"hours": 24}, + "optimization": { + "genetic": { + "interval_sec": 900, + "horizon_hours": 1, + "tail_horizon_hours": 0, + "terminal_value_mode": "FIXED", + } + }, + } + ) + start = to_datetime(timestamp).in_timezone("Europe/Berlin") + get_ems(init=True).set_start_datetime(start) + elapsed_slots = int((start - start.start_of("day")).total_seconds() // 900) + count = elapsed_slots + 4 + parameters = GeneticOptimizationParameters.model_validate( + { + "forecast_interval_seconds": 900, + "ems": { + "pv_forecast_wh": [0.0] * count, + "total_load": [25.0] * count, + "electricity_price_per_wh": [0.0003] * count, + "feed_in_tariff_per_wh": [0.00005] * count, + "price_per_wh_battery": 0.0, + }, + "pv_battery": { + "device_id": "owned_battery", + "capacity_wh": 1000, + "initial_soc_percentage": 50, + }, + "inverter": { + "device_id": "owned_inverter", + "battery_id": "owned_battery", + "max_power_wh": 1000, + }, + "ev": None, + } + ) + optimizer = GeneticOptimization(fixed_seed=42) + native = optimizer.optimize_ems(parameters, ngen=1, individuals=6) + repeat = GeneticOptimization(fixed_seed=42).optimize_ems(parameters, ngen=1, individuals=6) + assert native.start_solution == repeat.start_solution + assert native.interval_seconds == 900 + assert native.start_solution_datetime == start + assert native.controls_start_at_now + assert len(native.result.load_wh_per_hour) == 4 + assert native.parameters.ems.total_load == [25.0] * 4 + parameters.ems.total_load[elapsed_slots] = 999.0 + get_ems().set_start_datetime(start.add(days=1)) + config_eos.optimization.genetic.interval_sec = 3600 + with patch( + "akkudoktoreos.optimization.genetic.geneticsolution.get_prediction", + side_effect=AssertionError("Native serialization must not reread providers"), + ): + generic = await native.optimization_solution() + plan = native.energy_management_plan() + assert generic.valid_from == start + assert generic.valid_until == start.add(hours=1) + assert plan.valid_from == start + assert plan.valid_until is None + assert all( + start.timestamp() <= item.execution_time.timestamp() < start.add(hours=1).timestamp() + for item in plan.instructions + ) + forecast = generic.prediction.to_dataframe() + np.testing.assert_allclose(forecast["loadforecast_energy_wh"], [25.0] * 4) + np.testing.assert_allclose(forecast["elec_price_amt_kwh"], [0.3] * 4) + assert all( + (right - left).total_seconds() == 900 + for left, right in zip(forecast.index, forecast.index[1:]) + ) + assert "owned_battery_soc_factor" in generic.solution.to_dataframe().columns + + +def test_native_quarter_hour_temperatures_average_when_coarsened(config_eos): + config_eos.merge_settings_from_dict( + {"optimization": {"genetic": {"interval_sec": 3600, "horizon_hours": 1}}} + ) + get_ems(init=True).set_start_datetime( + to_datetime("2026-09-16T00:00:00+02:00", in_timezone="Europe/Berlin") + ) + parameters = GeneticOptimizationParameters.model_validate( + { + "forecast_interval_seconds": 900, + "temperature_forecast": [10, 12, 14, 16], + "ems": { + "pv_forecast_wh": [25.0] * 4, + "total_load": [50.0] * 4, + "electricity_price_per_wh": [0.0003] * 4, + "feed_in_tariff_per_wh": 0.00005, + "price_per_wh_battery": 0.0, + }, + "pv_battery": None, + "ev": None, + "inverter": None, + } + ) + normalized = GeneticOptimization()._parameters_for_slot_grid(parameters) + assert normalized.temperature_forecast == [13.0] + assert normalized.ems.pv_forecast_wh == [100.0] + assert normalized.ems.total_load == [200.0] + + +@pytest.mark.parametrize("host_timezone", ["UTC", "Europe/Berlin"]) +@pytest.mark.parametrize("offset", ["+02:00", "+01:00"]) +@pytest.mark.parametrize("as_json_string", [False, True]) +def test_snapshot_and_warm_start_preserve_aware_instants( + config_eos, set_other_timezone, host_timezone, offset, as_json_string +): + import json + from pathlib import Path + + from akkudoktoreos.optimization.genetic.configrequest import ( + ConfigOptimizationRequest, + ) + from akkudoktoreos.optimization.genetic.geneticsolution import GeneticSolution + + set_other_timezone(host_timezone) + expected = to_datetime(f"2026-10-25T02:30:00{offset}", in_timezone="Europe/Berlin") + supplied = expected.to_iso8601_string() if as_json_string else expected + payload = json.loads( + (Path(__file__).parent / "testdata/genetic/optimize_result_1.json").read_text() + ) + payload["start_solution_datetime"] = supplied + native = GeneticSolution.model_validate(payload) + parameters = GeneticOptimizationParameters.model_validate( + native.parameters.model_dump() | {"start_solution_datetime": supplied} + ) + request = ConfigOptimizationRequest.model_validate({"start_solution_datetime": supplied}) + for model in (native, parameters, request): + value = model.start_solution_datetime + assert value is not None + assert value.timestamp() == expected.timestamp() + assert value.utcoffset() == expected.utcoffset() + if not as_json_string: + assert value.timezone_name == "Europe/Berlin" + assert value.fold == expected.fold + restored = type(model).model_validate_json(model.model_dump_json()) + assert restored.start_solution_datetime is not None + assert restored.start_solution_datetime.timestamp() == expected.timestamp() + assert restored.start_solution_datetime.utcoffset() == expected.utcoffset() diff --git a/tests/test_genetic_end_to_end_devices.py b/tests/test_genetic_end_to_end_devices.py new file mode 100644 index 00000000..c3565f8e --- /dev/null +++ b/tests/test_genetic_end_to_end_devices.py @@ -0,0 +1,417 @@ +"""Real small optimizer runs covering device contracts across the public result.""" + +from collections.abc import AsyncGenerator, Callable +from typing import Any + +import numpy as np +import pytest +import pytest_asyncio + +from akkudoktoreos.config.config import ConfigEOS +from akkudoktoreos.core.coreabc import get_ems, get_measurement +from akkudoktoreos.core.emplan import DDBCInstruction +from akkudoktoreos.measurement.measurement import Measurement +from akkudoktoreos.optimization.genetic.configrequest import ConfigOptimizationRequest +from akkudoktoreos.optimization.genetic.genetic import GeneticOptimization +from akkudoktoreos.optimization.genetic.geneticparams import ( + GeneticOptimizationParameters, +) +from akkudoktoreos.optimization.genetic.geneticsolution import GeneticSolution +from akkudoktoreos.utils.datetimeutil import DateTime, to_datetime + + +@pytest.fixture(autouse=True, params=["UTC", "Europe/Berlin"]) +def local_clock(request: pytest.FixtureRequest, set_other_timezone: Callable[[str], str]) -> None: + """Run the same local-wall-clock schedules in UTC and a DST-observing zone. + + Scenario dates intentionally have no fixed offset: each names local midnight, + a local time window or a local departure in the selected timezone. + """ + set_other_timezone(request.param) + + +@pytest_asyncio.fixture +async def isolated_measurement(config_eos: ConfigEOS) -> AsyncGenerator[Measurement, None]: + """Keep synthetic records out of the process-wide measurement singleton.""" + measurement = get_measurement() + await measurement.delete_by_datetime(None, None) + try: + yield measurement + finally: + await measurement.delete_by_datetime(None, None) + + +def configure( + config: ConfigEOS, + *, + hours: int = 4, + start: str = "2026-09-16T00:00:00", + marketing: bool = False, +) -> None: + config.merge_settings_from_dict( + { + "prediction": {"hours": max(48, hours)}, + "optimization": { + "algorithm": "GENETIC", + "genetic": { + "horizon_hours": hours, + "tail_horizon_hours": 0, + "interval_sec": 900, + "individuals": 12, + "generations": 10, + "terminal_value_mode": "FIXED", + }, + }, + "feedintariff": {"direct_marketing_enabled": marketing}, + } + ) + get_ems(init=True).set_start_datetime(to_datetime(start)) + + +def parameters( + *, hours: int = 4, consumers: list[dict[str, Any]] | None = None, **kwargs: Any +) -> GeneticOptimizationParameters: + slots = hours * 4 + get_ems().start_datetime.hour * 4 + get_ems().start_datetime.minute // 15 + return GeneticOptimizationParameters.model_validate( + { + "ems": { + "pv_prognose_wh": [0.0] * slots, + "gesamtlast": [0.0] * slots, + "strompreis_euro_pro_wh": [0.0003] * slots, + "einspeiseverguetung_euro_pro_wh": [0.0001] * slots, + "preis_euro_pro_wh_akku": 0.0, + }, + "inverter": {"device_id": "inv", "max_power_wh": 10000}, + "forecast_interval_seconds": 900, + "home_appliances": consumers, + "pv_battery": None, + "ev": None, + **kwargs, + } + ) + + +def run(params: GeneticOptimizationParameters) -> tuple[GeneticOptimization, GeneticSolution]: + optimizer = GeneticOptimization(fixed_seed=42) + solution = optimizer.optimize_ems(params, ngen=1, individuals=12) + return optimizer, solution + + +def run_local_starts(solution: GeneticSolution) -> list[DateTime]: + """Compare scheduled instants in the run zone, independently of output zone.""" + timezone = get_ems().start_datetime.timezone_name + assert timezone is not None + return [moment.in_timezone(timezone) for moment in solution.appliance_starts["washer"]] + + +def profile(**kwargs: Any) -> dict[str, Any]: + return { + "device_id": "washer", + "load_profile_power_w": [1200.0, 600.0, 300.0], + "load_profile_interval_seconds": 600, + **kwargs, + } + + +def test_real_optimizer_keeps_reverse_cycle_window_identity(config_eos: ConfigEOS) -> None: + configure(config_eos) + consumer = profile( + num_cycles=2, + time_windows={ + "windows": [ + {"start_time": "02:00", "duration": "30 minutes", "value": 0}, + {"start_time": "01:00", "duration": "30 minutes", "value": 1}, + ] + }, + ) + _, solution = run(parameters(consumers=[consumer])) + assert [value.hour for value in run_local_starts(solution)] == [1, 2] + assert sum(solution.result.home_appliance_energy_wh["washer"]) == pytest.approx(700.0) + assert sum(solution.result.grid_consumption_wh_per_hour) == pytest.approx(700.0) + assert solution.result.total_costs == pytest.approx(0.21) + + +def test_real_daily_optimizer_skips_completed_cycles_only_on_first_day( + config_eos: ConfigEOS, +) -> None: + configure(config_eos, hours=28) + consumer = profile( + num_cycles=2, + completed_cycles=1, + schedule_mode="DAILY", + time_windows={ + "windows": [ + {"start_time": "01:00", "duration": "30 minutes", "value": 0}, + {"start_time": "02:00", "duration": "30 minutes", "value": 1}, + ] + }, + ) + _, solution = run(parameters(hours=28, consumers=[consumer])) + assert [(value.day, value.hour) for value in run_local_starts(solution)] == [ + (16, 2), + (17, 1), + (17, 2), + ] + assert sum(solution.result.home_appliance_energy_wh["washer"]) == pytest.approx(1050.0) + + +def test_real_best_effort_multicycle_prioritizes_delay_over_cheaper_prices( + config_eos: ConfigEOS, +) -> None: + configure(config_eos) + consumer = profile(num_cycles=2, min_cycle_gap_h=1, deadline_datetime="2026-09-15T23:00:00") + params = parameters(consumers=[consumer]) + params.ems.electricity_price_per_wh = [0.001] * 8 + [-0.001] * 8 + _, solution = run(params) + assert [(value.hour, value.minute) for value in run_local_starts(solution)] == [ + (0, 0), + (1, 30), + ] + assert solution.appliance_deadline_missed["washer"] + assert sum(solution.result.home_appliance_energy_wh["washer"]) == pytest.approx(700.0) + + +def test_real_mixed_best_effort_cycle_status_survives_later_strict_cycle( + config_eos: ConfigEOS, +) -> None: + configure(config_eos) + consumer = profile( + num_cycles=2, + deadline_datetime="2026-09-16T01:00:00", + time_windows={ + "windows": [ + {"start_time": "02:00", "duration": "2 hours", "value": 0}, + {"start_time": "00:00", "duration": "30 minutes", "value": 1}, + ] + }, + ) + params = parameters(consumers=[consumer]) + params.ems.electricity_price_per_wh = [0.001] * 12 + [-0.001] * 4 + _, solution = run(params) + assert [(value.hour, value.minute) for value in run_local_starts(solution)] == [ + (0, 0), + (2, 0), + ] + assert solution.appliance_deadline_missed["washer"] + + +def test_real_warm_start_handles_changed_completed_cycle_layout(config_eos: ConfigEOS) -> None: + configure(config_eos) + consumer = profile( + num_cycles=2, + time_windows={ + "windows": [ + {"start_time": "01:00", "duration": "30 minutes", "value": 0}, + {"start_time": "02:00", "duration": "30 minutes", "value": 1}, + ] + }, + ) + optimizer, first = run(parameters(consumers=[consumer])) + consumer["completed_cycles"] = 1 + followup = parameters( + consumers=[consumer], + start_solution=first.start_solution, + start_solution_datetime=first.start_solution_datetime, + ) + second = optimizer.optimize_ems(followup, ngen=1, individuals=12) + assert len(second.appliance_starts["washer"]) == 1 + assert run_local_starts(second)[0].hour == 2 + assert sum(second.result.home_appliance_energy_wh["washer"]) == pytest.approx(350.0) + assert first.start_solution is not None and second.start_solution is not None + assert len(second.start_solution) == len(first.start_solution) - 1 + + +@pytest.mark.parametrize("deadline", ["2026-09-15T23:00:00", "2026-09-16T00:01:00"]) +def test_real_ev_does_not_credit_energy_delivered_after_departure( + config_eos: ConfigEOS, deadline: str +) -> None: + configure(config_eos) + params = parameters( + ev={ + "device_id": "car", + "capacity_wh": 4000, + "initial_soc_percentage": 0, + "min_soc_percentage": 25, + "charging_efficiency": 1.0, + "max_charge_power_w": 4000, + "charge_rates": [0.0, 1.0], + "min_soc_deadline_datetime": deadline, + } + ) + optimizer, solution = run(params) + assert optimizer._ev_soc_deadline_slot == 0 + assert ( + optimizer._ev_soc_at_deadline({"EAuto_SoC_pro_Stunde": solution.result.ev_soc_per_hour}, 0) + == 0.0 + ) + + +def test_real_ev_target_across_midnight_uses_elapsed_slots(config_eos: ConfigEOS) -> None: + configure(config_eos, start="2026-09-16T23:30:00") + params = parameters( + ev={ + "device_id": "car", + "capacity_wh": 4000, + "initial_soc_percentage": 0, + "min_soc_percentage": 50, + "charging_efficiency": 1.0, + "max_charge_power_w": 4000, + "charge_rates": [0.0, 1.0], + "min_soc_deadline_datetime": "2026-09-17T00:00:00", + } + ) + params.ems.electricity_price_per_wh[-16:] = [0.001] * 2 + [0.00001] * 14 + optimizer, solution = run(params) + assert optimizer._ev_soc_deadline_slot == 2 + assert solution.result.ev_soc_per_hour[2] >= 50 + assert solution.ev_charge_hours_float is not None + assert solution.ev_charge_hours_float[:2] == [1.0, 1.0] + + +@pytest.mark.parametrize( + "marketing,lcos,export_expected", + [(False, 0.0, False), (True, 0.0, True), (True, 0.2, True), (True, 2.0, False)], +) +def test_real_export_respects_marketing_gate_and_storage_cost( + config_eos: ConfigEOS, marketing: bool, lcos: float, export_expected: bool +) -> None: + configure(config_eos, marketing=marketing) + params = parameters( + pv_battery={ + "device_id": "battery", + "capacity_wh": 1000, + "initial_soc_percentage": 100, + "charging_efficiency": 1.0, + "discharging_efficiency": 1.0, + "min_soc_percentage": 0, + "max_charge_power_w": 4000, + "grid_export_rates": [0.5, 1.0], + "levelized_cost_of_storage_kwh": lcos, + }, + inverter={"device_id": "inv", "max_power_wh": 10000, "battery_id": "battery"}, + ) + params.ems.feed_in_tariff_per_wh = [0.00001] + [0.001] * 4 + [0.00001] * 11 + _, solution = run(params) + exported = sum(solution.result.grid_feed_in_wh_per_hour) + if export_expected: + assert exported == pytest.approx(1000.0) + assert solution.result.total_revenue == pytest.approx(1.0) + assert solution.result.total_costs == pytest.approx(lcos) + assert any(solution.battery_grid_export_allowed) + else: + assert exported == pytest.approx(0.0) + assert not any(solution.battery_grid_export_allowed) + assert np.isfinite(solution.result.total_balance) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("custom_key", [None, "washer.completed_today"]) +async def test_real_measurement_completed_cycles_reach_request_and_optimizer( + config_eos: ConfigEOS, custom_key: str | None, isolated_measurement: Measurement +) -> None: + configure(config_eos) + config_eos.merge_settings_from_dict( + { + "devices": { + "max_batteries": 0, + "batteries": {}, + "max_electric_vehicles": 0, + "electric_vehicles": {}, + "max_inverters": 1, + "inverters": {"inv": {"max_power_w": 10000}}, + "max_home_appliances": 1, + "home_appliances": { + "washer": profile( + num_cycles=2, + cycles_completed_measurement_key=custom_key, + cycle_time_windows={ + "windows": [ + {"start_time": "01:00", "duration": "30 minutes", "value": 0}, + {"start_time": "02:00", "duration": "30 minutes", "value": 1}, + ] + }, + ) + }, + } + } + ) + key = custom_key or "washer.cycles_completed" + assert key in config_eos.devices.measurement_keys + measurement = isolated_measurement + zero = get_ems().start_datetime + await measurement.update_value(zero.subtract(days=1), key, 2.0) + await measurement.update_value(zero, key, 1.0) + await measurement.update_value(zero.add(seconds=1), key, 2.0) + dates, counts = await measurement.key_to_lists( + key=key, start_datetime=zero, end_datetime=zero.add(seconds=1) + ) + assert len(dates) == 1 and counts == [1.0] + prepared = await ConfigOptimizationRequest.model_validate( + { + "forecasts": { + "pv_forecast_wh": [0.0] * 16, + "total_load": [0.0] * 16, + "electricity_price_per_wh": [0.0003] * 16, + "feed_in_tariff_per_wh": [0.0001] * 16, + } + } + ).resolve() + assert prepared.home_appliances is not None + assert prepared.home_appliances[0].completed_cycles == 1 + _, solution = run(prepared) + assert [start.hour for start in run_local_starts(solution)] == [2] + assert sum(solution.result.home_appliance_energy_wh["washer"]) == pytest.approx(350.0) + + +@pytest.mark.asyncio +async def test_real_multiple_consumers_keep_separate_solution_channels( + config_eos: ConfigEOS, +) -> None: + configure(config_eos) + consumers = [ + profile( + shared_time_windows={"windows": [{"start_time": "01:00", "duration": "30 minutes"}]} + ), + profile( + device_id="dryer", + load_profile_power_w=[2000.0], + load_profile_interval_seconds=900, + shared_time_windows={"windows": [{"start_time": "01:00", "duration": "15 minutes"}]}, + ), + ] + _, solution = run(parameters(consumers=consumers)) + assert sum(solution.result.home_appliance_energy_wh["washer"]) == pytest.approx(350.0) + assert sum(solution.result.home_appliance_energy_wh["dryer"]) == pytest.approx(500.0) + assert sum(solution.result.grid_consumption_wh_per_hour) == pytest.approx(850.0) + exported = await solution.optimization_solution() + table = exported.solution.to_dataframe() + assert table["washer_energy_wh"].sum() == pytest.approx(350.0) + assert table["dryer_energy_wh"].sum() == pytest.approx(500.0) + assert table.index[4].hour == 1 + assert table["washer_run_op_mode"].tolist()[4:6] == [1.0, 1.0] + assert table["dryer_run_op_mode"].tolist()[4:6] == [1.0, 0.0] + + +@pytest.mark.asyncio +async def test_profile_zero_power_phase_keeps_device_running_until_complete( + config_eos: ConfigEOS, +) -> None: + configure(config_eos) + consumer = profile( + load_profile_power_w=[1200.0, 0.0, 1200.0], + load_profile_interval_seconds=900, + shared_time_windows={"windows": [{"start_time": "01:00", "duration": "45 minutes"}]}, + ) + _, solution = run(parameters(consumers=[consumer])) + exported = await solution.optimization_solution() + table = exported.solution.to_dataframe() + assert table["washer_energy_wh"].tolist()[4:7] == [300.0, 0.0, 300.0] + assert table["washer_run_op_mode"].tolist()[4:7] == [1.0, 1.0, 1.0] + plan = solution.energy_management_plan() + assert {item.resource_id for item in plan.instructions} == {"washer"} + commands = [ + (item.execution_time.hour, item.execution_time.minute, str(item.operation_mode_id)) + for item in plan.instructions + if isinstance(item, DDBCInstruction) + ] + assert commands == [(0, 0, "OFF"), (1, 0, "RUN"), (1, 45, "OFF")] diff --git a/tests/test_genetic_forecast_coverage.py b/tests/test_genetic_forecast_coverage.py new file mode 100644 index 00000000..e6c6e81d --- /dev/null +++ b/tests/test_genetic_forecast_coverage.py @@ -0,0 +1,66 @@ +"""Raw forecast coverage must survive resampling without inventing valid intervals.""" + +from unittest.mock import AsyncMock, Mock + +import numpy as np +import pandas as pd +import pytest + +from akkudoktoreos.optimization.genetic.forecast import bounded_forecast_array +from akkudoktoreos.utils.datetimeutil import to_datetime, to_duration + + +@pytest.mark.asyncio +@pytest.mark.parametrize("drop,missing", [(1, 4), (2, 8)]) +async def test_hourly_gaps_and_unavailable_tail_are_not_forward_filled(drop, missing): + index = pd.date_range("2026-09-16T00:00:00Z", periods=4, freq="h").delete(drop) + prediction = Mock(key_to_raw_series=AsyncMock(return_value=pd.Series([100.0] * 3, index=index))) + start = to_datetime("2026-09-16T00:00:00Z", in_timezone="UTC") + values = await bounded_forecast_array( + prediction, + key="pv", + start_datetime=start, + end_datetime=start.add(hours=5), + interval=to_duration(900), + ) + assert np.isnan(values[missing : missing + 4]).all() + assert np.isnan(values[16:]).all() + assert np.isfinite(values[:4]).all() + + +@pytest.mark.asyncio +async def test_downsampling_requires_complete_coverage_and_uses_interval_average(): + index = pd.date_range("2026-09-16T00:00:00Z", periods=8, freq="15min") + series = pd.Series([100.0, 200.0, 300.0, 400.0, 100.0, np.nan, 100.0, 100.0], index=index) + prediction = Mock(key_to_raw_series=AsyncMock(return_value=series)) + start = to_datetime("2026-09-16T00:00:00Z", in_timezone="UTC") + values = await bounded_forecast_array( + prediction, + key="pv", + start_datetime=start, + end_datetime=start.add(hours=2), + interval=to_duration(3600), + ) + assert values[0] == 250.0 + assert np.isnan(values[1]) + + +@pytest.mark.asyncio +async def test_non_aligned_source_intervals_are_weighted_by_actual_overlap(): + index = pd.date_range("2026-09-16T00:05:00Z", periods=4, freq="15min") + prediction = Mock( + key_to_raw_series=AsyncMock( + return_value=pd.Series([100.0, 400.0, 700.0, 1000.0], index=index) + ) + ) + start = to_datetime("2026-09-16T00:00:00Z", in_timezone="UTC") + values = await bounded_forecast_array( + prediction, + key="pv", + start_datetime=start, + end_datetime=start.add(hours=1), + interval=to_duration(900), + ) + assert np.isnan(values[0]) + assert values[1] == pytest.approx(300.0) + assert values[2] == pytest.approx(600.0) diff --git a/tests/test_genetic_seeding.py b/tests/test_genetic_seeding.py new file mode 100644 index 00000000..3e35ea74 --- /dev/null +++ b/tests/test_genetic_seeding.py @@ -0,0 +1,476 @@ +from types import SimpleNamespace +from unittest.mock import patch + +import numpy as np +import pytest +from deap import creator, tools + +from akkudoktoreos.config.config import ConfigEOS +from akkudoktoreos.core.coreabc import get_ems +from akkudoktoreos.optimization.genetic.genetic import GeneticOptimization +from akkudoktoreos.utils.datetimeutil import to_datetime + + +def _configure_hourly_grid(config_eos: ConfigEOS, *, start_hour: int = 0) -> None: + config_eos.merge_settings_from_dict( + { + "prediction": {"hours": 48}, + "optimization": { + "genetic": {"tail_horizon_hours": 0, "horizon_hours": 48, "interval_sec": 3600} + }, + } + ) + get_ems(init=True).set_start_datetime(to_datetime().set(hour=start_hour, minute=0)) + + +def test_ev_repair_is_resimulated_before_fitness_assignment(config_eos: ConfigEOS): + _configure_hourly_grid(config_eos) + opt = GeneticOptimization(fixed_seed=42) + opt.optimize_ev = True + opt.ev_possible_charge_values = [0.0, 1.0] + opt.setup_deap_environment({"home_appliance": 0}, start_hour=0) + individual = creator.Individual([0] * opt.control_slots + [1] * opt.control_slots) + + first_result = { + "Gesamtbilanz_Euro": 10.0, + "Gesamt_Verluste": 0.0, + "EAuto_SoC_pro_Stunde": np.full(opt.control_slots, 100.0), + } + repaired_result = { + "Gesamtbilanz_Euro": 1.0, + "Gesamt_Verluste": 0.0, + "EAuto_SoC_pro_Stunde": np.full(opt.control_slots, 100.0), + } + parameters = SimpleNamespace( + ems=SimpleNamespace(price_per_wh_battery=0.0), + ev=None, + ) + + with patch.object( + opt, "evaluate_inner", side_effect=[first_result, repaired_result] + ) as evaluate: + fitness = opt.evaluate(individual, parameters, start_hour=0, worst_case=False) # type: ignore[arg-type] + + assert evaluate.call_count == 2 + assert fitness == pytest.approx((1.0,)) + assert individual[opt.control_slots :] == [0] * opt.control_slots + + +def test_fitness_cache_restores_canonical_ev_genome(config_eos: ConfigEOS): + _configure_hourly_grid(config_eos) + opt = GeneticOptimization(fixed_seed=42) + opt.optimize_ev = True + opt.ev_possible_charge_values = [0.0, 1.0] + opt.setup_deap_environment({"home_appliance": 0}, start_hour=0) + parameters = SimpleNamespace( + ems=SimpleNamespace(price_per_wh_battery=0.0), + ev=None, + ) + result = { + "Gesamtbilanz_Euro": 1.0, + "Gesamt_Verluste": 0.0, + "EAuto_SoC_pro_Stunde": np.full(opt.control_slots, 100.0), + } + first = creator.Individual([0] * opt.control_slots + [1] * opt.control_slots) + duplicate = creator.Individual(first) + opt._fitness_cache_enabled = True + + with patch.object(opt, "evaluate_inner", return_value=result) as evaluate: + first_fitness = opt.evaluate(first, parameters, 0, False) # type: ignore[arg-type] + duplicate_fitness = opt.evaluate(duplicate, parameters, 0, False) # type: ignore[arg-type] + + # The miss evaluates and then re-evaluates the repaired EV plan. The duplicate + # is served directly from the original-key alias and receives the canonical genome. + assert evaluate.call_count == 2 + assert first_fitness == duplicate_fitness + assert duplicate == first + assert duplicate[opt.control_slots :] == [0] * opt.control_slots + assert duplicate.extra_data == first.extra_data + assert opt._fitness_cache_hits == 1 + assert opt._fitness_cache_misses == 1 + + +def test_fitness_cache_never_stores_failed_evaluations(config_eos: ConfigEOS): + _configure_hourly_grid(config_eos) + opt = GeneticOptimization(fixed_seed=42) + opt.optimize_ev = False + opt.setup_deap_environment({"home_appliance": 0}, start_hour=0) + parameters = SimpleNamespace( + ems=SimpleNamespace(price_per_wh_battery=0.0), + ev=None, + ) + first = creator.Individual([0] * opt.control_slots) + duplicate = creator.Individual(first) + opt._fitness_cache_enabled = True + + with patch.object(opt, "evaluate_inner", side_effect=RuntimeError("transient")) as evaluate: + assert opt.evaluate(first, parameters, 0, False) == (100000.0,) # type: ignore[arg-type] + assert opt.evaluate(duplicate, parameters, 0, False) == (100000.0,) # type: ignore[arg-type] + + assert evaluate.call_count == 2 + assert opt._fitness_cache_hits == 0 + assert opt._fitness_cache_misses == 2 + assert opt._fitness_cache == {} + + +def test_fitness_cache_includes_first_run_relative_control(config_eos: ConfigEOS): + _configure_hourly_grid(config_eos, start_hour=10) + opt = GeneticOptimization(fixed_seed=42) + opt.optimize_ev = False + opt.setup_deap_environment({"home_appliance": 0}, start_hour=10) + parameters = SimpleNamespace( + ems=SimpleNamespace(price_per_wh_battery=0.0), + ev=None, + ) + result = { + "Gesamtbilanz_Euro": 1.0, + "Gesamt_Verluste": 0.0, + "EAuto_SoC_pro_Stunde": np.zeros(opt.control_slots), + } + first = creator.Individual([0] * opt.control_slots) + elapsed_variant = creator.Individual(first) + elapsed_variant[0] = 1 + opt._fitness_cache_enabled = True + + with patch.object(opt, "evaluate_inner", return_value=result) as evaluate: + first_fitness = opt.evaluate(first, parameters, 10, False) # type: ignore[arg-type] + variant_fitness = opt.evaluate(elapsed_variant, parameters, 10, False) # type: ignore[arg-type] + + assert evaluate.call_count == 2 + assert first_fitness == variant_fitness + assert opt._fitness_cache_hits == 0 + + +def test_mutated_warm_start_neighbors_stay_within_control_horizon(config_eos: ConfigEOS): + _configure_hourly_grid(config_eos, start_hour=10) + opt = GeneticOptimization(fixed_seed=42) + opt.optimize_ev = False + opt.setup_deap_environment({"home_appliance": 0}, start_hour=10) + start_solution = [0.0] * opt.control_slots + + neighbors = opt._mutated_warm_start_neighbors(start_solution, count=5) + + assert len(neighbors) == 5 + assert len({tuple(neighbor) for neighbor in neighbors}) == 5 + assert all(len(neighbor) == opt.control_slots for neighbor in neighbors) + assert all(neighbor != start_solution for neighbor in neighbors) + + +def test_initial_population_uses_fixed_seed_budget_and_configured_population( + config_eos: ConfigEOS, +): + _configure_hourly_grid(config_eos) + config_eos.optimization.genetic.individuals = 300 + opt = GeneticOptimization(fixed_seed=42) + opt.optimize_ev = False + opt.setup_deap_environment({"home_appliance": 0}, start_hour=0) + start_solution = [5.0] * opt.control_slots + warm_neighbors = [[6] * opt.control_slots for _ in range(50)] + educated = [[7] * opt.control_slots for _ in range(100)] + captured: dict[str, object] = {} + + def fake_evolution(population, **kwargs): + captured["population"] = list(population) + captured["mu"] = kwargs["mu"] + captured["lambda"] = kwargs["lambda_"] + for individual in population: + individual.fitness.values = (float(sum(individual)),) + individual.extra_data = (0.0, 0.0, 0.0) + kwargs["halloffame"].update(population) + return population, SimpleNamespace(select=lambda _name: []) + + with ( + patch.object(opt, "_mutated_warm_start_neighbors", return_value=warm_neighbors), + patch.object(opt, "_educated_guess_individuals", return_value=educated), + patch.object( + opt.toolbox, + "population", + side_effect=lambda n: [creator.Individual([9] * opt.control_slots) for _ in range(n)], + ), + patch.object(opt, "_evolve_population_adaptive", side_effect=fake_evolution), + ): + opt.optimize(start_solution=start_solution, ngen=1) + + population = captured["population"] + assert isinstance(population, list) + first_genes = [individual[0] for individual in population] + assert len(population) == 300 + assert first_genes.count(5) == 10 + assert first_genes.count(6) == 50 + assert first_genes.count(7) == 100 + assert first_genes.count(9) == 140 + assert captured["mu"] == 300 + assert captured["lambda"] == 300 + + +def test_small_population_scales_warm_and_educated_seed_families(config_eos: ConfigEOS): + _configure_hourly_grid(config_eos) + config_eos.optimization.genetic.individuals = 100 + opt = GeneticOptimization(fixed_seed=42) + opt.optimize_ev = False + opt.setup_deap_environment({"home_appliance": 0}, start_hour=0) + start_solution = [5.0] * opt.control_slots + captured: dict[str, object] = {} + + def warm_neighbors(_solution, count): + captured["warm_count"] = count + return [[6] * opt.control_slots for _ in range(count)] + + def educated(count): + captured["educated_count"] = count + return [[7] * opt.control_slots for _ in range(count)] + + def fake_evolution(population, **kwargs): + captured["population"] = list(population) + captured["mu"] = kwargs["mu"] + captured["lambda"] = kwargs["lambda_"] + for individual in population: + individual.fitness.values = (float(sum(individual)),) + individual.extra_data = (0.0, 0.0, 0.0) + kwargs["halloffame"].update(population) + return population, SimpleNamespace(select=lambda _name: []) + + with ( + patch.object(opt, "_mutated_warm_start_neighbors", side_effect=warm_neighbors), + patch.object(opt, "_educated_guess_individuals", side_effect=educated), + patch.object( + opt.toolbox, + "population", + side_effect=lambda n: [creator.Individual([9] * opt.control_slots) for _ in range(n)], + ), + patch.object(opt, "_evolve_population_adaptive", side_effect=fake_evolution), + ): + opt.optimize(start_solution=start_solution, ngen=1) + + population = captured["population"] + assert isinstance(population, list) + first_genes = [individual[0] for individual in population] + assert len(population) == 100 + assert first_genes.count(5) == 10 + assert first_genes.count(6) == 20 + assert first_genes.count(7) == 40 + assert first_genes.count(9) == 30 + assert captured["warm_count"] == 20 + assert captured["educated_count"] == 40 + assert captured["mu"] == 100 + assert captured["lambda"] == 100 + + +def test_adaptive_evolution_soft_restarts_collapsed_population(config_eos: ConfigEOS): + _configure_hourly_grid(config_eos) + opt = GeneticOptimization(fixed_seed=42) + opt.optimize_ev = False + opt.setup_deap_environment({"home_appliance": 0}, start_hour=0) + opt.toolbox.register("evaluate", lambda individual: (float(sum(individual)),)) + population = [creator.Individual([0] * opt.control_slots) for _ in range(20)] + stats = tools.Statistics(lambda individual: individual.fitness.values) + stats.register("min", np.min) + stats.register("avg", np.mean) + stats.register("max", np.max) + halloffame = tools.HallOfFame(1) + + fresh = [creator.Individual([value] + [0] * (opt.control_slots - 1)) for value in range(1, 20)] + with patch.object(opt, "_fresh_population", return_value=fresh) as create_fresh: + evolved, log = opt._evolve_population_adaptive( + population, + mu=20, + lambda_=20, + ngen=1, + stats=stats, + halloffame=halloffame, + ) + + create_fresh.assert_called_once_with(19, educated_fraction=0.40) + assert log.select("restart") == [0, 1] + assert log.select("immigrants") == [0, 19] + assert opt._adaptive_evolution_metrics["soft_restarts"] == 1 + assert opt._population_diversity(evolved) == pytest.approx(1.0) + assert halloffame[0].fitness.values == (0.0,) + + +def test_local_search_moves_weak_export_to_later_expensive_import(config_eos: ConfigEOS): + _configure_hourly_grid(config_eos) + opt = GeneticOptimization(fixed_seed=42) + opt.optimize_ev = False + opt.optimize_dc_charge = True + opt.optimize_battery_grid_export = True + opt.bat_possible_charge_values = [1.0] + opt.setup_deap_environment({"home_appliance": 0}, start_hour=0) + + slots = opt.control_slots + export_state = 5 + self_consumption_state = 6 + discharge_state = 1 + source = 10 + targets = list(range(20, 32)) + base = [self_consumption_state] * slots + base[source] = export_state + for slot in targets: + base[slot] = 0 + + opt.simulation.elect_price_hourly = np.full(slots, 0.10) + opt.simulation.elect_price_hourly[targets] = 0.30 + opt.simulation.elect_revenue_per_hour_arr = np.full(slots, 0.05) + opt.simulation.elect_revenue_per_hour_arr[source] = 0.20 + opt.simulation.pv_prediction_wh = np.zeros(slots) + opt.simulation.load_energy_array = np.full(slots, 100.0) + + def evaluate(individual): + export_value = -0.20 if individual[source] == export_state else 0.0 + avoided_import = -0.05 * sum(individual[slot] == discharge_state for slot in targets) + return (export_value + avoided_import,) + + opt.toolbox.register("evaluate", evaluate) + incumbent = creator.Individual(base) + incumbent.fitness.values = evaluate(incumbent) + + best, evaluations, improvements, initial, final = opt._locally_improve_grid_export( + incumbent, + max_evaluations=96, + ) + + assert evaluations > 0 + assert improvements == 1 + assert final < initial + assert best[source] == self_consumption_state + assert sum(best[slot] == discharge_state for slot in targets) >= 6 + + +def test_educated_guesses_encode_high_price_direct_marketing(config_eos: ConfigEOS): + _configure_hourly_grid(config_eos) + opt = GeneticOptimization(fixed_seed=42) + opt.optimize_ev = False + opt.optimize_dc_charge = True + opt.optimize_battery_grid_export = True + opt.bat_possible_charge_values = [1.0] + opt.setup_deap_environment({"home_appliance": 0}, start_hour=0) + + slots = opt.control_slots + opt.simulation.elect_price_hourly = np.linspace(0.0001, 0.0004, slots) + opt.simulation.elect_revenue_per_hour_arr = np.linspace(0.00001, 0.0003, slots) + opt.simulation.pv_prediction_wh = np.full(slots, 1000.0) + opt.simulation.load_energy_array = np.full(slots, 500.0) + + guesses = opt._educated_guess_individuals() + + dc_allowed_state = 4 + export_state = 5 + self_consumption_state = 6 + assert len(guesses) == opt.EDUCATED_GUESS_TARGET + assert all(len(guess) == slots for guess in guesses) + assert any(dc_allowed_state in guess or self_consumption_state in guess for guess in guesses) + assert any(self_consumption_state in guess for guess in guesses) + assert any(guess[-1] == export_state for guess in guesses) + + +def test_flat_feed_in_tariff_does_not_seed_direct_marketing(config_eos: ConfigEOS): + _configure_hourly_grid(config_eos) + opt = GeneticOptimization(fixed_seed=42) + opt.optimize_ev = False + opt.optimize_dc_charge = True + opt.optimize_battery_grid_export = True + opt.bat_possible_charge_values = [1.0] + opt.setup_deap_environment({"home_appliance": 0}, start_hour=0) + + slots = opt.control_slots + opt.simulation.elect_price_hourly = np.linspace(0.0001, 0.0004, slots) + opt.simulation.elect_revenue_per_hour_arr = np.full(slots, 0.00005) + opt.simulation.pv_prediction_wh = np.full(slots, 1000.0) + opt.simulation.load_energy_array = np.full(slots, 500.0) + + guesses = opt._educated_guess_individuals() + + export_state = 5 + assert all(export_state not in guess for guess in guesses) + + +def _rated(genome: list[int], fitness: float, *, protection: int = 0): + """Build an evaluated individual, optionally a protected immigrant.""" + individual = creator.Individual(genome) + individual.fitness.values = (fitness,) + if protection: + individual.immigrant_protection = protection + return individual + + +def test_diversity_boost_threshold_stays_below_selection_floor(): + # The selection guarantees SELECTION_DIVERSITY_FLOOR unique genomes, so a + # boost threshold at or above the floor would fire in every converged + # generation and turn the boost into the normal operating state. + assert ( + GeneticOptimization.DIVERSITY_BOOST_THRESHOLD + < GeneticOptimization.SELECTION_DIVERSITY_FLOOR + ) + + +def _immigrant_selection_pool(opt: GeneticOptimization, protection: int): + """Converged incumbents plus fresh immigrants that the tournament dislikes. + + The incumbents already carry more unique genomes than + ``SELECTION_DIVERSITY_FLOOR`` demands, so the duplicate repair has no reason + to reach for an immigrant and only the protection can seat one. + """ + slots = opt.control_slots + incumbents = [ + _rated([1, index] + [0] * (slots - 2), -5.73 + index * 1e-4) for index in range(100) + ] + offspring = [ + _rated([2, index] + [0] * (slots - 2), -5.72 + index * 1e-4) for index in range(88) + ] + offspring.extend( + _rated([3, index, index] + [0] * (slots - 3), 3.0 + index, protection=protection) + for index in range(12) + ) + return incumbents, offspring + + +def test_protected_immigrants_survive_the_selection(config_eos: ConfigEOS): + _configure_hourly_grid(config_eos) + opt = GeneticOptimization(fixed_seed=42) + opt.optimize_ev = False + opt.setup_deap_environment({"home_appliance": 0}, start_hour=0) + + seated = {} + for protection in (0, opt.IMMIGRANT_PROTECTION_GENERATIONS): + incumbents, offspring = _immigrant_selection_pool(opt, protection) + selected = opt._select_diverse(incumbents + offspring, 100) + seated[protection] = sum(1 for candidate in selected if candidate[0] == 3) + # The incumbent is never evicted to make room for an immigrant. + assert min(candidate.fitness.values[0] for candidate in selected) == pytest.approx(-5.73) + + # Without protection the tournament removes every immigrant in the + # generation it is born, so its genes never get to recombine. + assert seated[0] == 0 + assert seated[opt.IMMIGRANT_PROTECTION_GENERATIONS] == 12 + + +def test_immigrant_protection_expires_after_its_generations(config_eos: ConfigEOS): + _configure_hourly_grid(config_eos) + opt = GeneticOptimization(fixed_seed=42) + opt.optimize_ev = False + opt.setup_deap_environment({"home_appliance": 0}, start_hour=0) + + immigrants = [_rated([3, 0, 0], 3.0, protection=opt.IMMIGRANT_PROTECTION_GENERATIONS)] + for _ in range(opt.IMMIGRANT_PROTECTION_GENERATIONS): + assert immigrants[0].immigrant_protection > 0 + opt._age_immigrant_protection(immigrants) + assert immigrants[0].immigrant_protection == 0 + + # Aging is idempotent once the protection is spent. + opt._age_immigrant_protection(immigrants) + assert immigrants[0].immigrant_protection == 0 + + +def test_offspring_do_not_inherit_immigrant_protection(config_eos: ConfigEOS): + _configure_hourly_grid(config_eos) + opt = GeneticOptimization(fixed_seed=42) + opt.optimize_ev = False + opt.setup_deap_environment({"home_appliance": 0}, start_hour=0) + opt.toolbox.register("evaluate", lambda individual: (float(sum(individual)),)) + + parents = [ + _rated([0] * opt.control_slots, 0.0, protection=opt.IMMIGRANT_PROTECTION_GENERATIONS) + for _ in range(4) + ] + offspring = opt._make_offspring(parents, 8, mutation_probability=1.0) + assert all(getattr(child, "immigrant_protection", 0) == 0 for child in offspring) diff --git a/tests/test_genetic_warm_start_alignment.py b/tests/test_genetic_warm_start_alignment.py new file mode 100644 index 00000000..4b32e1ce --- /dev/null +++ b/tests/test_genetic_warm_start_alignment.py @@ -0,0 +1,146 @@ +"""A warm start from an earlier run is aligned with the slot this run starts in. + +Genomes are run-relative: gene 0 controls the slot the run starts in. Reusing +the previous solution unchanged after a slot boundary describes every decision +one slot too late, and a search that keeps the seed postpones a planned action +by one slot per run. +""" + +from types import SimpleNamespace + +import pytest + +from akkudoktoreos.config.config import ConfigEOS +from akkudoktoreos.core.coreabc import get_ems +from akkudoktoreos.optimization.genetic.genetic import GeneticOptimization +from akkudoktoreos.optimization.genetic.geneticparams import ( + GeneticEnergyManagementParameters, + GeneticOptimizationParameters, +) +from akkudoktoreos.utils.datetimeutil import DateTime, compare_datetimes, to_datetime + + +def _optimizer( + config_eos: ConfigEOS, + *, + interval: int, + optimize_ev: bool = False, + n_appliance_genes: int = 0, +) -> tuple[GeneticOptimization, DateTime]: + config_eos.merge_settings_from_dict( + { + "prediction": {"hours": 48}, + "optimization": { + "genetic": {"tail_horizon_hours": 0, "horizon_hours": 24, "interval_sec": interval} + }, + } + ) + slot0 = get_ems(init=True).set_start_datetime(to_datetime().set(hour=8, minute=0, second=0)) + opt = GeneticOptimization(fixed_seed=42) + opt.optimize_ev = optimize_ev + opt.appliance_layout = SimpleNamespace(n_genes=n_appliance_genes, genes=[]) # type: ignore[assignment] + opt._slot0_datetime = slot0 + return opt, slot0 + + +def _genes(start: int, stop: int) -> list[float]: + return [float(value) for value in range(start, stop)] + + +def test_quarter_hour_warm_start_moves_one_slot_forward(config_eos: ConfigEOS): + # 07:45 run: export genes (32) at 08:00 and 08:15. Unshifted, the 08:00 run + # would read them as 08:15 and 08:30. + opt, slot0 = _optimizer(config_eos, interval=900) + previous = [19.0, 32.0, 32.0, 14.0] + [36.0] * (opt.control_slots - 4) + + aligned = opt._start_solution_for_run_start(previous, slot0.subtract(minutes=15)) + + assert aligned == [32, 32, 14] + [36] * (opt.control_slots - 3) + + +def test_elapsed_slots_are_dropped_per_block_and_appliance_genes_kept(config_eos: ConfigEOS): + opt, slot0 = _optimizer(config_eos, interval=3600, optimize_ev=True, n_appliance_genes=1) + slots = opt.control_slots + battery = _genes(0, slots) + ev = _genes(100, 100 + slots) + appliance = [3.0] + + aligned = opt._start_solution_for_run_start(battery + ev + appliance, slot0.subtract(hours=3)) + + assert aligned == ( + _genes(3, slots) + + [slots - 1.0] * 3 + + _genes(103, 100 + slots) + + [100 + slots - 1.0] * 3 + + appliance + ) + + +def test_same_slot_or_unknown_start_keeps_warm_start(config_eos: ConfigEOS): + opt, slot0 = _optimizer(config_eos, interval=900) + previous = _genes(0, opt.control_slots) + + assert opt._start_solution_for_run_start(previous, slot0) == previous + assert opt._start_solution_for_run_start(previous, None) == previous + assert opt._start_solution_for_run_start(None, slot0) is None + + +@pytest.mark.parametrize( + "offset_minutes", + [ + pytest.param(-24 * 60, id="all-control-slots-elapsed"), + pytest.param(15, id="starts-after-this-run"), + ], +) +def test_unusable_warm_start_is_dropped(config_eos: ConfigEOS, offset_minutes: int): + opt, slot0 = _optimizer(config_eos, interval=900) + previous = _genes(0, opt.control_slots) + + assert opt._start_solution_for_run_start(previous, slot0.add(minutes=offset_minutes)) is None + + +def test_start_datetime_falls_back_to_last_solution_of_this_server( + config_eos: ConfigEOS, monkeypatch: pytest.MonkeyPatch +): + opt, slot0 = _optimizer(config_eos, interval=900) + last_start = slot0.subtract(minutes=15) + monkeypatch.setattr( + type(get_ems()), + "_genetic_solution", + SimpleNamespace(start_solution=[19.0, 32.0, 14.0], start_solution_datetime=last_start), + ) + + same = SimpleNamespace(start_solution=[19, 32, 14], start_solution_datetime=None) + other = SimpleNamespace(start_solution=[19, 32, 15], start_solution_datetime=None) + explicit_start = slot0.subtract(minutes=30) + explicit = SimpleNamespace(start_solution=[19, 32, 14], start_solution_datetime=explicit_start) + + assert opt._resolve_start_solution_datetime(same) == last_start # type: ignore[arg-type] + assert opt._resolve_start_solution_datetime(other) is None # type: ignore[arg-type] + assert opt._resolve_start_solution_datetime(explicit) == explicit_start # type: ignore[arg-type] + + +def test_parameters_accept_iso_start_solution_datetime(config_eos: ConfigEOS): + parameters = GeneticOptimizationParameters.model_validate( + dict( + ems=GeneticEnergyManagementParameters.model_validate( + dict( + pv_prognose_wh=[0.0, 0.0], + strompreis_euro_pro_wh=[0.0, 0.0], + einspeiseverguetung_euro_pro_wh=0.0, + preis_euro_pro_wh_akku=0.0, + gesamtlast=[0.0, 0.0], + ) + ), + pv_akku=None, + inverter=None, + eauto=None, + start_solution=[1.0, 2.0], + start_solution_datetime="2026-09-14T07:45:00+02:00", + ) + ) + + assert parameters.start_solution_datetime is not None + assert compare_datetimes( + parameters.start_solution_datetime, to_datetime("2026-09-14T07:45:00+02:00") + ).equal diff --git a/tests/test_geneticoptimize.py b/tests/test_geneticoptimize.py index b9a72b7d..9077be12 100644 --- a/tests/test_geneticoptimize.py +++ b/tests/test_geneticoptimize.py @@ -2,11 +2,9 @@ import json from io import BytesIO from pathlib import Path from typing import Any -from unittest.mock import patch import numpy as np import pytest -from pydantic import ValidationError from pypdf import PdfReader from akkudoktoreos.config.config import ConfigEOS @@ -22,7 +20,7 @@ from akkudoktoreos.optimization.genetic.geneticvisualize import ( ) from akkudoktoreos.utils.datetimeutil import to_datetime -ems_eos = get_ems(init=True) # init once +ems_eos = get_ems(init=True) # init once DIR_TESTDATA = Path(__file__).parent / "testdata" / "genetic" @@ -40,6 +38,7 @@ def compare_dict(actual: dict[str, Any], expected: dict[str, Any]): else: assert actual[key] == pytest.approx(value) + @pytest.mark.asyncio @pytest.mark.parametrize( "fn_in, fn_out, ngen, break_even", @@ -67,30 +66,30 @@ async def test_optimize( # Assure configuration holds the correct values config_eos.merge_settings_from_dict( { - "prediction": { - "hours": 48 - }, + "prediction": {"hours": 48}, "optimization": { "algorithm": "GENETIC", "genetic": { - "horizon_hours": 48, + "horizon_hours": 38, + "tail_horizon_hours": 0, + "terminal_value_mode": "FIXED", "individuals": 300, "generations": 10, "penalties": { "ev_soc_miss": 10, "ac_charge_break_even": break_even, - } - } + }, + }, }, "devices": { "max_electric_vehicles": 1, - "electric_vehicles": { "ev1": - { + "electric_vehicles": { + "ev1": { "charge_rates": [0.0, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0], } }, - } - } + }, + } ) # Load input and output data @@ -99,7 +98,9 @@ async def test_optimize( input_data = GeneticOptimizationParameters(**json.load(f_in)) # Fake energy management run start datetime - ems_eos.set_start_datetime(to_datetime().set(hour=fixed_start_hour)) + ems_eos.set_start_datetime( + to_datetime("2026-09-16T10:00:00+02:00", in_timezone="Europe/Berlin") + ) # Throw away any cached results of the last energy management run. CacheEnergyManagementStore().clear() @@ -115,31 +116,9 @@ async def test_optimize( parameters=input_data, start_hour=fixed_start_hour, ngen=ngen ) - # Write test output to file, so we can take it as new data on intended change - TESTDATA_FILE = DIR_TESTDATA / f"new_{fn_out}" - with TESTDATA_FILE.open("w", encoding="utf-8", newline="\n") as f_out: - f_out.write(genetic_solution.model_dump_json(indent=4, exclude_unset=True)) - - solution_file = DIR_TESTDATA / fn_out - # In case a new test case is added, we don't want to fail here, so the new output is written - # to disk before - try: - with solution_file.open("r") as f_out: - expected_data = json.load(f_out) - expected_result = GeneticSolution(**expected_data) - except ValidationError: - # Expected genetic solution data does not fit to GeneticSolution data schema - # Possibly the GeneticSolution class changed. - pytest.fail( - f"ValidationError: Can not load expected solution from {solution_file}\n" - f"cp {TESTDATA_FILE} {solution_file}\n" - ) - except FileNotFoundError: - # Should not happen - pytest.fail( - f"FileNotFoundError: Can not load expected solution from {solution_file}\n" - f"cp {TESTDATA_FILE} {solution_file}\n" - ) + # Historical payloads still deserialize with deprecated English/German aliases. + with (DIR_TESTDATA / fn_out).open("r") as expected_file: + expected_result = GeneticSolution.model_validate(json.load(expected_file)) # Keep the output contract, but do not demand an identical stochastic # schedule or monetary golden from the previous direct-consumption model. @@ -148,10 +127,8 @@ async def test_optimize( expected_slots = len(input_data.ems.pv_forecast_wh) - fixed_start_hour assert len(result.grid_consumption_wh_per_hour) == expected_slots assert len(result.grid_feed_in_wh_per_hour) == expected_slots - prices = np.asarray(genetic_solution.parameters.ems.electricity_price_per_wh)[fixed_start_hour:] - tariffs = np.asarray(genetic_solution.parameters.ems.feed_in_tariff_per_wh) - if tariffs.ndim > 0: - tariffs = tariffs[fixed_start_hour:] + prices = np.asarray(genetic_solution.parameters.ems.electricity_price_per_wh) + tariffs = np.asarray(genetic_solution.parameters.ems.feed_in_tariff_per_wh)[:expected_slots] expected_costs = np.asarray(result.grid_consumption_wh_per_hour) * prices expected_revenues = np.asarray(result.grid_feed_in_wh_per_hour) * tariffs np.testing.assert_allclose(result.costs_per_hour, expected_costs) @@ -167,11 +144,25 @@ async def test_optimize( # Check the correct generic optimization solution is created optimization_solution = await genetic_solution.optimization_solution() - # @TODO + dataframe = optimization_solution.solution.to_dataframe() + assert len(dataframe) == expected_slots + assert optimization_solution.valid_from == genetic_solution.start_solution_datetime + assert optimization_solution.valid_until == ems_eos.start_datetime.add(hours=expected_slots) + assert genetic_solution.controls_start_at_now + assert len(genetic_solution.ac_charge) == expected_slots + assert len(genetic_solution.dc_charge) == expected_slots + assert len(genetic_solution.discharge_allowed) == expected_slots # Check the correct generic energy management plan is created plan = genetic_solution.energy_management_plan() - # @TODO + assert plan.valid_from == optimization_solution.valid_from + assert plan.valid_until is None + assert optimization_solution.valid_from is not None + assert optimization_solution.valid_until is not None + assert all( + optimization_solution.valid_from <= item.execution_time < optimization_solution.valid_until + for item in plan.instructions + ) # Check visualization works pdf = genetic_prepare_visualize( @@ -180,8 +171,4 @@ async def test_optimize( assert pdf.startswith(b"%PDF-") reader = PdfReader(BytesIO(pdf)) - assert len(reader.pages) == 6 - - - # Everything passed, remove generated files - TESTDATA_FILE.unlink() + assert len(reader.pages) >= 6 diff --git a/tests/test_geneticparams_feedin.py b/tests/test_geneticparams_feedin.py index 6dabe105..78218fcf 100644 --- a/tests/test_geneticparams_feedin.py +++ b/tests/test_geneticparams_feedin.py @@ -12,6 +12,7 @@ import pandas as pd import pytest from akkudoktoreos.devices.genetic.inverter import Inverter, InverterParameters +from akkudoktoreos.optimization.genetic.configrequest import ConfigOptimizationRequest from akkudoktoreos.optimization.genetic.genetic import GeneticSimulation from akkudoktoreos.optimization.genetic.geneticparams import ( GeneticOptimizationParameters, @@ -35,7 +36,8 @@ def prepare_tariffs(config_eos): "devices": { "max_batteries": 0, "max_electric_vehicles": 0, - "max_inverters": 0, + "max_inverters": 1, + "inverters": {"inverter1": {"max_power_w": 10000}}, "max_home_appliances": 0, }, } @@ -49,20 +51,27 @@ def prepare_tariffs(config_eos): "feed_in_tariff_wh": revenues, } - async def read_array(key, **kwargs): + async def read_series(key, **kwargs): if key == "feed_in_tariff_wh" and tariff_reader is not None: return await tariff_reader(key=key, **kwargs) value = arrays[key] if isinstance(value, Exception): raise value - return np.asarray(value) + return pd.Series( + value, index=pd.date_range("2026-08-01T00:00:00Z", periods=len(value), freq="h") + ) - prediction = Mock(update_data=AsyncMock(), key_to_array=AsyncMock(side_effect=read_array)) - ems = Mock(start_datetime=to_datetime("2026-08-01T00:00:00+00:00")) + prediction = Mock( + update_data=AsyncMock(), key_to_raw_series=AsyncMock(side_effect=read_series) + ) + ems = Mock( + start_datetime=to_datetime("2026-08-01T00:00:00+00:00", in_timezone="UTC"), + observation_datetime=to_datetime("2026-08-01T00:00:00+00:00", in_timezone="UTC"), + ) ems.genetic_solution.return_value = None with ( - patch("akkudoktoreos.optimization.genetic.geneticparams.get_ems", return_value=ems), - patch.object(GeneticOptimizationParameters, "prediction", prediction), + patch("akkudoktoreos.optimization.genetic.configrequest.get_ems", return_value=ems), + patch.object(ConfigOptimizationRequest, "prediction", prediction), ): parameters = await GeneticOptimizationParameters.prepare() return parameters, prices, prediction @@ -100,7 +109,7 @@ async def test_provider_revenues_survive_preparation_and_simulation( # No battery or household load is needed to expose tariff substitution/unit bugs. inverter = Inverter(InverterParameters(device_id="inverter1", max_power_wh=10000)) inverter.self_consumption_predictor = Mock() - inverter.self_consumption_predictor.calculate_self_consumption.return_value = 1.0 + inverter.self_consumption_predictor.calculate_expected_direct_consumption.return_value = 0.0 simulation = GeneticSimulation() simulation.prepare( parameters.ems, optimization_hours=24, prediction_hours=24, inverter=inverter @@ -121,7 +130,6 @@ async def test_provider_revenues_survive_preparation_and_simulation( RuntimeError("import unavailable"), [], [0.00007] * 23, - [0.00007] * 25, [np.nan] * 24, [0.00007] * 23 + [np.nan], [np.inf] * 24, @@ -152,14 +160,14 @@ def test_prediction_record_prices_are_already_per_wh(): async def test_timestamped_import_records_are_read_in_order(prepare_tariffs, values): provider = FeedInTariffImport() provider._db_reset_state() - start = to_datetime("2026-08-01T00:00:00+00:00").set(hour=0) + start = to_datetime("2026-08-01T00:00:00+00:00", in_timezone="UTC").set(hour=0) try: await provider.key_from_series( "feed_in_tariff_wh", pd.Series(values, index=pd.date_range(start=start, periods=24, freq="h")), ) parameters, _, _ = await prepare_tariffs( - "FeedInTariffImport", [], tariff_reader=provider.key_to_array + "FeedInTariffImport", [], tariff_reader=provider.key_to_raw_series ) if values[0] is None: assert parameters is None diff --git a/tests/test_geneticvisualize_intervals.py b/tests/test_geneticvisualize_intervals.py new file mode 100644 index 00000000..fc7ecc79 --- /dev/null +++ b/tests/test_geneticvisualize_intervals.py @@ -0,0 +1,217 @@ +"""Semantic report tests with synthetic data; no forecast provider or live EMS run.""" + +from io import BytesIO +from pathlib import Path +from types import SimpleNamespace + +import matplotlib.dates as mdates +import matplotlib.pyplot as plt +import numpy as np +import pendulum +import pytest +from pypdf import PdfReader + +from akkudoktoreos.optimization.genetic import geneticvisualize as visualize +from akkudoktoreos.optimization.genetic.geneticsolution import GeneticSolution +from akkudoktoreos.optimization.genetic.terminalvalue import ( + TailDiagnostics, + TailPlanSlot, + TerminalValueCurve, + TerminalValueResult, +) + + +@pytest.fixture +def report_solution(): + start = pendulum.datetime(2026, 10, 25, 1, 45, tz="Europe/Berlin") + energy = [100.0, 150.0, 200.0, 250.0] + result = SimpleNamespace( + load_wh_per_hour=energy, + home_appliance_wh_per_hour=energy, + grid_feed_in_wh_per_hour=energy, + grid_consumption_wh_per_hour=energy, + losses_per_hour=[1.0] * 4, + battery_soc_per_hour=[20.0, 25.0, 30.0, 35.0], + ev_soc_per_hour=[0.0] * 4, + costs_per_hour=[0.03] * 4, + revenue_per_hour=[0.01] * 4, + total_costs=0.12, + total_revenue=0.04, + total_balance=0.08, + home_appliance_energy_wh={"water-heater": energy}, + ) + return SimpleNamespace( + parameters=SimpleNamespace( + ems=SimpleNamespace( + total_load=energy, + pv_forecast_wh=energy, + feed_in_tariff_per_wh=[0.00007] * 4, + electricity_price_per_wh=[0.0003] * 4, + ), + temperature_forecast=[12.0] * 4, + ), + interval_seconds=900, + start_solution_datetime=start, + controls_start_at_now=True, + start_hour=1, + ac_charge=[0.0, 0.25, 0.5, 1.0], + dc_charge=[1.0] * 4, + discharge_allowed=[0, 0, 1, 1], + battery_grid_export_allowed=[0, 0, 0, 1], + battery_grid_export_factor=[0.0, 0.0, 0.0, 0.5], + result=result, + extra_data=None, + fitness_history=None, + fixed_seed=None, + appliance_starts={"water-heater": [start.add(minutes=15)]}, + appliance_deadline_missed={"water-heater": True}, + terminal_value=TerminalValueResult( + mode="TAIL", control_horizon_hours=1, requested_tail_hours=2, + effective_tail_hours=0.25, tail_end_hour=1.25, + battery_energy_wh=500, credited_euro=0.13, + tail_operating_euro=0.1, continuation_value_euro=0.03, + continuation_mode="AUTO", reason="forecast gap limits tail", + curve=TerminalValueCurve(energy_wh=[0, 1000], value_euro=[0, 0.26]), + tail_diagnostics=TailDiagnostics(slots=1, slot_hours=0.25), + tail_plan=[TailPlanSlot( + slot=0, hour_from_start=1, action="discharge", + soc_start_percentage=40, soc_end_percentage=35, + pv_wh=0, load_wh=100, grid_import_wh=0, grid_export_wh=0, + battery_charge_wh=0, battery_discharge_wh=100, + import_price_euro_per_kwh=0.3, feed_in_tariff_euro_per_kwh=0.07, + slot_value_euro=0.03, remaining_value_euro=0.1, + ac_charge_factor=0, dc_charge_allowed=1, discharge_allowed=1, + battery_grid_export_factor=0, + )], + ), + ) + + +@pytest.mark.parametrize("interval_seconds", [900, 3600]) +def test_report_preserves_run_timing_energy_and_export_controls( + config_eos, monkeypatch, report_solution, interval_seconds +): + report_solution.interval_seconds = interval_seconds + captured = [] + original = visualize.GeneticVisualizationReport.create_line_chart_date + + def capture(self, start_date, y_list, **kwargs): + captured.append((start_date, y_list, kwargs, self.interval_seconds)) + return original(self, start_date, y_list, **kwargs) + + monkeypatch.setattr(visualize.GeneticVisualizationReport, "create_line_chart_date", capture) + monkeypatch.setattr(visualize, "get_ems", lambda: pytest.fail("Snapshot must own report timing")) + pdf = visualize.genetic_prepare_visualize(report_solution) + assert pdf.startswith(b"%PDF-") + assert all(row[0] == report_solution.start_solution_datetime for row in captured) + assert all(row[3] == interval_seconds for row in captured) + load = next(row for row in captured if row[2].get("title") == "Load Profile") + assert load[1] == [[100, 150, 200, 250]] # No repeated wall-clock trimming or Wh scaling. + controls = next(row for row in captured if row[2].get("title") == "Executable Battery Controls") + assert controls[1][-1] == [0, 0, 0, 0.5] + assert len(controls[1]) == 5 + text = " ".join(page.extract_text() for page in PdfReader(BytesIO(pdf)).pages) + for expected in ( + "Energy Flow per Interval", "water-heater", "Deadline missed: True", + "forecast gap limits tail", "effective tail: 0.25 h", "0.13 EUR", + "not executable controls", "not realized revenue", "Residual Battery Value Curve", + ): + assert expected in text + assert "Energy Flow per Hour" not in text + + +def test_short_quarterhour_chart_has_elapsed_fractional_hours(config_eos): + report = visualize.GeneticVisualizationReport(interval_seconds=900) + start = pendulum.datetime(2026, 10, 25, 2, 45, tz="Europe/Berlin", fold=0) + report.create_line_chart_date(start, [[100.0, 200.0, 300.0, 400.0]], ylabel="Wh") + fig, axis = plt.subplots() + try: + report.current_group[0]() + dates = mdates.num2date(axis.lines[0].get_xdata()) + assert np.diff([value.timestamp() for value in dates]).tolist() == [900] * 3 + assert np.asarray(axis.lines[0].get_ydata()).tolist() == [100, 200, 300, 400] + assert [label.get_text() for label in fig.axes[1].get_xticklabels()] == ["0", "0.25", "0.5", "0.75"] + finally: + plt.close(fig) + + +def test_empty_and_mismatched_date_series(config_eos): + report = visualize.GeneticVisualizationReport() + start = pendulum.now("UTC") + report.create_line_chart_date(start, [[]], ylabel="Wh") + assert report.current_group == [] + with pytest.raises(ValueError, match="same intervals"): + report.create_line_chart_date(start, [[1.0, 2.0], [1.0]], ylabel="Wh") + + +def test_existing_hourly_solution_renders_without_terminal_metadata(config_eos, monkeypatch): + payload = (Path(__file__).parent / "testdata/genetic/optimize_result_1.json").read_text() + solution = GeneticSolution.model_validate_json(payload) + assert not solution.controls_start_at_now + assert solution.terminal_value is None + start = pendulum.datetime(2026, 9, 16, solution.start_hour, tz="UTC") + monkeypatch.setattr(visualize, "get_ems", lambda: SimpleNamespace(start_datetime=start)) + pdf = visualize.genetic_prepare_visualize(solution) + assert len(PdfReader(BytesIO(pdf)).pages) >= 4 + + +def test_fixed_fallback_does_not_invent_tail_charts(config_eos, report_solution): + report_solution.terminal_value = TerminalValueResult( + mode="FIXED", reason="No complete tail forecast", credited_euro=0.05, + battery_energy_wh=500, requested_tail_hours=24, + ) + report_solution.result.home_appliance_energy_wh = {} + pdf = visualize.genetic_prepare_visualize(report_solution) + text = " ".join(page.extract_text() for page in PdfReader(BytesIO(pdf)).pages) + assert "No complete tail forecast" in text + assert "effective tail: 0 h" in text + assert "Deadline missed: True" in text # Report unscheduled consumers even with no energy series. + assert "Residual Battery Value Curve" not in text + assert "Tail Lookahead:" not in text + + +def test_tail_chart_starts_after_control_horizon(config_eos, report_solution): + report = visualize.GeneticVisualizationReport(interval_seconds=900) + start = report_solution.start_solution_datetime + visualize._add_solution_diagnostics(report, report_solution, start) + fig, axis = plt.subplots() + try: + report.groups[-1][0]() + expected = mdates.date2num(start.add(hours=1)) + assert np.asarray(axis.lines[0].get_xdata()).tolist() == [expected] + assert np.asarray(axis.lines[3].get_ydata()).tolist() == [100] + assert all(line.get_marker() == "o" for line in axis.lines) + left, right = axis.get_xlim() + assert left < expected < right + assert (right - left) * 86400 == pytest.approx(900) + assert "Not Executable" in axis.get_title() + assert report_solution.discharge_allowed == [0, 0, 1, 1] + finally: + plt.close(fig) + + +@pytest.mark.parametrize("tariff", [0.00007, [0.00007] * 6]) +def test_pdf_clips_forecast_tail_and_accepts_historical_diagnostics( + config_eos, monkeypatch, report_solution, tariff +): + report_solution.parameters.ems.total_load = [100.0] * 6 + report_solution.parameters.ems.pv_forecast_wh = [200.0] * 6 + report_solution.parameters.ems.feed_in_tariff_per_wh = tariff + report_solution.extra_data = { + "verluste": [1.0, 2.0, 3.0], "bilanz": [0.1, 0.2, 0.3], + "nebenbedingung": [0.0, 0.0, 0.0], + } + captured = [] + original = visualize.GeneticVisualizationReport.create_line_chart_date + + def capture(self, start_date, y_list, **kwargs): + captured.append((y_list, kwargs)) + return original(self, start_date, y_list, **kwargs) + + monkeypatch.setattr(visualize.GeneticVisualizationReport, "create_line_chart_date", capture) + pdf = visualize.genetic_prepare_visualize(report_solution) + assert pdf.startswith(b"%PDF-") + tariff_plot = next(series for series, kwargs in captured if kwargs.get("title") == "Remuneration") + assert np.asarray(tariff_plot[0]).tolist() == [0.00007] * 4 + assert all(len(series[0]) == 4 for series, _ in captured) + assert "verluste" in report_solution.extra_data # Rendering must not mutate the retained run. diff --git a/tests/test_optimization_compatibility.py b/tests/test_optimization_compatibility.py new file mode 100644 index 00000000..1a87fca0 --- /dev/null +++ b/tests/test_optimization_compatibility.py @@ -0,0 +1,115 @@ +"""Contracts shared by the configuration, tariff and optimizer PRs.""" + +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from akkudoktoreos.optimization.genetic0.genetic0params import ( + Genetic0EnergyManagementParameters, +) +from akkudoktoreos.optimization.genetic.geneticparams import ( + GeneticEnergyManagementParameters, +) +from akkudoktoreos.optimization.optimization import OptimizationAlgorithm + + +@pytest.mark.parametrize( + "model", [GeneticEnergyManagementParameters, Genetic0EnergyManagementParameters] +) +@pytest.mark.parametrize("tariff", [0.00008, [0.00008, -0.00002]]) +def test_energy_parameter_aliases_preserve_wh_prices(model, tariff): + raw = { + "pv_prognose_wh": [250.0, 0.0], + "gesamtlast": [125.0, 125.0], + "strompreis_euro_pro_wh": [0.0003, -0.0001], + "einspeiseverguetung_euro_pro_wh": tariff, + "preis_euro_pro_wh_akku": 0.00005, + } + params = model.model_validate(raw) + data = params.model_dump(mode="json") + pairs = { + "pv_prognose_wh": "pv_forecast_wh", + "gesamtlast": "total_load", + "strompreis_euro_pro_wh": "electricity_price_per_wh", + "einspeiseverguetung_euro_pro_wh": "feed_in_tariff_per_wh", + "preis_euro_pro_wh_akku": "price_per_wh_battery", + } + for deprecated, canonical in pairs.items(): + assert data[deprecated] == data[canonical] == raw[deprecated] + assert ( + model.model_validate({pairs[k]: v for k, v in raw.items()}).model_dump(mode="json") == data + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("algorithm", list(OptimizationAlgorithm)) +async def test_algorithm_result_endpoint_keeps_solution_types_separate(monkeypatch, algorithm): + from akkudoktoreos.server import eos + + genetic, genetic0 = object(), object() + fake = SimpleNamespace( + genetic_solution=Mock(return_value=genetic), + genetic0_solution=Mock(return_value=genetic0), + ) + monkeypatch.setattr(eos, "get_ems", lambda: fake) + monkeypatch.setattr( + eos, + "get_config", + lambda: SimpleNamespace(optimization=SimpleNamespace(algorithms=["GENETIC", "GENETIC0"])), + ) + result = await eos.fastapi_energy_management_optimization_solution_algorithm_get(algorithm) + if algorithm == OptimizationAlgorithm.GENETIC: + assert result is genetic + fake.genetic_solution.assert_called_once_with() + fake.genetic0_solution.assert_not_called() + else: + assert result is genetic0 + fake.genetic0_solution.assert_called_once_with() + fake.genetic_solution.assert_not_called() + + +@pytest.mark.parametrize("algorithm", ["genetic", "genetic0"]) +def test_device_maps_feed_algorithm_specific_converters(algorithm): + from akkudoktoreos.devices.devices import DevicesCommonSettings + + settings = DevicesCommonSettings.model_validate( + { + "batteries": {"storage": {"capacity_wh": 12000, "max_charge_power_w": 3200}}, + "electric_vehicles": {"car": {"capacity_wh": 60000, "max_charge_power_w": 7000}}, + "inverters": {"inverter": {"battery_id": "storage", "max_power_w": 5000}}, + "home_appliances": { + "washer": { + "consumption_wh": 1800, + "duration_h": 2, + "num_cycles": 2, + "min_cycle_gap_h": 1, + } + }, + } + ) + assert settings.batteries is not None + assert settings.electric_vehicles is not None + assert settings.inverters is not None + battery = getattr(settings.batteries["storage"], "to_" + algorithm + "_pv_bat_param")() + ev = getattr(settings.electric_vehicles["car"], "to_" + algorithm + "_ev_bat_param")() + inverter = getattr(settings.inverters["inverter"], "to_" + algorithm + "_param")() + appliance = getattr(settings.home_appliances["washer"], "to_" + algorithm + "_param")() + assert (battery.device_id, ev.device_id, inverter.device_id, appliance.device_id) == ( + "storage", + "car", + "inverter", + "washer", + ) + assert battery.capacity_wh == 12000 + assert battery.max_charge_power_w == 3200 + assert ev.capacity_wh == 60000 + assert ev.max_charge_power_w == 7000 + assert inverter.max_power_wh == 5000 + assert inverter.battery_id == "storage" + assert appliance.consumption_wh == 1800 + assert appliance.duration_h == 2 + assert settings.batteries["storage"].measurement_key_soc_factor == "storage-soc-factor" + if algorithm == "genetic": + assert appliance.num_cycles == 2 + assert appliance.min_cycle_gap_h == 1 diff --git a/tests/test_optimization_pdf_api.py b/tests/test_optimization_pdf_api.py new file mode 100644 index 00000000..de0562dd --- /dev/null +++ b/tests/test_optimization_pdf_api.py @@ -0,0 +1,78 @@ +"""The report endpoint selects retained results and renders outside the event loop.""" + +import threading +from io import BytesIO +from pathlib import Path +from types import SimpleNamespace + +import pytest +from fastapi.testclient import TestClient +from pypdf import PdfReader + +from akkudoktoreos.optimization.genetic.geneticsolution import GeneticSolution +from akkudoktoreos.server import eos +from akkudoktoreos.utils.datetimeutil import to_datetime + + +@pytest.fixture +def retained_result(): + path = Path(__file__).parent / "testdata/genetic/optimize_result_1.json" + solution = GeneticSolution.model_validate_json(path.read_text()) + solution.start_solution_datetime = to_datetime("2026-09-16T10:00:00+02:00") + return solution + + +def test_missing_algorithm_report_returns_404(config_eos, monkeypatch): + monkeypatch.setattr(eos, "get_ems", lambda: SimpleNamespace(genetic_solution=lambda: None)) + response = TestClient(eos.app).get("/v1/energy-management/optimization/solution/GENETIC/pdf") + assert response.status_code == 404 + assert response.headers["content-type"].startswith("application/problem+json") + + +def test_retained_algorithm_result_renders_real_pdf(config_eos, monkeypatch, retained_result): + solution = retained_result + monkeypatch.setattr(eos, "get_ems", lambda: SimpleNamespace(genetic_solution=lambda: solution)) + # A later run and changed configuration must not retime the retained report. + from akkudoktoreos.core.coreabc import get_ems + from akkudoktoreos.optimization.genetic import geneticvisualize + + get_ems().set_start_datetime(to_datetime("2027-01-01T00:00:00+01:00")) + config_eos.optimization.genetic.interval_sec = 900 + monkeypatch.setattr( + geneticvisualize, "get_ems", lambda: pytest.fail("Report must retain its own time grid") + ) + before = solution.model_dump_json() + response = TestClient(eos.app).get("/v1/energy-management/optimization/solution/GENETIC/pdf") + assert response.status_code == 200 + assert response.headers["content-type"] == "application/pdf" + assert "genetic" in response.headers["content-disposition"] + assert response.content.startswith(b"%PDF-") + assert len(PdfReader(BytesIO(response.content)).pages) >= 4 + assert solution.model_dump_json() == before + + +@pytest.mark.asyncio +async def test_render_is_offloaded_and_uses_an_owned_copy(config_eos, monkeypatch, retained_result): + solution = retained_result + caller_thread = threading.get_ident() + captured = [] + monkeypatch.setattr( + eos, + "get_ems", + lambda: SimpleNamespace( + genetic_solution=lambda: solution, genetic0_solution=lambda: solution + ), + ) + original_controls = list(solution.ac_charge) + + def render(*, solution): + captured.append((threading.get_ident(), solution)) + solution.ac_charge[0] = 0.123 + return b"%PDF-isolated-render" + + monkeypatch.setattr(eos, "genetic_prepare_visualize", render) + response = await eos.fastapi_energy_management_optimization_solution_genetic_pdf_get() + assert response.body == b"%PDF-isolated-render" + assert captured[0][0] != caller_thread + assert captured[0][1] is not solution + assert solution.ac_charge == original_controls diff --git a/tests/test_optimize_run_result.py b/tests/test_optimize_run_result.py index 00acba98..0e90a201 100644 --- a/tests/test_optimize_run_result.py +++ b/tests/test_optimize_run_result.py @@ -22,6 +22,7 @@ def offline_ems(monkeypatch): cls = ems_module.EnergyManagement for name in ( "_start_datetime", + "_observation_datetime", "_last_run_datetime", "_plan", "_optimization_solution", @@ -34,10 +35,11 @@ def offline_ems(monkeypatch): monkeypatch.setattr(ems_module, "CacheEnergyManagementStore", Mock()) return SimpleNamespace( config=SimpleNamespace( + general=SimpleNamespace(timezone="Europe/Berlin"), ems=SimpleNamespace(mode=EnergyManagementMode.OPTIMIZATION), optimization=SimpleNamespace( algorithm=OptimizationAlgorithm.GENETIC, - genetic=SimpleNamespace(generations=3, seed=17), + genetic=SimpleNamespace(generations=3, seed=17, interval_sec=3600, individuals=31), genetic0=SimpleNamespace(generations=5, seed=29), ), server=SimpleNamespace(verbose=False), @@ -48,6 +50,42 @@ def offline_ems(monkeypatch): ) +@pytest.mark.asyncio +@pytest.mark.parametrize("algorithm", list(OptimizationAlgorithm)) +@pytest.mark.parametrize("explicit_start", [False, True]) +async def test_implicit_genetic_start_uses_site_timezone( + offline_ems, monkeypatch, set_other_timezone, algorithm, explicit_start +): + set_other_timezone("UTC") + now = to_datetime("2026-09-16T22:47:23Z", in_timezone="UTC") + + def frozen_datetime(value=None, **kwargs): + return to_datetime(now if value is None else value, **kwargs) + + monkeypatch.setattr(ems_module, "to_datetime", frozen_datetime) + offline_ems.config.optimization.genetic.interval_sec = 900 + await ems_module.EnergyManagement.run( + offline_ems, + mode=EnergyManagementMode.PREDICTION, + algorithm=algorithm, + start_datetime=now if explicit_start else None, + ) + start = ems_module.EnergyManagement._start_datetime + observed = ems_module.EnergyManagement._observation_datetime + assert start is not None and observed is not None + assert observed.timestamp() == now.timestamp() + if algorithm == OptimizationAlgorithm.GENETIC: + assert start.minute == 45 + assert start.hour == (22 if explicit_start else 0) + assert start.day == (16 if explicit_start else 17) + assert start.timezone_name == ("UTC" if explicit_start else "Europe/Berlin") + else: + assert start.minute == 0 + assert start.hour == 22 + assert start.day == 16 + assert start.timezone_name == "UTC" + + @pytest.mark.asyncio @pytest.mark.parametrize("algorithm", list(OptimizationAlgorithm)) @pytest.mark.parametrize("selection", ["configured", "explicit"]) @@ -90,6 +128,8 @@ async def test_optimization_routes_only_selected_algorithm( kwargs[suffix + "_parameters"] = sentinel_parameters kwargs[suffix + "_generations"] = 7 kwargs[suffix + "_seed"] = 43 + if algorithm == OptimizationAlgorithm.GENETIC: + kwargs["genetic_individuals"] = 11 run_result = await ems_module.EnergyManagement.run(offline_ems, **kwargs) assert run_result is solution selected = constructors[selected_name] @@ -97,11 +137,14 @@ async def test_optimization_routes_only_selected_algorithm( selected.assert_called_once_with( verbose=False, fixed_seed=43 if supplied else expected_config.seed ) - selected.return_value.optimize_ems.assert_called_once_with( - start_hour=expected_hour, - parameters=sentinel_parameters, - ngen=7 if supplied else expected_config.generations, - ) + expected_arguments: dict[str, Any] = { + "start_hour": expected_hour, + "parameters": sentinel_parameters, + "ngen": 7 if supplied else expected_config.generations, + } + if algorithm == OptimizationAlgorithm.GENETIC: + expected_arguments["individuals"] = 11 if supplied else None + selected.return_value.optimize_ems.assert_called_once_with(**expected_arguments) other_name = "Genetic0" if selected_name == "Genetic" else "Genetic" constructors[other_name].assert_not_called() preparers[other_name].assert_not_awaited() diff --git a/tests/test_tailvalue_physics.py b/tests/test_tailvalue_physics.py new file mode 100644 index 00000000..1e2321aa --- /dev/null +++ b/tests/test_tailvalue_physics.py @@ -0,0 +1,199 @@ +"""Economic tail scenarios and hard control/forecast boundaries.""" + +from unittest.mock import patch + +import numpy as np +import pandas as pd +import pytest + +from akkudoktoreos.config.config import SettingsEOSDefaults +from akkudoktoreos.core.coreabc import get_ems +from akkudoktoreos.devices.genetic.battery import Battery, SolarPanelBatteryParameters +from akkudoktoreos.devices.genetic.inverter import Inverter, InverterParameters +from akkudoktoreos.optimization.genetic.forecast import bounded_forecast_array +from akkudoktoreos.optimization.genetic.genetic import GeneticOptimization +from akkudoktoreos.optimization.genetic.geneticparams import ( + GeneticOptimizationParameters, +) +from akkudoktoreos.optimization.genetic.tailvalue import build_tail_value_curve +from akkudoktoreos.optimization.genetic.terminalvalue import TerminalValueCurve +from akkudoktoreos.utils.datetimeutil import to_datetime, to_duration + + +def devices(power=1000, efficiency=1.0, lcos=0, ac_limit=None, export_power=5000): + bat = Battery( + SolarPanelBatteryParameters( + device_id="battery1", + capacity_wh=1000, + max_charge_power_w=power, + charging_efficiency=efficiency, + discharging_efficiency=efficiency, + initial_soc_percentage=50, + levelized_cost_of_storage_kwh=lcos, + charge_rates=[0, 0.5, 1], + ), + prediction_hours=1, + ) + inv = Inverter( + InverterParameters( + device_id="inverter1", + battery_id="battery1", + max_power_wh=export_power, + dc_to_ac_efficiency=1, + ac_to_dc_efficiency=1, + max_ac_charge_power_w=ac_limit, + ), + battery=bat, + ) + return bat, inv + + +def curve( + prices=(-0.1, 0.3), + tariffs=(0, 0.3), + direct=True, + continuation=None, + load=None, + pv=None, + **kwargs, +): + bat, inv = devices(**kwargs) + return build_tail_value_curve( + battery=bat, + inverter=inv, + prices_euro_per_wh=np.array(prices) / 1000, + feed_in_euro_per_wh=np.array(tariffs) / 1000, + load_wh=np.zeros(len(prices)) if load is None else np.array(load), + pv_wh=np.zeros(len(prices)) if pv is None else np.array(pv), + continuation=continuation or TerminalValueCurve(), + charge_rates=[0.5, 1], + export_rates=[1], + direct_marketing=direct, + ) + + +def test_headroom_has_value_and_empty_state_can_earn(): + c = curve() + assert c.value(0) == pytest.approx(0.4) + tail, continuation = c.component_values(0) + assert tail == pytest.approx(0.4) + assert continuation == pytest.approx(0.0) + assert c.value(0) == pytest.approx(tail + continuation) + assert c.value(500) > c.value(1000) + assert any(v < 0 for v in c.marginal_euro_per_kwh) + + +def test_chronology_changes_arbitrage(): + forward = curve() + reverse = curve(prices=(0.3, -0.1), tariffs=(0.3, 0)) + assert forward.value(0) > reverse.value(0) + + +def test_discharge_and_ac_power_limits(): + limited = curve(prices=(1,), tariffs=(1,), power=100) + assert limited.value(1000) == pytest.approx(0.1) + limited_ac = curve(ac_limit=100) + assert limited_ac.value(0) == pytest.approx(0.04) + limited_inverter = curve(prices=(1,), tariffs=(1,), export_power=50) + assert limited_inverter.value(1000) == pytest.approx(0.05) + + +def test_losses_and_lcos_reduce_arbitrage(): + ideal = curve(prices=(0.1, 0.3)) + lossy = curve(prices=(0.1, 0.3), efficiency=0.8) + assert 0 < lossy.value(0) < ideal.value(0) + assert curve(prices=(0.1, 0.3), lcos=0.25).value(0) == pytest.approx(0) + + +def test_no_battery_export_without_permission(): + assert curve(prices=(0.1, 0.3), direct=False).value(0) == pytest.approx(0) + + +def test_pv_surplus_can_be_stored_for_local_load(): + c = curve(prices=(0.2, 0.3), tariffs=(0, 0), pv=[1000, 0], load=[0, 1000], direct=False) + assert c.value(0) == pytest.approx(0) + # Without PV the same empty battery must buy energy to serve the load. + assert curve(prices=(0.2, 0.3), tariffs=(0, 0), load=[0, 1000], direct=False).value( + 0 + ) < c.value(0) + + +def test_continuation_survives_tail_end(): + continuation = TerminalValueCurve(energy_wh=[0, 1000], value_euro=[0, 0.2]) + c = curve(prices=(0.5,), tariffs=(0,), continuation=continuation) + assert c.value(1000) == pytest.approx(0.2) + tail, continuation_credit = c.component_values(1000) + assert tail == pytest.approx(0.0) + assert continuation_credit == pytest.approx(0.2) + + +def test_tail_diagnostic_plan_explains_the_selected_path(): + c = curve() + plan = c.diagnostic_plan(0, control_horizon_hours=24) + assert len(plan) == 2 + assert plan[0].hour_from_start == 24 + assert plan[0].action == "GRID_CHARGE" + assert plan[0].soc_end_percentage > plan[0].soc_start_percentage + assert plan[0].grid_import_wh > 0 + assert plan[1].action == "BATTERY_EXPORT" + assert plan[1].soc_end_percentage < plan[1].soc_start_percentage + assert plan[1].grid_export_wh > 0 + assert sum(slot.slot_value_euro for slot in plan) == pytest.approx(c.value(0)) + + +@pytest.mark.asyncio +async def test_provider_values_are_not_extrapolated(): + from types import SimpleNamespace + + start = to_datetime("2026-09-05T00:00:00Z") + series = pd.Series([1.0, 2.0], index=pd.date_range(start=start, periods=2, freq="h")) + from unittest.mock import AsyncMock + + provider = SimpleNamespace(key_to_raw_series=AsyncMock(return_value=series)) + result = await bounded_forecast_array( + provider, + key="price", + start_datetime=start, + end_datetime=start.add(hours=3), + interval=to_duration("15 minutes"), + ) + assert result[:8].tolist() == [1.0] * 4 + [2.0] * 4 + assert np.isnan(result[8:]).all() + + +def test_disabled_ac_conversion_cannot_earn_negative_price_revenue(): + bat, inv = devices() + inv.parameters.ac_to_dc_efficiency = 0 + c = build_tail_value_curve( + battery=bat, + inverter=inv, + prices_euro_per_wh=np.array([-0.001, 0.001]), + feed_in_euro_per_wh=np.array([0.0, 0.001]), + load_wh=np.zeros(2), + pv_wh=np.zeros(2), + continuation=TerminalValueCurve(), + charge_rates=[1], + export_rates=[1], + direct_marketing=True, + ) + assert c.value(0) == pytest.approx(0) + assert bat.soc_wh == 500 # Building the tail never mutates the real battery. + + +@pytest.mark.asyncio +async def test_missing_provider_key_stays_missing(): + from types import SimpleNamespace + + async def unavailable(*a, **kw): + raise KeyError("price unavailable") + + start = to_datetime("2026-09-05T00:00:00Z") + result = await bounded_forecast_array( + SimpleNamespace(key_to_raw_series=unavailable), + key="price", + start_datetime=start, + end_datetime=start.add(hours=2), + interval=to_duration("1 hour"), + ) + assert np.isnan(result).all() + assert len(result) == 2 diff --git a/tests/test_terminalvalue.py b/tests/test_terminalvalue.py new file mode 100644 index 00000000..acc2f002 --- /dev/null +++ b/tests/test_terminalvalue.py @@ -0,0 +1,129 @@ +"""Tests for the concave terminal value of the energy left in the battery.""" + +from typing import Any + +import numpy as np +import pytest + +from akkudoktoreos.optimization.genetic.terminalvalue import ( + build_terminal_value_curve, + trailing_window, +) + + +def _curve(**overrides): + """Two expensive slots, one cheap one, no PV, 10 kWh of usable battery.""" + params: dict[str, Any] = dict( + prices_euro_per_wh=np.array([0.0004, 0.0003, 0.0001]), + load_wh=np.array([1000.0, 1000.0, 1000.0]), + pv_wh=np.array([0.0, 0.0, 0.0]), + feed_in_euro_per_wh=np.array([0.00008, 0.00008, 0.00008]), + max_energy_wh=10000.0, + lcos_euro_per_kwh=0.0, + dc_to_ac_efficiency=1.0, + grid_export_allowed=False, + ) + params.update(overrides) + return build_terminal_value_curve(**params) + + +def test_marginal_value_follows_the_most_expensive_hours_first(): + """The first stored kWh replaces the most expensive slot, then the next.""" + curve = _curve() + + # 0.40, 0.30 and 0.10 EUR/kWh, in that order. + assert curve.marginal_euro_per_kwh == pytest.approx([0.4, 0.3, 0.1]) + assert curve.energy_wh == pytest.approx([0.0, 1000.0, 2000.0, 3000.0]) + assert curve.value_euro == pytest.approx([0.0, 0.4, 0.7, 0.8]) + + +def test_curve_is_concave_and_saturates(): + """Marginal values only decrease, and beyond the last breakpoint nothing is added.""" + curve = _curve() + marginals = curve.marginal_euro_per_kwh + + assert all(a >= b for a, b in zip(marginals, marginals[1:])) + # The residual load of the window is 3 kWh - more energy replaces nothing. + assert curve.value(3000.0) == pytest.approx(0.8) + assert curve.value(9000.0) == pytest.approx(0.8) + + +def test_value_interpolates_within_a_segment(): + """Half of the first slot is worth half of the first segment.""" + curve = _curve() + assert curve.value(500.0) == pytest.approx(0.2) + + +def test_pv_reduces_the_residual_load(): + """Only load that PV cannot cover can be replaced by stored energy.""" + curve = _curve(pv_wh=np.array([600.0, 1000.0, 0.0])) + + # Slot 0 keeps 400 Wh, slot 1 is fully covered by PV, slot 2 keeps 1000 Wh. + assert curve.energy_wh == pytest.approx([0.0, 400.0, 1400.0]) + assert curve.marginal_euro_per_kwh == pytest.approx([0.4, 0.1]) + + +def test_lcos_is_subtracted_from_the_marginal_value(): + """Storage cost is already charged on discharge and must not be credited twice.""" + curve = _curve(lcos_euro_per_kwh=0.05, dc_to_ac_efficiency=1.0) + assert curve.marginal_euro_per_kwh == pytest.approx([0.35, 0.25, 0.05]) + + +def test_negative_prices_do_not_create_value(): + """Storing energy for an hour that pays nothing is not worth anything.""" + curve = _curve(prices_euro_per_wh=np.array([0.0004, -0.0001, 0.0])) + assert curve.marginal_euro_per_kwh == pytest.approx([0.4]) + assert curve.value(5000.0) == pytest.approx(0.4) + + +def test_export_tail_only_with_direct_marketing(): + """Surplus beyond the residual load is worth an export - if export is allowed.""" + without = _curve(grid_export_allowed=False) + with_export = _curve(grid_export_allowed=True) + + assert without.value(10000.0) == pytest.approx(0.8) + # 7 kWh beyond the residual load at the median feed-in tariff of 0.08 EUR/kWh. + assert with_export.value(10000.0) == pytest.approx(0.8 + 7.0 * 0.08) + assert with_export.marginal_euro_per_kwh[-1] == pytest.approx(0.08) + + +def test_residual_energy_marks_the_knee(): + """The knee separates load-backed value from the export tail.""" + without = _curve(grid_export_allowed=False) + with_export = _curve(grid_export_allowed=True) + + # 3 kWh of residual load in the window, whether or not export is allowed. + assert without.residual_energy_wh == pytest.approx(3000.0) + assert with_export.residual_energy_wh == pytest.approx(3000.0) + # Only the export tail reaches beyond it. + assert without.energy_wh[-1] == pytest.approx(3000.0) + assert with_export.energy_wh[-1] == pytest.approx(10000.0) + + +def test_curve_is_capped_by_the_usable_battery_energy(): + """A battery smaller than the residual load ends the curve early.""" + curve = _curve(max_energy_wh=1500.0) + assert curve.energy_wh[-1] == pytest.approx(1500.0) + assert curve.value(5000.0) == pytest.approx(0.4 + 0.5 * 0.3) + + +def test_empty_window_yields_an_empty_curve(): + """Without data there is no curve, and no credit.""" + curve = build_terminal_value_curve( + prices_euro_per_wh=np.zeros(0), + load_wh=np.zeros(0), + pv_wh=np.zeros(0), + feed_in_euro_per_wh=np.zeros(0), + max_energy_wh=10000.0, + ) + assert curve.energy_wh == [] + assert curve.value(5000.0) == 0.0 + + +def test_trailing_window_takes_the_end_of_the_horizon(): + values = np.arange(10, dtype=float) + + assert list(trailing_window(values, end_slot=8, window_slots=3)) == [5.0, 6.0, 7.0] + # A window longer than the horizon yields what there is. + assert list(trailing_window(values, end_slot=2, window_slots=5)) == [0.0, 1.0] + assert list(trailing_window(None, end_slot=8, window_slots=3)) == [] diff --git a/tests/test_typingmodels.py b/tests/test_typingmodels.py index 5938a65b..50c8688f 100644 --- a/tests/test_typingmodels.py +++ b/tests/test_typingmodels.py @@ -130,11 +130,17 @@ def test_backup_operations_report_an_uninitialized_path_as_an_invariant_failure( def test_energy_management_initializes_its_start_datetime_once( + config_eos: ConfigEOS, monkeypatch: pytest.MonkeyPatch, ) -> None: clock = MagicMock(return_value=to_datetime("2024-01-01T12:34:56+01:00")) - monkeypatch.setattr("akkudoktoreos.core.ems.to_datetime", clock) + + def fixed_datetime(*args, **kwargs): + return to_datetime(*args, **kwargs) if args or kwargs else clock() + + monkeypatch.setattr("akkudoktoreos.core.ems.to_datetime", fixed_datetime) monkeypatch.setattr(EnergyManagement, "_start_datetime", None) + monkeypatch.setattr(EnergyManagement, "_observation_datetime", None) ems = EnergyManagement() first = ems.start_datetime @@ -159,7 +165,7 @@ async def test_energy_calculation_requires_resolvable_record_times( monkeypatch: pytest.MonkeyPatch, ) -> None: measurement = Measurement() - measurement.records = [MeasurementDataRecord()] + monkeypatch.setattr(measurement, "records", [MeasurementDataRecord()]) monkeypatch.setattr(Measurement, "min_datetime", AsyncMock(return_value=None)) monkeypatch.setattr(Measurement, "max_datetime", AsyncMock(return_value=None)) with pytest.raises(ValueError, match="Start and end datetimes are required"): diff --git a/tests/testdata/docs/_generated/configdevices.md b/tests/testdata/docs/_generated/configdevices.md index 5a9b4fb5..ce09b543 100644 --- a/tests/testdata/docs/_generated/configdevices.md +++ b/tests/testdata/docs/_generated/configdevices.md @@ -12,14 +12,14 @@ config path from ``self.device_id`` without needing an external index. | Name | Environment Variable | Type | Read-Only | Default | Description | | ---- | -------------------- | ---- | --------- | ------- | ----------- | -| batteries | `EOS_DEVICES__BATTERIES` | `dict[str, akkudoktoreos.devices.settings.batterysettings.BatteriesCommonSettings] | None` | `rw` | `None` | Stationary battery storage devices, keyed by device_id. | -| electric_vehicles | `EOS_DEVICES__ELECTRIC_VEHICLES` | `dict[str, akkudoktoreos.devices.settings.batterysettings.BatteriesCommonSettings] | None` | `rw` | `None` | Electric vehicle battery packs, keyed by device_id. | +| batteries | `EOS_DEVICES__BATTERIES` | `Optional[dict[str, akkudoktoreos.devices.settings.batterysettings.BatteriesCommonSettings]]` | `rw` | `None` | Stationary battery storage devices, keyed by device_id. | +| electric_vehicles | `EOS_DEVICES__ELECTRIC_VEHICLES` | `Optional[dict[str, akkudoktoreos.devices.settings.batterysettings.BatteriesCommonSettings]]` | `rw` | `None` | Electric vehicle battery packs, keyed by device_id. | | home_appliances | `EOS_DEVICES__HOME_APPLIANCES` | `dict[str, akkudoktoreos.devices.settings.homeappliancesettings.HomeApplianceCommonSettings]` | `rw` | `required` | Shiftable home appliance devices, keyed by device_id. | -| inverters | `EOS_DEVICES__INVERTERS` | `dict[str, akkudoktoreos.devices.settings.invertersettings.InverterCommonSettings] | None` | `rw` | `None` | Inverter devices, keyed by device_id. | -| max_batteries | `EOS_DEVICES__MAX_BATTERIES` | `int | None` | `rw` | `None` | Maximum number of batteries allowed. | -| max_electric_vehicles | `EOS_DEVICES__MAX_ELECTRIC_VEHICLES` | `int | None` | `rw` | `None` | Maximum number of EVs allowed. | -| max_home_appliances | `EOS_DEVICES__MAX_HOME_APPLIANCES` | `int | None` | `rw` | `None` | Maximum number of home appliances allowed. | -| max_inverters | `EOS_DEVICES__MAX_INVERTERS` | `int | None` | `rw` | `None` | Maximum number of inverters allowed. | +| inverters | `EOS_DEVICES__INVERTERS` | `Optional[dict[str, akkudoktoreos.devices.settings.invertersettings.InverterCommonSettings]]` | `rw` | `None` | Inverter devices, keyed by device_id. | +| max_batteries | `EOS_DEVICES__MAX_BATTERIES` | `Optional[int]` | `rw` | `None` | Maximum number of batteries allowed. | +| max_electric_vehicles | `EOS_DEVICES__MAX_ELECTRIC_VEHICLES` | `Optional[int]` | `rw` | `None` | Maximum number of EVs allowed. | +| max_home_appliances | `EOS_DEVICES__MAX_HOME_APPLIANCES` | `Optional[int]` | `rw` | `None` | Maximum number of home appliances allowed. | +| max_inverters | `EOS_DEVICES__MAX_INVERTERS` | `Optional[int]` | `rw` | `None` | Maximum number of inverters allowed. | | measurement_keys | | `list[str]` | `ro` | `N/A` | All measurement keys across all configured devices. | ::: @@ -35,6 +35,8 @@ config path from ``self.device_id`` without needing an external index. "batteries": { "bat0": { "device_id": "bat0", + "capacity_estimation": null, + "capacity_estimate": null, "capacity_wh": 8000, "charging_efficiency": 0.88, "discharging_efficiency": 0.88, @@ -55,13 +57,21 @@ config path from ``self.device_id`` without needing an external index. 1.0 ], "min_soc_percentage": 0, - "max_soc_percentage": 100 + "max_soc_percentage": 100, + "grid_export_rates": [ + 0.25, + 0.5, + 0.75, + 1.0 + ] } }, "max_batteries": 1, "electric_vehicles": { "ev0": { "device_id": "ev0", + "capacity_estimation": null, + "capacity_estimate": null, "capacity_wh": 60000, "charging_efficiency": 0.88, "discharging_efficiency": 0.88, @@ -82,7 +92,13 @@ config path from ``self.device_id`` without needing an external index. 1.0 ], "min_soc_percentage": 0, - "max_soc_percentage": 100 + "max_soc_percentage": 100, + "grid_export_rates": [ + 0.25, + 0.5, + 0.75, + 1.0 + ] } }, "max_electric_vehicles": 1, @@ -116,6 +132,8 @@ config path from ``self.device_id`` without needing an external index. "batteries": { "bat0": { "device_id": "bat0", + "capacity_estimation": null, + "capacity_estimate": null, "capacity_wh": 8000, "charging_efficiency": 0.88, "discharging_efficiency": 0.88, @@ -137,6 +155,12 @@ config path from ``self.device_id`` without needing an external index. ], "min_soc_percentage": 0, "max_soc_percentage": 100, + "grid_export_rates": [ + 0.25, + 0.5, + 0.75, + 1.0 + ], "measurement_key_soc_factor": "bat0-soc-factor", "measurement_key_power_l1_w": "bat0-power-l1-w", "measurement_key_power_l2_w": "bat0-power-l2-w", @@ -155,6 +179,8 @@ config path from ``self.device_id`` without needing an external index. "electric_vehicles": { "ev0": { "device_id": "ev0", + "capacity_estimation": null, + "capacity_estimate": null, "capacity_wh": 60000, "charging_efficiency": 0.88, "discharging_efficiency": 0.88, @@ -176,6 +202,12 @@ config path from ``self.device_id`` without needing an external index. ], "min_soc_percentage": 0, "max_soc_percentage": 100, + "grid_export_rates": [ + 0.25, + 0.5, + 0.75, + 1.0 + ], "measurement_key_soc_factor": "ev0-soc-factor", "measurement_key_power_l1_w": "ev0-power-l1-w", "measurement_key_power_l2_w": "ev0-power-l2-w", diff --git a/tests/testdata/docs/_generated/configexample.md b/tests/testdata/docs/_generated/configexample.md index 02bf87bf..908f3ab1 100644 --- a/tests/testdata/docs/_generated/configexample.md +++ b/tests/testdata/docs/_generated/configexample.md @@ -39,6 +39,8 @@ "batteries": { "bat0": { "device_id": "bat0", + "capacity_estimation": null, + "capacity_estimate": null, "capacity_wh": 8000, "charging_efficiency": 0.88, "discharging_efficiency": 0.88, @@ -59,13 +61,21 @@ 1.0 ], "min_soc_percentage": 0, - "max_soc_percentage": 100 + "max_soc_percentage": 100, + "grid_export_rates": [ + 0.25, + 0.5, + 0.75, + 1.0 + ] } }, "max_batteries": 1, "electric_vehicles": { "ev0": { "device_id": "ev0", + "capacity_estimation": null, + "capacity_estimate": null, "capacity_wh": 60000, "charging_efficiency": 0.88, "discharging_efficiency": 0.88, @@ -86,7 +96,13 @@ 1.0 ], "min_soc_percentage": 0, - "max_soc_percentage": 100 + "max_soc_percentage": 100, + "grid_export_rates": [ + 0.25, + 0.5, + 0.75, + 1.0 + ] } }, "max_electric_vehicles": 1, @@ -205,6 +221,9 @@ }, "measurement": { "historic_hours": 17520, + "channels": {}, + "household": null, + "energy_context_seconds": 86400, "load_emr_keys": [ "load0_emr" ], diff --git a/tests/testdata/docs/_generated/configmeasurement.md b/tests/testdata/docs/_generated/configmeasurement.md index f6bc7c66..c0d58f09 100644 --- a/tests/testdata/docs/_generated/configmeasurement.md +++ b/tests/testdata/docs/_generated/configmeasurement.md @@ -7,12 +7,15 @@ | Name | Environment Variable | Type | Read-Only | Default | Description | | ---- | -------------------- | ---- | --------- | ------- | ----------- | -| grid_export_emr_keys | `EOS_MEASUREMENT__GRID_EXPORT_EMR_KEYS` | `list[str] | None` | `rw` | `None` | The keys of the measurements that are energy meter readings of energy export to grid [kWh]. | -| grid_import_emr_keys | `EOS_MEASUREMENT__GRID_IMPORT_EMR_KEYS` | `list[str] | None` | `rw` | `None` | The keys of the measurements that are energy meter readings of energy import from grid [kWh]. | -| historic_hours | `EOS_MEASUREMENT__HISTORIC_HOURS` | `int | None` | `rw` | `17520` | Number of hours into the past for measurement data | +| channels | `EOS_MEASUREMENT__CHANNELS` | `dict[str, akkudoktoreos.measurement.measurement.MeasurementChannelSettings]` | `rw` | `required` | Typed raw measurement channels keyed by measurement key. | +| energy_context_seconds | `EOS_MEASUREMENT__ENERGY_CONTEXT_SECONDS` | `int` | `rw` | `86400` | None | +| grid_export_emr_keys | `EOS_MEASUREMENT__GRID_EXPORT_EMR_KEYS` | `Optional[list[str]]` | `rw` | `None` | The keys of the measurements that are energy meter readings of energy export to grid [kWh]. | +| grid_import_emr_keys | `EOS_MEASUREMENT__GRID_IMPORT_EMR_KEYS` | `Optional[list[str]]` | `rw` | `None` | The keys of the measurements that are energy meter readings of energy import from grid [kWh]. | +| historic_hours | `EOS_MEASUREMENT__HISTORIC_HOURS` | `Optional[int]` | `rw` | `17520` | Number of hours into the past for measurement data | +| household | `EOS_MEASUREMENT__HOUSEHOLD` | `Optional[akkudoktoreos.measurement.household.HouseholdSettings]` | `rw` | `None` | Optional household energy balance definition. | | keys | | `list[str]` | `ro` | `N/A` | The keys of the measurements that can be stored. | -| load_emr_keys | `EOS_MEASUREMENT__LOAD_EMR_KEYS` | `list[str] | None` | `rw` | `None` | The keys of the measurements that are energy meter readings of a load [kWh]. | -| pv_production_emr_keys | `EOS_MEASUREMENT__PV_PRODUCTION_EMR_KEYS` | `list[str] | None` | `rw` | `None` | The keys of the measurements that are PV production energy meter readings [kWh]. | +| load_emr_keys | `EOS_MEASUREMENT__LOAD_EMR_KEYS` | `Optional[list[str]]` | `rw` | `None` | The keys of the measurements that are energy meter readings of a load [kWh]. | +| pv_production_emr_keys | `EOS_MEASUREMENT__PV_PRODUCTION_EMR_KEYS` | `Optional[list[str]]` | `rw` | `None` | The keys of the measurements that are PV production energy meter readings [kWh]. | ::: @@ -25,6 +28,9 @@ { "measurement": { "historic_hours": 17520, + "channels": {}, + "household": null, + "energy_context_seconds": 86400, "load_emr_keys": [ "load0_emr" ], @@ -51,6 +57,9 @@ { "measurement": { "historic_hours": 17520, + "channels": {}, + "household": null, + "energy_context_seconds": 86400, "load_emr_keys": [ "load0_emr" ],