Files
EOS/src/akkudoktoreos/optimization/genetic/terminalvalue.py
T
04f28997ea feat: deliver complete GENETIC optimization to main (#1330)
* feat: adapt configuration for multi optimization algorithms

Decouple configuration from optimization algorithm parameters. Add to_[algorithm]_param() methods
to the configuration that derive optimization algorithm specific parameters from the configuration.
Add x-scope tags to the configuration options that describe for which specific algorithms the
configuration option is for.

The whole device settings are restructured. There are now general settings for the device classes
with the afore mentioned to_[algorithm]_param() methods. The general device settings got their own
directory `devices/settings`. By this the parameter class also does not have to be a pydantic model
which can be used for future optimization/ simulations speed up.

Also the parameter class for a device is now part of the device module. This better decouples and
also is the natural place for parameters of a device.

Besides this feature there are also fixes and improvements:

* feat: extend home appliance time window settings and simulation

  Home appliance can now be configured for multiple runs with per-cycle allowed time windows. The
  number of remaining cycles to plan is determined at runtime by reading the
  ``cycles_completed_measurement_key`` from the measurement store.

* feat: specialiced CycleTimeWindowSequence for time window sequences

  Sequence of time windows associated to cycles.

  This model specializes ``ValueTimeWindowSequence`` so that the ``value``
  field of each ``ValueTimeWindow`` encodes the **cycle index** (0-based
  integer) the window belongs to.

  Typical use: an appliance that must run ``n`` times per day, each run
  constrained to a distinct time window.  Assign ``value=0`` to windows
  for the first cycle, ``value=1`` for the second, and so on.  Multiple
  windows may share the same cycle index (their allowed regions are unioned).
  Windows with ``value=None`` are silently ignored by all cycle-aware methods.

* fix: Make test_configmigrate also regard the _ANY_SENTENIEL in key values

* chore: Make devices configurations a map instead of a list

  This makes config paths stable regardless of declaration order and lets each device settings
  class build its own config path from ``self.device_id`` without needing an external index.
  Tests are adapted likewise.

  Devices configurations are automatically migrated from lists to maps.

* chore: rename levelized_cost_of_storage_kwh to levelized_cost_of_storage_amt kwh

  This better fits in the naming scheme and also makes clear the costs are money.

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>

* fix: runtime config update ignored by config file

Runtime settings were handed back to pydantic-settings as init settings,
which rank below the config file and the environment. Any key already
present in EOS.config.json or in the environment silently discarded the
update, so a bulk PUT /v1/config returned 200 without applying anything,
while the granular PUT /v1/config/{path} endpoint kept working.

Add a dedicated runtime settings source ranked directly below the command
line arguments and record granular updates there as well, so both
endpoints share one store that survives re-evaluation of the settings
sources. Environment variables keep precedence over the config file for
all keys that were not set at runtime.

Also repairs revert_settings() and update(), which passed their data
through the same init settings.

Closes #1303

* fix: env vars ignored on first config build

ConfigEOS.__init__ passed self as first positional argument to _setup,
which forwards it to pydantic_settings.BaseSettings.__init__. Its first
positional parameter is _case_sensitive, so the environment source
matched the upper case variable names against the lower case field names
and returned nothing. Environment settings only took effect after the
next configuration setup.

* docs: changelog for config priority fixes

* fix(config): preserve device identities and storage costs during migration

* fix(measurement): restore JSON records into the existing singleton

* fix(devices): preserve charge-rate typing and public import compatibility

* feat(measurement): integrate typed energy quality and capacity APIs

Port the locally backed-up measurement extensions to main async storage and PR #1256 device maps. Preserve runtime capacity estimates across #1305 bulk updates. Confirm JSON singleton restore defect on unchanged main and add regression. No production configuration or measurements included.

Co-authored-by: Andreas <drbacke@gmx.de>

* docs(measurement): describe household settings and consolidate regression coverage

* docs(measurement): regenerate configuration and API contracts

* test(measurement): isolate capacity database state between tests

* ruff format fix

* fix(measurement): restore JSON records into the existing singleton

* test(measurement): assert restored timestamps before timezone conversion

* test(measurement): assert restored timestamps before timezone conversion

* fix: preserve imported feed-in revenue during parameter preparation

Cancel GENETIC preparation when imported revenue cannot be read or contains invalid values, preserving the chosen provider instead of replacing it with demo tariffs. Keep valid positive, zero and negative amount/Wh series unchanged.

Adapt the revenue-preservation regressions from PRs #1224 and #1304 to the async main API, including real timestamped imports and simulation repricing. The feature-only direct-marketing override remains outside this main fix.

Co-authored-by: Christin <info@bikinibottom.capital>

Co-authored-by: Normann <github@koldrack.com>

* feat(devices): port slot-aware battery export and direct-use physics

Port scoped device changes from d2e2d58237. Keep PR #1256 parameter conversion structure and separate GENETIC0 devices. Validate physical flows and reprice changed simulation results.

Co-authored-by: Andreas <drbacke@gmx.de>

Co-authored-by: Christin <info@bikinibottom.capital>

* docs(measurement): align API version with refreshed prerequisites

* fix: return only completed optimization results per run

* feat(pvforecast): add calibrated local Akkudoktor backend

Port local PV modeling and outage calibration from feature commits f976335, 6dc58c3 and faed0fd by Andreas. Keep PVForecastAkkudoktor identity and remote default, adapt to async storage, and migrate legacy provider settings.

* fix(cache): distinguish callables in the shared EMS cache

Include the function object in cache keys so methods of one interpolator cannot reuse a probability as a power value. Cover both call orders, keyword arguments, cache hits and separate closures with identical qualified names.

* fix(devices): constrain the physics port and validate export levels

Defer inactive EV deadline fields to the optimizer port, reject nonfinite export rates, and document the hourly Optimize boundary. Verify converter IDs, rates and LCOS, separate GENETIC0 interpolation, physical boundary flows and independent GENETIC repricing.

* docs(pvforecast): regenerate local backend configuration schema

* docs(devices): regenerate slot-physics configuration and OpenAPI schemas

* test: type dynamic Optimize regression arguments

* style: wrap imported tariff test parameter import

* style(pvforecast): apply CI import formatting

* docs(pvforecast): refresh API version after CI formatting

* fix(config): satisfy typed device conversion and migration contracts

* docs(config): refresh validated configuration prerequisite schemas

* fix(measurement): enforce typed capacity and sample validation

* test(devices): align physics regressions with strict type checking

* style(measurement): normalize imports for CI

* docs(measurement): refresh typed measurement API schemas

* docs(devices): refresh API version after prerequisite merge

* test: make optimization dispatch timezones explicit

* docs(interpolator): use portable reStructuredText markup

* docs(devices): refresh API version after docstring compatibility fix

* feat: complete configuration-driven GENETIC optimization and reports (#1329)

* feat(devices): port slot-aware battery export and direct-use physics

Port scoped device changes from d2e2d58237. Keep PR #1256 parameter conversion structure and separate GENETIC0 devices. Validate physical flows and reprice changed simulation results.

Co-authored-by: Andreas <drbacke@gmx.de>

Co-authored-by: Christin <info@bikinibottom.capital>

* feat(optimization): port tested terminal and tail value primitives

Source d2e2d58237. 22 primitive tests pass; integration with the optimizer, forecast horizon and API is still pending.

Co-authored-by: Andreas <drbacke@gmx.de>

Co-authored-by: Christin <info@bikinibottom.capital>

* fix(devices): preserve charge-rate typing and public import compatibility

* feat(measurement): integrate typed energy quality and capacity APIs

Port the locally backed-up measurement extensions to main async storage and PR #1256 device maps. Preserve runtime capacity estimates across #1305 bulk updates. Confirm JSON singleton restore defect on unchanged main and add regression. No production configuration or measurements included.

Co-authored-by: Andreas <drbacke@gmx.de>

* test(integration): validate optimizer economics and document measurement settings

* docs(integration): record tested checkpoint and remaining consolidation work

* docs(development): define isolated PR packages and remaining porting gates

* docs(integration): refresh API version after measurement reconciliation

* docs(integration): record PR readiness verification results

* docs(development): record publication and verification of PR 1322

* test(measurement): assert restored timestamps before timezone conversion

* docs(development): record corrected PR head and CI progress

* docs(integration): refresh API version after prerequisite alignment

* docs(integration): define parallel packages and Optimize compatibility gates

* fix: preserve imported feed-in revenue during parameter preparation

Cancel GENETIC preparation when imported revenue cannot be read or contains invalid values, preserving the chosen provider instead of replacing it with demo tariffs. Keep valid positive, zero and negative amount/Wh series unchanged.

Adapt the revenue-preservation regressions from PRs #1224 and #1304 to the async main API, including real timestamped imports and simulation repricing. The feature-only direct-marketing override remains outside this main fix.

Co-authored-by: Christin <info@bikinibottom.capital>

Co-authored-by: Normann <github@koldrack.com>

* test(integration): verify tariff protection with mapped device physics

* fix: return only completed optimization results per run

* test(integration): verify algorithm aliases and mapped-device contracts

* fix(cache): distinguish callables in the shared EMS cache

Include the function object in cache keys so methods of one interpolator cannot reuse a probability as a power value. Cover both call orders, keyword arguments, cache hits and separate closures with identical qualified names.

* fix(devices): constrain the physics port and validate export levels

Defer inactive EV deadline fields to the optimizer port, reject nonfinite export rates, and document the hourly Optimize boundary. Verify converter IDs, rates and LCOS, separate GENETIC0 interpolation, physical boundary flows and independent GENETIC repricing.

* feat(pvforecast): add calibrated local Akkudoktor backend

Port local PV modeling and outage calibration from feature commits f976335, 6dc58c3 and faed0fd by Andreas. Keep PVForecastAkkudoktor identity and remote default, adapt to async storage, and migrate legacy provider settings.

* docs(integration): record combined compatibility checks and green JSON PR CI

* test: type dynamic Optimize regression arguments

* docs(integration): record Optimize fix PR publication

* docs(integration): record imported tariff protection PR

* style(pvforecast): apply CI import formatting

* style(integration): align combined regression imports

* test: make optimization dispatch timezones explicit

* docs(interpolator): use portable reStructuredText markup

* chore: validate combined integration with locked mypy

* docs: hand off six validated pull requests for manual review

* feat: report genetic interval and terminal value diagnostics

* feat(devices): reconcile flexible profiles and EV deadlines with cycle scheduling

Adapt the flexible consumer primitives from d2e2d582 while retaining the keyed settings and per-cycle scheduling introduced by #1256. Preserve slot battery physics and GENETIC0 flat-load conversion. Cover energy conservation, deadlines, window intersections, DST, completed cycles and EV converters.

* test: satisfy typed genetic PDF chart contracts

* feat(optimization): resolve quarter-hour GENETIC requests from configuration

* test(genetic): verify real device scheduling, measurement and export contracts

Register appliance completed-cycle measurement keys so the real store accepts both default and custom counters. Exercise complete low-budget optimizer runs, persisted measurements, generic solution output and instructions, including zero-power phases, EV departure boundaries, per-cycle windows and LCOS.

* fix: bound genetic report forecasts to executable horizon

* feat: complete native genetic scheduling and retained result contracts

* fix: retain missing raw samples when dropna is disabled

* fix: align local optimization slots and measurement instants

* docs: explain complete genetic rollout and PR dependencies

* feat: expose retained GENETIC report through the versioned API

* docs: regenerate complete genetic configuration and API schema

* docs: format consolidation and review handoff markdown

* test: align isolated EMS fixture with native genetic run options

* test(genetic): clean up singleton measurements after device integration tests

* test: freeze the clock without replacing timestamp conversion

* fix: preserve explicit warmstart timezones in runtime requests

* test(genetic): validate device schedules in UTC and Berlin

Use explicit Berlin origins for Berlin wall-clock windows, compare absolute deadline instants correctly, and run all real device optimizer scenarios under both UTC and Europe/Berlin. Compare exported starts in the run timezone instead of assuming the output timezone matches the host.

* fix: start automatic genetic runs in the site timezone

* Preserve aware GENETIC snapshot times across host timezones

* docs: specify site clock and rehearsed merge resolutions

* test: isolate invalid measurement records and refresh API version

* fix: render single-slot genetic tail diagnostics

* docs: refresh schema version after report fix

* fix: preserve configuration-only Optimize API contract

* docs: refresh configuration request schema

---------

Co-authored-by: Christin <info@bikinibottom.capital>
Co-authored-by: Normann <github@koldrack.com>

---------

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: r0b2g1t <r0b2g1t@users.noreply.github.com>
Co-authored-by: Normann <github@koldrack.com>
Co-authored-by: Christin <info@bikinibottom.capital>
2026-09-17 20:14:24 +02:00

360 lines
13 KiB
Python

"""Terminal value of the energy left in the battery at the end of the horizon.
The optimizer stops at the horizon, but the energy still stored in the battery
keeps its worth: it replaces grid imports that would otherwise be paid for
afterwards. Crediting that worth with a single price per kWh - the historical
``preis_euro_pro_wh_akku`` - cannot describe it, because the value of stored
energy is **not linear in the amount stored**:
- The first kWh replaces the most expensive hour after the horizon.
- The next one replaces the second most expensive hour, and so on.
- Once every hour that PV cannot cover is served, further energy replaces
nothing; it is worth an export at best, and nothing at worst.
The resulting value function is monotone and concave. A scalar has to pick one
slope: high enough for the first kWh means hoarding a full battery, low enough
for the last kWh means running it empty by midnight. This module builds the
curve instead.
There is no forecast beyond the horizon, so the trailing window of the horizon
itself stands in for the day that follows: same season, same household rhythm,
same tariff structure. That approximation is the reason the curve is a planning
aid, not a prediction - which is also why the marginal values are deliberately
conservative wherever a choice exists.
"""
from typing import Optional
import numpy as np
from loguru import logger
from pydantic import Field
from akkudoktoreos.core.pydantic import PydanticBaseModel
class TerminalValueCurve(PydanticBaseModel):
"""Piecewise linear, concave value of battery energy left at the horizon.
``energy_wh`` and ``value_euro`` are the breakpoints of the cumulative
value, ``marginal_euro_per_kwh`` the slope of each segment. Both arrays
start at the origin; the curve is flat beyond its last breakpoint.
"""
energy_wh: list[float] = Field(
default_factory=list,
json_schema_extra={
"description": "Breakpoints of usable AC energy left in the battery [Wh]."
},
)
value_euro: list[float] = Field(
default_factory=list,
json_schema_extra={"description": "Cumulative credit at each breakpoint [EUR]."},
)
operating_value_euro: list[float] = Field(
default_factory=list,
json_schema_extra={
"description": "Tail operating component at each breakpoint [EUR]; empty for a proxy curve."
},
)
continuation_value_euro: list[float] = Field(
default_factory=list,
json_schema_extra={
"description": "Continuation component at each breakpoint [EUR]; empty for a proxy curve."
},
)
marginal_euro_per_kwh: list[float] = Field(
default_factory=list,
json_schema_extra={
"description": (
"Marginal value of the segment that starts at each breakpoint "
"[EUR/kWh]. May be negative or non-monotone in TAIL mode."
)
},
)
residual_energy_wh: float = Field(
default=0.0,
json_schema_extra={
"description": (
"Energy up to which the curve is backed by residual load - the "
"knee. Everything beyond it is only worth an export."
)
},
)
window_slots: int = Field(
default=0,
json_schema_extra={
"description": (
"Number of trailing horizon slots the curve was derived from. "
"Fewer slots than a full day mean a shorter proxy period."
)
},
)
def value(self, energy_wh: float) -> float:
"""Return the credit for ``energy_wh`` of usable AC energy [EUR].
Args:
energy_wh: Usable AC energy left in the battery.
Returns:
Interpolated value of the curve; 0.0 for an empty curve.
"""
if not self.energy_wh or energy_wh <= 0.0:
return 0.0
return float(np.interp(energy_wh, self.energy_wh, self.value_euro))
class TailDiagnostics(PydanticBaseModel):
"""Forecast summary used by the deterministic tail optimization."""
slots: int = 0
slot_hours: float = 0.0
soc_grid_points: int = 0
min_import_price_euro_per_kwh: float = 0.0
max_import_price_euro_per_kwh: float = 0.0
min_feed_in_tariff_euro_per_kwh: float = 0.0
max_feed_in_tariff_euro_per_kwh: float = 0.0
negative_import_price_slots: int = 0
positive_battery_export_slots: int = 0
class TailPlanSlot(PydanticBaseModel):
"""One diagnostic slot of the optimal tail path.
These values explain the lookahead used for fitness. They are diagnostics
only and are never copied into the executable control arrays.
"""
slot: int
hour_from_start: float
action: str
alternative_action: str = ""
decision_margin_euro: float = 0.0
soc_start_percentage: float
soc_end_percentage: float
pv_wh: float
load_wh: float
grid_import_wh: float
grid_export_wh: float
battery_charge_wh: float
battery_discharge_wh: float
import_price_euro_per_kwh: float
feed_in_tariff_euro_per_kwh: float
slot_value_euro: float
remaining_value_euro: float
ac_charge_factor: float
dc_charge_allowed: int
discharge_allowed: int
battery_grid_export_factor: float
class TerminalValueResult(PydanticBaseModel):
"""What the optimizer credited for the energy left in the battery."""
control_horizon_hours: float = 0
requested_tail_hours: float = 0
effective_tail_hours: float = 0
tail_end_hour: float = 0
continuation_mode: str = "FIXED"
mode: str = Field(
json_schema_extra={
"description": "Terminal value mode the run used: TAIL, AUTO or FIXED.",
"examples": ["TAIL", "AUTO", "FIXED"],
}
)
battery_energy_wh: float = Field(
default=0.0,
json_schema_extra={
"description": "Usable AC energy left in the battery at the end of the horizon [Wh]."
},
)
credited_euro: float = Field(
default=0.0,
json_schema_extra={"description": "Credit applied to the total balance [EUR]."},
)
tail_operating_euro: float = Field(
default=0.0,
json_schema_extra={
"description": (
"Optimal net cash flow within the effective tail for the selected "
"control-end battery state [EUR]."
)
},
)
continuation_value_euro: float = Field(
default=0.0,
json_schema_extra={
"description": (
"Continuation credit remaining at the end of the optimal tail path [EUR]."
)
},
)
curve: Optional[TerminalValueCurve] = Field(
default=None,
json_schema_extra={
"description": (
"Combined tail value curve (tail operation plus continuation) read by fitness; "
"None in FIXED mode."
)
},
)
continuation_curve: Optional[TerminalValueCurve] = Field(
default=None,
json_schema_extra={
"description": "Conservative AUTO proxy constructed at the effective tail end."
},
)
tail_diagnostics: Optional[TailDiagnostics] = None
tail_plan: list[TailPlanSlot] = Field(
default_factory=list,
json_schema_extra={
"description": (
"Diagnostic optimal battery path inside the tail. It explains the "
"lookahead but is never an executable control plan."
)
},
)
reason: str = Field(
default="",
json_schema_extra={
"description": (
"Why this mode applied. Empty in AUTO mode; in FIXED mode it "
"says whether FIXED was configured or whether AUTO fell back "
"because no curve could be derived."
),
"examples": ["", "terminal_value_mode is FIXED"],
},
)
def build_terminal_value_curve(
*,
prices_euro_per_wh: np.ndarray,
load_wh: np.ndarray,
pv_wh: np.ndarray,
feed_in_euro_per_wh: np.ndarray,
max_energy_wh: float,
lcos_euro_per_kwh: float = 0.0,
dc_to_ac_efficiency: float = 1.0,
grid_export_allowed: bool = False,
) -> TerminalValueCurve:
"""Build the terminal value curve from the trailing horizon window.
Every slot of the window contributes its residual load - the part of the
load that PV does not cover - at its import price. Sorting those slots by
price and accumulating them yields the marginal value of the first, second,
... kWh in the battery. Energy beyond the residual load can only be
exported, and only when direct marketing allows it.
Args:
prices_euro_per_wh: Import prices of the window [EUR/Wh].
load_wh: Load per slot of the window [Wh].
pv_wh: PV generation per slot of the window [Wh].
feed_in_euro_per_wh: Feed-in tariff of the window [EUR/Wh].
max_energy_wh: Usable AC energy of a full battery [Wh]; the curve ends here.
lcos_euro_per_kwh: Levelized cost of storage, already charged per
delivered DC energy in the simulation and therefore subtracted here
so stored energy is not credited twice.
dc_to_ac_efficiency: Inverter efficiency, used to convert the LCOS from
delivered DC energy to the AC energy of the curve.
grid_export_allowed: Whether the battery may feed the grid (direct
marketing). Without it, energy beyond the residual load gets no
credit: it can neither be exported nor is its use covered by the
proxy window.
Returns:
The curve; empty when the window carries no usable information.
"""
window = min(len(prices_euro_per_wh), len(load_wh), len(pv_wh))
if window <= 0 or max_energy_wh <= 0.0:
return TerminalValueCurve()
residual = np.maximum(load_wh[:window] - pv_wh[:window], 0.0)
prices = np.asarray(prices_euro_per_wh[:window], dtype=float)
# LCOS is charged on delivered DC energy; the curve is in AC energy.
lcos_per_wh_ac = (lcos_euro_per_kwh / 1000.0) / max(dc_to_ac_efficiency, 1e-9)
order = np.argsort(-prices)
energy_points: list[float] = [0.0]
value_points: list[float] = [0.0]
marginals: list[float] = []
cumulative_energy = 0.0
cumulative_value = 0.0
for index in order:
slot_energy = float(residual[index])
if slot_energy <= 0.0:
continue
# Negative or very cheap hours are not worth storing energy for.
marginal = max(float(prices[index]) - lcos_per_wh_ac, 0.0)
if marginal <= 0.0:
continue
slot_energy = min(slot_energy, max_energy_wh - cumulative_energy)
if slot_energy <= 0.0:
break
cumulative_energy += slot_energy
cumulative_value += slot_energy * marginal
energy_points.append(cumulative_energy)
value_points.append(cumulative_value)
marginals.append(marginal * 1000.0)
# Everything beyond the residual load can only be sold. A median feed-in
# tariff rather than the best one: exporting all of it in the single best
# slot is not something the horizon can promise.
residual_energy_wh = cumulative_energy
if grid_export_allowed and cumulative_energy < max_energy_wh:
positive_feed_in = [
float(value) for value in feed_in_euro_per_wh[:window] if float(value) > 0.0
]
export_marginal = max(
(float(np.median(positive_feed_in)) if positive_feed_in else 0.0) - lcos_per_wh_ac,
0.0,
)
if export_marginal > 0.0:
remaining = max_energy_wh - cumulative_energy
cumulative_energy += remaining
cumulative_value += remaining * export_marginal
energy_points.append(cumulative_energy)
value_points.append(cumulative_value)
marginals.append(export_marginal * 1000.0)
if len(energy_points) <= 1:
logger.debug("Terminal value curve is empty - no priced residual load in the window.")
return TerminalValueCurve(window_slots=window)
# The segment slopes are decreasing by construction (prices were sorted),
# so the curve is concave; the export tail is the flattest segment.
return TerminalValueCurve(
energy_wh=energy_points,
value_euro=value_points,
marginal_euro_per_kwh=marginals,
residual_energy_wh=residual_energy_wh,
window_slots=window,
)
def trailing_window(
values: Optional[np.ndarray],
end_slot: int,
window_slots: int,
) -> np.ndarray:
"""Return the ``window_slots`` values in front of ``end_slot``.
Args:
values: Full slot array, or None.
end_slot: Exclusive end of the window (end of the optimization horizon).
window_slots: Desired window length; a shorter horizon yields less.
Returns:
The window as a float array, empty when no data is available.
"""
if values is None:
return np.zeros(0, dtype=float)
end = min(int(end_slot), len(values))
start = max(end - int(window_slots), 0)
if end <= start:
return np.zeros(0, dtype=float)
return np.asarray(values[start:end], dtype=float)