EOS/CHANGELOG.md

# Changelog

All notable changes to the akkudoktoreos project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

## Unreleased

### Added
- Add direct `ElecPriceSMARD` and `FeedInTariffSMARD` providers for German/Luxembourg
  day-ahead prices without relying on a supplier account or the rate-limited Energy-Charts API.
  The providers retrieve SMARD's native quarter-hour prices, cache weekly chart-data chunks, and
  extend missing horizon slots with the existing seasonal ETS forecast. Feed-in prices remain raw,
  while import prices can include retail charges.
- Add named net electricity-price components and recurring time-window network fees. Constant
  taxes, levies, supplier charges, and the matching dynamic grid fee are added to the underlying
  market price before VAT; seasonal forecasting continues to model only the market-price component.
- Add `FeedInTariffAkkudoktor`, using raw hourly Akkudoktor/aWATTar day-ahead market prices as
  feed-in tariff data without import charges or VAT. Quarter-hour optimization holds each hourly
  value constant for four slots.
- Add `FeedInTariffTibber`, using Tibber's native `QUARTER_HOURLY` spot-price component as a
  strict 15-minute feed-in tariff. Hourly API responses are rejected instead of expanded.
- Flexible consumers (home appliances): schedule any number of consumers via
  `devices.home_appliances`, each with a unique `device_id`. Every consumer defines its
  load **either** as an explicit power profile (`load_profile_power_w` at
  `load_profile_interval_seconds`, energy-preservingly resampled onto the optimization
  slot grid, including the 15-minute interval and non-integer ratios such as 10→15 min)
  **or** as the flat `consumption_wh` + `duration_h` fallback. A `schedule_mode` selects
  `ONCE` (a single run in the horizon) or `DAILY` (one run per local calendar day that
  still has a feasible full run). Allowed start times honour `time_windows` (including
  weekday/date restrictions) and the horizon; ONCE without any valid start is rejected.
  Results are reported per device (`result.home_appliance_energy_wh`, `appliance_starts`,
  per-device solution columns and `DDBCInstruction`s emitted only on RUN/OFF transitions).
- Flexible consumers can be given absolute time bounds: `earliest_start_datetime` and
  `deadline_datetime`, where the deadline demands that the complete run has *finished* before
  that moment ("clean dishes by 03:00 tonight"). When no start can meet the deadline,
  `deadline_policy` decides between `BEST_EFFORT` (run as early as possible, minimizing the
  delay) and `STRICT` (keep the deadline; a `ONCE` consumer then fails the optimization).
  The solution reports `appliance_deadline_missed` per device so callers can warn instead of
  silently trusting a late schedule.
- Battery-to-grid export (direct marketing) is no longer all-or-nothing: `grid_export_rates`
  configures the selectable export levels as a factor of the rated discharge power
  (default `[0.25, 0.5, 0.75, 1.0]`), settable per battery in `devices.batteries[].
  grid_export_rates` or per request in `pv_akku.grid_export_rates`. The optimizer picks one
  level per slot and reports it in `battery_grid_export_factor` and as the
  `GRID_SUPPORT_EXPORT` operation factor. `[1.0]` restores the previous behaviour.
- The EV charging target can be given a deadline: `min_soc_deadline_datetime` (absolute, e.g.
  the next departure) and/or `min_soc_max_duration_h` ("full in 6 hours"), the earlier of the
  two applies. The `ev_soc_miss` penalty is then evaluated at that slot instead of at the end
  of the horizon, and the seeding heuristics only propose charge slots before it. Without a
  deadline the behaviour is unchanged.
- Fix: with grid charging disabled (`inverter.max_ac_charge_power_w = 0`) the returned
  `ac_charge` array kept the optimizer's unused gene values. The simulation ignored them, so
  they were never costed - but a controller acting on the plan would grid-charge the battery
  anyway. The disabled AC charge is now cleared in the reported plan as well.
- The energy left in the battery at the end of the horizon is now valued with a concave curve
  derived from the trailing horizon window (`optimization.terminal_value_mode = AUTO`, the new
  default): the first stored kWh replaces the most expensive hour that PV cannot cover, the next
  one the second most expensive, and energy beyond the residual load is credited only when it can
  be exported. A single price per kWh could not express this - with the previous default of 0 the
  optimizer emptied the battery towards the end of the horizon, with a high value it hoarded it.
  The curve is built once per run and reported as `terminal_value` in the solution.
  `terminal_value_mode = FIXED` restores the old scalar behaviour.
- EV Bug (wrong output in genetic.py / no senseful results)
- Direktvermarktung active / Battery discharge into grid (new state / action battery_grid_export_allowed) + (new simulation output Feed_in_tariff)
- New PV forecast providers giving operators more cloud forecast sources to choose from in
  addition to Akkudoktor, VRM and Import:
  - `PVForecastPVNode` — native 15-minute forecasts from the pvnode.com API.
  - `PVForecastForecastSolar` — forecasts from the free Forecast.Solar API.
  - `PVForecastSolcast` — forecasts from the Solcast rooftop-site API. (THX to @chloepriceless)
- 15-minute optimization interval for the genetic optimizer. `optimization.interval`
  now accepts 900 (15 min) in addition to the default 3600 (1 hour), letting the
  optimizer schedule on a quarter-hour grid for 15-minute dynamic electricity
  tariffs. Device power caps and the solution/plan serializers are slot-aware; the
  default 3600 s interval keeps the previous hourly behaviour. The new sub-hourly PV
  providers (pvnode, Forecast.Solar, Solcast) feed their native resolution straight
  into the quarter-hour grid.
  - Legacy hourly API inputs are normalized onto the quarter-hour grid: PV/load energy
    is distributed across four slots, prices are held constant, and hourly warm-start
    solutions are expanded to slot controls. Native slot arrays are preserved and
    ambiguous lengths are rejected.
  - Home-appliance (flexible consumer) scheduling now runs on the same slot grid and
    supports the 15-minute interval (see the flexible consumers entry below).
  - The Tibber electricity price provider now requests native 15-minute exchange prices
    (`priceInfoRange(resolution: QUARTER_HOURLY)`) and stores them at their native
    resolution, so both the hourly and the 15-minute optimizer are fed the correct
    grid. The seasonal price extrapolation is resolution-agnostic and stays identical
    at the default hourly resolution.
- Separate battery economics into two independent settings: battery
  `levelized_cost_of_storage_kwh` is now charged once on delivered DC energy, while
  `optimization.terminal_value_euro_per_kwh` values usable battery energy left at the end of the
  optimization horizon. LCOS applies to both local battery supply and battery-to-grid export and is
  included in hourly and total costs.
- Model direct PV-to-load consumption probabilistically from the bundled conditional minute-load
  table. The expected direct flow is used consistently for PV bypass, residual load, battery
  charging, and grid export on hourly and 15-minute optimization grids.
- Add a dynamic `FeedInTariffEnergyCharts` forecast for direct marketing. Published Energy-Charts
  day-ahead market prices are retained at their native hourly or quarter-hourly resolution and
  missing slots at the end of the optimization horizon are extended with weekly or daily seasonal
  ETS forecasts. A median fallback is used when the available history is too short for ETS.
- Add the `PVForecastAkkudoktorLocal` PV forecast provider, which runs the whole modelling chain
  inside EOS with `pvlib` on raw Open-Meteo irradiance instead of calling a forecast service:
  solar position, horizon shading, plane transposition, incidence-angle modifier, cell temperature,
  PVWatts DC and inverter AC. It needs no API key, serves up to 16 days at 15-minute resolution
  from one hourly request - enough to feed `optimization.tail_horizon_hours` - and exposes albedo,
  inverter efficiency and the module temperature coefficient as real configuration. Several
  Open-Meteo models can be listed in `weather_models` and are averaged per variable at no extra
  request cost.
- The local provider can calibrate itself against measured PV production (`calibration_enabled`).
  It compares its own model against `measurement.pv_production_emr_keys` over the past
  `calibration_days` and fits a global scale factor plus optional per-solar-azimuth factors, each
  weighted by modelled energy, shrunk toward the global factor by `calibration_prior_kwh` and
  clamped to `[calibration_min_factor, calibration_max_factor]`. The comparison runs on past
  intervals, where Open-Meteo serves analysed rather than forecast weather, so it corrects the
  error of the PV model and not that of the weather forecast. Setting
  `calibration_azimuth_bin_degrees` to 0 fits the global factor alone, which is what a short
  window supports.
- Add `scripts/pvforecast_backtest.py`, which scores PV forecast configuration variants against
  the stored meter readings straight away instead of waiting for new forecasts to come true, and
  `Measurement.pv_production_total_kwh()` alongside the existing load total.
- The local PV provider's calibration now excludes probable outage and curtailment days instead of
  learning them as permanent model losses: `calibration_outage_filter_enabled` (default on),
  `calibration_outage_threshold`, `calibration_reference_days` and `calibration_min_healthy_days`
  estimate the healthy plant ratio and fall back to the most recent healthy days. Calibration also
  uses native 15-minute meter readings when every configured PV meter supplies them, interpolates
  azimuth factors smoothly between bin centres instead of stepping, and normalizes the fitted
  shape per forecast day so it redistributes energy without changing that day's kWh correction.
  The default `calibration_azimuth_bin_degrees` moves from 15 to 45, which is what a typical
  calibration window actually supports.
- Separate the control horizon from the battery lookahead. `optimization.horizon_hours` remains
  the only span that receives control commands; the new `optimization.tail_horizon_hours`
  (default 48 h) is a forecast lookahead that never produces a command. In `AUTO` terminal-value
  mode a deterministic dynamic program now solves that tail backwards on a 101-point SoC grid,
  using the production battery and inverter models with their SoC bounds, power caps, conversion
  losses, configured charge/export rates and LCOS, and applies the existing AUTO proxy as the
  continuation value at the tail end. Genetic fitness reads the resulting curve instead of a
  single price per kWh, so the optimizer stops treating the horizon boundary as the end of the
  world. `tail_horizon_hours: 0` restores the plain AUTO proxy at the control end; `FIXED` is
  unchanged. See `docs/akkudoktoreos/optimization_horizons.md`.
- The `terminal_value` result reports the split explicitly: `mode` (`TAIL`, `AUTO` or `FIXED`),
  `tail_operating_euro` plus `continuation_value_euro` (which always sum to `credited_euro`),
  the combined `curve` fitness reads, the `continuation_curve` proxy at the tail end,
  `requested_tail_hours` versus `effective_tail_hours`, and `tail_diagnostics`. The optional
  `tail_plan` replays the optimal battery path inside the tail for debugging. None of it is
  executable - tail actions are never copied into the returned control arrays.
- Raise the default `prediction.hours` from 48 to 72 so the default control horizon plus the
  default tail are covered out of the box. A shorter prediction horizon is never rejected: a tail
  that does not fit is cut to what the forecast covers (logged once and reported as
  `effective_tail_hours`), and a control horizon that does not fit is warned about at
  configuration time and rejected by the optimizer at run time, naming the series that ends too
  early. Existing configurations therefore keep starting after an upgrade.
- Genetic solutions carry `controls_start_at_now`. Index zero of every returned control array and
  warm-start genome is now the run timestamp rather than midnight of the run's day. The solution
  and plan adapters still read older, midnight-indexed solutions, and warm starts with an
  incompatible genome length are discarded instead of misapplied.
- The optimization request accepts `forecast_interval_seconds` (900 or 3600), which declares the
  resolution of shortened native quarter-hour input arrays. Fully sized native arrays are still
  auto-detected, and a scalar feed-in tariff still means an explicitly constant tariff.
- Add operator tooling for the split horizon: `docs/akkudoktoreos/grafana_tail_debugging.md`
  explains how to read control plan versus tail in Grafana, and
  `scripts/update_nodered_tail_flow.py` plus the `nodered_tail_*`/`nodered_pv_*` helpers build an
  importable Node-RED flow for it.

### Changed
- Replace the fixed DEAP variation loop with adaptive genetic evolution. Crossover offspring may
  now also mutate; population diversity and stagnation are tracked per generation; diversity
  boosts inject fresh educated/random candidates; and incumbent-preserving soft restarts recover
  from collapsed populations without stopping the run early.
- Apply small point mutations only to future, fitness-relevant controls and choose point, coherent
  block, energy-shift, or flexible-device mutations as alternatives instead of stacking random
  changes on top of every specialized move. Tournament selection retains useful duplicates while
  enforcing a 30% minimum diversity floor.
- Scale the diverse genetic start population with the configured population size while preserving
  the established 300-member mix: exact warm starts, locally mutated neighbours, randomized
  domain-informed battery/direct-marketing/EV/appliance schedules, and a guaranteed random
  remainder. Survivor and offspring counts now follow `optimization.genetic.individuals` instead
  of remaining fixed at 150.
- Add coherent battery block mutations and energy-shift mutations that move weak battery exports
  into several later expensive self-consumption slots in one step. A bounded, fitness-checked
  local search applies the same neighbourhood to the final incumbent, avoiding local minima that
  cannot be crossed by an individually disadvantageous single-slot mutation.
- Memoize successful canonical fitness evaluations within one optimization run, including repaired
  EV genomes and auxiliary metrics. Log cache hits, misses, key count, and hit rate after each run;
  failed evaluations and results from previous runs are never reused.
- `max_home_appliances` is now purely an upper bound. No demo appliance is created when
  no `home_appliances` are configured, and the number is no longer used as an on/off switch.
- The genetic diversity boost is an intervention again instead of the steady state. Its trigger
  (`DIVERSITY_BOOST_THRESHOLD`) sat above the floor the selection guarantees
  (`SELECTION_DIVERSITY_FLOOR`), so on a converged population it was permanently true; it now sits
  below the floor. Freshly injected immigrants are also the worst individuals in the pool and were
  removed by the very tournament of the generation that created them, so their genes never
  recombined - a bounded share of seats is now reserved for them for two selections. The log line
  is edge-triggered on the boost itself rather than on the last fitness improvement, and the end
  of a boost is logged too.
- Required forecasts are no longer silently replaced by demo providers. Previously a missing PV,
  price, load, feed-in or weather forecast rewrote the configured provider to a demo one and
  retried, so a run could quietly optimize against invented data. Missing values now stay missing:
  a gap inside the control horizon fails the run with the series that ends too early, and a gap
  after it shortens the tail. Provider values are held only within their own source interval and
  the last observed value is never extended indefinitely.

### Deprecated
- The single-appliance genetic optimization input `dishwasher` is deprecated in favour of
  the `home_appliances` list; a lone `dishwasher` is mapped to a one-element list, and
  setting both at once is rejected. In the solution, `washingstart` (start slot of a single
  hourly appliance) and `result.Home_appliance_wh_per_hour` (aggregate over all appliances)
  are deprecated in favour of `appliance_starts` and `result.home_appliance_energy_wh`.

### Fixed
- Exclude elapsed control slots from fitness-cache keys and clear cached genome tuples after run
  metrics are captured, avoiding false misses and delayed memory retention in long-lived API
  processes. Random EV individuals now also keep the fixed horizon tail switched off.
- Respect `optimization.genetic.individuals` and `optimization.genetic.generations` independently
  in automatic and `/optimize` runs. Previously the individual count was accidentally passed as
  the generation count, the configured generation count was ignored, and every generation still
  generated 150 offspring. The deprecated `?ngen=` query remains a generation-count override;
  `?individuals=` can override the population for one API run.
- Allow the direct-marketing optimizer to select a true battery self-consumption state with DC
  charging and local-load discharge enabled in the same slot. Existing warm-start state numbers
  remain compatible, and educated guesses now use the combined state for PV/load overlap instead
  of unnecessarily bypassing PV while serving loads such as EV charging.
- Account for EV charging losses in the AC load seen by the inverter and grid, so fitness and
  energy costs use the charger's raw input rather than only the energy stored in the EV battery.
- Treat fitness memoization as disabled for lightweight optimizer instances constructed without
  the normal initializer, preserving isolated penalty evaluation and test callers.
- Re-simulate genetic candidates after removing EV charging genes from slots that begin at full
  SoC, keeping the repaired genome and its assigned fitness consistent.
- FeedInTariffEnergyCharts no longer aborts the whole prediction/optimization when the
  Energy-Charts API is briefly unreachable: transient timeouts/connection errors are
  retried (with a (connect, read) timeout of (5, 60) s), and if a fetch still fails while
  historical data exists, the existing history is kept and the remaining slots are
  extrapolated via ETS instead of failing. A genuine cold start (no data at all) still
  fails.
- A day-ahead price source that has not published the next day yet no longer fails the whole
  prediction update. `ElecPriceEnergyCharts` and its `ElecPriceSMARD` subclass ask for prices
  starting at the run day, while the optimization horizon always reaches past the last published
  price, so every morning before the auction is published the request came back empty and the
  provider raised - answering `/v1/prediction/update` with 400 until the source caught up. The
  provider now keeps its existing history and extrapolates the remaining slots via ETS, the same
  way `FeedInTariffEnergyCharts` already did. A cold start with no history at all still fails.
- `ElecPriceSMARD` now distinguishes a lagging publication from a broken response. A window the
  source cannot serve yet reports the latest value it does have, instead of claiming the response
  contained no usable prices.
- `cache_in_file` no longer leaves an empty cache entry behind when the wrapped function raises.
  The entry was claimed before the call, so every later call within the TTL first failed to read
  it ("Ran out of input") before refetching. The entry is now created only after the call returns.
- The deprecated `/gesamtlast` endpoint no longer forces a full provider refresh on every
  call. Forcing bypassed the provider caches and hammered external APIs, so a single flaky
  provider could 404 the whole load prediction. It now defaults to a cache-aware update and
  accepts an optional `force_update` flag in the request body for callers that still want
  to force.
- The local PV provider derived its calibration window from the measurement store as a whole
  instead of from the configured PV production meters. A load meter reaching further than the PV
  meter placed the window where no PV reading exists, so calibration silently fell back to hourly
  fitting or skipped itself entirely. The window now follows the PV meters.
- `Measurement.load()` silently discarded every stored record. It validated the file into a
  temporary `Measurement`, but `Measurement` is a singleton, so the "temporary" instance was the
  already initialized one and the parsed records were dropped. The records are now validated
  individually and inserted directly.
- A rejected configuration update no longer damages the running configuration.
  `merge_settings_from_dict` validated the merged candidate only while reinitializing the
  singleton, so an invalid update could leave EOS half-updated. The candidate is validated first.

## 0.3.0 (2026-03-17)

Akkudoktor-EOS can now be run as Home Assistant add-on and standalone.
As Home Assistant add-on EOS uses ingress to fully integrate the EOSdash dashboard
in Home Assistant.

Adapters for Home Assistant and NodeRed integration are added. These adapters
provide a simplified interface to these HEMS besides the standard REST interface.

The prediction and measurement data can now be backed by a database. The database allows
to keep historic prediction data and measurement data for long time without keeping
it in memory. The database supports backend selection, compression, incremental data load,
automatic data saving to storage, automatic vacuum and compaction. Two database backends
are integrated and can be configured, LMDB and SQLight3.

New prediction providers allow to access OpenMeteo weather data and to define fixed
electricity prices for configurable time windows.

An anoying bug in the genetic algorithm that created unfeasable battery charge and
discharge amounts is now hopefully fixed.

In addition, bugs were fixed and new features were added.

### Feat

- add inverter AC/DC efficiency and break-even penalty
- add database support for measurements and historic prediction data.
  The prediction and measurement data can now be backed by a database. The database allows
  to keep historic prediction data and measurement data for long time without keeping
  it in memory. Two database backends are integrated and can be configured, LMDB and SQLight3.
- add adapters for integrations
  Adapters for Home Assistant and NodeRED integration are added.
  Akkudoktor-EOS can now be run as Home Assistant add-on and standalone.
  As Home Assistant add-on EOS uses ingress to fully integrate the EOSdash dashboard
  in Home Assistant.
- add make repeated task function
  make_repeated_task allows to wrap a function to be repeated cyclically.
- allow eos to be started with root permissions and drop priviledges
  Home assistant starts all add-ons with root permissions. Eos now drops
  root permissions if an applicable user is defined by paramter --run_as_user.
  The docker image defines the user eos to be used.
- make home assistant add-on run optimization by default
  When running as Home Assistant add-on the only viable usage is running with
  cyclic optimization. Make this the default to als propvide a better experience
  for first time users. The optimization will start with demo data, which also
  helps to configure Akkudoktor-EOS to the personal usage.
- make eos supervise and monitor EOSdash
  Eos now not only starts EOSdash but also monitors EOSdash during runtime
  and restarts EOSdash on fault. EOSdash logging is captured by EOS
  and forwarded to the EOS log to provide better visibility.
- add openmeteo weather provider
- add fixed electricity prediction with time window support
- add duration to string conversion
  Make to_duration to also return the duration as string on request.

### Fixed

- genetic optimizer charge rates and soc accuracy
- energy charts bidding zone in request
- prevent exception when load prediction data is missing
- eosdash startup
  Ensure that EOSdash is only started after EOS configuration is available.
- config eos test setup
  Make the config_eos fixture generate a new instance of the config_eos singleton.
  Use correct env names to setup data folder path.
- startup with no config
  Make cache and measurements complain about missing data path configuration but
  do not bail out.
- soc data preparation and usage for genetic optimization.
  Search for soc measurments 48 hours around the optimization start time.
  Only clamp soc to maximum in battery device simulation.
- dashboard bailout on zero value solution display
  Do not use zero values to calculate the chart values adjustment for display.
- openapi generation script
  Make the script also replace data_folder_path and data_output_path to hide
  real (test) environment pathes.
- development version scheme
  The development versioning scheme is adaptet to fit to docker and
  home assistant expectations. The new scheme is x.y.z and x.y.z.dev'date''hash'.
  Hash is only digits as expected by home assistant. Development version
  is appended by .dev as expected by docker.
- use mean value in interval on resampling for array
  When downsampling data use the mean value of all values within the new
  sampling interval.
- default battery ev soc and appliance wh
  Make the genetic simulation return default values for the
  battery SoC, electric vehicle SoC and appliance load if these
  assets are not used.
- import json string
  Strip outer quotes from JSON strings on import to be compliant to json.loads()
  expectation.
- default interval definition for import data
  Default interval must be defined in lowercase human definition to
  be accepted by pendulum.
- clearoutside schema change

### Chore

- removed index based data sequence access
  Index based data sequence access does not make sense as the sequence can be backed
  by the database. The sequence is now purely time series data.
- refactor eos startup to avoid module import startup
  Avoid module import initialisation expecially of the EOS configuration.
  Config mutation, singleton initialization, logging setup, argparse parsing,
  background task definitions depending on config and environment-dependent behavior
  is now done at function startup.
- introduce retention manager
  A single long-running background task that owns the scheduling of all periodic
  server-maintenance jobs (cache cleanup, DB autosave, …)
- guard against visualization errors in genetic optimization
- improve provider update error handling and add VRM provider settings validation
- canonicalize timezone name for UTC
  Timezone names that are semantically identical to UTC are canonicalized to UTC.
- extend config file migration for default value handling
- extend datetime util test cases
- make version test check for untracked files
  Check for files that are not tracked by git. Version calculation will be
  wrong if these files will not be commited.
- bump pandas to 3.0.0
  Pandas 3.0 now performs inference on the appropriate resolution (a.k.a. unit)
  for the output dtype which may become datetime64[us] (before it was ns). Also
  numeric dtype detection is now more strict which needs a different detection for
  numerics.
- bump pydantic-settings to 2.12.0
  pydantic-settings 2.12.0 under pytest creates a different behaviour. The tests
  were adapted and a workaround was introduced. Also ConfigEOS was adapted
  to allow for fine grain initialization control to be able to switch
  off certain settings such as file settings during test.
- remove sci learn kit from dependencies
  The sci learn kit is not strictly necessary as long as we have scipy.
- add documentation mode guarding for sphinx autosummary
  Sphinx autosummary excecutes functions. Prevent exceptions in case of pure doc
  mode.
- adapt docker-build CI workflow to stricter GitHub handling
- add CodeQL analysis workflow to CI
- Use info logging to report missing optimization parameters
  In parameter preparation for automatic optimization an error was logged for missing paramters.
  Log is now down using the info level.
- make EOSdash use the EOS data directory for file import/ export
  EOSdash use the EOS data directory for file import/ export by default.
  This allows to use the configuration import/ export function also
  within docker images.
- improve EOSdash config tab display
  Improve display of JSON code and add more forms for config value update.
- make docker image file system layout similar to home assistant
  Only use /data directory for persistent data. This is handled as a
  docker volume. The /data volume is mapped to ~/.local/share/net.akkudoktor.eos
  if using docker compose.
- add home assistant add-on development environment
  Add VSCode devcontainer and task definition for home assistant add-on
  development.
- Use uv to manage the virtual environment for development.
  This enormously increases dependency updates.
- improve documentation

## 0.2.0 (2025-11-09)

The most important new feature is **automatic optimization**.
EOS can now independently perform optimization at regular intervals.
This is based on the configured system parameters and forecasts, and also uses supplied
measurement data, such as the current battery SoC.
The result is an energy-management plan as well as the optimization output.
The existing optimization interface using `POST /optimize` remains available and can still
be used as before.

In addition, bugs were fixed and new features were added:

- Automatic optimization creates a **default configuration** if none is provided.
  This is intended to make it easier to create a custom configuration by adapting the default.
- The parameters of the genetic optimization algorithm (number of generations, etc.) are now
  configurable.
- For home appliances, start windows can now be specified (experimental).
- Configuration files from previous versions are converted to the current format on first launch.
- There are now measurement keys that are permanently assigned to a specific device simulation.
  This simplifies providing measurement values for device simulations (e.g. battery SoC).
- The infrastructure and first applications for **feed-in tariff forecasting**
  (currently only fixed tariffs) are now integrated.
- EOSdash has been expanded with new tabs for displaying the **energy-management plan**
  and **predictions**.
- The documentation has been updated and expanded in many places.

### Feat

- Energy-management plan generation based on S2 standard instructions
- Feed-in-tariff prediction support (incl. tests & docs)
- `LoadAkkudoktorAdjusted` load prediction variant
- Standardized measurement keys for battery/EV SoC
- Measurement keys configurable via EOS configuration
- Setup default device configuration for automatic optimization
- Health endpoints show version + last optimization timestamps
- Configuration of genetic algorithm parameters
- Configuration options for home-appliance time windows
- Mitigation of legacy configuration
- Config backup enhancements:

  - Timestamp-based backup IDs
  - API to list backups
  - API to revert to a specific backup
  - EOSdash Admin tab integration

- Pendulum date types via `pydantic_extra_types.pendulum_dt`
- `Time`, `TimeWindow`, `TimeWindowSequence`, and `to_time` helpers in `datetimeutil`
- Extended `DataRecord` with configurable field-like semantics
- EOSdash: Solution view now displays genetic optimization results and aggregated totals
- EOSdash UI:

  - Plan tab
  - Predictions tab
  - Cache management in Admin tab
  - About tab

- Pydantic merge model tests
- Developer profiling entry in Makefile
- Changelog & docs updated for commitizen release flow
- Developer documentation updated
- Improved install & development documentation

### Changed

- Battery simulation

  - Performance improvements
  - Charge + start times now reflect realistic simulation

- Appliance simulation:

  - Time windows may roll over to next day

- Revised load prediction by splitting original `LoadAkkudoktor` into:

  - `LoadAkkudoktor`
  - `LoadAkkudoktorAdjusted`

### Fixed

- Correct URL/path for Akkudoktor forum in README
- Automatic optimization:

  - Reuses previous start solution
  - Interval execution + locking + new endpoints
  - Properly loads required data
  - EV charge-rate migration for proper availability

- Genetic common settings consistently available
- Config markdown generation
- Recognize environment variables on EOS server startup
- Remove `0.0.0.0 → localhost` translation on Windows
- Allow hostnames as well as IPs
- Access Pydantic model fields via class instead of instance
- Down-sampling in `key_to_array`
- `/v1/admin/cache/clear` clears all cache files; added `/clear-expired`
- Use `tzfpy` instead of timezonefinder for more accurate EU timezones
- Explicit provider settings in config instead of union
- ClearOutside weather prediction irradiance calculation
- Test config file priority without `config_eos` fixture
- Complete optimization sample-request documentation
- Replace gitlint with commitizen
- Synchronize pre-commit config with real dependencies
- Add missing `babel` to requirements
- Fix documentation, tests, and implementation around optimization + predictions

### Chore

- Use memory cache for inverter interpolation
- Refactor genetic modules (split config, remove device singleton)
- Rename memory cache to `CacheEnergyManagementStore`
- Use class properties for config/EMS/prediction mixins
- Skip matplotlib debug logs
- Auto-sync Bokeh JS CDN version
- Rename `hello.py` → `about.py` in EOSdash
- Remove EOSdash demo page
- Split server test from system test
- Move doc utils to `generate_config_md.py`
- Improve documentation for pydantic merge models
- Remove pendulum warning from README
- Drop GitHub Discussions from contributing docs
- Rename or reorganize files / classes during refactors

### BREAKING CHANGES

EOS configuration + v1 API have changed:

- `available_charge_rates_percent` removed → replaced by `charge_rate`
- Optimization parameter `hours` → renamed to `horizon_hours`
- Device config must explicitly list devices + properties
- Prediction providers now explicit (instead of union)
- Measurement keys provided as lists
- Feed-in-tariff providers must be explicitly configured
- `/v1/measurement/loadxxx` endpoints removed → use generic measurement endpoints
- `/v1/admin/cache/clear` now clears **all*- cache files;
  `/v1/admin/cache/clear-expired` only clears expired entries

## v0.1.0 (2025-09-30)

### Feat

- added Changelog for 0.0.0 and 0.1.0

## v0.0.0 (2025-09-30)

This version represents one year of development of EOS (Energy Optimization System). From this point forward, release management will be introduced.

### Feat

#### Core Features
- energy Management System (EMS) with battery optimization
- PV (Photovoltaic) forecast integration with multiple providers
- load prediction and forecasting capabilities
- electricity price integration
- VRM API integration for load and PV forecasting
- battery State of Charge (SoC) prediction and optimization
- inverter class with AC/DC charging logic
- electric vehicle (EV) charging optimization with configurable currents
- home appliance scheduling optimization
- horizon validation for shading calculations

#### API & Server
- migration from Flask to FastAPI
- RESTful API with comprehensive endpoints
- EOSdash web interface for configuration and visualization
- Docker support with multi-architecture builds
- web-based visualization with interactive charts
- OpenAPI/Swagger documentation
- configurable server settings (port, host)

#### Configuration & Data Management
- JSON-based configuration system with nested support
- configuration validation with Pydantic
- device registry for managing multiple devices
- persistent caching for predictions and prices
- manual prediction updates
- timezone support with automatic detection
- configurable VAT rates for electricity prices

#### Optimization
- DEAP-based genetic algorithm optimization
- multi-objective optimization (cost, battery usage, self-consumption)
- 48-hour prediction and optimization window
- AC/DC charging decision optimization
- discharge hour optimization
- start solution enforcement
- fitness visualization with violin plots
- self-consumption probability interpolator

#### Testing & Quality
- comprehensive test suite with pytest
- unit tests for major components (EMS, battery, inverter, load, optimization)
- integration tests for server endpoints
- pre-commit hooks for code quality
- type checking with mypy
- code formatting with ruff and isort
- markdown linting

#### Documentation
- conceptual documentation
- API documentation with Sphinx
- ReadTheDocs integration
- Docker setup instructions
- contributing guidelines
- English README translation

#### Providers & Integrations
- PVForecast.Akkudoktor provider
- BrightSky weather provider
- ClearOutside weather provider
- electricity price provider

### Refactor

- optimized Inverter class for improved SCR calculation performance
- improved caching mechanisms for better performance
- enhanced visualization with proper timestamp handling
- updated dependency management with automatic Dependabot updates
- restructured code into logical submodules
- package directory structure reorganization
- improved error handling and logging
- Windows compatibility improvements

### Fix

- cross-site scripting (XSS) vulnerabilities
- ReDoS vulnerability in duration parsing
- timezone and daylight saving time handling
- BrightSky provider with None humidity data
- negative values in load mean adjusted calculations
- SoC calculation bugs
- AC charge efficiency in price calculations
- optimization timing bugs
- Docker BuildKit compatibility
- float value handling in user horizon configuration
- circular runtime import issues
- load simulation data return issues
- multiple optimization-related bugs

### Build

- Python version requirement updated to 3.10+
- added Bandit security checks
- improved credential management with environment variables

#### Dependencies
Major dependencies included in this release:
- FastAPI 0.115.14
- Pydantic 2.11.9
- NumPy 2.3.3
- Pandas 2.3.2
- Scikit-learn 1.7.2
- Uvicorn 0.36.0
- Bokeh 3.8.0
- Matplotlib 3.10.6
- PVLib 0.13.1
- Python-FastHTML 0.12.29

### Notes

#### Development Notes
This version encompasses all development from the initial commit (February 16, 2024) through September 29, 2025. The project evolved from a basic energy optimization concept to a comprehensive energy management system with:
- 698+ commits
- multiple contributor involvement
- continuous integration/deployment setup
- automated dependency updates
- comprehensive testing infrastructure

#### Migration Notes
As this is the initial versioned release, no migration is required. Future releases will include migration guides as needed.