EOS/CHANGELOG.md

# Changelog

All notable changes to the akkudoktoreos project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

## Unreleased

### Added
- Add `FeedInTariffAkkudoktor`, using raw hourly Akkudoktor/aWATTar day-ahead market prices as
  feed-in tariff data without import charges or VAT. Quarter-hour optimization holds each hourly
  value constant for four slots.
- Add `FeedInTariffTibber`, using Tibber's native `QUARTER_HOURLY` spot-price component as a
  strict 15-minute feed-in tariff. Hourly API responses are rejected instead of expanded.
- Flexible consumers (home appliances): schedule any number of consumers via
  `devices.home_appliances`, each with a unique `device_id`. Every consumer defines its
  load **either** as an explicit power profile (`load_profile_power_w` at
  `load_profile_interval_seconds`, energy-preservingly resampled onto the optimization
  slot grid, including the 15-minute interval and non-integer ratios such as 10→15 min)
  **or** as the flat `consumption_wh` + `duration_h` fallback. A `schedule_mode` selects
  `ONCE` (a single run in the horizon) or `DAILY` (one run per local calendar day that
  still has a feasible full run). Allowed start times honour `time_windows` (including
  weekday/date restrictions) and the horizon; ONCE without any valid start is rejected.
  Results are reported per device (`result.home_appliance_energy_wh`, `appliance_starts`,
  per-device solution columns and `DDBCInstruction`s emitted only on RUN/OFF transitions).
- EV Bug (wrong output in genetic.py / no senseful results)
- Direktvermarktung active / Battery discharge into grid (new state / action battery_grid_export_allowed) + (new simulation output Feed_in_tariff)
- New PV forecast providers giving operators more cloud forecast sources to choose from in
  addition to Akkudoktor, VRM and Import:
  - `PVForecastPVNode` — native 15-minute forecasts from the pvnode.com API.
  - `PVForecastForecastSolar` — forecasts from the free Forecast.Solar API.
  - `PVForecastSolcast` — forecasts from the Solcast rooftop-site API. (THX to @chloepriceless)
- 15-minute optimization interval for the genetic optimizer. `optimization.interval`
  now accepts 900 (15 min) in addition to the default 3600 (1 hour), letting the
  optimizer schedule on a quarter-hour grid for 15-minute dynamic electricity
  tariffs. Device power caps and the solution/plan serializers are slot-aware; the
  default 3600 s interval keeps the previous hourly behaviour. The new sub-hourly PV
  providers (pvnode, Forecast.Solar, Solcast) feed their native resolution straight
  into the quarter-hour grid.
  - Legacy hourly API inputs are normalized onto the quarter-hour grid: PV/load energy
    is distributed across four slots, prices are held constant, and hourly warm-start
    solutions are expanded to slot controls. Native slot arrays are preserved and
    ambiguous lengths are rejected.
  - Home-appliance (flexible consumer) scheduling now runs on the same slot grid and
    supports the 15-minute interval (see the flexible consumers entry below).
  - The Tibber electricity price provider now requests native 15-minute exchange prices
    (`priceInfoRange(resolution: QUARTER_HOURLY)`) and stores them at their native
    resolution, so both the hourly and the 15-minute optimizer are fed the correct
    grid. The seasonal price extrapolation is resolution-agnostic and stays identical
    at the default hourly resolution.
- Separate battery economics into two independent settings: battery
  `levelized_cost_of_storage_kwh` is now charged once on delivered DC energy, while
  `optimization.terminal_value_euro_per_kwh` values usable battery energy left at the end of the
  optimization horizon. LCOS applies to both local battery supply and battery-to-grid export and is
  included in hourly and total costs.
- Model direct PV-to-load consumption probabilistically from the bundled conditional minute-load
  table. The expected direct flow is used consistently for PV bypass, residual load, battery
  charging, and grid export on hourly and 15-minute optimization grids.
- Add a dynamic `FeedInTariffEnergyCharts` forecast for direct marketing. Published Energy-Charts
  day-ahead market prices are retained at their native hourly or quarter-hourly resolution and
  missing slots at the end of the optimization horizon are extended with weekly or daily seasonal
  ETS forecasts. A median fallback is used when the available history is too short for ETS.

### Changed
- Seed genetic optimization runs with ten exact warm-start copies, twenty locally mutated
  warm-start neighbours, and diverse domain-informed battery, direct-marketing, EV, and flexible
  appliance schedules to improve early convergence without discarding the previous solution.
- `max_home_appliances` is now purely an upper bound. No demo appliance is created when
  no `home_appliances` are configured, and the number is no longer used as an on/off switch.

### Deprecated
- The single-appliance genetic optimization input `dishwasher` is deprecated in favour of
  the `home_appliances` list; a lone `dishwasher` is mapped to a one-element list, and
  setting both at once is rejected. In the solution, `washingstart` (start slot of a single
  hourly appliance) and `result.Home_appliance_wh_per_hour` (aggregate over all appliances)
  are deprecated in favour of `appliance_starts` and `result.home_appliance_energy_wh`.

### Fixed
- Re-simulate genetic candidates after removing EV charging genes from slots that begin at full
  SoC, keeping the repaired genome and its assigned fitness consistent.
- FeedInTariffEnergyCharts no longer aborts the whole prediction/optimization when the
  Energy-Charts API is briefly unreachable: transient timeouts/connection errors are
  retried (with a (connect, read) timeout of (5, 60) s), and if a fetch still fails while
  historical data exists, the existing history is kept and the remaining slots are
  extrapolated via ETS instead of failing. A genuine cold start (no data at all) still
  fails.
- The deprecated `/gesamtlast` endpoint no longer forces a full provider refresh on every
  call. Forcing bypassed the provider caches and hammered external APIs, so a single flaky
  provider could 404 the whole load prediction. It now defaults to a cache-aware update and
  accepts an optional `force_update` flag in the request body for callers that still want
  to force.

## 0.3.0 (2026-03-17)

Akkudoktor-EOS can now be run as Home Assistant add-on and standalone.
As Home Assistant add-on EOS uses ingress to fully integrate the EOSdash dashboard
in Home Assistant.

Adapters for Home Assistant and NodeRed integration are added. These adapters
provide a simplified interface to these HEMS besides the standard REST interface.

The prediction and measurement data can now be backed by a database. The database allows
to keep historic prediction data and measurement data for long time without keeping
it in memory. The database supports backend selection, compression, incremental data load,
automatic data saving to storage, automatic vacuum and compaction. Two database backends
are integrated and can be configured, LMDB and SQLight3.

New prediction providers allow to access OpenMeteo weather data and to define fixed
electricity prices for configurable time windows.

An anoying bug in the genetic algorithm that created unfeasable battery charge and
discharge amounts is now hopefully fixed.

In addition, bugs were fixed and new features were added.

### Feat

- add inverter AC/DC efficiency and break-even penalty
- add database support for measurements and historic prediction data.
  The prediction and measurement data can now be backed by a database. The database allows
  to keep historic prediction data and measurement data for long time without keeping
  it in memory. Two database backends are integrated and can be configured, LMDB and SQLight3.
- add adapters for integrations
  Adapters for Home Assistant and NodeRED integration are added.
  Akkudoktor-EOS can now be run as Home Assistant add-on and standalone.
  As Home Assistant add-on EOS uses ingress to fully integrate the EOSdash dashboard
  in Home Assistant.
- add make repeated task function
  make_repeated_task allows to wrap a function to be repeated cyclically.
- allow eos to be started with root permissions and drop priviledges
  Home assistant starts all add-ons with root permissions. Eos now drops
  root permissions if an applicable user is defined by paramter --run_as_user.
  The docker image defines the user eos to be used.
- make home assistant add-on run optimization by default
  When running as Home Assistant add-on the only viable usage is running with
  cyclic optimization. Make this the default to als propvide a better experience
  for first time users. The optimization will start with demo data, which also
  helps to configure Akkudoktor-EOS to the personal usage.
- make eos supervise and monitor EOSdash
  Eos now not only starts EOSdash but also monitors EOSdash during runtime
  and restarts EOSdash on fault. EOSdash logging is captured by EOS
  and forwarded to the EOS log to provide better visibility.
- add openmeteo weather provider
- add fixed electricity prediction with time window support
- add duration to string conversion
  Make to_duration to also return the duration as string on request.

### Fixed

- genetic optimizer charge rates and soc accuracy
- energy charts bidding zone in request
- prevent exception when load prediction data is missing
- eosdash startup
  Ensure that EOSdash is only started after EOS configuration is available.
- config eos test setup
  Make the config_eos fixture generate a new instance of the config_eos singleton.
  Use correct env names to setup data folder path.
- startup with no config
  Make cache and measurements complain about missing data path configuration but
  do not bail out.
- soc data preparation and usage for genetic optimization.
  Search for soc measurments 48 hours around the optimization start time.
  Only clamp soc to maximum in battery device simulation.
- dashboard bailout on zero value solution display
  Do not use zero values to calculate the chart values adjustment for display.
- openapi generation script
  Make the script also replace data_folder_path and data_output_path to hide
  real (test) environment pathes.
- development version scheme
  The development versioning scheme is adaptet to fit to docker and
  home assistant expectations. The new scheme is x.y.z and x.y.z.dev'date''hash'.
  Hash is only digits as expected by home assistant. Development version
  is appended by .dev as expected by docker.
- use mean value in interval on resampling for array
  When downsampling data use the mean value of all values within the new
  sampling interval.
- default battery ev soc and appliance wh
  Make the genetic simulation return default values for the
  battery SoC, electric vehicle SoC and appliance load if these
  assets are not used.
- import json string
  Strip outer quotes from JSON strings on import to be compliant to json.loads()
  expectation.
- default interval definition for import data
  Default interval must be defined in lowercase human definition to
  be accepted by pendulum.
- clearoutside schema change

### Chore

- removed index based data sequence access
  Index based data sequence access does not make sense as the sequence can be backed
  by the database. The sequence is now purely time series data.
- refactor eos startup to avoid module import startup
  Avoid module import initialisation expecially of the EOS configuration.
  Config mutation, singleton initialization, logging setup, argparse parsing,
  background task definitions depending on config and environment-dependent behavior
  is now done at function startup.
- introduce retention manager
  A single long-running background task that owns the scheduling of all periodic
  server-maintenance jobs (cache cleanup, DB autosave, …)
- guard against visualization errors in genetic optimization
- improve provider update error handling and add VRM provider settings validation
- canonicalize timezone name for UTC
  Timezone names that are semantically identical to UTC are canonicalized to UTC.
- extend config file migration for default value handling
- extend datetime util test cases
- make version test check for untracked files
  Check for files that are not tracked by git. Version calculation will be
  wrong if these files will not be commited.
- bump pandas to 3.0.0
  Pandas 3.0 now performs inference on the appropriate resolution (a.k.a. unit)
  for the output dtype which may become datetime64[us] (before it was ns). Also
  numeric dtype detection is now more strict which needs a different detection for
  numerics.
- bump pydantic-settings to 2.12.0
  pydantic-settings 2.12.0 under pytest creates a different behaviour. The tests
  were adapted and a workaround was introduced. Also ConfigEOS was adapted
  to allow for fine grain initialization control to be able to switch
  off certain settings such as file settings during test.
- remove sci learn kit from dependencies
  The sci learn kit is not strictly necessary as long as we have scipy.
- add documentation mode guarding for sphinx autosummary
  Sphinx autosummary excecutes functions. Prevent exceptions in case of pure doc
  mode.
- adapt docker-build CI workflow to stricter GitHub handling
- add CodeQL analysis workflow to CI
- Use info logging to report missing optimization parameters
  In parameter preparation for automatic optimization an error was logged for missing paramters.
  Log is now down using the info level.
- make EOSdash use the EOS data directory for file import/ export
  EOSdash use the EOS data directory for file import/ export by default.
  This allows to use the configuration import/ export function also
  within docker images.
- improve EOSdash config tab display
  Improve display of JSON code and add more forms for config value update.
- make docker image file system layout similar to home assistant
  Only use /data directory for persistent data. This is handled as a
  docker volume. The /data volume is mapped to ~/.local/share/net.akkudoktor.eos
  if using docker compose.
- add home assistant add-on development environment
  Add VSCode devcontainer and task definition for home assistant add-on
  development.
- Use uv to manage the virtual environment for development.
  This enormously increases dependency updates.
- improve documentation

## 0.2.0 (2025-11-09)

The most important new feature is **automatic optimization**.
EOS can now independently perform optimization at regular intervals.
This is based on the configured system parameters and forecasts, and also uses supplied
measurement data, such as the current battery SoC.
The result is an energy-management plan as well as the optimization output.
The existing optimization interface using `POST /optimize` remains available and can still
be used as before.

In addition, bugs were fixed and new features were added:

- Automatic optimization creates a **default configuration** if none is provided.
  This is intended to make it easier to create a custom configuration by adapting the default.
- The parameters of the genetic optimization algorithm (number of generations, etc.) are now
  configurable.
- For home appliances, start windows can now be specified (experimental).
- Configuration files from previous versions are converted to the current format on first launch.
- There are now measurement keys that are permanently assigned to a specific device simulation.
  This simplifies providing measurement values for device simulations (e.g. battery SoC).
- The infrastructure and first applications for **feed-in tariff forecasting**
  (currently only fixed tariffs) are now integrated.
- EOSdash has been expanded with new tabs for displaying the **energy-management plan**
  and **predictions**.
- The documentation has been updated and expanded in many places.

### Feat

- Energy-management plan generation based on S2 standard instructions
- Feed-in-tariff prediction support (incl. tests & docs)
- `LoadAkkudoktorAdjusted` load prediction variant
- Standardized measurement keys for battery/EV SoC
- Measurement keys configurable via EOS configuration
- Setup default device configuration for automatic optimization
- Health endpoints show version + last optimization timestamps
- Configuration of genetic algorithm parameters
- Configuration options for home-appliance time windows
- Mitigation of legacy configuration
- Config backup enhancements:

  - Timestamp-based backup IDs
  - API to list backups
  - API to revert to a specific backup
  - EOSdash Admin tab integration

- Pendulum date types via `pydantic_extra_types.pendulum_dt`
- `Time`, `TimeWindow`, `TimeWindowSequence`, and `to_time` helpers in `datetimeutil`
- Extended `DataRecord` with configurable field-like semantics
- EOSdash: Solution view now displays genetic optimization results and aggregated totals
- EOSdash UI:

  - Plan tab
  - Predictions tab
  - Cache management in Admin tab
  - About tab

- Pydantic merge model tests
- Developer profiling entry in Makefile
- Changelog & docs updated for commitizen release flow
- Developer documentation updated
- Improved install & development documentation

### Changed

- Battery simulation

  - Performance improvements
  - Charge + start times now reflect realistic simulation

- Appliance simulation:

  - Time windows may roll over to next day

- Revised load prediction by splitting original `LoadAkkudoktor` into:

  - `LoadAkkudoktor`
  - `LoadAkkudoktorAdjusted`

### Fixed

- Correct URL/path for Akkudoktor forum in README
- Automatic optimization:

  - Reuses previous start solution
  - Interval execution + locking + new endpoints
  - Properly loads required data
  - EV charge-rate migration for proper availability

- Genetic common settings consistently available
- Config markdown generation
- Recognize environment variables on EOS server startup
- Remove `0.0.0.0 → localhost` translation on Windows
- Allow hostnames as well as IPs
- Access Pydantic model fields via class instead of instance
- Down-sampling in `key_to_array`
- `/v1/admin/cache/clear` clears all cache files; added `/clear-expired`
- Use `tzfpy` instead of timezonefinder for more accurate EU timezones
- Explicit provider settings in config instead of union
- ClearOutside weather prediction irradiance calculation
- Test config file priority without `config_eos` fixture
- Complete optimization sample-request documentation
- Replace gitlint with commitizen
- Synchronize pre-commit config with real dependencies
- Add missing `babel` to requirements
- Fix documentation, tests, and implementation around optimization + predictions

### Chore

- Use memory cache for inverter interpolation
- Refactor genetic modules (split config, remove device singleton)
- Rename memory cache to `CacheEnergyManagementStore`
- Use class properties for config/EMS/prediction mixins
- Skip matplotlib debug logs
- Auto-sync Bokeh JS CDN version
- Rename `hello.py` → `about.py` in EOSdash
- Remove EOSdash demo page
- Split server test from system test
- Move doc utils to `generate_config_md.py`
- Improve documentation for pydantic merge models
- Remove pendulum warning from README
- Drop GitHub Discussions from contributing docs
- Rename or reorganize files / classes during refactors

### BREAKING CHANGES

EOS configuration + v1 API have changed:

- `available_charge_rates_percent` removed → replaced by `charge_rate`
- Optimization parameter `hours` → renamed to `horizon_hours`
- Device config must explicitly list devices + properties
- Prediction providers now explicit (instead of union)
- Measurement keys provided as lists
- Feed-in-tariff providers must be explicitly configured
- `/v1/measurement/loadxxx` endpoints removed → use generic measurement endpoints
- `/v1/admin/cache/clear` now clears **all*- cache files;
  `/v1/admin/cache/clear-expired` only clears expired entries

## v0.1.0 (2025-09-30)

### Feat

- added Changelog for 0.0.0 and 0.1.0

## v0.0.0 (2025-09-30)

This version represents one year of development of EOS (Energy Optimization System). From this point forward, release management will be introduced.

### Feat

#### Core Features
- energy Management System (EMS) with battery optimization
- PV (Photovoltaic) forecast integration with multiple providers
- load prediction and forecasting capabilities
- electricity price integration
- VRM API integration for load and PV forecasting
- battery State of Charge (SoC) prediction and optimization
- inverter class with AC/DC charging logic
- electric vehicle (EV) charging optimization with configurable currents
- home appliance scheduling optimization
- horizon validation for shading calculations

#### API & Server
- migration from Flask to FastAPI
- RESTful API with comprehensive endpoints
- EOSdash web interface for configuration and visualization
- Docker support with multi-architecture builds
- web-based visualization with interactive charts
- OpenAPI/Swagger documentation
- configurable server settings (port, host)

#### Configuration & Data Management
- JSON-based configuration system with nested support
- configuration validation with Pydantic
- device registry for managing multiple devices
- persistent caching for predictions and prices
- manual prediction updates
- timezone support with automatic detection
- configurable VAT rates for electricity prices

#### Optimization
- DEAP-based genetic algorithm optimization
- multi-objective optimization (cost, battery usage, self-consumption)
- 48-hour prediction and optimization window
- AC/DC charging decision optimization
- discharge hour optimization
- start solution enforcement
- fitness visualization with violin plots
- self-consumption probability interpolator

#### Testing & Quality
- comprehensive test suite with pytest
- unit tests for major components (EMS, battery, inverter, load, optimization)
- integration tests for server endpoints
- pre-commit hooks for code quality
- type checking with mypy
- code formatting with ruff and isort
- markdown linting

#### Documentation
- conceptual documentation
- API documentation with Sphinx
- ReadTheDocs integration
- Docker setup instructions
- contributing guidelines
- English README translation

#### Providers & Integrations
- PVForecast.Akkudoktor provider
- BrightSky weather provider
- ClearOutside weather provider
- electricity price provider

### Refactor

- optimized Inverter class for improved SCR calculation performance
- improved caching mechanisms for better performance
- enhanced visualization with proper timestamp handling
- updated dependency management with automatic Dependabot updates
- restructured code into logical submodules
- package directory structure reorganization
- improved error handling and logging
- Windows compatibility improvements

### Fix

- cross-site scripting (XSS) vulnerabilities
- ReDoS vulnerability in duration parsing
- timezone and daylight saving time handling
- BrightSky provider with None humidity data
- negative values in load mean adjusted calculations
- SoC calculation bugs
- AC charge efficiency in price calculations
- optimization timing bugs
- Docker BuildKit compatibility
- float value handling in user horizon configuration
- circular runtime import issues
- load simulation data return issues
- multiple optimization-related bugs

### Build

- Python version requirement updated to 3.10+
- added Bandit security checks
- improved credential management with environment variables

#### Dependencies
Major dependencies included in this release:
- FastAPI 0.115.14
- Pydantic 2.11.9
- NumPy 2.3.3
- Pandas 2.3.2
- Scikit-learn 1.7.2
- Uvicorn 0.36.0
- Bokeh 3.8.0
- Matplotlib 3.10.6
- PVLib 0.13.1
- Python-FastHTML 0.12.29

### Notes

#### Development Notes
This version encompasses all development from the initial commit (February 16, 2024) through September 29, 2025. The project evolved from a basic energy optimization concept to a comprehensive energy management system with:
- 698+ commits
- multiple contributor involvement
- continuous integration/deployment setup
- automated dependency updates
- comprehensive testing infrastructure

#### Migration Notes
As this is the initial versioned release, no migration is required. Future releases will include migration guides as needed.