Add new provider class for electricity fees providers.
Add the generic providers:
- ElecFeeFixed
- ElecFeeImport
The providers provide predictions for:
- elecfee_consumption_amt_wh:
Total fixed fee for consumed energy per Wh [amount/Wh]. This is the accumulation of all
fixed per-Wh fees payable on "consumed energy - such as network charge, concession fee,
and electricity charge - into a single amount.
- elecfee_consumption_percent_amt:
Total fixed surcharge on consumed energy, given as a percentage of the monetary amount
already charged for that energy [%]. This is the accumulation of all percentage-based
surcharges payable on top of the consumed-energy fee - such as VAT - into a single
percentage. This is a percentage of the fee amount, not a per-Wh rate.
- elecfee_feedin_amt_wh:
Total fixed deduction from feed-in energy per Wh [amount/Wh]. This is the accumulation of
all fixed per-Wh charges deducted from feed-in energy - such as metering fees or
grid-operator handling "charges - into a single amount. Applied after the percentage-based
deduction, i.e. it reduces the price by a flat amount per Wh rather than by a share of the
raw price.
- elecfee_feedin_percent_amt:
Total percentage deducted from the raw feed-in price (spot price) [%]. This is the
accumulation of all percentage-based deductions payable on the feed-in tariff - such as a
marketing or balancing fee retained by the aggregator - into a single percentage. It is
applied as `raw_price * (100 - percent) / 100`, i.e. it scales down the raw price rather
than adding a surcharge to it.
A new _apply_fee() method is added to the base class for ElecPrice and FeedInTariff to be used to
add the fees in a consistent way. Fees are taken from the active ElecFee provider and applied
to the raw prices given to the _apply_fee() method.
The optional application of fees is added to:
- ElecPriceAkkudoktor
- ElecPriceFixed
- ElecPriceEnergyCharts
- ElecPriceSMARD
- FeedInTariffEnergyCharts
- FeedInTariffFixed
- FeedInTariffSMARD
The import providers ElecPriceImport and FeedInTariffImport do not apply fees by intentention.
The following providers currently do not handle fees defined by ElecFee:
- ElecPriceTibber
- FeedInTariffAkkudoktor
- FeedInTariffDvhubOnline
- FeedInTariffTibber
The tests for this feature are either added or existing tests are extended.
The documentation was extended for the electricity fee provider settings.
Besides this feature further improvements are added:
* feat: add SMARD quarter-hour electricty price and feed-in tariff provider
* feat: to_series method for TimeWindows and ValueTimeWindows
Additional to to_array the time window sequence can now also produce a pandas series.
Test have been extended to cover the series generation.
* feat: use time windows in fixed feedin tariff provider
Feedin tariff can now be configured by time windows - not a single value.
* feat: EOSdash select for PVLib inverters and modules
Provide PVLib inverter and module names in config selection.
* feat: EOSdash lazy select for big option sets
Add a new form for lazy selection of big option sets. Filtering and
generation of the option set is done server-side.
* fix: use raw data for ETS/ median prediction
Use to raw time series data for ETS/ median prediction to avoid interference
by e.g. dynamic grid charges.
* fix: EOSdash config drops by type only on details resolve
Drop configuration by type and path. Prevents dropping of configuration items
with same type and level but different path.
* fix: EOSdash configuration section closes on update
Open section if searching or if last update touched this category — including
updates on deeply nested sub-fields.
* chore: make elecfeefixed, elecpricefixed and feedintarifffixed warn about no windows and default to 0
Missining configuration creates default 0 value and a warning instead of an exception.
* fix: test setup for providers
Reset db state on each test run.
* chore: improve config option naming for elecpricefixed.
* chore: adapt elecpricefixed test to changed time_windows naming
* chore: factorized common price provider helpers to priceabc.py
Factorized common price provider helpers to priceabc.py. Add tests for these helpers.
Reduce/ change testing of elecpriceabc.py and feedintariffabc.py to cover
only specifics. Rest of testing is already covered by test_priceabc.py.
* chore: update version
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
### Significant changes / EOS impact
The dependency bumps are mostly patch/tooling updates, but a few are worth explicit review:
- **`pydantic-settings` 2.14.2 → 2.15.0 — highest runtime impact.** EOS directly subclasses `pydantic_settings.BaseSettings` for `SettingsEOS` / `ConfigEOS`, with `case_sensitive` left at its default. In 2.15.0, `case_sensitive` now also applies to init kwargs and config-file sources, so top-level setting keys are now matched case-insensitively by default where those sources previously did not behave that way. This can change how differently-cased config keys are accepted/resolved. The release also adds warnings for unresolved forward references and changes strict non-JSON env-value failures to `ValidationError`. **Recommended:** exercise JSON config loading, init/update paths, env overrides, and config round-trips.
- **`tzfpy` 1.3.2 → 1.3.3 — runtime correctness change.** EOS uses `tzfpy.get_tz()` in `to_timezone()` and exposes the result through `GeneralSettings.timezone`. Queries exactly on timezone polygon borders now resolve instead of returning an empty result, and the `America/Argentina/Ushuaia` boundary is corrected. This can intentionally change timezone output for users on/near affected boundaries.
- **`cachebox` 6.2.2 → 6.2.5 — runtime cache correctness/safety.** EOS uses `cachebox.LRUCache` and `cachebox.cached` for the energy-management cache. The update fixes iterator lifetime safety, LRU iterator invalidation when reads promote entries, and a potential `setdefault_with` locking issue. EOS does not appear to rely on those edge cases directly, so this is expected to be low-risk and mostly corrective; existing cache tests are the relevant regression coverage.
- **`GitPython` 3.1.58 → 3.1.59 — dev/tooling security hardening.** This release blocks file-reading Git options, separate git-directory use during clone, and hardens config parsing. It appears to be a dev/docs dependency rather than EOS runtime code, but CI/tooling that intentionally passes unusual Git options could be affected.
- **`pre-commit` 4.6.1 → 4.6.2 — dev-only bug fix.** Fixes a regression in Node-language hooks using npm 11.x build scripts.
- **`mypy` 2.3.1, `commitizen` 4.17.1, and `types-PyYaml` stub update** are tooling/type-checking changes with no expected EOS runtime behavior change.
Overall, the main compatibility focus should be **configuration handling (`pydantic-settings`)**, followed by **timezone edge cases (`tzfpy`)**. The remaining updates are primarily correctness, security, or developer-tooling fixes.
Consolidates the currently applicable dependency updates into one PR, including the closed Dependabot backlog such as #1241, plus dependency surfaces that were not covered by the repository's previous pip-only Dependabot configuration.
Cleanup of test warnings.
Most importent:
* GitPython + pypdf security hardening.
* Uvicorn WebSocket close/backpressure/header fixes for server/dashboard reliability.
* FastAPI dependency-memory/OpenAPI improvements for the API process.
* Bokeh WebSocket/resource-leak/prefix fixes for EOSdash and proxied deployments.
* cachebox cancellation/lock cleanup fixes for long-running/concurrent work.
* pandas 3.0.5 avoiding the yanked 3.0.4 datetime/segfault build.
* Ruff security-lint and pydocstyle correctness fixes, plus faster release builds via PGO.
* platformdirs malformed-XDG and duplicate-directory fixes for deployment portability.
* CI action modernization, regenerated uv.lock, and expanded Dependabot coverage.
Runtime dependencies
cachebox: 6.1.2 → 6.2.2
fastapi: 0.139.2 → 0.141.1
python-fasthtml: 0.14.9 → 0.14.11
MonsterUI: 1.0.46 → 1.0.47
bokeh: 3.9.1 → 3.9.2
uvicorn: 0.51.0 → 0.52.4 (build(deps): bump uvicorn from 0.51.0 to 0.52.3 #1241, refreshed to latest patch)
pandas: 3.0.3 → 3.0.5
platformdirs: 4.11.0 → 4.11.3
Development/test dependencies
pandas-stubs: 3.0.3.260530 → 3.0.5.260730
types-PyYAML: 6.0.12.20260518 → 6.0.12.20260724
GitPython: 3.1.53 → 3.1.58 (security/fix releases)
coverage: 7.15.2 → 7.15.4
pypdf: 6.14.2 → 6.16.1 (includes security fixes)
Pre-commit/tooling
ruff-pre-commit: v0.15.21 → v0.16.3
synchronize pandas-stubs, types-docutils, and types-PyYAML pins with pyproject.toml
CI / repository dependencies
Python 3.13.9 → 3.13.15 in CI, Docker, .env, and local Docker Make targets
actions/checkout → v7 in pytest, pre-commit, CodeQL, and release workflows
actions/setup-python → v7 in pytest, pre-commit, and release workflows
actions/upload-artifact → v7 in pytest workflow
actions/stale: v9.1.0 → v11.0.0 (SHA-pinned)
regenerate uv.lock from the final dependency pins so locked/frozen installs match pyproject.toml
Future update coverage
Expand Dependabot from pip-only to also monitor:
GitHub Actions
Docker
The existing open docutils 0.23 update (#1085) is intentionally excluded because it has separate compatibility/ignore handling and should remain isolated.
docker-build.yml was audited and is already using the newer action generations, so no changes were needed there.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Change configuration source priorities to:
- cli
- environment vars
- dotenv settings
- config file settings
- init settings
By this the environment vars supersede any configuration var
provided by the configuration file or by the initialisation
with pydantic.
The test_config.py::test_computed_path was fixed to to not
use the defaul env var overwrite defined by conftest.py.
This seemed to indicate non working env vars, but in fact
was a test setupt fault.
Besides this fix there are other fixes and changes added:
* fix: exclude computed fields when merging settings
Pydantic may overwrite settings by values given for computed
fields and use these values instead of re-computing the field.
Avoid computed fields in merging settings.
* chore: improve Windows compatability of development setup
Improve scripts to better run also on Windows. When doing path
checks keep compatibility also to Windows pathes. A lot of
changes to avoid the famous Windows CRLF handling and keep
line endings to LF.
* chore: add development hint for Windows
Windows developers should set core.autocrlf to false.
* chore: update version
Signed-off-by: b0661 <b0661n0e17e@gmail.com>
Change PDF visualization to be created on demand and per optimization algorithm. The PDF
for the GENETIC0 optimization is provided by the /visualization_results.pdf endpoint.
There is no change in the interface.
By this the optimization algorithm is offloaded from the PDF generation which spares some
time.
To cope with several users may call the /visualization_results.pdf endpoint at the same
time the PDF is generated on the fly without any intermediate file taking the stored
GENETIC0 solution as an input. SVG picture generation is removed as this would again
create intermediate files. Chart pictures can easily be taken from the PDF.
To allow on demand creation of the optimization results visualization the optimisation
solution stored is extended by several new attributes. To keep the deprecated
/optimize endpoint compatible the optimization solution is stripped to the legacy
content before returned. Due to the extension of the solution the optimization tests were
adapted to cover the extended content.
The optimization tests are adapted to test the generated visualization report by
the pypdf reader. Pypdf is added to the development dependencies.
Besides the adaptation several fixes and improvements are added:
* feat: extend /v1/prediction/series endpoint by resampling and filling
Add parameters for resampling and filling. Add the processing parameter
to control wether raw data or resampled data shall be returned.
* feat: extend /v1/measurement/series endpoint by resampling and filling
Add parameters for resampling and filling: Add the processing parameter
to control wether raw data or resampled data shall be returned.
* feat: standardize and improve API error response
Use FASTApi exception handlers to provide a standardized API exception handling.
All exceptions are logged.
Exception traces are only returned if the new logging configuration parameter
logging.api_logging_level is set to "DEBUG" or "TRACE". Avoids unwanted leackage
of server internals on exceptions.
* fix: align to intervall when resampling
Ensure resampling is aligned to interval also when the buckets are shifted due to the
align_to_intervall parameter is set.
* chore: make dropna mandatory and default to True
* chore: refactor key_to_xxx data management methods
Make key_to_series the central method for data resampling and fill.
Add a new key_to_raw_series to retrieve the data as it is stored
(without resampling and filling).
Users of key_to_series were mostly moved to key_to_raw_series as this resembles
the former interface. Especially in predictions and tests this was done.
* chore: create test data sub-directory for each optimization algorithm
To prevent cluttering the test data directory and ease test data management for
optimization algorithms each algorithm got it's own sub-directory. The current
test data was moved to these sub-directories.
* chore: update version
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
The `FeedInTariffTibber` provider requests `priceInfo` and `priceInfoRange` with
`resolution: QUARTER_HOURLY` and preserves the native 15-minute timestamps. It uses Tibber's
`energy` spot-price component without the `tax` part or EOS electricity-price charges. The
end-customer `total` component is deliberately ignored.
The provider deliberately rejects hourly API responses instead of silently repeating them. It
reuses `elecprice.tibber.access_token` and `elecprice.tibber.home_id`, so no duplicate credentials
are needed.
Signed-off-by: Andreas Schmitz <akkudoktor.net>
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
The `FeedInTariffAkkudoktor` provider uses raw day-ahead market prices from
`https://api.akkudoktor.net/prices` as `feed_in_tariff_wh`. It does not add electricity import
charges or VAT. Published prices are extended to the configured prediction horizon with the same
seasonal ETS or median fallback used by the Akkudoktor electricity-price provider.
The Akkudoktor endpoint currently forwards hourly market prices from aWATTar. With a 15-minute
optimization interval, EOS holds each hourly price constant for its four quarter-hour slots. This
keeps the slot grid consistent but does not create genuine quarter-hour market prices.
Signed-off-by: Andreas Schmitz <akkudoktor.net>
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
The `FeedInTariffEnergyCharts` provider uses the raw Energy-Charts day-ahead market price as the
feed-in tariff. It stores prices in `feed_in_tariff_wh` without adding electricity import charges
or VAT. The data is loaded from the Energy-Charts `/price` endpoint for the configured bidding
zone. The native Energy-Charts resolution, including quarter-hour data, is retained.
Energy-Charts usually supplies prices only for the published day-ahead period. If that data does
not cover the complete configured prediction horizon, the provider extends it as follows:
- With more than 800 hours of history, an ETS (Holt-Winters exponential smoothing) forecast with
weekly seasonality is used.
- With more than 168 hours of history, an ETS forecast with daily seasonality is used.
- With less history, the median of the available values is used as a constant fallback.
The seasonal periods are adjusted to the source resolution. For example, quarter-hour data uses
four values per hour. Values already supplied by Energy-Charts are kept unchanged; only missing
future slots after the last published price are forecast. Consequently, a 15-minute optimization
uses four forecast values per hour without converting them to hourly averages.
Signed-off-by: Andreas Schmitz <akkudoktor.net>
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
When running as a Home Assistant add-on the ports shall not be changed
as this would break config.yaml that is used by Home Assistant. Prevent
the change and return an error message.
Extra fixes:
* fix: EOS configuration initialisation by cli under EOSdash.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Fix test warnings left over from data management conversion to async design.
Mostly tagging async tests.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
The akkudoktor api for PV forecast seems to be changed and does not support to
concatenate several planes into one request anymore.
Make a request for every plane and add up the power results. Do not use the
new 15 minutes slots as the returned data does not include the hourly values
for windspeed and temperature as does the hourly slots.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Configuration for some providers was given in the sub-field `provider_settings`
combining the settings of several providers. Pydantic does not understand
this very well and the configuration became cumbersome, especially in EOSdash.
All provider settings from the `provider-settings` sub-field are now lifted
to the top level field of the configuration.
The changes are automatically migrated in the configuration.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
FAstAPI is an async framework. Data may be imported and exported, load and save, set and get
asynchronously. Prevent interleaving data operations to corrupt the data. In the previous design
sync and async data access was intermixed leading to data corruption.
The basic data classes DataSequence and DataContainer and the derived classes like Provider and
Measurement now are async. Data access is protected by several async locks.
To support the async design of the data classes the database interface became async.
The energy management is also adapted to the new async design. Optimization is still off-loaded
to another thread, but the prepration for the optimization and the post optimization actions now
follow the async design.
Adapter operations are now also protected by async locks.
Tests were adapted to the async design and new tests were created.
Besides this major fix several other improvements and fixes are included in this PR.
* fix: key_to_dict/list/array only regard data records with key value set.
Before the exclusion of no value data records was only done if the dropna flag was set.
* fix: test for visual result pdf generation
Due to updates in the library the generated charts text was a little bit different.
Adapt the test to create the comaprison pdf in the test data durectory and
update the reference pdf.
* chore: Remove MutableMapping from DataSequence and DataContainer.
Mutable Mapping does not fit to the now async design.
* chore: Add NoDB database backend
This backend implements the full database backend interface but performs
no actual persistence. It is intended for configurations where database
persistence is disabled (`provider=None`).
* chore: Improve measurement data import testing with real world scenarios.
Added two new endpoints to support testing.
* chore: Add mermaid to supported documentation tools
* chore: Add documentation about async design
* chore: Add documentation about generic data handling
Covers the basics of measurement and prediction time series data handling.
* chore: Add empty lines around markdown lists.
* chore: sync pre-commit config to updated package versions
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Fix documentation for the loadforecast_power_w key.
Fix documentation to explain the usage of import file/ JSON string to
primarily initialise prediction data.
Fix code scanning alert no. 6: URL redirection from remote source
Enable to automatically save the configuration to the configuration file
by default, which is a widespread user expectation.
Make the genetic parameters non optional for better pydantic compliance.
Update:
- bump pytest to 9.0.3
- bump pillow to 12.2.0
- bump platformdirs to 4.9.6
- bump typespyyaml to 6.0.12.20260408
- bump tzfpy to 1.2.0
- bump pydantic to 2.13.0
- bump types-requests to 2.33.0.20260408
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: Normann <github@koldrack.com>
Make openmeteo test robust against timing issues.
Also update:
- types-docutils==0.22.3.20260322
- pytest-cov==7.1.0
- ruff-pre-commit v0.15.7
- uvlock
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
This pull request prepares release **v0.3.0**.
Akkudoktor-EOS can now be run as Home Assistant add-on and standalone.
As Home Assistant add-on EOS uses ingress to fully integrate the EOSdash dashboard
in Home Assistant.
Adapters for Home Assistant and NodeRed integration are added. These adapters
provide a simplified interface to these HEMS besides the standard REST interface.
The prediction and measurement data can now be backed by a database. The database allows
to keep historic prediction data and measurement data for long time without keeping
it in memory. The database supports backend selection, compression, incremental data load,
automatic data saving to storage, automatic vacuum and compaction. Two database backends
are integrated and can be configured, LMDB and SQLight3.
New prediction providers allow to access OpenMeteo weather data and to define fixed
electricity prices for configurable time windows.
An anoying bug in the genetic algorithm that created unfeasable battery charge and
discharge amounts is now hopefully fixed.
In addition, bugs were fixed and new features were added.
See CHANGELOG for full details.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Ensure that the bidding zone in the request is correctly set to a
string value (not an enum).
This seems to be also an issue with python version < 3.11. Add safeguards
to only use python >= 3.11. Still keep a regression test for the enum
conversion to string.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Add a fixed electricity prediction that supports prices per time window.
The time windows may flexible be defined by day or date.
The prediction documentation is updated to also cover the ElecPriceFixed
provider.
The feature includes several changes that are not directly related to the
electricity price prediction implementation but are necessary to keep
EOS running properly and to test and document the changes.
* feat: add value time windows
Add time windows with an associated float value.
* feat: harden eos measurements endpoints error detection and reporting
Cover more errors that may be raised during endpoint access. Report the
errors including trace information to ease debugging.
* feat: extend server configuration to cover all arguments
Make the argument controlled options also available in server configuration.
* fix: eos config configuration by cli arguments
Move the command line argument handling to config eos so that it is
excuted whenever eos config is rebuild or reset.
* chore: extend measurement endpoint system test
* chore: refactor time windows
Move time windows to configabc as they are only used in configurations.
Also move all tests to test_configabc.
* chore: provide config update errors in eosdash with summarized error text
If there is an update error provide the error text as a summary. On click
provide the full error text.
* chore: force eosdash ip address and port in makefile dev run
Ensure eosdash ip address and port are correctly set for development runs.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Validate solution prediction data before processing.
If required prediction data is missing, the prediction is skipped
instead of raising an exception.
Introduce a new configuration file saving policy to improve loading robustness:
- Exclude computed fields
- Exclude fields set to their default values
- Exclude fields with value None
- Use field aliases
- Recursively remove empty dictionaries and lists
- Ensure general.version is always present and correctly set
When loading older configuration files, computed fields are now stripped
before migration. This further improves backward compatibility and loading
robustness.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Adapt uvicorn log level to allowed levels.
Ensure that EOSdash is started after EOS configuration is available.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Upgrade to:
- pandas==3.01
- fastapi[standard-no-fastapi-cloud-cli]==0.132.0
- fastapi_cli==0.0.23
- MonsterUI==1.0.44
- uvicorn==0.41.0
Close database in database fixture on teardown.
Fix file exclusion in hash and version date generation.
Update version information in documentation.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Home Assistant expects versioning always increases numbers. Add
a date component to the development version to comply with this
expectation. The scheme is now 0.0.0.dev<date><hash>.
Use uv for creating and managing the virtual environment for developement.
This enourmously speeds up dependency updates. For this change
dependency requirements are now solely handled in pyproject.toml.
requirements.tx and requirements-dev.txt are deleted.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Bobby NoelteandGitHub
NormannandGitHub
