15 Commits

Include renames

Author	SHA1	Message	Date
Bobby NoelteandGitHub	886c93c92b	fix: default server settings prevent env var config (#1234) ... Change configuration source priorities to: - cli - environment vars - dotenv settings - config file settings - init settings By this the environment vars supersede any configuration var provided by the configuration file or by the initialisation with pydantic. The test_config.py::test_computed_path was fixed to to not use the defaul env var overwrite defined by conftest.py. This seemed to indicate non working env vars, but in fact was a test setupt fault. Besides this fix there are other fixes and changes added: * fix: exclude computed fields when merging settings Pydantic may overwrite settings by values given for computed fields and use these values instead of re-computing the field. Avoid computed fields in merging settings. * chore: improve Windows compatability of development setup Improve scripts to better run also on Windows. When doing path checks keep compatibility also to Windows pathes. A lot of changes to avoid the famous Windows CRLF handling and keep line endings to LF. * chore: add development hint for Windows Windows developers should set core.autocrlf to false. * chore: update version Signed-off-by: b0661 <b0661n0e17e@gmail.com>	2026-08-16 13:56:25 +02:00
Bobby NoelteandGitHub	894790f577	feat: add pvlib pv forecast provider (#1214) ... Add a PV forecast provider that calculates the forecast using a PVLib system model and weather forecast from the EOS weather forecast provider. Additional module and inverter models can be easily added as the database is build from PVLib and SAM databases and a bundled csv file. The module model and inververt model names are provided by new endpoints to be used in configuration. The provider is based on the fantastic work of EMHASS. See https://github.com/davidusb-geek/emhass/blob/master/src/emhass/forecast.py A short description of the provider is added to the documentation. Besides the new features there are the fixes and improvements: * feat: improve EOSdash config page * fix: kex_to_series for start_datetime Make key_to_series always start the series at start_datetime. * fix: default provider for GENETIC and GENETIC0 optimization To make the default less dependent on internet servers (with API changes and availability issues) the default for PVForecast is set to PVForecastPVLib and for ElecPrice to ElecPriceFixed. The default weather provider is changed to OpenMeteo. * fix: EOSdash display resampled prediction values Make EOSdash display resampled prediction values where resampling fits to the prediction value type. Use bar width that fits to 15 minutes value samples. * chore: add a UI hints system to EOSdash The UI hints system eases the definition of forms for configuration items. There are also forms for items in maps and lists. These forms allow to add and delete items to/ from maps and lists. The forms ensure that all required fields of newly added items are filled. * chore: Create an enum for valid optimization algorithms * chore. Make config also provide the available energy management modes. Used for configuration hints. * chore: Randomize default device id in configuration Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-08-07 13:13:17 +02:00
Bobby NoelteandGitHub	1905682113	chore: adapt pdf visualization (#1205) ... Change PDF visualization to be created on demand and per optimization algorithm. The PDF for the GENETIC0 optimization is provided by the /visualization_results.pdf endpoint. There is no change in the interface. By this the optimization algorithm is offloaded from the PDF generation which spares some time. To cope with several users may call the /visualization_results.pdf endpoint at the same time the PDF is generated on the fly without any intermediate file taking the stored GENETIC0 solution as an input. SVG picture generation is removed as this would again create intermediate files. Chart pictures can easily be taken from the PDF. To allow on demand creation of the optimization results visualization the optimisation solution stored is extended by several new attributes. To keep the deprecated /optimize endpoint compatible the optimization solution is stripped to the legacy content before returned. Due to the extension of the solution the optimization tests were adapted to cover the extended content. The optimization tests are adapted to test the generated visualization report by the pypdf reader. Pypdf is added to the development dependencies. Besides the adaptation several fixes and improvements are added: * feat: extend /v1/prediction/series endpoint by resampling and filling Add parameters for resampling and filling. Add the processing parameter to control wether raw data or resampled data shall be returned. * feat: extend /v1/measurement/series endpoint by resampling and filling Add parameters for resampling and filling: Add the processing parameter to control wether raw data or resampled data shall be returned. * feat: standardize and improve API error response Use FASTApi exception handlers to provide a standardized API exception handling. All exceptions are logged. Exception traces are only returned if the new logging configuration parameter logging.api_logging_level is set to "DEBUG" or "TRACE". Avoids unwanted leackage of server internals on exceptions. * fix: align to intervall when resampling Ensure resampling is aligned to interval also when the buckets are shifted due to the align_to_intervall parameter is set. * chore: make dropna mandatory and default to True * chore: refactor key_to_xxx data management methods Make key_to_series the central method for data resampling and fill. Add a new key_to_raw_series to retrieve the data as it is stored (without resampling and filling). Users of key_to_series were mostly moved to key_to_raw_series as this resembles the former interface. Especially in predictions and tests this was done. * chore: create test data sub-directory for each optimization algorithm To prevent cluttering the test data directory and ease test data management for optimization algorithms each algorithm got it's own sub-directory. The current test data was moved to these sub-directories. * chore: update version Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-08-01 12:45:19 +02:00
Bobby NoelteandGitHub	52fe489d4e	feat: add dvhubonline feed-in tariff provider (#1193) ... Fetches GET /api/prices?start&end&zone (15-min slots, EUR/MWh) and stores the raw market price as feed_in_tariff_wh (EUR/Wh) — no import charges/VAT. Slots beyond the day-ahead horizon are left to the consumer's forward-fill (FeedInTariffImport behaviour). 5 unit tests + opt-in live smoke (EOS_DVHUB_ONLINE_LIVE=1, passing). Signed-off-by: Christin <info@bikinibottom.capital> Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-29 22:21:17 +02:00
Bobby NoelteandGitHub	e23bb7b497	chore: prepare for update of genetic algorithm (#1190) ... Andreas will update the genetic algorithm for 15-minutes optimization intervals. Copy the current GENETIC optimization algorithm to GENETIC0 to enable to keep the algorithm with the current functionality. Also copy resources like the load interpolator to the GENETIC0 algorithm to keep them despite possible later changes to the interpolator. Make the deprecated legacy /optimize endpoint use the GENETIC0 optimization algorithm to in-fact behave the same way even if there will later be changes to the GENETIC algorithm by Andreas. Add a new REST endpoint to provide the unprocessed optimisation results of the GENETIC and GENETIC0 algorithm in case one wants to use them as done with the deprecated /optimize endpoint. Adapt the optimization configuration to have distinct configurations for the GENETIC and the GENETIC0 algorithm. Create a copy of the current tests for the GENETIC algorithm to be used for the GENETIC0 algorithm. This avoids the tests for the GENETIC0 algorithm to be influenced by later changes by Andreas. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-29 12:56:08 +02:00
Bobby NoelteandGitHub	7e5aa2f218	feat: add Tibber feed-in tariff provider (#1189) ... The `FeedInTariffTibber` provider requests `priceInfo` and `priceInfoRange` with `resolution: QUARTER_HOURLY` and preserves the native 15-minute timestamps. It uses Tibber's `energy` spot-price component without the `tax` part or EOS electricity-price charges. The end-customer `total` component is deliberately ignored. The provider deliberately rejects hourly API responses instead of silently repeating them. It reuses `elecprice.tibber.access_token` and `elecprice.tibber.home_id`, so no duplicate credentials are needed. Signed-off-by: Andreas Schmitz <akkudoktor.net> Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-24 15:52:26 +02:00
Bobby NoelteandGitHub	4c8a8e40f5	feat: add Akkudoktor feed-in provider (#1187) ... The `FeedInTariffAkkudoktor` provider uses raw day-ahead market prices from `https://api.akkudoktor.net/prices` as `feed_in_tariff_wh`. It does not add electricity import charges or VAT. Published prices are extended to the configured prediction horizon with the same seasonal ETS or median fallback used by the Akkudoktor electricity-price provider. The Akkudoktor endpoint currently forwards hourly market prices from aWATTar. With a 15-minute optimization interval, EOS holds each hourly price constant for its four quarter-hour slots. This keeps the slot grid consistent but does not create genuine quarter-hour market prices. Signed-off-by: Andreas Schmitz <akkudoktor.net> Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-23 22:37:38 +02:00
Bobby NoelteandGitHub	ed61918fe0	feat: add EnergyCharts feed-in tariff provider (#1165) ... The `FeedInTariffEnergyCharts` provider uses the raw Energy-Charts day-ahead market price as the feed-in tariff. It stores prices in `feed_in_tariff_wh` without adding electricity import charges or VAT. The data is loaded from the Energy-Charts `/price` endpoint for the configured bidding zone. The native Energy-Charts resolution, including quarter-hour data, is retained. Energy-Charts usually supplies prices only for the published day-ahead period. If that data does not cover the complete configured prediction horizon, the provider extends it as follows: - With more than 800 hours of history, an ETS (Holt-Winters exponential smoothing) forecast with weekly seasonality is used. - With more than 168 hours of history, an ETS forecast with daily seasonality is used. - With less history, the median of the available values is used as a constant fallback. The seasonal periods are adjusted to the source resolution. For example, quarter-hour data uses four values per hour. Values already supplied by Energy-Charts are kept unchanged; only missing future slots after the last published price are forecast. Consequently, a 15-minute optimization uses four forecast values per hour without converting them to hourly averages. Signed-off-by: Andreas Schmitz <akkudoktor.net> Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-23 15:48:29 +02:00
Bobby NoelteandGitHub	6093d8d348	chore: improve error msg on home assistant add-on port config (#1186) ... When running as a Home Assistant add-on the ports shall not be changed as this would break config.yaml that is used by Home Assistant. Prevent the change and return an error message. Extra fixes: * fix: EOS configuration initialisation by cli under EOSdash. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-23 15:11:18 +02:00
Bobby NoelteandGitHub	641873d867	fix: akkudoktor api requests (#1181) ... The akkudoktor api for PV forecast seems to be changed and does not support to concatenate several planes into one request anymore. Make a request for every plane and add up the power results. Do not use the new 15 minutes slots as the returned data does not include the hourly values for windspeed and temperature as does the hourly slots. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-20 21:46:01 +02:00
Bobby NoelteandGitHub	3bb0e02aed	fix: provider settings in top level field of configuration (#1161) ... Configuration for some providers was given in the sub-field `provider_settings` combining the settings of several providers. Pydantic does not understand this very well and the configuration became cumbersome, especially in EOSdash. All provider settings from the `provider-settings` sub-field are now lifted to the top level field of the configuration. The changes are automatically migrated in the configuration. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-20 14:37:03 +02:00
Bobby NoelteandGitHub	2ed04d5573	fix: db compaction run (#1156) ... These fixes were taken from https://github.com/arneman/EOS/tree/refactor/economic-objective. 1. Fix compaction job registration bug in eos.py: - compact_eos_database was registered with save_eos_database function - Now correctly calls compact_eos_database() - Root cause: compaction/vacuum never ran, allowing records to grow unbounded 2. Optimize db_iterate_records() from O(n) to O(log n): - Previous: linear scan from index 0 for every key_to_array() call - Now: uses bisect_left on _db_sorted_timestamps to skip to start position - Critical for large datasets: 9400 measurement records → 94s/call overhead → 5 calls/optimization 3. Reduce compaction_interval_sec default from 604800s (7 days) to 3600s (1 hour): - With 7-day interval: 65000 records accumulate before first cleanup - Bridge pushes ~1 record/9s (grid_export_emr) - Hourly compaction + 2h data window → stable ~950 records Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-20 12:59:08 +02:00
3 people	6933b33542	feat: rename genetic optimization API fields to English (#675) ... Rename the German field names of the genetic optimization API to English with full backward compatibility: - English names are canonical and documented in the OpenAPI schema; the German names are still accepted on input via validation aliases and re-emitted in responses as deprecated computed fields - fix visualization receiving the raw German-keyed simulation dict (KeyError: load_wh_per_hour) - rename internal German identifiers (optimize_ems, total_balance, battery_residual_value, self_consumption, extra_data keys) - use amount instead of currency/euro in chart labels and schema descriptions; document currency handling (whole currency units, never cents) in the API description - regenerate openapi.json and generated docs Co-authored-by: Tobias Welz <tobias.wizneteu@gmail.com>	2026-07-18 08:08:25 +02:00
3 people	75548990e1	feat: add cloud PV forecast providers: pvnode.com, forecast-solar and solcast (#1150) ... This PR adds three native PV power forecast providers, giving operators more cloud forecast sources to choose from via pvforecast.provider, alongside the existing PVForecastAkkudoktor, PVForecastVrm and PVForecastImport: PVForecastPVNode — native 15-minute forecasts from the pvnode.com V2 API. Saved-site mode (GET /v2/forecast/{site_id}) where the operator enters their API key + site id, or inline mode (POST /v2/forecast/inline) using the configured planes. PVForecastForecastSolar — the free Forecast.Solar API (no key required for the public endpoint). Multi-plane systems issue one request per plane and the instantaneous powers are summed per timestamp. PVForecastSolcast — the Solcast rooftop-site API (API key + resource id). Implementation notes All three populate the existing pvforecast_ac_power (and mirror pvforecast_dc_power) prediction keys, so they slot into the optimizer unchanged. Timezone handling: pvnode and Forecast.Solar return site-local wall-clock timestamps with an IANA timezone field, which are resolved to absolute instants before resampling; Solcast returns UTC period_end and is normalised to the period start. Each provider follows the existing provider pattern (provider_id(), _request_forecast() with @cache_in_file, _update_data()), is registered in pvforecast.py and prediction.py, and is upstream-neutral. Validation 27 unit tests (timezone resolution, null/zero handling, kW→W and period-start conversion, azimuth conversion, multi-plane summation, request URL/auth, HTTP-error handling). ruff check (F/D/S/bandit) and ruff format clean. Each provider was additionally validated against its live API with a real plant, confirming the response shapes (pvnode: 288 native 15-min slots; Forecast.Solar: instantaneous watts; Solcast: kW estimates with period_end/PT30M). Documentation Provider descriptions and configuration examples added to docs/akkudoktoreos/prediction.md. CHANGELOG.md entry under Unreleased. Regenerated docs/_generated/configpvforecast.md and configexample.md. Notes for reviewers Forecast.Solar's free endpoint is rate-limited (12 req/hour) and Solcast's free tier limits daily calls; both providers rely on the standard 1-hour cache_in_file TTL to stay within budget. Authors: The code is created by Christin. Only minor adaptions by Bobby. Signed-off-by: Christin <info@bikinibottom.capital> Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Christin <info@bikinibottom.capital>	2026-07-17 16:51:00 +02:00
Bobby NoelteandGitHub	eb9e966de9	fix: move data management to async (#1015) ... FAstAPI is an async framework. Data may be imported and exported, load and save, set and get asynchronously. Prevent interleaving data operations to corrupt the data. In the previous design sync and async data access was intermixed leading to data corruption. The basic data classes DataSequence and DataContainer and the derived classes like Provider and Measurement now are async. Data access is protected by several async locks. To support the async design of the data classes the database interface became async. The energy management is also adapted to the new async design. Optimization is still off-loaded to another thread, but the prepration for the optimization and the post optimization actions now follow the async design. Adapter operations are now also protected by async locks. Tests were adapted to the async design and new tests were created. Besides this major fix several other improvements and fixes are included in this PR. * fix: key_to_dict/list/array only regard data records with key value set. Before the exclusion of no value data records was only done if the dropna flag was set. * fix: test for visual result pdf generation Due to updates in the library the generated charts text was a little bit different. Adapt the test to create the comaprison pdf in the test data durectory and update the reference pdf. * chore: Remove MutableMapping from DataSequence and DataContainer. Mutable Mapping does not fit to the now async design. * chore: Add NoDB database backend This backend implements the full database backend interface but performs no actual persistence. It is intended for configurations where database persistence is disabled (`provider=None`). * chore: Improve measurement data import testing with real world scenarios. Added two new endpoints to support testing. * chore: Add mermaid to supported documentation tools * chore: Add documentation about async design * chore: Add documentation about generic data handling Covers the basics of measurement and prediction time series data handling. * chore: Add empty lines around markdown lists. * chore: sync pre-commit config to updated package versions Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-15 16:38:53 +02:00