9 Commits

Include renames

Author	SHA1	Message	Date
Bobby NoelteandGitHub	1905682113	chore: adapt pdf visualization (#1205) ... Change PDF visualization to be created on demand and per optimization algorithm. The PDF for the GENETIC0 optimization is provided by the /visualization_results.pdf endpoint. There is no change in the interface. By this the optimization algorithm is offloaded from the PDF generation which spares some time. To cope with several users may call the /visualization_results.pdf endpoint at the same time the PDF is generated on the fly without any intermediate file taking the stored GENETIC0 solution as an input. SVG picture generation is removed as this would again create intermediate files. Chart pictures can easily be taken from the PDF. To allow on demand creation of the optimization results visualization the optimisation solution stored is extended by several new attributes. To keep the deprecated /optimize endpoint compatible the optimization solution is stripped to the legacy content before returned. Due to the extension of the solution the optimization tests were adapted to cover the extended content. The optimization tests are adapted to test the generated visualization report by the pypdf reader. Pypdf is added to the development dependencies. Besides the adaptation several fixes and improvements are added: * feat: extend /v1/prediction/series endpoint by resampling and filling Add parameters for resampling and filling. Add the processing parameter to control wether raw data or resampled data shall be returned. * feat: extend /v1/measurement/series endpoint by resampling and filling Add parameters for resampling and filling: Add the processing parameter to control wether raw data or resampled data shall be returned. * feat: standardize and improve API error response Use FASTApi exception handlers to provide a standardized API exception handling. All exceptions are logged. Exception traces are only returned if the new logging configuration parameter logging.api_logging_level is set to "DEBUG" or "TRACE". Avoids unwanted leackage of server internals on exceptions. * fix: align to intervall when resampling Ensure resampling is aligned to interval also when the buckets are shifted due to the align_to_intervall parameter is set. * chore: make dropna mandatory and default to True * chore: refactor key_to_xxx data management methods Make key_to_series the central method for data resampling and fill. Add a new key_to_raw_series to retrieve the data as it is stored (without resampling and filling). Users of key_to_series were mostly moved to key_to_raw_series as this resembles the former interface. Especially in predictions and tests this was done. * chore: create test data sub-directory for each optimization algorithm To prevent cluttering the test data directory and ease test data management for optimization algorithms each algorithm got it's own sub-directory. The current test data was moved to these sub-directories. * chore: update version Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-08-01 12:45:19 +02:00
Bobby NoelteandGitHub	52fe489d4e	feat: add dvhubonline feed-in tariff provider (#1193) ... Fetches GET /api/prices?start&end&zone (15-min slots, EUR/MWh) and stores the raw market price as feed_in_tariff_wh (EUR/Wh) — no import charges/VAT. Slots beyond the day-ahead horizon are left to the consumer's forward-fill (FeedInTariffImport behaviour). 5 unit tests + opt-in live smoke (EOS_DVHUB_ONLINE_LIVE=1, passing). Signed-off-by: Christin <info@bikinibottom.capital> Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-29 22:21:17 +02:00
Bobby NoelteandGitHub	e23bb7b497	chore: prepare for update of genetic algorithm (#1190) ... Andreas will update the genetic algorithm for 15-minutes optimization intervals. Copy the current GENETIC optimization algorithm to GENETIC0 to enable to keep the algorithm with the current functionality. Also copy resources like the load interpolator to the GENETIC0 algorithm to keep them despite possible later changes to the interpolator. Make the deprecated legacy /optimize endpoint use the GENETIC0 optimization algorithm to in-fact behave the same way even if there will later be changes to the GENETIC algorithm by Andreas. Add a new REST endpoint to provide the unprocessed optimisation results of the GENETIC and GENETIC0 algorithm in case one wants to use them as done with the deprecated /optimize endpoint. Adapt the optimization configuration to have distinct configurations for the GENETIC and the GENETIC0 algorithm. Create a copy of the current tests for the GENETIC algorithm to be used for the GENETIC0 algorithm. This avoids the tests for the GENETIC0 algorithm to be influenced by later changes by Andreas. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-29 12:56:08 +02:00
Bobby NoelteandGitHub	ed61918fe0	feat: add EnergyCharts feed-in tariff provider (#1165) ... The `FeedInTariffEnergyCharts` provider uses the raw Energy-Charts day-ahead market price as the feed-in tariff. It stores prices in `feed_in_tariff_wh` without adding electricity import charges or VAT. The data is loaded from the Energy-Charts `/price` endpoint for the configured bidding zone. The native Energy-Charts resolution, including quarter-hour data, is retained. Energy-Charts usually supplies prices only for the published day-ahead period. If that data does not cover the complete configured prediction horizon, the provider extends it as follows: - With more than 800 hours of history, an ETS (Holt-Winters exponential smoothing) forecast with weekly seasonality is used. - With more than 168 hours of history, an ETS forecast with daily seasonality is used. - With less history, the median of the available values is used as a constant fallback. The seasonal periods are adjusted to the source resolution. For example, quarter-hour data uses four values per hour. Values already supplied by Energy-Charts are kept unchanged; only missing future slots after the last published price are forecast. Consequently, a 15-minute optimization uses four forecast values per hour without converting them to hourly averages. Signed-off-by: Andreas Schmitz <akkudoktor.net> Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-23 15:48:29 +02:00
Bobby NoelteandGitHub	641873d867	fix: akkudoktor api requests (#1181) ... The akkudoktor api for PV forecast seems to be changed and does not support to concatenate several planes into one request anymore. Make a request for every plane and add up the power results. Do not use the new 15 minutes slots as the returned data does not include the hourly values for windspeed and temperature as does the hourly slots. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-20 21:46:01 +02:00
Bobby NoelteandGitHub	3bb0e02aed	fix: provider settings in top level field of configuration (#1161) ... Configuration for some providers was given in the sub-field `provider_settings` combining the settings of several providers. Pydantic does not understand this very well and the configuration became cumbersome, especially in EOSdash. All provider settings from the `provider-settings` sub-field are now lifted to the top level field of the configuration. The changes are automatically migrated in the configuration. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-20 14:37:03 +02:00
Bobby NoelteandGitHub	2ed04d5573	fix: db compaction run (#1156) ... These fixes were taken from https://github.com/arneman/EOS/tree/refactor/economic-objective. 1. Fix compaction job registration bug in eos.py: - compact_eos_database was registered with save_eos_database function - Now correctly calls compact_eos_database() - Root cause: compaction/vacuum never ran, allowing records to grow unbounded 2. Optimize db_iterate_records() from O(n) to O(log n): - Previous: linear scan from index 0 for every key_to_array() call - Now: uses bisect_left on _db_sorted_timestamps to skip to start position - Critical for large datasets: 9400 measurement records → 94s/call overhead → 5 calls/optimization 3. Reduce compaction_interval_sec default from 604800s (7 days) to 3600s (1 hour): - With 7-day interval: 65000 records accumulate before first cleanup - Bridge pushes ~1 record/9s (grid_export_emr) - Hourly compaction + 2h data window → stable ~950 records Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-20 12:59:08 +02:00
3 people	75548990e1	feat: add cloud PV forecast providers: pvnode.com, forecast-solar and solcast (#1150) ... This PR adds three native PV power forecast providers, giving operators more cloud forecast sources to choose from via pvforecast.provider, alongside the existing PVForecastAkkudoktor, PVForecastVrm and PVForecastImport: PVForecastPVNode — native 15-minute forecasts from the pvnode.com V2 API. Saved-site mode (GET /v2/forecast/{site_id}) where the operator enters their API key + site id, or inline mode (POST /v2/forecast/inline) using the configured planes. PVForecastForecastSolar — the free Forecast.Solar API (no key required for the public endpoint). Multi-plane systems issue one request per plane and the instantaneous powers are summed per timestamp. PVForecastSolcast — the Solcast rooftop-site API (API key + resource id). Implementation notes All three populate the existing pvforecast_ac_power (and mirror pvforecast_dc_power) prediction keys, so they slot into the optimizer unchanged. Timezone handling: pvnode and Forecast.Solar return site-local wall-clock timestamps with an IANA timezone field, which are resolved to absolute instants before resampling; Solcast returns UTC period_end and is normalised to the period start. Each provider follows the existing provider pattern (provider_id(), _request_forecast() with @cache_in_file, _update_data()), is registered in pvforecast.py and prediction.py, and is upstream-neutral. Validation 27 unit tests (timezone resolution, null/zero handling, kW→W and period-start conversion, azimuth conversion, multi-plane summation, request URL/auth, HTTP-error handling). ruff check (F/D/S/bandit) and ruff format clean. Each provider was additionally validated against its live API with a real plant, confirming the response shapes (pvnode: 288 native 15-min slots; Forecast.Solar: instantaneous watts; Solcast: kW estimates with period_end/PT30M). Documentation Provider descriptions and configuration examples added to docs/akkudoktoreos/prediction.md. CHANGELOG.md entry under Unreleased. Regenerated docs/_generated/configpvforecast.md and configexample.md. Notes for reviewers Forecast.Solar's free endpoint is rate-limited (12 req/hour) and Solcast's free tier limits daily calls; both providers rely on the standard 1-hour cache_in_file TTL to stay within budget. Authors: The code is created by Christin. Only minor adaptions by Bobby. Signed-off-by: Christin <info@bikinibottom.capital> Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Christin <info@bikinibottom.capital>	2026-07-17 16:51:00 +02:00
Bobby NoelteandGitHub	eb9e966de9	fix: move data management to async (#1015) ... FAstAPI is an async framework. Data may be imported and exported, load and save, set and get asynchronously. Prevent interleaving data operations to corrupt the data. In the previous design sync and async data access was intermixed leading to data corruption. The basic data classes DataSequence and DataContainer and the derived classes like Provider and Measurement now are async. Data access is protected by several async locks. To support the async design of the data classes the database interface became async. The energy management is also adapted to the new async design. Optimization is still off-loaded to another thread, but the prepration for the optimization and the post optimization actions now follow the async design. Adapter operations are now also protected by async locks. Tests were adapted to the async design and new tests were created. Besides this major fix several other improvements and fixes are included in this PR. * fix: key_to_dict/list/array only regard data records with key value set. Before the exclusion of no value data records was only done if the dropna flag was set. * fix: test for visual result pdf generation Due to updates in the library the generated charts text was a little bit different. Adapt the test to create the comaprison pdf in the test data durectory and update the reference pdf. * chore: Remove MutableMapping from DataSequence and DataContainer. Mutable Mapping does not fit to the now async design. * chore: Add NoDB database backend This backend implements the full database backend interface but performs no actual persistence. It is intended for configurations where database persistence is disabled (`provider=None`). * chore: Improve measurement data import testing with real world scenarios. Added two new endpoints to support testing. * chore: Add mermaid to supported documentation tools * chore: Add documentation about async design * chore: Add documentation about generic data handling Covers the basics of measurement and prediction time series data handling. * chore: Add empty lines around markdown lists. * chore: sync pre-commit config to updated package versions Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-15 16:38:53 +02:00