Files
EOS/docs/akkudoktoreos/prediction.md

635 lines
26 KiB
Markdown
Raw Normal View History

% SPDX-License-Identifier: Apache-2.0
(prediction-page)=
# Predictions
Predictions, along with simulations and measurements, form the foundation upon which energy
optimization is executed. In EOS, a standard set of predictions is managed, including:
- Household Load Prediction
- Electricity Price Prediction
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
- Feed In Tariff Prediction
- PV Power Prediction
- Weather Prediction
## Storing Predictions
EOS stores predictions in a **key-value store**, where the term `prediction key` refers to the
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
unique key used to retrieve specific prediction data.
## Prediction Providers
Most predictions can be sourced from various providers. The specific provider to use is configured
in the EOS configuration and can be set by prediction type. For example:
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
```json
{
"weather": {
"provider": "ClearOutside"
}
}
```
Some providers offer multiple prediction keys. For instance, a weather provider might provide data
to prediction keys like:
- `weather_temp_air` (air temperature)
- `weather_wind_speed` (wind speed)
### Prediction Import Providers
The prediction import providers are designed to import prediction data from a file or a JSON
string. An external entity should update the file or JSON string whenever new prediction data
becomes available.
The prediction data must be provided in one of the following formats:
#### 1. DateTimeData
A dictionary with the following structure:
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
```json
{
"start_datetime": "2024-01-01 00:00:00",
"interval": "1 Hour",
"<prediction key>": [value, value, ...],
"<prediction key>": [value, value, ...],
...
}
```
#### 2. DateTimeDataFrame
A JSON string created from a [pandas](https://pandas.pydata.org/docs/index.html) dataframe with a
`DatetimeIndex`. Use
[pandas.DataFrame.to_json(orient="index")](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html#pandas.DataFrame.to_json).
The column name of the data must be the same as the names of the `prediction key`s.
#### 3. DateTimeSeries
A JSON string created from a [pandas](https://pandas.pydata.org/docs/index.html) series with a
`DatetimeIndex`. Use
[pandas.Series.to_json(orient="index")](https://pandas.pydata.org/docs/reference/api/pandas.Series.to_json.html#pandas.Series.to_json).
## Adjusted Predictions
Certain prediction keys include an `_adjusted` suffix, such as `load_total_adjusted`. These
predictions are adjusted by real data from your system's measurements if given to enhance accuracy.
For example, the load prediction provider `LoadAkkudoktor` takes generic load data assembled by
Akkudoktor.net, maps that to the yearly energy consumption given in the configuration option
`loadakkudoktor_year_energy`, and finally adjusts the predicted load by the `loads`
of your system.
## Prediction Updates
Predictions are updated at the start of each energy management run, i.e., when EOS performs
optimization. Key considerations for updates include:
- Predictions sourced from online providers are usually rate-limited to one retrieval per hour.
- Only predictions with a configured provider are updated.
- Some providers may not support all generic prediction keys, leading to potential gaps
in updated predictions even after update.
## Accessing Predictions
Prediction data can be accessed using the EOS **REST API** via the `/v1/prediction/<...>` endpoints.
In a standard configuration, the [**REST API**](http://localhost:8503/docs) of a running EOS instance
is available at [http://localhost:8503/docs](http://localhost:8503/docs). This link provides access to
the API documentation and allows you to explore available endpoints interactively.
To view all available prediction keys, use the **GET** `/v1/prediction/keys` endpoint.
If no keys are displayed, or if the ones you need are missing, it indicates that your configuration
lacks the necessary prediction provider settings. You can configure prediction providers by using
the **PUT** `/v1/config` endpoint. You may save your configuration to the EOS configuration file.
## Electricity Price Prediction
Prediction keys:
- `elecprice_marketprice_wh`: Electricity market price per Wh (€/Wh).
- `elecprice_marketprice_kwh`: Electricity market price per kWh (€/kWh).
Configuration options:
- `elecprice`: Electricity price configuration.
- `provider`: Electricity price provider id of provider to be used.
- `ElecPriceAkkudoktor`: Retrieves from Akkudoktor.net.
- `ElecPriceEnergyCharts`: Retrieves from Energy-Charts.info.
- `ElecPriceImport`: Imports from a file or JSON string.
- `charges_kwh`: Electricity price charges (€/kWh).
- `vat_rate`: VAT rate factor applied to electricity price when charges are used (default: 1.19).
- `provider_settings.import_file_path`: Path to the file to import electricity price forecast data from.
- `provider_settings.import_json`: JSON string, dictionary of electricity price forecast value lists.
### ElecPriceAkkudoktor Provider
The `ElecPriceAkkudoktor` provider retrieves electricity prices directly from **Akkudoktor.net**,
which supplies price data for the next 24 hours. For periods beyond 24 hours, the provider generates
prices by extrapolating historical price data combined with the most recent actual prices obtained
from Akkudoktor.net. Electricity price charges given in the `charges_kwh` configuration
option are added.
### ElecPriceEnergyCharts Provider
The `ElecPriceEnergyCharts` provider retrieves day-ahead electricity market prices from
[Energy-Charts.info](https://www.Energy-Charts.info). It supports both short-term and extended forecasting by combining
real-time market data with historical price trends.
- For the next 24 hours, market prices are fetched directly from Energy-Charts.info.
- For periods beyond 24 hours, prices are estimated using extrapolation based on historical data and the latest
available market values.
Charges and VAT
- If `charges_kwh` configuration option is greater than 0, the electricity price is calculated as:
`(market price + charges_kwh) * vat_rate` where `vat_rate` is configurable (default: 1.19 for 19% VAT).
- If `charges_kwh` is set to 0, the electricity price is simply: `market_price` (no VAT applied).
**Note:** For the most accurate forecasts, it is recommended to set the `historic_hours` parameter to 840.
### ElecPriceImport Provider
The `ElecPriceImport` provider is designed to import electricity prices from a file or a JSON
string. An external entity should update the file or JSON string whenever new prediction data
becomes available.
The prediction key for the electricity price forecast data is:
- `elecprice_marketprice_wh`: Electricity market price per Wh (€/Wh).
The electricity proce forecast data must be provided in one of the formats described in
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
<project:#prediction-import-providers>. The data source can be given in the
`import_file_path` or `import_json` configuration option.
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
The data may additionally or solely be provided by the
**PUT** `/v1/prediction/import/ElecPriceImport` endpoint.
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
## Feed In Tariff Prediction
Prediction keys:
- `feed_in_tarif_wh`: Feed in tarif per Wh (€/Wh).
- `feed_in_tarif_kwh`: Feed in tarif per kWh (€/kWh)
Configuration options:
- `feedintarif`: Feed in tariff configuration.
- `provider`: Feed in tariff provider id of provider to be used.
- `FeedInTariffFixed`: Provides fixed feed in tariff values.
- `FeedInTariffImport`: Imports from a file or JSON string.
- `provider_settings.feed_in_tariff_kwh`: Fixed feed in tariff (€/kWh).
- `provider_settings.import_file_path`: Path to the file to import feed in tariff forecast data from.
- `provider_settings.import_json`: JSON string, dictionary of feed in tariff value lists.
## Load Prediction
Prediction keys:
- `load_mean`: Predicted load mean value (W).
- `load_std`: Predicted load standard deviation (W).
- `load_mean_adjusted`: Predicted load mean value adjusted by load measurement (W).
Configuration options:
- `load`: Load configuration.
- `provider`: Load provider id of provider to be used.
- `LoadAkkudoktor`: Retrieves from local database.
- `LoadVrm`: Retrieves data from the VRM API by Victron Energy.
- `LoadImport`: Imports from a file or JSON string.
- `provider_settings.loadakkudoktor_year_energy`: Yearly energy consumption (kWh).
- `provider_settings.loadimport_file_path`: Path to the file to import load forecast data from.
- `provider_settings.loadimport_json`: JSON string, dictionary of load forecast value lists.
### LoadAkkudoktor Provider
The `LoadAkkudoktor` provider retrieves generic load data from a local database and tailors it to
align with the annual energy consumption specified in the `loadakkudoktor_year_energy` configuration
option.
### LoadVrm Provider
The `LoadVrm` provider retrieves load forecast data from the VRM API by Victron Energy.
To receive forecasts, the system data must be configured under Dynamic ESS in the VRM portal.
To query the forecasts, an API token is required, which can also be created in the VRM portal under Preferences.
This token must be stored in the EOS configuration along with the VRM-Installations-ID.
```python
{
"load": {
"provider": "LoadVrm",
"provider_settings": {
"load_vrm_token": "dummy-token",
"load_vrm_idsite": 12345
}
```
The prediction keys for the load forecast data are:
- `load_mean`: Predicted load mean value (W).
### LoadImport Provider
The `LoadImport` provider is designed to import load forecast data from a file or a JSON
string. An external entity should update the file or JSON string whenever new prediction data
becomes available.
The prediction keys for the load forecast data are:
- `load_mean`: Predicted load mean value (W).
- `load_std`: Predicted load standard deviation (W).
- `load_mean_adjusted`: Predicted load mean value adjusted by load measurement (W).
The load forecast data must be provided in one of the formats described in
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
<project:#prediction-import-providers>. The data source can be given in the `loadimport_file_path`
or `loadimport_json` configuration option.
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
The data may additionally or solely be provided by the
**PUT** `/v1/prediction/import/LoadImport` endpoint.
## PV Power Prediction
Prediction keys:
- `pvforecast_ac_power`: Total DC power (W).
- `pvforecast_dc_power`: Total AC power (W).
Configuration options:
2025-01-24 21:14:37 +01:00
- `general`: General configuration.
- `latitude`: Latitude in decimal degrees, between -90 and 90, north is positive (ISO 19115) (°)"
- `longitude`: Longitude in decimal degrees, within -180 to 180 (°)
- `pvforecast`: PV forecast configuration.
- `provider`: PVForecast provider id of provider to be used.
- `PVForecastAkkudoktor`: Retrieves from Akkudoktor.net.
- `PVForecastVrm`: Retrieves data from the VRM API by Victron Energy.
- `PVForecastImport`: Imports from a file or JSON string.
- `planes[].surface_tilt`: Tilt angle from horizontal plane. Ignored for two-axis tracking.
- `planes[].surface_azimuth`: Orientation (azimuth angle) of the (fixed) plane.
Clockwise from north (north=0, east=90, south=180, west=270).
- `planes[].userhorizon`: Elevation of horizon in degrees, at equally spaced azimuth clockwise from north.
- `planes[].peakpower`: Nominal power of PV system in kW.
- `planes[].pvtechchoice`: PV technology. One of 'crystSi', 'CIS', 'CdTe', 'Unknown'.
- `planes[].mountingplace`: Type of mounting for PV system.
Options are 'free' for free-standing and 'building' for building-integrated.
- `planes[].loss`: Sum of PV system losses in percent
- `planes[].trackingtype`: Type of suntracking.
0=fixed,
1=single horizontal axis aligned north-south,
2=two-axis tracking,
3=vertical axis tracking,
4=single horizontal axis aligned east-west,
5=single inclined axis aligned north-south.
- `planes[].optimal_surface_tilt`: Calculate the optimum tilt angle. Ignored for two-axis tracking.
- `planes[].optimalangles`: Calculate the optimum tilt and azimuth angles. Ignored for two-axis tracking.
- `planes[].albedo`: Proportion of the light hitting the ground that it reflects back.
- `planes[].module_model`: Model of the PV modules of this plane.
- `planes[].inverter_model`: Model of the inverter of this plane.
- `planes[].inverter_paco`: AC power rating of the inverter. [W]
- `planes[].modules_per_string`: Number of the PV modules of the strings of this plane.
- `planes[].strings_per_inverter`: Number of the strings of the inverter of this plane.
- `provider_settings.import_file_path`: Path to the file to import PV forecast data from.
- `provider_settings.import_json`: JSON string, dictionary of PV forecast value lists.
---
Detailed definitions taken from
[PVGIS](https://joint-research-centre.ec.europa.eu/photovoltaic-geographical-information-system-pvgis/getting-started-pvgis/pvgis-user-manual_en).
- `pvtechchoice`
The performance of PV modules depends on the temperature and on the solar irradiance, but the exact
dependence varies between different types of PV modules. At the moment we can estimate the losses
due to temperature and irradiance effects for the following types of modules: crystalline silicon
cells; thin film modules made from CIS or CIGS and thin film modules made from Cadmium Telluride
(CdTe).
For other technologies (especially various amorphous technologies), this correction cannot be
calculated here. If you choose one of the first three options here the calculation of performance
will take into account the temperature dependence of the performance of the chosen technology. If
you choose the other option (other/unknown), the calculation will assume a loss of 8% of power due
to temperature effects (a generic value which has found to be reasonable for temperate climates).
PV power output also depends on the spectrum of the solar radiation. PVGIS can calculate how the
variations of the spectrum of sunlight affects the overall energy production from a PV system. At
the moment this calculation can be done for crystalline silicon and CdTe modules. Note that this
calculation is not yet available when using the NSRDB solar radiation database.
- `peakpower`
This is the power that the manufacturer declares that the PV array can produce under standard test
conditions (STC), which are a constant 1000W of solar irradiation per square meter in the plane of
the array, at an array temperature of 25°C. The peak power should be entered in kilowatt-peak (kWp).
If you do not know the declared peak power of your modules but instead know the area of the modules
and the declared conversion efficiency (in percent), you can calculate the peak power as
power = area \* efficiency / 100.
Bifacial modules: PVGIS doesn't make specific calculations for bifacial modules at present. Users
who wish to explore the possible benefits of this technology can input the power value for Bifacial
Nameplate Irradiance. This can also be can also be estimated from the front side peak power P_STC
value and the bifaciality factor, φ (if reported in the module data sheet) as:
P_BNPI = P_STC \* (1 + φ \* 0.135). NB this bifacial approach is not appropriate for BAPV or BIPV
installations or for modules mounting on a N-S axis i.e. facing E-W.
- `loss`
The estimated system losses are all the losses in the system, which cause the power actually
delivered to the electricity grid to be lower than the power produced by the PV modules. There are
several causes for this loss, such as losses in cables, power inverters, dirt (sometimes snow) on
the modules and so on. Over the years the modules also tend to lose a bit of their power, so the
average yearly output over the lifetime of the system will be a few percent lower than the output
in the first years.
We have given a default value of 14% for the overall losses. If you have a good idea that your value
will be different (maybe due to a really high-efficiency inverter) you may reduce this value a little.
- `mountingplace`
For fixed (non-tracking) systems, the way the modules are mounted will have an influence on the
temperature of the module, which in turn affects the efficiency. Experiments have shown that if the
movement of air behind the modules is restricted, the modules can get considerably hotter
(up to 15°C at 1000W/m2 of sunlight).
In PVGIS there are two possibilities: free-standing, meaning that the modules are mounted on a rack
with air flowing freely behind the modules; and building- integrated, which means that the modules
are completely built into the structure of the wall or roof of a building, with no air movement
behind the modules.
Some types of mounting are in between these two extremes, for instance if the modules are mounted on
a roof with curved roof tiles, allowing air to move behind the modules. In such cases, the
performance will be somewhere between the results of the two calculations that are possible here.
- `userhorizon`
Elevation of horizon in degrees, at equally spaced azimuth clockwise from north. In the user horizon
data each number represents the horizon height in degrees in a certain compass direction around the
point of interest. The horizon heights should be given in a clockwise direction starting at North;
that is, from North, going to East, South, West, and back to North. The values are assumed to
represent equal angular distance around the horizon. For instance, if you have 36 values, the first
point is due north, the next is 10 degrees east of north, and so on, until the last point, 10
degrees west of north.
---
Most of the configuration options are in line with the
[PVLib](https://pvlib-python.readthedocs.io/en/stable/_modules/pvlib/iotools/pvgis.html) definition for PVGIS data.
Detailed definitions from **PVLib** for PVGIS data.
- `surface_tilt`:
Tilt angle from horizontal plane.
- `surface_azimuth`
Orientation (azimuth angle) of the (fixed) plane. Clockwise from north (north=0, east=90, south=180,
west=270). This is offset 180 degrees from the convention used by PVGIS.
---
### PVForecastAkkudoktor Provider
The `PVForecastAkkudoktor` provider retrieves the PV power forecast data directly from
**Akkudoktor.net**.
The following prediction configuration options of the PV system must be set:
- `general.latitude`: Latitude in decimal degrees, between -90 and 90, north is positive (ISO 19115) (°)"
- `general.longitude`: Longitude in decimal degrees, within -180 to 180 (°)
For each plane of the PV system the following configuration options must be set:
- `pvforecast.planes[].surface_tilt`: Tilt angle from horizontal plane. Ignored for two-axis tracking.
- `pvforecast.planes[].surface_azimuth`: Orientation (azimuth angle) of the (fixed) plane.
Clockwise from north (north=0, east=90, south=180, west=270).
- `pvforecast.planes[].userhorizon`: Elevation of horizon in degrees, at equally spaced azimuth clockwise from north.
- `pvforecast.planes[].inverter_paco`: AC power rating of the inverter. [W]
- `pvforecast.planes[].peakpower`: Nominal power of PV system in kW.
Example:
```Python
{
"general": {
"latitude": 50.1234,
"longitude": 9.7654,
},
"pvforecast": {
"provider": "PVForecastAkkudoktor",
"planes": [
{
"peakpower": 5.0,
"surface_azimuth": -10,
"surface_tilt": 7,
"userhorizon": [20, 27, 22, 20],
"inverter_paco": 10000
},
{
"peakpower": 4.8,
"surface_azimuth": -90,
"surface_tilt": 7,
"userhorizon": [30, 30, 30, 50],
"inverter_paco": 10000
},
{
"peakpower": 1.4,
"surface_azimuth": -40,
"surface_tilt": 60,
"userhorizon": [60, 30, 0, 30],
"inverter_paco": 2000
},
{
"peakpower": 1.6,
"surface_azimuth": 5,
"surface_tilt": 45,
"userhorizon": [45, 25, 30, 60],
"inverter_paco": 1400
}
]
}
}
```
### PVForecastVrm Provider
The `PVForecastVrm` provider retrieves pv power forecast data from the VRM API by Victron Energy.
To receive forecasts, the system data must be configured under Dynamic ESS in the VRM portal.
To query the forecasts, an API token is required, which can also be created in the VRM portal under Preferences.
This token must be stored in the EOS configuration along with the VRM-Installations-ID.
```python
{
"pvforecast": {
"provider": "PVForecastVrm",
"provider_settings": {
"pvforecast_vrm_token": "dummy-token",
"pvforecast_vrm_idsite": 12345
}
}
```
The prediction keys for the PV forecast data are:
- `pvforecast_dc_power`: Total DC power (W).
### PVForecastImport Provider
The `PVForecastImport` provider is designed to import PV forecast data from a file or a JSON
string. An external entity should update the file or JSON string whenever new prediction data
becomes available.
The prediction keys for the PV forecast data are:
- `pvforecast_ac_power`: Total AC power (W).
- `pvforecast_dc_power`: Total DC power (W).
The PV forecast data must be provided in one of the formats described in
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
<project:#prediction-import-providers>. The data source can be given in the
`import_file_path` or `import_json` configuration option.
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
The data may additionally or solely be provided by the
**PUT** `/v1/prediction/import/PVForecastImport` endpoint.
## Weather Prediction
Prediction keys:
- `weather_dew_point`: Dew Point (°C)
- `weather_dhi`: Diffuse Horizontal Irradiance (W/m2)
- `weather_dni`: Direct Normal Irradiance (W/m2)
- `weather_feels_like`: Feels Like (°C)
- `weather_fog`: Fog (%)
- `weather_frost_chance`: Chance of Frost
- `weather_ghi`: Global Horizontal Irradiance (W/m2)
- `weather_high_clouds`: High Clouds (% Sky Obscured)
- `weather_low_clouds`: Low Clouds (% Sky Obscured)
- `weather_medium_clouds`: Medium Clouds (% Sky Obscured)
- `weather_ozone`: Ozone (du)
- `weather_precip_amt`: Precipitation Amount (mm)
- `weather_precip_prob`: Precipitation Probability (%)
- `weather_preciptable_water`: Precipitable Water (cm)
- `weather_precip_type`: Precipitation Type
- `weather_pressure`: Pressure (mb)
- `weather_relative_humidity`: Relative Humidity (%)
- `weather_temp_air`: Temperature (°C)
- `weather_total_clouds`: Total Clouds (% Sky Obscured)
- `weather_visibility`: Visibility (m)
- `weather_wind_direction`: "Wind Direction (°)
- `weather_wind_speed`: Wind Speed (kmph)
Configuration options:
- `weather`: General weather configuration.
- `provider`: Load provider id of provider to be used.
- `BrightSky`: Retrieves from [BrightSky](https://api.brightsky.dev).
- `ClearOutside`: Retrieves from [ClearOutside](https://clearoutside.com/forecast).
- `LoadImport`: Imports from a file or JSON string.
- `provider_settings.import_file_path`: Path to the file to import weatherforecast data from.
- `provider_settings.import_json`: JSON string, dictionary of weather forecast value lists.
### BrightSky Provider
The `BrightSky` provider retrieves the PV power forecast data directly from
[**BrightSky**](https://api.brightsky.dev).
The provider provides forecast data for the following prediction keys:
- `weather_dew_point`: Dew Point (°C)
- `weather_ghi`: Global Horizontal Irradiance (W/m2)
- `weather_precip_amt`: Precipitation Amount (mm)
- `weather_precip_prob`: Precipitation Probability (%)
- `weather_pressure`: Pressure (mb)
- `weather_relative_humidity`: Relative Humidity (%)
- `weather_temp_air`: Temperature (°C)
- `weather_total_clouds`: Total Clouds (% Sky Obscured)
- `weather_visibility`: Visibility (m)
- `weather_wind_direction`: "Wind Direction (°)
- `weather_wind_speed`: Wind Speed (kmph)
### ClearOutside Provider
The `ClearOutside` provider retrieves the PV power forecast data directly from
[**ClearOutside**](https://clearoutside.com/forecast).
The provider provides forecast data for the following prediction keys:
- `weather_dew_point`: Dew Point (°C)
- `weather_dhi`: Diffuse Horizontal Irradiance (W/m2)
- `weather_dni`: Direct Normal Irradiance (W/m2)
- `weather_feels_like`: Feels Like (°C)
- `weather_fog`: Fog (%)
- `weather_frost_chance`: Chance of Frost
- `weather_ghi`: Global Horizontal Irradiance (W/m2)
- `weather_high_clouds`: High Clouds (% Sky Obscured)
- `weather_low_clouds`: Low Clouds (% Sky Obscured)
- `weather_medium_clouds`: Medium Clouds (% Sky Obscured)
- `weather_ozone`: Ozone (du)
- `weather_precip_amt`: Precipitation Amount (mm)
- `weather_precip_prob`: Precipitation Probability (%)
- `weather_preciptable_water`: Precipitable Water (cm)
- `weather_precip_type`: Precipitation Type
- `weather_pressure`: Pressure (mb)
- `weather_relative_humidity`: Relative Humidity (%)
- `weather_temp_air`: Temperature (°C)
- `weather_total_clouds`: Total Clouds (% Sky Obscured)
- `weather_visibility`: Visibility (m)
- `weather_wind_direction`: "Wind Direction (°)
- `weather_wind_speed`: Wind Speed (kmph)
### WeatherImport Provider
The `WeatherImport` provider is designed to import weather forecast data from a file or a JSON
string. An external entity should update the file or JSON string whenever new prediction data
becomes available.
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
The prediction keys for the weather forecast data are:
- `weather_dew_point`: Dew Point (°C)
- `weather_dhi`: Diffuse Horizontal Irradiance (W/m2)
- `weather_dni`: Direct Normal Irradiance (W/m2)
- `weather_feels_like`: Feels Like (°C)
- `weather_fog`: Fog (%)
- `weather_frost_chance`: Chance of Frost
- `weather_ghi`: Global Horizontal Irradiance (W/m2)
- `weather_high_clouds`: High Clouds (% Sky Obscured)
- `weather_low_clouds`: Low Clouds (% Sky Obscured)
- `weather_medium_clouds`: Medium Clouds (% Sky Obscured)
- `weather_ozone`: Ozone (du)
- `weather_precip_amt`: Precipitation Amount (mm)
- `weather_precip_prob`: Precipitation Probability (%)
- `weather_preciptable_water`: Precipitable Water (cm)
- `weather_precip_type`: Precipitation Type
- `weather_pressure`: Pressure (mb)
- `weather_relative_humidity`: Relative Humidity (%)
- `weather_temp_air`: Temperature (°C)
- `weather_total_clouds`: Total Clouds (% Sky Obscured)
- `weather_visibility`: Visibility (m)
- `weather_wind_direction`: "Wind Direction (°)
- `weather_wind_speed`: Wind Speed (kmph)
The PV forecast data must be provided in one of the formats described in
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
<project:#prediction-import-providers>. The data source can be given in the
`import_file_path` or `import_json` configuration option.
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
The data may additionally or solely be provided by the
**PUT** `/v1/prediction/import/WeatherImport` endpoint.