Files
EOS/tests/test_config.py

533 lines
18 KiB
Python
Raw Normal View History

import tempfile
from pathlib import Path
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
from typing import Any, Optional, Union
from unittest.mock import patch
import pytest
from loguru import logger
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
from pydantic import IPvAnyAddress, ValidationError
2025-01-24 21:14:37 +01:00
from akkudoktoreos.config.config import ConfigEOS, GeneralSettings
# overwrite config_mixin fixture from conftest
@pytest.fixture(autouse=True)
def config_mixin():
pass
def test_fixture_new_config_file(config_default_dirs):
"""Assure fixture stash_config_file is working."""
config_default_dir_user, config_default_dir_cwd, _, _ = config_default_dirs
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
config_file_user = config_default_dir_user.joinpath(ConfigEOS.CONFIG_FILE_NAME)
config_file_cwd = config_default_dir_cwd.joinpath(ConfigEOS.CONFIG_FILE_NAME)
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
assert not config_file_user.exists()
assert not config_file_cwd.exists()
def test_config_constants(config_eos):
"""Test config constants are the way expected by the tests."""
assert config_eos.APP_NAME == "net.akkudoktor.eos"
assert config_eos.APP_AUTHOR == "akkudoktor"
assert config_eos.EOS_DIR == "EOS_DIR"
assert config_eos.ENCODING == "UTF-8"
assert config_eos.CONFIG_FILE_NAME == "EOS.config.json"
def test_computed_paths(config_eos):
"""Test computed paths for output and cache."""
# Don't actually try to create the data folder
with patch("pathlib.Path.mkdir"):
config_eos.merge_settings_from_dict(
{
"general": {
"data_folder_path": "/base/data",
"data_output_subpath": "extra/output",
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
},
"cache": {
"subpath": "somewhere/cache",
},
}
)
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
assert config_eos.general.data_folder_path == Path("/base/data")
assert config_eos.general.data_output_path == Path("/base/data/extra/output")
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
assert config_eos.cache.path() == Path("/base/data/somewhere/cache")
# Check non configurable pathes
assert config_eos.package_root_path == Path(__file__).parent.parent.resolve().joinpath(
"src/akkudoktoreos"
)
# reset settings so the config_eos fixture can verify the default paths
config_eos.reset_settings()
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
def test_config_from_env(monkeypatch, config_eos):
"""Test configuration from env."""
assert config_eos.server.port == 8503
assert config_eos.server.eosdash_port is None
monkeypatch.setenv("EOS_SERVER__PORT", "8553")
monkeypatch.setenv("EOS_SERVER__EOSDASH_PORT", "8555")
config_eos.reset_settings()
assert config_eos.server.port == 8553
assert config_eos.server.eosdash_port == 8555
def test_config_ipaddress(monkeypatch, config_eos):
"""Test configuration for IP adresses."""
assert config_eos.server.host == "127.0.0.1"
monkeypatch.setenv("EOS_SERVER__HOST", "0.0.0.0")
config_eos.reset_settings()
assert config_eos.server.host == "0.0.0.0"
monkeypatch.setenv("EOS_SERVER__HOST", "mail.akkudoktor.net")
config_eos.reset_settings()
assert config_eos.server.host == "mail.akkudoktor.net"
# keep last
monkeypatch.setenv("EOS_SERVER__HOST", "localhost")
config_eos.reset_settings()
assert config_eos.server.host == "localhost"
def test_singleton_behavior(config_eos, config_default_dirs):
"""Test that ConfigEOS behaves as a singleton."""
initial_cfg_file = config_eos.general.config_file_path
with patch(
"akkudoktoreos.config.config.user_config_dir", return_value=str(config_default_dirs[0])
):
instance1 = ConfigEOS()
instance2 = ConfigEOS()
assert instance1 is config_eos
assert instance1 is instance2
assert instance1.general.config_file_path == initial_cfg_file
def test_default_config_path(config_eos, config_default_dirs):
"""Test that the default config file path is computed correctly."""
_, _, config_default_dir_default, _ = config_default_dirs
expected_path = config_default_dir_default.joinpath("default.config.json")
assert config_eos.config_default_file_path == expected_path
assert config_eos.config_default_file_path.is_file()
def test_config_file_priority(config_default_dirs):
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
"""Test config file priority.
Priority is:
1. environment variable directory
2. user configuration directory
3. current working directory
"""
config_default_dir_user, config_default_dir_cwd, _, _ = config_default_dirs
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
config_file_cwd = Path(config_default_dir_cwd) / ConfigEOS.CONFIG_FILE_NAME
config_file_user = Path(config_default_dir_user) / ConfigEOS.CONFIG_FILE_NAME
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
assert not config_file_cwd.exists()
assert not config_file_user.exists()
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
# current working directory (prio 3)
config_file_cwd.write_text("{}")
config_eos = ConfigEOS()
config_eos.update()
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
assert config_eos.general.config_file_path == config_file_cwd
# user configuration directory (prio 2)
config_file_user.parent.mkdir()
config_file_user.write_text("{}")
config_eos.update()
assert config_eos.general.config_file_path == config_file_user
@patch("akkudoktoreos.config.config.user_config_dir")
def test_get_config_file_path(user_config_dir_patch, config_eos, config_default_dirs, monkeypatch):
"""Test that _get_config_file_path identifies the correct config file."""
config_default_dir_user, _, _, _ = config_default_dirs
user_config_dir_patch.return_value = str(config_default_dir_user)
def cfg_file(dir: Path) -> Path:
return dir.joinpath(ConfigEOS.CONFIG_FILE_NAME)
# Config newly created from fixture with fresh user config directory
assert config_eos._get_config_file_path() == (cfg_file(config_default_dir_user), True)
cfg_file(config_default_dir_user).unlink()
with tempfile.TemporaryDirectory() as temp_dir:
temp_dir_path = Path(temp_dir)
monkeypatch.setenv("EOS_DIR", str(temp_dir_path))
assert config_eos._get_config_file_path() == (cfg_file(temp_dir_path), False)
monkeypatch.setenv("EOS_CONFIG_DIR", "config")
assert config_eos._get_config_file_path() == (
cfg_file(temp_dir_path / "config"),
False,
)
monkeypatch.setenv("EOS_CONFIG_DIR", str(temp_dir_path / "config2"))
assert config_eos._get_config_file_path() == (
cfg_file(temp_dir_path / "config2"),
False,
)
monkeypatch.delenv("EOS_DIR")
monkeypatch.setenv("EOS_CONFIG_DIR", "config3")
assert config_eos._get_config_file_path() == (cfg_file(config_default_dir_user), False)
monkeypatch.setenv("EOS_CONFIG_DIR", str(temp_dir_path / "config3"))
assert config_eos._get_config_file_path() == (
cfg_file(temp_dir_path / "config3"),
False,
)
def test_config_copy(config_eos, monkeypatch):
"""Test if the config is copied to the provided path."""
with tempfile.TemporaryDirectory() as temp_dir:
temp_folder_path = Path(temp_dir)
temp_config_file_path = temp_folder_path.joinpath(config_eos.CONFIG_FILE_NAME).resolve()
monkeypatch.setenv(config_eos.EOS_DIR, str(temp_folder_path))
assert not temp_config_file_path.exists()
with patch("akkudoktoreos.config.config.user_config_dir", return_value=temp_dir):
assert config_eos._get_config_file_path() == (temp_config_file_path, False)
config_eos.update()
assert temp_config_file_path.exists()
@pytest.mark.parametrize(
"latitude, longitude, expected_timezone",
[
(40.7128, -74.0060, "America/New_York"), # Valid latitude/longitude
(None, None, None), # No location
(51.5074, -0.1278, "Europe/London"), # Another valid location
],
)
def test_config_common_settings_valid(latitude, longitude, expected_timezone):
2025-01-24 21:14:37 +01:00
"""Test valid settings for GeneralSettings."""
general_settings = GeneralSettings(
latitude=latitude,
longitude=longitude,
)
assert general_settings.latitude == latitude
assert general_settings.longitude == longitude
assert general_settings.timezone == expected_timezone
@pytest.mark.parametrize(
"field_name, invalid_value, expected_error",
[
("latitude", -91.0, "Input should be greater than or equal to -90"),
("latitude", 91.0, "Input should be less than or equal to 90"),
("longitude", -181.0, "Input should be greater than or equal to -180"),
("longitude", 181.0, "Input should be less than or equal to 180"),
],
)
def test_config_common_settings_invalid(field_name, invalid_value, expected_error):
"""Test invalid settings for PredictionCommonSettings."""
valid_data = {
"latitude": 40.7128,
"longitude": -74.0060,
}
2025-01-24 21:14:37 +01:00
assert GeneralSettings(**valid_data) is not None
valid_data[field_name] = invalid_value
with pytest.raises(ValidationError, match=expected_error):
2025-01-24 21:14:37 +01:00
GeneralSettings(**valid_data)
def test_config_common_settings_no_location():
"""Test that timezone is None when latitude and longitude are not provided."""
2025-01-24 21:14:37 +01:00
settings = GeneralSettings(latitude=None, longitude=None)
assert settings.timezone is None
def test_config_common_settings_with_location():
"""Test that timezone is correctly computed when latitude and longitude are provided."""
2025-01-24 21:14:37 +01:00
settings = GeneralSettings(latitude=34.0522, longitude=-118.2437)
assert settings.timezone == "America/Los_Angeles"
def test_config_common_settings_timezone_none_when_coordinates_missing():
"""Test that timezone is None when latitude or longitude is missing."""
2025-01-24 21:14:37 +01:00
config_no_latitude = GeneralSettings(latitude=None, longitude=-74.0060)
config_no_longitude = GeneralSettings(latitude=40.7128, longitude=None)
config_no_coords = GeneralSettings(latitude=None, longitude=None)
assert config_no_latitude.timezone is None
assert config_no_longitude.timezone is None
assert config_no_coords.timezone is None
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
# Test partial assignments and possible side effects
@pytest.mark.parametrize(
"path, value, expected, exception",
[
# Correct value assignment
(
"general/latitude",
42.0,
[("general.latitude", 42.0), ("general.longitude", 13.405)],
None,
),
# Correct value assignment (trailing /)
(
"general/latitude/",
41,
[("general.latitude", 41.0), ("general.longitude", 13.405)],
None,
),
# Correct value assignment (cast)
(
"general/latitude",
"43.0",
[("general.latitude", 43.0), ("general.longitude", 13.405)],
None,
),
# Invalid value assignment (constraint)
(
"general/latitude",
91.0,
[("general.latitude", 52.52), ("general.longitude", 13.405)],
ValueError,
),
# Invalid value assignment (type)
(
"general/latitude",
"test",
[("general.latitude", 52.52), ("general.longitude", 13.405)],
ValueError,
),
# Invalid path
(
"general/latitude/test",
"",
[("general.latitude", 52.52), ("general.longitude", 13.405)],
KeyError,
),
# Correct value nested assignment
(
"general",
{"latitude": 22},
[("general.latitude", 22.0), ("general.longitude", 13.405)],
None,
),
# Invalid value nested assignment
(
"general",
{"latitude": "test"},
[("general.latitude", 52.52), ("general.longitude", 13.405)],
ValueError,
),
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
# Correct value assignment - preparation for list
(
"devices/max_electric_vehicles",
1,
[("devices.max_electric_vehicles", 1), ],
None,
),
# Correct value for list
(
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
"devices/electric_vehicles/0/charge_rates",
[0.1, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0],
[
(
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
"devices.electric_vehicles[0].charge_rates",
[0.1, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0],
)
],
None,
),
# Invalid value for list
(
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
"devices/electric_vehicles/0/charge_rates",
"invalid",
[
(
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
"devices.electric_vehicles[0].charge_rates",
[0.0, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0],
)
],
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
ValueError,
),
# Invalid index (out of bound)
(
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
"devices/electric_vehicles/0/charge_rates/10",
0,
[
(
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
"devices.electric_vehicles[0].charge_rates",
[0.0, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0],
)
],
TypeError,
),
# Invalid index (no number)
(
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
"devices/electric_vehicles/0/charge_rates/test",
0,
[
(
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
"devices.electric_vehicles[0].charge_rates",
[0.0, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0],
)
],
IndexError,
),
# Unset value (set None)
(
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
"devices/electric_vehicles/0/charge_rates",
None,
[
(
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
"devices.electric_vehicles[0].charge_rates",
None,
)
],
None,
),
],
)
def test_set_nested_key(path, value, expected, exception, config_eos):
if not exception:
config_eos.set_nested_value(path, value)
for expected_path, expected_value in expected:
actual_value = eval(f"config_eos.{expected_path}")
assert actual_value == expected_value, (
f"Expected {expected_value} at {expected_path}, but got {actual_value}"
)
else:
try:
config_eos.set_nested_value(path, value)
for expected_path, expected_value in expected:
actual_value = eval(f"config_eos.{expected_path}")
assert actual_value == expected_value, (
f"Expected {expected_value} at {expected_path}, but got {actual_value}"
)
pytest.fail(
f"Expected exception {exception} but none was raised. Set '{expected_path}' to '{actual_value}'"
)
except Exception as e:
assert isinstance(e, exception), (
f"Expected exception {exception}, but got {type(e)}: {e}"
)
@pytest.mark.parametrize(
"path, expected_value, exception",
[
("general/latitude", 52.52, None),
("general/latitude/", 52.52, None),
("general/latitude/test", None, KeyError),
],
)
def test_get_nested_key(path, expected_value, exception, config_eos):
if not exception:
assert config_eos.get_nested_value(path) == expected_value
else:
with pytest.raises(exception):
config_eos.get_nested_value(path)
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
def test_merge_settings_from_dict_invalid(config_eos):
"""Test merging invalid data."""
invalid_settings = {
"general": {
"latitude": "invalid_latitude" # Should be a float
},
}
with pytest.raises(Exception): # Pydantic ValidationError expected
config_eos.merge_settings_from_dict(invalid_settings)
def test_merge_settings_partial(config_eos):
"""Test merging only a subset of settings."""
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
partial_settings: dict[str, Any] = {
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
"general": {
"latitude": 51.1657 # Only latitude is updated
},
}
config_eos.merge_settings_from_dict(partial_settings)
assert config_eos.general.latitude == 51.1657
assert config_eos.general.longitude == 13.405 # Should remain unchanged
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
#-----------------
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
partial_settings = {
"weather": {
"provider": "BrightSky",
},
}
config_eos.merge_settings_from_dict(partial_settings)
assert config_eos.weather.provider == "BrightSky"
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
#-----------------
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
partial_settings = {
"general": {
"latitude": None,
},
"weather": {
"provider": "ClearOutside",
},
}
config_eos.merge_settings_from_dict(partial_settings)
assert config_eos.general.latitude is None
assert config_eos.weather.provider == "ClearOutside"
# Assure update keeps same values
config_eos.update()
assert config_eos.general.latitude is None
assert config_eos.weather.provider == "ClearOutside"
fix: automatic optimization (#596) This fix implements the long term goal to have the EOS server run optimization (or energy management) on regular intervals automatically. Thus clients can request the current energy management plan at any time and it is updated on regular intervals without interaction by the client. This fix started out to "only" make automatic optimization (or energy management) runs working. It turned out there are several endpoints that in some way update predictions or run the optimization. To lock against such concurrent attempts the code had to be refactored to allow control of execution. During refactoring it became clear that some classes and files are named without a proper reference to their usage. Thus not only refactoring but also renaming became necessary. The names are still not the best, but I hope they are more intuitive. The fix includes several bug fixes that are not directly related to the automatic optimization but are necessary to keep EOS running properly to do the automatic optimization and to test and document the changes. This is a breaking change as the configuration structure changed once again and the server API was also enhanced and streamlined. The server API that is used by Andreas and Jörg in their videos has not changed. * fix: automatic optimization Allow optimization to automatically run on configured intervals gathering all optimization parameters from configuration and predictions. The automatic run can be configured to only run prediction updates skipping the optimization. Extend documentaion to also cover automatic optimization. Lock automatic runs against runs initiated by the /optimize or other endpoints. Provide new endpoints to retrieve the energy management plan and the genetic solution of the latest automatic optimization run. Offload energy management to thread pool executor to keep the app more responsive during the CPU heavy optimization run. * fix: EOS servers recognize environment variables on startup Force initialisation of EOS configuration on server startup to assure all sources of EOS configuration are properly set up and read. Adapt server tests and configuration tests to also test for environment variable configuration. * fix: Remove 0.0.0.0 to localhost translation under Windows EOS imposed a 0.0.0.0 to localhost translation under Windows for convenience. This caused some trouble in user configurations. Now, as the default IP address configuration is 127.0.0.1, the user is responsible for to set up the correct Windows compliant IP address. * fix: allow names for hosts additional to IP addresses * fix: access pydantic model fields by class Access by instance is deprecated. * fix: down sampling key_to_array * fix: make cache clear endpoint clear all cache files Make /v1/admin/cache/clear clear all cache files. Before it only cleared expired cache files by default. Add new endpoint /v1/admin/clear-expired to only clear expired cache files. * fix: timezonefinder returns Europe/Paris instead of Europe/Berlin timezonefinder 8.10 got more inaccurate for timezones in europe as there is a common timezone. Use new package tzfpy instead which is still returning Europe/Berlin if you are in Germany. tzfpy also claims to be faster than timezonefinder. * fix: provider settings configuration Provider configuration used to be a union holding the settings for several providers. Pydantic union handling does not always find the correct type for a provider setting. This led to exceptions in specific configurations. Now provider settings are explicit comfiguration items for each possible provider. This is a breaking change as the configuration structure was changed. * fix: ClearOutside weather prediction irradiance calculation Pvlib needs a pandas time index. Convert time index. * fix: test config file priority Do not use config_eos fixture as this fixture already creates a config file. * fix: optimization sample request documentation Provide all data in documentation of optimization sample request. * fix: gitlint blocking pip dependency resolution Replace gitlint by commitizen. Gitlint is not actively maintained anymore. Gitlint dependencies blocked pip from dependency resolution. * fix: sync pre-commit config to actual dependency requirements .pre-commit-config.yaml was out of sync, also requirements-dev.txt. * fix: missing babel in requirements.txt Add babel to requirements.txt * feat: setup default device configuration for automatic optimization In case the parameters for automatic optimization are not fully defined a default configuration is setup to allow the automatic energy management run. The default configuration may help the user to correctly define the device configuration. * feat: allow configuration of genetic algorithm parameters The genetic algorithm parameters for number of individuals, number of generations, the seed and penalty function parameters are now avaliable as configuration options. * feat: allow configuration of home appliance time windows The time windows a home appliance is allowed to run are now configurable by the configuration (for /v1 API) and also by the home appliance parameters (for the classic /optimize API). If there is no such configuration the time window defaults to optimization hours, which was the standard before the change. Documentation on how to configure time windows is added. * feat: standardize mesaurement keys for battery/ ev SoC measurements The standardized measurement keys to report battery SoC to the device simulations can now be retrieved from the device configuration as a read-only config option. * feat: feed in tariff prediction Add feed in tarif predictions needed for automatic optimization. The feed in tariff can be retrieved as fixed feed in tarif or can be imported. Also add tests for the different feed in tariff providers. Extend documentation to cover the feed in tariff providers. * feat: add energy management plan based on S2 standard instructions EOS can generate an energy management plan as a list of simple instructions. May be retrieved by the /v1/energy-management/plan endpoint. The instructions loosely follow the S2 energy management standard. * feat: make measurement keys configurable by EOS configuration. The fixed measurement keys are replaced by configurable measurement keys. * feat: make pendulum DateTime, Date, Duration types usable for pydantic models Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are added to the datetimeutil utility. Remove custom made pendulum adaptations from EOS pydantic module. Make EOS modules use the pydantic pendulum types managed by the datetimeutil module instead of the core pendulum types. * feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil. The time windows are are added to support home appliance time window configuration. All time classes are also pydantic models. Time is the base class for time definition derived from pendulum.Time. * feat: Extend DataRecord by configurable field like data. Configurable field like data was added to support the configuration of measurement records. * feat: Add additional information to health information Version information is added to the health endpoints of eos and eosDash. The start time of the last optimization and the latest run time of the energy management is added to the EOS health information. * feat: add pydantic merge model tests * feat: add plan tab to EOSdash The plan tab displays the current energy management instructions. * feat: add predictions tab to EOSdash The predictions tab displays the current predictions. * feat: add cache management to EOSdash admin tab The admin tab is extended by a section for cache management. It allows to clear the cache. * feat: add about tab to EOSdash The about tab resembles the former hello tab and provides extra information. * feat: Adapt changelog and prepare for release management Release management using commitizen is added. The changelog file is adapted and teh changelog and a description for release management is added in the documentation. * feat(doc): Improve install and devlopment documentation Provide a more concise installation description in Readme.md and add extra installation page and development page to documentation. * chore: Use memory cache for interpolation instead of dict in inverter Decorate calculate_self_consumption() with @cachemethod_until_update to cache results in memory during an energy management/ optimization run. Replacement of dict type caching in inverter is now possible because all optimization runs are properly locked and the memory cache CacheUntilUpdateStore is properly cleared at the start of any energy management/ optimization operation. * chore: refactor genetic Refactor the genetic algorithm modules for enhanced module structure and better readability. Removed unnecessary and overcomplex devices singleton. Also split devices configuration from genetic algorithm parameters to allow further development independently from genetic algorithm parameter format. Move charge rates configuration for electric vehicles from optimization to devices configuration to allow to have different charge rates for different cars in the future. * chore: Rename memory cache to CacheEnergyManagementStore The name better resembles the task of the cache to chache function and method results for an energy management run. Also the decorator functions are renamed accordingly: cachemethod_energy_management, cache_energy_management * chore: use class properties for config/ems/prediction mixin classes * chore: skip debug logs from mathplotlib Mathplotlib is very noisy in debug mode. * chore: automatically sync bokeh js to bokeh python package bokeh was updated to 3.8.0, make JS CDN automatically follow the package version. * chore: rename hello.py to about.py Make hello.py the adapted EOSdash about page. * chore: remove demo page from EOSdash As no the plan and prediction pages are working without configuration, the demo page is no longer necessary * chore: split test_server.py for system test Split test_server.py to create explicit test_system.py for system tests. * chore: move doc utils to generate_config_md.py The doc utils are only used in scripts/generate_config_md.py. Move it there to attribute for strong cohesion. * chore: improve pydantic merge model documentation * chore: remove pendulum warning from readme * chore: remove GitHub discussions from contributing documentation Github discussions is to be replaced by Akkudoktor.net. * chore(release): bump version to 0.1.0+dev for development * build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1 bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break. * build(deps): bump uvicorn from 0.36.0 to 0.37.0 BREAKING CHANGE: EOS configuration changed. V1 API changed. - The available_charge_rates_percent configuration is removed from optimization. Use the new charge_rate configuration for the electric vehicle - Optimization configuration parameter hours renamed to horizon_hours - Device configuration now has to provide the number of devices and device properties per device. - Specific prediction provider configuration to be provided by explicit configuration item (no union for all providers). - Measurement keys to be provided as a list. - New feed in tariff providers have to be configured. - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints. - /v1/admin/cache/clear now clears all cache files. Use /v1/admin/cache/clear-expired to only clear all expired cache files. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-10-28 02:50:31 +01:00
#-----------------
partial_settings = {
"devices": {
"max_electric_vehicles": 1,
"electric_vehicles": [
{
"charge_rates": [0.0, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0],
}
],
}
}
config_eos.merge_settings_from_dict(partial_settings)
assert config_eos.devices.max_electric_vehicles == 1
assert len(config_eos.devices.electric_vehicles) == 1
assert config_eos.devices.electric_vehicles[0].charge_rates == [0.0, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0]
# Assure re-apply generates the same config
config_eos.merge_settings_from_dict(partial_settings)
assert config_eos.devices.max_electric_vehicles == 1
assert len(config_eos.devices.electric_vehicles) == 1
assert config_eos.devices.electric_vehicles[0].charge_rates == [0.0, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0]
# Assure update keeps same values
config_eos.update()
assert config_eos.devices.max_electric_vehicles == 1
assert len(config_eos.devices.electric_vehicles) == 1
assert config_eos.devices.electric_vehicles[0].charge_rates == [0.0, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0]
Improve caching. (#431) * Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Improve startup of EOS EOS now has got an energy management task that runs shortly after startup. It tries to execute energy management runs with predictions newly fetched or initialized from cached data on first run. * Improve shutdown of EOS EOS has now a shutdown task that shuts EOS down gracefully with some time delay to allow REST API requests for shutdwon or restart to be fully serviced. * Improve EMS Add energy management task for repeated energy management controlled by startup delay and interval configuration parameters. Translate EnergieManagementSystem to english EnergyManagement. * Add administration endpoints - endpoints to control caching from REST API. - endpoints to control server restart (will not work on Windows) and shutdown from REST API * Improve doc generation Use "\n" linenend convention also on Windows when generating doc files. Replace Windows specific 127.0.0.1 address by standard 0.0.0.0. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class and test function - Make kill signal adapt to Windows/ Linux - Use consistently "\n" for lineends when writing text files in doc test - Fix test_logging under Windows - Fix conftest config_default_dirs test fixture under Windows From @Lasall * Improve Windows support - Use 127.0.0.1 as default config host (model defaults) and addionally redirect 0.0.0.0 to localhost on Windows (because default config file still has 0.0.0.0). - Update install/startup instructions as package installation is required atm. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2025-02-12 21:35:51 +01:00
def test_merge_settings_empty(config_eos):
"""Test merging an empty dictionary does not change settings."""
original_latitude = config_eos.general.latitude
config_eos.merge_settings_from_dict({}) # No changes
assert config_eos.general.latitude == original_latitude # Should remain unchanged