mirror of
https://github.com/Akkudoktor-EOS/EOS.git
synced 2025-11-25 06:46:25 +00:00
fix: automatic optimization (#596)
This fix implements the long term goal to have the EOS server run optimization (or
energy management) on regular intervals automatically. Thus clients can request
the current energy management plan at any time and it is updated on regular
intervals without interaction by the client.
This fix started out to "only" make automatic optimization (or energy management)
runs working. It turned out there are several endpoints that in some way
update predictions or run the optimization. To lock against such concurrent attempts
the code had to be refactored to allow control of execution. During refactoring it
became clear that some classes and files are named without a proper reference
to their usage. Thus not only refactoring but also renaming became necessary.
The names are still not the best, but I hope they are more intuitive.
The fix includes several bug fixes that are not directly related to the automatic optimization
but are necessary to keep EOS running properly to do the automatic optimization and
to test and document the changes.
This is a breaking change as the configuration structure changed once again and
the server API was also enhanced and streamlined. The server API that is used by
Andreas and Jörg in their videos has not changed.
* fix: automatic optimization
Allow optimization to automatically run on configured intervals gathering all
optimization parameters from configuration and predictions. The automatic run
can be configured to only run prediction updates skipping the optimization.
Extend documentaion to also cover automatic optimization. Lock automatic runs
against runs initiated by the /optimize or other endpoints. Provide new
endpoints to retrieve the energy management plan and the genetic solution
of the latest automatic optimization run. Offload energy management to thread
pool executor to keep the app more responsive during the CPU heavy optimization
run.
* fix: EOS servers recognize environment variables on startup
Force initialisation of EOS configuration on server startup to assure
all sources of EOS configuration are properly set up and read. Adapt
server tests and configuration tests to also test for environment
variable configuration.
* fix: Remove 0.0.0.0 to localhost translation under Windows
EOS imposed a 0.0.0.0 to localhost translation under Windows for
convenience. This caused some trouble in user configurations. Now, as the
default IP address configuration is 127.0.0.1, the user is responsible
for to set up the correct Windows compliant IP address.
* fix: allow names for hosts additional to IP addresses
* fix: access pydantic model fields by class
Access by instance is deprecated.
* fix: down sampling key_to_array
* fix: make cache clear endpoint clear all cache files
Make /v1/admin/cache/clear clear all cache files. Before it only cleared
expired cache files by default. Add new endpoint /v1/admin/clear-expired
to only clear expired cache files.
* fix: timezonefinder returns Europe/Paris instead of Europe/Berlin
timezonefinder 8.10 got more inaccurate for timezones in europe as there is
a common timezone. Use new package tzfpy instead which is still returning
Europe/Berlin if you are in Germany. tzfpy also claims to be faster than
timezonefinder.
* fix: provider settings configuration
Provider configuration used to be a union holding the settings for several
providers. Pydantic union handling does not always find the correct type
for a provider setting. This led to exceptions in specific configurations.
Now provider settings are explicit comfiguration items for each possible
provider. This is a breaking change as the configuration structure was
changed.
* fix: ClearOutside weather prediction irradiance calculation
Pvlib needs a pandas time index. Convert time index.
* fix: test config file priority
Do not use config_eos fixture as this fixture already creates a config file.
* fix: optimization sample request documentation
Provide all data in documentation of optimization sample request.
* fix: gitlint blocking pip dependency resolution
Replace gitlint by commitizen. Gitlint is not actively maintained anymore.
Gitlint dependencies blocked pip from dependency resolution.
* fix: sync pre-commit config to actual dependency requirements
.pre-commit-config.yaml was out of sync, also requirements-dev.txt.
* fix: missing babel in requirements.txt
Add babel to requirements.txt
* feat: setup default device configuration for automatic optimization
In case the parameters for automatic optimization are not fully defined a
default configuration is setup to allow the automatic energy management
run. The default configuration may help the user to correctly define
the device configuration.
* feat: allow configuration of genetic algorithm parameters
The genetic algorithm parameters for number of individuals, number of
generations, the seed and penalty function parameters are now avaliable
as configuration options.
* feat: allow configuration of home appliance time windows
The time windows a home appliance is allowed to run are now configurable
by the configuration (for /v1 API) and also by the home appliance parameters
(for the classic /optimize API). If there is no such configuration the
time window defaults to optimization hours, which was the standard before
the change. Documentation on how to configure time windows is added.
* feat: standardize mesaurement keys for battery/ ev SoC measurements
The standardized measurement keys to report battery SoC to the device
simulations can now be retrieved from the device configuration as a
read-only config option.
* feat: feed in tariff prediction
Add feed in tarif predictions needed for automatic optimization. The feed in
tariff can be retrieved as fixed feed in tarif or can be imported. Also add
tests for the different feed in tariff providers. Extend documentation to
cover the feed in tariff providers.
* feat: add energy management plan based on S2 standard instructions
EOS can generate an energy management plan as a list of simple instructions.
May be retrieved by the /v1/energy-management/plan endpoint. The instructions
loosely follow the S2 energy management standard.
* feat: make measurement keys configurable by EOS configuration.
The fixed measurement keys are replaced by configurable measurement keys.
* feat: make pendulum DateTime, Date, Duration types usable for pydantic models
Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are
added to the datetimeutil utility. Remove custom made pendulum adaptations
from EOS pydantic module. Make EOS modules use the pydantic pendulum types
managed by the datetimeutil module instead of the core pendulum types.
* feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil.
The time windows are are added to support home appliance time window
configuration. All time classes are also pydantic models. Time is the base
class for time definition derived from pendulum.Time.
* feat: Extend DataRecord by configurable field like data.
Configurable field like data was added to support the configuration of
measurement records.
* feat: Add additional information to health information
Version information is added to the health endpoints of eos and eosDash.
The start time of the last optimization and the latest run time of the energy
management is added to the EOS health information.
* feat: add pydantic merge model tests
* feat: add plan tab to EOSdash
The plan tab displays the current energy management instructions.
* feat: add predictions tab to EOSdash
The predictions tab displays the current predictions.
* feat: add cache management to EOSdash admin tab
The admin tab is extended by a section for cache management. It allows to
clear the cache.
* feat: add about tab to EOSdash
The about tab resembles the former hello tab and provides extra information.
* feat: Adapt changelog and prepare for release management
Release management using commitizen is added. The changelog file is adapted and
teh changelog and a description for release management is added in the
documentation.
* feat(doc): Improve install and devlopment documentation
Provide a more concise installation description in Readme.md and add extra
installation page and development page to documentation.
* chore: Use memory cache for interpolation instead of dict in inverter
Decorate calculate_self_consumption() with @cachemethod_until_update to cache
results in memory during an energy management/ optimization run. Replacement
of dict type caching in inverter is now possible because all optimization
runs are properly locked and the memory cache CacheUntilUpdateStore is properly
cleared at the start of any energy management/ optimization operation.
* chore: refactor genetic
Refactor the genetic algorithm modules for enhanced module structure and better
readability. Removed unnecessary and overcomplex devices singleton. Also
split devices configuration from genetic algorithm parameters to allow further
development independently from genetic algorithm parameter format. Move
charge rates configuration for electric vehicles from optimization to devices
configuration to allow to have different charge rates for different cars in
the future.
* chore: Rename memory cache to CacheEnergyManagementStore
The name better resembles the task of the cache to chache function and method
results for an energy management run. Also the decorator functions are renamed
accordingly: cachemethod_energy_management, cache_energy_management
* chore: use class properties for config/ems/prediction mixin classes
* chore: skip debug logs from mathplotlib
Mathplotlib is very noisy in debug mode.
* chore: automatically sync bokeh js to bokeh python package
bokeh was updated to 3.8.0, make JS CDN automatically follow the package version.
* chore: rename hello.py to about.py
Make hello.py the adapted EOSdash about page.
* chore: remove demo page from EOSdash
As no the plan and prediction pages are working without configuration, the demo
page is no longer necessary
* chore: split test_server.py for system test
Split test_server.py to create explicit test_system.py for system tests.
* chore: move doc utils to generate_config_md.py
The doc utils are only used in scripts/generate_config_md.py. Move it there to
attribute for strong cohesion.
* chore: improve pydantic merge model documentation
* chore: remove pendulum warning from readme
* chore: remove GitHub discussions from contributing documentation
Github discussions is to be replaced by Akkudoktor.net.
* chore(release): bump version to 0.1.0+dev for development
* build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1
bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break.
* build(deps): bump uvicorn from 0.36.0 to 0.37.0
BREAKING CHANGE: EOS configuration changed. V1 API changed.
- The available_charge_rates_percent configuration is removed from optimization.
Use the new charge_rate configuration for the electric vehicle
- Optimization configuration parameter hours renamed to horizon_hours
- Device configuration now has to provide the number of devices and device
properties per device.
- Specific prediction provider configuration to be provided by explicit
configuration item (no union for all providers).
- Measurement keys to be provided as a list.
- New feed in tariff providers have to be configured.
- /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints.
- /v1/admin/cache/clear now clears all cache files. Use
/v1/admin/cache/clear-expired to only clear all expired cache files.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
This commit is contained in:
Bobby Noelte
GitHub
@@ -14,28 +14,28 @@ import shutil
|
||||
from pathlib import Path
|
||||
from typing import Any, ClassVar, Optional, Type
|
||||
|
||||
import pydantic_settings
|
||||
from loguru import logger
|
||||
from platformdirs import user_config_dir, user_data_dir
|
||||
from pydantic import Field, computed_field
|
||||
from pydantic_settings import (
|
||||
BaseSettings,
|
||||
JsonConfigSettingsSource,
|
||||
PydanticBaseSettingsSource,
|
||||
SettingsConfigDict,
|
||||
)
|
||||
from pydantic import Field, computed_field, field_validator
|
||||
|
||||
# settings
|
||||
from akkudoktoreos.config.configabc import SettingsBaseModel
|
||||
from akkudoktoreos.config.configmigrate import migrate_config_file
|
||||
from akkudoktoreos.core.cachesettings import CacheCommonSettings
|
||||
from akkudoktoreos.core.coreabc import SingletonMixin
|
||||
from akkudoktoreos.core.decorators import classproperty
|
||||
from akkudoktoreos.core.emsettings import EnergyManagementCommonSettings
|
||||
from akkudoktoreos.core.emsettings import (
|
||||
EnergyManagementCommonSettings,
|
||||
)
|
||||
from akkudoktoreos.core.logsettings import LoggingCommonSettings
|
||||
from akkudoktoreos.core.pydantic import PydanticModelNestedValueMixin, merge_models
|
||||
from akkudoktoreos.devices.settings import DevicesCommonSettings
|
||||
from akkudoktoreos.core.version import __version__
|
||||
from akkudoktoreos.devices.devices import DevicesCommonSettings
|
||||
from akkudoktoreos.measurement.measurement import MeasurementCommonSettings
|
||||
from akkudoktoreos.optimization.optimization import OptimizationCommonSettings
|
||||
from akkudoktoreos.prediction.elecprice import ElecPriceCommonSettings
|
||||
from akkudoktoreos.prediction.feedintariff import FeedInTariffCommonSettings
|
||||
from akkudoktoreos.prediction.load import LoadCommonSettings
|
||||
from akkudoktoreos.prediction.prediction import PredictionCommonSettings
|
||||
from akkudoktoreos.prediction.pvforecast import PVForecastCommonSettings
|
||||
@@ -83,6 +83,10 @@ class GeneralSettings(SettingsBaseModel):
|
||||
_config_folder_path: ClassVar[Optional[Path]] = None
|
||||
_config_file_path: ClassVar[Optional[Path]] = None
|
||||
|
||||
version: str = Field(
|
||||
default=__version__, description="Configuration file version. Used to check compatibility."
|
||||
)
|
||||
|
||||
data_folder_path: Optional[Path] = Field(
|
||||
default=None, description="Path to EOS data directory.", examples=[None, "/home/eos/data"]
|
||||
)
|
||||
@@ -131,11 +135,25 @@ class GeneralSettings(SettingsBaseModel):
|
||||
"""Path to EOS configuration file."""
|
||||
return self._config_file_path
|
||||
|
||||
compatible_versions: ClassVar[list[str]] = [__version__]
|
||||
|
||||
class SettingsEOS(BaseSettings, PydanticModelNestedValueMixin):
|
||||
@field_validator("version")
|
||||
@classmethod
|
||||
def check_version(cls, v: str) -> str:
|
||||
if v not in cls.compatible_versions:
|
||||
error = (
|
||||
f"Incompatible configuration version '{v}'. "
|
||||
f"Expected one of: {', '.join(cls.compatible_versions)}."
|
||||
)
|
||||
logger.error(error)
|
||||
raise ValueError(error)
|
||||
return v
|
||||
|
||||
|
||||
class SettingsEOS(pydantic_settings.BaseSettings, PydanticModelNestedValueMixin):
|
||||
"""Settings for all EOS.
|
||||
|
||||
Used by updating the configuration with specific settings only.
|
||||
Only used to update the configuration with specific settings.
|
||||
"""
|
||||
|
||||
general: Optional[GeneralSettings] = Field(
|
||||
@@ -174,6 +192,10 @@ class SettingsEOS(BaseSettings, PydanticModelNestedValueMixin):
|
||||
default=None,
|
||||
description="Electricity Price Settings",
|
||||
)
|
||||
feedintariff: Optional[FeedInTariffCommonSettings] = Field(
|
||||
default=None,
|
||||
description="Feed In Tariff Settings",
|
||||
)
|
||||
load: Optional[LoadCommonSettings] = Field(
|
||||
default=None,
|
||||
description="Load Settings",
|
||||
@@ -195,7 +217,7 @@ class SettingsEOS(BaseSettings, PydanticModelNestedValueMixin):
|
||||
description="Utilities Settings",
|
||||
)
|
||||
|
||||
model_config = SettingsConfigDict(
|
||||
model_config = pydantic_settings.SettingsConfigDict(
|
||||
env_nested_delimiter="__",
|
||||
nested_model_default_partial_update=True,
|
||||
env_prefix="EOS_",
|
||||
@@ -218,12 +240,18 @@ class SettingsEOSDefaults(SettingsEOS):
|
||||
optimization: OptimizationCommonSettings = OptimizationCommonSettings()
|
||||
prediction: PredictionCommonSettings = PredictionCommonSettings()
|
||||
elecprice: ElecPriceCommonSettings = ElecPriceCommonSettings()
|
||||
feedintariff: FeedInTariffCommonSettings = FeedInTariffCommonSettings()
|
||||
load: LoadCommonSettings = LoadCommonSettings()
|
||||
pvforecast: PVForecastCommonSettings = PVForecastCommonSettings()
|
||||
weather: WeatherCommonSettings = WeatherCommonSettings()
|
||||
server: ServerCommonSettings = ServerCommonSettings()
|
||||
utils: UtilsCommonSettings = UtilsCommonSettings()
|
||||
|
||||
def __hash__(self) -> int:
|
||||
# Just for usage in configmigrate, finally overwritten when used by ConfigEOS.
|
||||
# This is mutable, so pydantic does not set a hash.
|
||||
return id(self)
|
||||
|
||||
|
||||
class ConfigEOS(SingletonMixin, SettingsEOSDefaults):
|
||||
"""Singleton configuration handler for the EOS application.
|
||||
@@ -290,33 +318,34 @@ class ConfigEOS(SingletonMixin, SettingsEOSDefaults):
|
||||
@classmethod
|
||||
def settings_customise_sources(
|
||||
cls,
|
||||
settings_cls: Type[BaseSettings],
|
||||
init_settings: PydanticBaseSettingsSource,
|
||||
env_settings: PydanticBaseSettingsSource,
|
||||
dotenv_settings: PydanticBaseSettingsSource,
|
||||
file_secret_settings: PydanticBaseSettingsSource,
|
||||
) -> tuple[PydanticBaseSettingsSource, ...]:
|
||||
"""Customizes the order and handling of settings sources for a Pydantic BaseSettings subclass.
|
||||
settings_cls: Type[pydantic_settings.BaseSettings],
|
||||
init_settings: pydantic_settings.PydanticBaseSettingsSource,
|
||||
env_settings: pydantic_settings.PydanticBaseSettingsSource,
|
||||
dotenv_settings: pydantic_settings.PydanticBaseSettingsSource,
|
||||
file_secret_settings: pydantic_settings.PydanticBaseSettingsSource,
|
||||
) -> tuple[pydantic_settings.PydanticBaseSettingsSource, ...]:
|
||||
"""Customizes the order and handling of settings sources for a pydantic_settings.BaseSettings subclass.
|
||||
|
||||
This method determines the sources for application configuration settings, including
|
||||
environment variables, dotenv files and JSON configuration files.
|
||||
It ensures that a default configuration file exists and creates one if necessary.
|
||||
|
||||
Args:
|
||||
settings_cls (Type[BaseSettings]): The Pydantic BaseSettings class for which sources are customized.
|
||||
init_settings (PydanticBaseSettingsSource): The initial settings source, typically passed at runtime.
|
||||
env_settings (PydanticBaseSettingsSource): Settings sourced from environment variables.
|
||||
dotenv_settings (PydanticBaseSettingsSource): Settings sourced from a dotenv file.
|
||||
file_secret_settings (PydanticBaseSettingsSource): Unused (needed for parent class interface).
|
||||
settings_cls (Type[pydantic_settings.BaseSettings]): The Pydantic BaseSettings class for
|
||||
which sources are customized.
|
||||
init_settings (pydantic_settings.PydanticBaseSettingsSource): The initial settings source, typically passed at runtime.
|
||||
env_settings (pydantic_settings.PydanticBaseSettingsSource): Settings sourced from environment variables.
|
||||
dotenv_settings (pydantic_settings.PydanticBaseSettingsSource): Settings sourced from a dotenv file.
|
||||
file_secret_settings (pydantic_settings.PydanticBaseSettingsSource): Unused (needed for parent class interface).
|
||||
|
||||
Returns:
|
||||
tuple[PydanticBaseSettingsSource, ...]: A tuple of settings sources in the order they should be applied.
|
||||
tuple[pydantic_settings.PydanticBaseSettingsSource, ...]: A tuple of settings sources in the order they should be applied.
|
||||
|
||||
Behavior:
|
||||
1. Checks for the existence of a JSON configuration file in the expected location.
|
||||
2. If the configuration file does not exist, creates the directory (if needed) and attempts to copy a
|
||||
default configuration file to the location. If the copy fails, uses the default configuration file directly.
|
||||
3. Creates a `JsonConfigSettingsSource` for both the configuration file and the default configuration file.
|
||||
3. Creates a `pydantic_settings.JsonConfigSettingsSource` for both the configuration file and the default configuration file.
|
||||
4. Updates class attributes `GeneralSettings._config_folder_path` and
|
||||
`GeneralSettings._config_file_path` to reflect the determined paths.
|
||||
5. Returns a tuple containing all provided and newly created settings sources in the desired order.
|
||||
@@ -325,13 +354,7 @@ class ConfigEOS(SingletonMixin, SettingsEOSDefaults):
|
||||
- This method logs a warning if the default configuration file cannot be copied.
|
||||
- It ensures that a fallback to the default configuration file is always possible.
|
||||
"""
|
||||
setting_sources = [
|
||||
init_settings,
|
||||
env_settings,
|
||||
dotenv_settings,
|
||||
]
|
||||
|
||||
file_settings: Optional[JsonConfigSettingsSource] = None
|
||||
# Ensure we know and have the config folder path and the config file
|
||||
config_file, exists = cls._get_config_file_path()
|
||||
config_dir = config_file.parent
|
||||
if not exists:
|
||||
@@ -342,20 +365,38 @@ class ConfigEOS(SingletonMixin, SettingsEOSDefaults):
|
||||
logger.warning(f"Could not copy default config: {exc}. Using default config...")
|
||||
config_file = cls.config_default_file_path
|
||||
config_dir = config_file.parent
|
||||
try:
|
||||
file_settings = JsonConfigSettingsSource(settings_cls, json_file=config_file)
|
||||
setting_sources.append(file_settings)
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Error reading config file '{config_file}' (falling back to default config): {e}"
|
||||
)
|
||||
default_settings = JsonConfigSettingsSource(
|
||||
settings_cls, json_file=cls.config_default_file_path
|
||||
)
|
||||
# Remember config_dir and config file
|
||||
GeneralSettings._config_folder_path = config_dir
|
||||
GeneralSettings._config_file_path = config_file
|
||||
|
||||
# All the settings sources in priority sequence
|
||||
setting_sources = [
|
||||
init_settings,
|
||||
env_settings,
|
||||
dotenv_settings,
|
||||
]
|
||||
|
||||
# Apend file settings to sources
|
||||
file_settings: Optional[pydantic_settings.JsonConfigSettingsSource] = None
|
||||
try:
|
||||
backup_file = config_file.with_suffix(".bak")
|
||||
if migrate_config_file(config_file, backup_file):
|
||||
# If correct version add it as settings source
|
||||
file_settings = pydantic_settings.JsonConfigSettingsSource(
|
||||
settings_cls, json_file=config_file
|
||||
)
|
||||
setting_sources.append(file_settings)
|
||||
except Exception as ex:
|
||||
logger.error(
|
||||
f"Error reading config file '{config_file}' (falling back to default config): {ex}"
|
||||
)
|
||||
|
||||
# Append default settings to sources
|
||||
default_settings = pydantic_settings.JsonConfigSettingsSource(
|
||||
settings_cls, json_file=cls.config_default_file_path
|
||||
)
|
||||
setting_sources.append(default_settings)
|
||||
|
||||
return tuple(setting_sources)
|
||||
|
||||
@classproperty
|
||||
@@ -374,28 +415,24 @@ class ConfigEOS(SingletonMixin, SettingsEOSDefaults):
|
||||
Configuration data is loaded from a configuration file or a default one is created if none
|
||||
exists.
|
||||
"""
|
||||
logger.debug("Config init with parameters {} {}", args, kwargs)
|
||||
# Check for singleton guard
|
||||
if hasattr(self, "_initialized"):
|
||||
return
|
||||
self._setup(self, *args, **kwargs)
|
||||
|
||||
def _setup(self, *args: Any, **kwargs: Any) -> None:
|
||||
"""Re-initialize global settings."""
|
||||
# Check for config file content/ version type
|
||||
config_file, exists = self._get_config_file_path()
|
||||
if exists:
|
||||
with config_file.open("r", encoding="utf-8", newline=None) as f_config:
|
||||
config_txt = f_config.read()
|
||||
if '"directories": {' in config_txt or '"server_eos_host": ' in config_txt:
|
||||
error_msg = f"Configuration file '{config_file}' is outdated. Please remove or update manually."
|
||||
logger.error(error_msg)
|
||||
raise ValueError(error_msg)
|
||||
# Assure settings base knows EOS configuration
|
||||
logger.debug("Config setup with parameters {} {}", args, kwargs)
|
||||
# Assure settings base knows the singleton EOS configuration
|
||||
SettingsBaseModel.config = self
|
||||
# (Re-)load settings
|
||||
# (Re-)load settings - call base class init
|
||||
SettingsEOSDefaults.__init__(self, *args, **kwargs)
|
||||
# Init config file and data folder pathes
|
||||
self._create_initial_config_file()
|
||||
self._update_data_folder_path()
|
||||
self._initialized = True
|
||||
logger.debug("Config setup:\n{}", self)
|
||||
|
||||
def merge_settings(self, settings: SettingsEOS) -> None:
|
||||
"""Merges the provided settings into the global settings for EOS, with optional overwrite.
|
||||
@@ -488,6 +525,11 @@ class ConfigEOS(SingletonMixin, SettingsEOSDefaults):
|
||||
def _get_config_file_path(cls) -> tuple[Path, bool]:
|
||||
"""Find a valid configuration file or return the desired path for a new config file.
|
||||
|
||||
Searches:
|
||||
1. environment variable directory
|
||||
2. user configuration directory
|
||||
3. current working directory
|
||||
|
||||
Returns:
|
||||
tuple[Path, bool]: The path to the configuration file and if there is already a config file there
|
||||
"""
|
||||
|
||||
225
src/akkudoktoreos/config/configmigrate.py
Normal file
225
src/akkudoktoreos/config/configmigrate.py
Normal file
@@ -0,0 +1,225 @@
|
||||
"""Migrate config file to actual version."""
|
||||
|
||||
import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Dict, List, Set, Tuple, Union
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from akkudoktoreos.core.version import __version__
|
||||
|
||||
# -----------------------------
|
||||
# Global migration map constant
|
||||
# -----------------------------
|
||||
# key: old JSON path, value: either
|
||||
# - str (new model path)
|
||||
# - tuple[str, Callable[[Any], Any]] (new path + transform)
|
||||
# - None (drop)
|
||||
MIGRATION_MAP: Dict[str, Union[str, Tuple[str, Callable[[Any], Any]], None]] = {
|
||||
# 0.1.0 -> now
|
||||
"devices/batteries/0/initial_soc_percentage": None,
|
||||
"devices/electric_vehicles/0/initial_soc_percentage": None,
|
||||
"elecprice/provider_settings/import_file_path": "elecprice/provider_settings/ElecPriceImport/import_file_path",
|
||||
"elecprice/provider_settings/import_json": "elecprice/provider_settings/ElecPriceImport/import_json",
|
||||
"load/provider_settings/import_file_path": "load/provider_settings/LoadImport/import_file_path",
|
||||
"load/provider_settings/import_json": "load/provider_settings/LoadImport/import_json",
|
||||
"load/provider_settings/loadakkudoktor_year_energy": "load/provider_settings/LoadAkkudoktor/loadakkudoktor_year_energy",
|
||||
"load/provider_settings/load_vrm_idsite": "load/provider_settings/LoadVrm/load_vrm_idsite",
|
||||
"load/provider_settings/load_vrm_token": "load/provider_settings/LoadVrm/load_vrm_token",
|
||||
"logging/level": "logging/console_level",
|
||||
"logging/root_level": None,
|
||||
"measurement/load0_name": "measurement/load_emr_keys/0",
|
||||
"measurement/load1_name": "measurement/load_emr_keys/1",
|
||||
"measurement/load2_name": "measurement/load_emr_keys/2",
|
||||
"measurement/load3_name": "measurement/load_emr_keys/3",
|
||||
"measurement/load4_name": "measurement/load_emr_keys/4",
|
||||
"optimization/ev_available_charge_rates_percent": (
|
||||
"devices/electric_vehicles/0/charge_rates",
|
||||
lambda v: [x / 100 for x in v],
|
||||
),
|
||||
"optimization/hours": "optimization/horizon_hours",
|
||||
"optimization/penalty": ("optimization/genetic/penalties/ev_soc_miss", lambda v: float(v)),
|
||||
"pvforecast/provider_settings/import_file_path": "pvforecast/provider_settings/PVForecastImport/import_file_path",
|
||||
"pvforecast/provider_settings/import_json": "pvforecast/provider_settings/PVForecastImport/import_json",
|
||||
"pvforecast/provider_settings/load_vrm_idsite": "pvforecast/provider_settings/PVForecastVrm/load_vrm_idsite",
|
||||
"pvforecast/provider_settings/load_vrm_token": "pvforecast/provider_settings/PVForecastVrm/load_vrm_token",
|
||||
"weather/provider_settings/import_file_path": "weather/provider_settings/WeatherImport/import_file_path",
|
||||
"weather/provider_settings/import_json": "weather/provider_settings/WeatherImport/import_json",
|
||||
}
|
||||
|
||||
# -----------------------------
|
||||
# Global migration stats
|
||||
# -----------------------------
|
||||
migrated_source_paths: Set[str] = set()
|
||||
mapped_count: int = 0
|
||||
auto_count: int = 0
|
||||
skipped_paths: List[str] = []
|
||||
|
||||
|
||||
def migrate_config_file(config_file: Path, backup_file: Path) -> bool:
|
||||
"""Migrate configuration file to the current version.
|
||||
|
||||
Returns:
|
||||
bool: True if up-to-date or successfully migrated, False on failure.
|
||||
"""
|
||||
global migrated_source_paths, mapped_count, auto_count, skipped_paths
|
||||
|
||||
# Reset globals at the start of each migration
|
||||
migrated_source_paths = set()
|
||||
mapped_count = 0
|
||||
auto_count = 0
|
||||
skipped_paths = []
|
||||
|
||||
try:
|
||||
with config_file.open("r", encoding="utf-8") as f:
|
||||
config_data: Dict[str, Any] = json.load(f)
|
||||
except (FileNotFoundError, json.JSONDecodeError) as e:
|
||||
logger.error(f"Failed to read configuration file '{config_file}': {e}")
|
||||
return False
|
||||
|
||||
match config_data:
|
||||
case {"general": {"version": v}} if v == __version__:
|
||||
logger.debug(f"Configuration file '{config_file}' is up to date (v{v}).")
|
||||
return True
|
||||
case _:
|
||||
logger.info(
|
||||
f"Configuration file '{config_file}' is missing current version info. "
|
||||
f"Starting migration to v{__version__}..."
|
||||
)
|
||||
|
||||
try:
|
||||
# Backup existing file - we already know it is existing
|
||||
try:
|
||||
config_file.replace(backup_file)
|
||||
logger.info(f"Backed up old configuration to '{backup_file}'.")
|
||||
except Exception as e_replace:
|
||||
try:
|
||||
shutil.copy(config_file, backup_file)
|
||||
logger.info(
|
||||
f"Could not replace; copied old configuration to '{backup_file}' instead."
|
||||
)
|
||||
except Exception as e_copy:
|
||||
logger.warning(
|
||||
f"Failed to backup existing config (replace: {e_replace}; copy: {e_copy}). Continuing without backup."
|
||||
)
|
||||
|
||||
from akkudoktoreos.config.config import SettingsEOSDefaults
|
||||
|
||||
new_config = SettingsEOSDefaults()
|
||||
|
||||
# 1) Apply explicit migration map
|
||||
for old_path, mapping in MIGRATION_MAP.items():
|
||||
new_path = None
|
||||
transform = None
|
||||
if mapping is None:
|
||||
migrated_source_paths.add(old_path.strip("/"))
|
||||
logger.debug(f"🗑️ Migration map: dropping '{old_path}'")
|
||||
continue
|
||||
if isinstance(mapping, tuple):
|
||||
new_path, transform = mapping
|
||||
else:
|
||||
new_path = mapping
|
||||
|
||||
old_value = _get_json_nested_value(config_data, old_path)
|
||||
if old_value is None:
|
||||
continue
|
||||
|
||||
try:
|
||||
if transform:
|
||||
old_value = transform(old_value)
|
||||
new_config.set_nested_value(new_path, old_value)
|
||||
migrated_source_paths.add(old_path.strip("/"))
|
||||
mapped_count += 1
|
||||
logger.debug(f"✅ Migrated mapped '{old_path}' → '{new_path}' = {old_value!r}")
|
||||
except Exception as e:
|
||||
logger.opt(exception=True).warning(
|
||||
f"Failed mapped migration '{old_path}' -> '{new_path}': {e}", exc_info=True
|
||||
)
|
||||
|
||||
# 2) Automatic migration for remaining fields
|
||||
auto_count += _migrate_matching_fields(
|
||||
config_data, new_config, migrated_source_paths, skipped_paths
|
||||
)
|
||||
|
||||
# 3) Ensure version
|
||||
try:
|
||||
new_config.set_nested_value("general/version", __version__)
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not set version on new configuration model: {e}")
|
||||
|
||||
# 4) Write migrated configuration
|
||||
try:
|
||||
with config_file.open("w", encoding="utf-8", newline=None) as f_out:
|
||||
json_str = new_config.model_dump_json(indent=4)
|
||||
f_out.write(json_str)
|
||||
except Exception as e_write:
|
||||
logger.error(f"Failed to write migrated configuration to '{config_file}': {e_write}")
|
||||
return False
|
||||
|
||||
# 5) Log final migration summary
|
||||
logger.info(
|
||||
f"Migration summary for '{config_file}': "
|
||||
f"mapped fields: {mapped_count}, automatically migrated: {auto_count}, skipped: {len(skipped_paths)}"
|
||||
)
|
||||
if skipped_paths:
|
||||
logger.debug(f"Skipped paths: {', '.join(skipped_paths)}")
|
||||
|
||||
logger.success(f"Configuration successfully migrated to version {__version__}.")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Unexpected error during migration: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def _get_json_nested_value(data: dict, path: str) -> Any:
|
||||
"""Retrieve a nested value from a JSON-like dict using '/'-separated path."""
|
||||
current: Any = data
|
||||
for part in path.strip("/").split("/"):
|
||||
if isinstance(current, list):
|
||||
try:
|
||||
part_idx = int(part)
|
||||
current = current[part_idx]
|
||||
except (ValueError, IndexError):
|
||||
return None
|
||||
elif isinstance(current, dict):
|
||||
if part not in current:
|
||||
return None
|
||||
current = current[part]
|
||||
else:
|
||||
return None
|
||||
return current
|
||||
|
||||
|
||||
def _migrate_matching_fields(
|
||||
source: Dict[str, Any],
|
||||
target_model: Any,
|
||||
migrated_source_paths: Set[str],
|
||||
skipped_paths: List[str],
|
||||
prefix: str = "",
|
||||
) -> int:
|
||||
"""Recursively copy matching keys from source dict into target_model using set_nested_value.
|
||||
|
||||
Returns:
|
||||
int: number of fields successfully auto-migrated
|
||||
"""
|
||||
count: int = 0
|
||||
for key, value in source.items():
|
||||
full_path = f"{prefix}/{key}".strip("/")
|
||||
|
||||
if full_path in migrated_source_paths:
|
||||
continue
|
||||
|
||||
if isinstance(value, dict):
|
||||
count += _migrate_matching_fields(
|
||||
value, target_model, migrated_source_paths, skipped_paths, full_path
|
||||
)
|
||||
else:
|
||||
try:
|
||||
target_model.set_nested_value(full_path, value)
|
||||
count += 1
|
||||
except Exception:
|
||||
skipped_paths.append(full_path)
|
||||
continue
|
||||
return count
|
||||
Reference in New Issue
Block a user