Mirrors/EOS
1
0
Fork 0
mirror of https://github.com/Akkudoktor-EOS/EOS.git synced 2026-02-24 09:56:20 +00:00
Code Issues Actions 5 Packages Projects Releases Wiki Activity

Add database support for measurements and historic prediction data. (#848)

Browse Source 
The database supports backend selection, compression, incremental data load,
automatic data saving to storage, automatic vaccum and compaction.

Make SQLite3 and LMDB database backends available.

Update tests for new interface conventions regarding data sequences,
data containers, data providers. This includes the measurements provider and
the prediction providers.

Add database documentation.

The fix includes several bug fixes that are not directly related to the database
implementation but are necessary to keep EOS running properly and to test and
document the changes.

* fix: config eos test setup

  Make the config_eos fixture generate a new instance of the config_eos singleton.
  Use correct env names to setup data folder path.

* fix: startup with no config

  Make cache and measurements complain about missing data path configuration but
  do not bail out.

* fix: soc data preparation and usage for genetic optimization.

  Search for soc measurments 48 hours around the optimization start time.
  Only clamp soc to maximum in battery device simulation.

* fix: dashboard bailout on zero value solution display

  Do not use zero values to calculate the chart values adjustment for display.

* fix: openapi generation script

  Make the script also replace data_folder_path and data_output_path to hide
  real (test) environment pathes.

* feat: add make repeated task function

  make_repeated_task allows to wrap a function to be repeated cyclically.

* chore: removed index based data sequence access

  Index based data sequence access does not make sense as the sequence can be backed
  by the database. The sequence is now purely time series data.

* chore: refactor eos startup to avoid module import startup

  Avoid module import initialisation expecially of the EOS configuration.
  Config mutation, singleton initialization, logging setup, argparse parsing,
  background task definitions depending on config and environment-dependent behavior
  is now done at function startup.

* chore: introduce retention manager

  A single long-running background task that owns the scheduling of all periodic
  server-maintenance jobs (cache cleanup, DB autosave, …)

* chore: canonicalize timezone name for UTC

  Timezone names that are semantically identical to UTC are canonicalized to UTC.

* chore: extend config file migration for default value handling

  Extend the config file migration handling values None or nonexisting values
  that will invoke a default value generation in the new config file. Also
  adapt test to handle this situation.

* chore: extend datetime util test cases

* chore: make version test check for untracked files

  Check for files that are not tracked by git. Version calculation will be
  wrong if these files will not be commited.

* chore: bump pandas to 3.0.0

  Pandas 3.0 now performs inference on the appropriate resolution (a.k.a. unit)
  for the output dtype which may become datetime64[us] (before it was ns). Also
  numeric dtype detection is now more strict which needs a different detection for
  numerics.

* chore: bump pydantic-settings to 2.12.0

  pydantic-settings 2.12.0 under pytest creates a different behaviour. The tests
  were adapted and a workaround was introduced. Also ConfigEOS was adapted
  to allow for fine grain initialization control to be able to switch
  off certain settings such as file settings during test.

* chore: remove sci learn kit from dependencies

  The sci learn kit is not strictly necessary as long as we have scipy.

* chore: add documentation mode guarding for sphinx autosummary

  Sphinx autosummary excecutes functions. Prevent exceptions in case of pure doc
  mode.

* chore: adapt docker-build CI workflow to stricter GitHub handling

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
This commit is contained in:
Bobby Noelte
2026-02-22 14:12:42 +01:00
committed by GitHub
parent 5f66591d21
commit 6498c7dc32
92 changed files with 12710 additions and 2173 deletions
Download Patch File Download Diff File Expand all files Collapse all files

64
src/akkudoktoreos/core/cache.py
View File

@@ -13,6 +13,7 @@ import os
import pickle
import tempfile
import threading
from pathlib import Path
from typing import (
IO,
Any,
@@ -236,6 +237,24 @@ Param = ParamSpec("Param")
RetType = TypeVar("RetType")
def cache_clear(clear_all: Optional[bool] = None) -> None:
"""Cleanup expired cache files."""
if clear_all:
CacheFileStore().clear(clear_all=True)
else:
CacheFileStore().clear(before_datetime=to_datetime())
def cache_load() -> dict:
"""Load cache from cachefilestore.json."""
return CacheFileStore().load_store()
def cache_save() -> dict:
"""Save cache to cachefilestore.json."""
return CacheFileStore().save_store()
class CacheFileRecord(PydanticBaseModel):
cache_file: Any = Field(
..., json_schema_extra={"description": "File descriptor of the cache file."}
@@ -284,9 +303,16 @@ class CacheFileStore(ConfigMixin, SingletonMixin):
return
self._store: Dict[str, CacheFileRecord] = {}
self._store_lock = threading.RLock()
self._store_file = self.config.cache.path().joinpath("cachefilestore.json")
super().__init__(*args, **kwargs)
def _store_file(self) -> Optional[Path]:
"""Get file to store the cache."""
try:
return self.config.cache.path().joinpath("cachefilestore.json")
except Exception:
logger.error("Path for cache files missing. Please configure!")
return None
def _until_datetime_by_options(
self,
until_date: Optional[Any] = None,
@@ -496,10 +522,18 @@ class CacheFileStore(ConfigMixin, SingletonMixin):
# File already available
cache_file_obj = cache_item.cache_file
else:
self.config.cache.path().mkdir(parents=True, exist_ok=True)
cache_file_obj = tempfile.NamedTemporaryFile(
mode=mode, delete=delete, suffix=suffix, dir=self.config.cache.path()
)
# Create cache file
store_file = self._store_file()
if store_file:
store_file.parent.mkdir(parents=True, exist_ok=True)
cache_file_obj = tempfile.NamedTemporaryFile(
mode=mode, delete=delete, suffix=suffix, dir=store_file.parent
)
else:
# Cache storage not configured, use temporary path
cache_file_obj = tempfile.NamedTemporaryFile(
mode=mode, delete=delete, suffix=suffix
)
self._store[cache_file_key] = CacheFileRecord(
cache_file=cache_file_obj,
until_datetime=until_datetime_dt,
@@ -766,10 +800,14 @@ class CacheFileStore(ConfigMixin, SingletonMixin):
Returns:
data (dict): cache management data that was saved.
"""
store_file = self._store_file()
if store_file is None:
return {}
with self._store_lock:
self._store_file.parent.mkdir(parents=True, exist_ok=True)
store_file.parent.mkdir(parents=True, exist_ok=True)
store_to_save = self.current_store()
with self._store_file.open("w", encoding="utf-8", newline="\n") as f:
with store_file.open("w", encoding="utf-8", newline="\n") as f:
try:
json.dump(store_to_save, f, indent=4)
except Exception as e:
@@ -782,18 +820,22 @@ class CacheFileStore(ConfigMixin, SingletonMixin):
Returns:
data (dict): cache management data that was loaded.
"""
store_file = self._store_file()
if store_file is None:
return {}
with self._store_lock:
store_loaded = {}
if self._store_file.exists():
with self._store_file.open("r", encoding="utf-8", newline=None) as f:
if store_file.exists():
with store_file.open("r", encoding="utf-8", newline=None) as f:
try:
store_to_load = json.load(f)
except Exception as e:
logger.error(
f"Error loading cache file store: {e}\n"
+ f"Deleting the store file {self._store_file}."
+ f"Deleting the store file {store_file}."
)
self._store_file.unlink()
store_file.unlink()
return {}
for key, record in store_to_load.items():
if record is None:

3
src/akkudoktoreos/core/cachesettings.py
View File

@@ -20,7 +20,8 @@ class CacheCommonSettings(SettingsBaseModel):
)
cleanup_interval: float = Field(
default=5 * 60,
default=5.0 * 60,
ge=5.0,
json_schema_extra={"description": "Intervall in seconds for EOS file cache cleanup."},
)

453
src/akkudoktoreos/core/coreabc.py
View File

@@ -1,28 +1,76 @@
"""Abstract and base classes for EOS core.
This module provides foundational classes for handling configuration and prediction functionality
in EOS. It includes base classes that provide convenient access to global
configuration and prediction instances through properties.
Classes:
- ConfigMixin: Mixin class for managing and accessing global configuration.
- PredictionMixin: Mixin class for managing and accessing global prediction data.
- SingletonMixin: Mixin class to create singletons.
This module provides foundational classes and functions to access global EOS resources.
"""
from __future__ import (
annotations, # use types lazy as strings, helps to prevent circular dependencies
)
import threading
from typing import Any, ClassVar, Dict, Optional, Type
from typing import TYPE_CHECKING, Any, ClassVar, Dict, Optional, Type, Union
from loguru import logger
from akkudoktoreos.core.decorators import classproperty
from akkudoktoreos.utils.datetimeutil import DateTime
adapter_eos: Any = None
config_eos: Any = None
measurement_eos: Any = None
prediction_eos: Any = None
ems_eos: Any = None
if TYPE_CHECKING:
# Prevents circular dependies
from akkudoktoreos.adapter.adapter import Adapter
from akkudoktoreos.config.config import ConfigEOS
from akkudoktoreos.core.database import Database
from akkudoktoreos.core.ems import EnergyManagement
from akkudoktoreos.devices.devices import ResourceRegistry
from akkudoktoreos.measurement.measurement import Measurement
from akkudoktoreos.prediction.prediction import Prediction
# Module level singleton cache
_adapter_eos: Optional[Adapter] = None
_config_eos: Optional[ConfigEOS] = None
_ems_eos: Optional[EnergyManagement] = None
_database_eos: Optional[Database] = None
_measurement_eos: Optional[Measurement] = None
_prediction_eos: Optional[Prediction] = None
_resource_registry_eos: Optional[ResourceRegistry] = None
def get_adapter(init: bool = False) -> Adapter:
"""Retrieve the singleton EOS Adapter instance.
This function provides access to the global EOS Adapter instance. The Adapter
object is created on first access if `init` is True. If the instance is
accessed before initialization and `init` is False, a RuntimeError is raised.
Args:
init (bool): If True, create the Adapter instance if it does not exist.
Default is False.
Returns:
Adapter: The global EOS Adapter instance.
Raises:
RuntimeError: If accessed before initialization with `init=False`.
Usage:
.. code-block:: python
adapter = get_adapter(init=True) # Initialize and retrieve
adapter.do_something()
"""
global _adapter_eos
if _adapter_eos is None:
from akkudoktoreos.config.config import ConfigEOS
if not init and not ConfigEOS.documentation_mode():
raise RuntimeError("Adapter access before init.")
from akkudoktoreos.adapter.adapter import Adapter
_adapter_eos = Adapter()
return _adapter_eos
class AdapterMixin:
@@ -49,20 +97,84 @@ class AdapterMixin:
"""
@classproperty
def adapter(cls) -> Any:
def adapter(cls) -> Adapter:
"""Convenience class method/ attribute to retrieve the EOS adapters.
Returns:
Adapter: The adapters.
"""
# avoid circular dependency at import time
global adapter_eos
if adapter_eos is None:
from akkudoktoreos.adapter.adapter import get_adapter
return get_adapter()
adapter_eos = get_adapter()
return adapter_eos
def get_config(init: Union[bool, dict[str, bool]] = False) -> ConfigEOS:
"""Retrieve the singleton EOS configuration instance.
This function provides controlled access to the global EOS configuration
singleton (`ConfigEOS`). The configuration is created lazily on first
access and can be initialized with a configurable set of settings sources.
By default, accessing the configuration without prior initialization
raises a `RuntimeError`. Passing `init=True` or an initialization
configuration dictionary enables creation of the singleton.
Args:
init (Union[bool, dict[str, bool]]):
Controls initialization of the configuration.
- ``False`` (default): Do not initialize. Raises ``RuntimeError``
if the configuration does not yet exist.
- ``True``: Initialize the configuration using default
initialization behavior (all settings sources enabled).
- ``dict[str, bool]``: Initialize the configuration with fine-grained
control over which settings sources are enabled. Missing keys
default to ``True``.
Supported keys include:
- ``with_init_settings``
- ``with_env_settings``
- ``with_dotenv_settings``
- ``with_file_settings``
- ``with_file_secret_settings``
Returns:
ConfigEOS: The global EOS configuration singleton instance.
Raises:
RuntimeError:
If the configuration has not been initialized and ``init`` is
``False``.
Usage:
.. code-block:: python
# Initialize with default behavior (all sources enabled)
config = get_config(init=True)
# Initialize with explicit source control
config = get_config(init={
"with_init_settings": True,
"with_env_settings": True,
"with_dotenv_settings": True,
"with_file_settings": False,
"with_file_secret_settings": False,
})
# Access existing configuration
host = get_config().server.host
"""
global _config_eos
if _config_eos is None:
from akkudoktoreos.config.config import ConfigEOS
if not init and not ConfigEOS.documentation_mode():
raise RuntimeError("Config access before init.")
if isinstance(init, dict):
ConfigEOS._init_config_eos = init
_config_eos = ConfigEOS()
return _config_eos
class ConfigMixin:
@@ -89,20 +201,51 @@ class ConfigMixin:
"""
@classproperty
def config(cls) -> Any:
def config(cls) -> ConfigEOS:
"""Convenience class method/ attribute to retrieve the EOS configuration data.
Returns:
ConfigEOS: The configuration.
"""
# avoid circular dependency at import time
global config_eos
if config_eos is None:
from akkudoktoreos.config.config import get_config
return get_config()
config_eos = get_config()
return config_eos
def get_measurement(init: bool = False) -> Measurement:
"""Retrieve the singleton EOS Measurement instance.
This function provides access to the global EOS Measurement object. The
Measurement instance is created on first access if `init` is True. If the
instance is accessed before initialization and `init` is False, a RuntimeError
is raised.
Args:
init (bool): If True, create the Measurement instance if it does not exist.
Default is False.
Returns:
Measurement: The global EOS Measurement instance.
Raises:
RuntimeError: If accessed before initialization with `init=False`.
Usage:
.. code-block:: python
measurement = get_measurement(init=True) # Initialize and retrieve
measurement.read_sensor_data()
"""
global _measurement_eos
if _measurement_eos is None:
from akkudoktoreos.config.config import ConfigEOS
if not init and not ConfigEOS.documentation_mode():
raise RuntimeError("Measurement access before init.")
from akkudoktoreos.measurement.measurement import Measurement
_measurement_eos = Measurement()
return _measurement_eos
class MeasurementMixin:
@@ -130,20 +273,51 @@ class MeasurementMixin:
"""
@classproperty
def measurement(cls) -> Any:
def measurement(cls) -> Measurement:
"""Convenience class method/ attribute to retrieve the EOS measurement data.
Returns:
Measurement: The measurement.
"""
# avoid circular dependency at import time
global measurement_eos
if measurement_eos is None:
from akkudoktoreos.measurement.measurement import get_measurement
return get_measurement()
measurement_eos = get_measurement()
return measurement_eos
def get_prediction(init: bool = False) -> Prediction:
"""Retrieve the singleton EOS Prediction instance.
This function provides access to the global EOS Prediction object. The
Prediction instance is created on first access if `init` is True. If the
instance is accessed before initialization and `init` is False, a RuntimeError
is raised.
Args:
init (bool): If True, create the Prediction instance if it does not exist.
Default is False.
Returns:
Prediction: The global EOS Prediction instance.
Raises:
RuntimeError: If accessed before initialization with `init=False`.
Usage:
.. code-block:: python
prediction = get_prediction(init=True) # Initialize and retrieve
prediction.forecast_next_hour()
"""
global _prediction_eos
if _prediction_eos is None:
from akkudoktoreos.config.config import ConfigEOS
if not init and not ConfigEOS.documentation_mode():
raise RuntimeError("Prediction access before init.")
from akkudoktoreos.prediction.prediction import Prediction
_prediction_eos = Prediction()
return _prediction_eos
class PredictionMixin:
@@ -171,20 +345,50 @@ class PredictionMixin:
"""
@classproperty
def prediction(cls) -> Any:
def prediction(cls) -> Prediction:
"""Convenience class method/ attribute to retrieve the EOS prediction data.
Returns:
Prediction: The prediction.
"""
# avoid circular dependency at import time
global prediction_eos
if prediction_eos is None:
from akkudoktoreos.prediction.prediction import get_prediction
return get_prediction()
prediction_eos = get_prediction()
return prediction_eos
def get_ems(init: bool = False) -> EnergyManagement:
"""Retrieve the singleton EOS Energy Management System (EMS) instance.
This function provides access to the global EOS EMS instance. The instance
is created on first access if `init` is True. If the instance is accessed
before initialization and `init` is False, a RuntimeError is raised.
Args:
init (bool): If True, create the EMS instance if it does not exist.
Default is False.
Returns:
EnergyManagement: The global EOS EMS instance.
Raises:
RuntimeError: If accessed before initialization with `init=False`.
Usage:
.. code-block:: python
ems = get_ems(init=True) # Initialize and retrieve
ems.start_energy_management_loop()
"""
global _ems_eos
if _ems_eos is None:
from akkudoktoreos.config.config import ConfigEOS
if not init and not ConfigEOS.documentation_mode():
raise RuntimeError("EMS access before init.")
from akkudoktoreos.core.ems import EnergyManagement
_ems_eos = EnergyManagement()
return _ems_eos
class EnergyManagementSystemMixin:
@@ -200,7 +404,7 @@ class EnergyManagementSystemMixin:
global EnergyManagementSystem instance lazily to avoid import-time circular dependencies.
Attributes:
ems (EnergyManagementSystem): Property to access the global EOS energy management system.
ems (EnergyManagement): Property to access the global EOS energy management system.
Example:
.. code-block:: python
@@ -213,20 +417,120 @@ class EnergyManagementSystemMixin:
"""
@classproperty
def ems(cls) -> Any:
def ems(cls) -> EnergyManagement:
"""Convenience class method/ attribute to retrieve the EOS energy management system.
Returns:
EnergyManagementSystem: The energy management system.
"""
# avoid circular dependency at import time
global ems_eos
if ems_eos is None:
from akkudoktoreos.core.ems import get_ems
return get_ems()
ems_eos = get_ems()
return ems_eos
def get_database(init: bool = False) -> Database:
"""Retrieve the singleton EOS database instance.
This function provides access to the global EOS Database instance. The
instance is created on first access if `init` is True. If the instance is
accessed before initialization and `init` is False, a RuntimeError is raised.
Args:
init (bool): If True, create the Database instance if it does not exist.
Default is False.
Returns:
Database: The global EOS database instance.
Raises:
RuntimeError: If accessed before initialization with `init=False`.
Usage:
.. code-block:: python
db = get_database(init=True) # Initialize and retrieve
db.insert_measurement(...)
"""
global _database_eos
if _database_eos is None:
from akkudoktoreos.config.config import ConfigEOS
if not init and not ConfigEOS.documentation_mode():
raise RuntimeError("Database access before init.")
from akkudoktoreos.core.database import Database
_database_eos = Database()
return _database_eos
class DatabaseMixin:
"""Mixin class for managing EOS database access.
This class serves as a foundational component for EOS-related classes requiring access
to the EOS database. It provides a `database` property that dynamically retrieves
the database instance.
Usage:
Subclass this base class to gain access to the `database` attribute, which retrieves the
global database instance lazily to avoid import-time circular dependencies.
Attributes:
database (Database): Property to access the global EOS database.
Example:
.. code-block:: python
class MyOptimizationClass(PredictionMixin):
def store something(self):
db = self.database
"""
@classproperty
def database(cls) -> Database:
"""Convenience class method/ attribute to retrieve the EOS database.
Returns:
Database: The database.
"""
return get_database()
def get_resource_registry(init: bool = False) -> ResourceRegistry:
"""Retrieve the singleton EOS Resource Registry instance.
This function provides access to the global EOS ResourceRegistry instance.
The instance is created on first access if `init` is True. If the instance
is accessed before initialization and `init` is False, a RuntimeError is raised.
Args:
init (bool): If True, create the ResourceRegistry instance if it does not exist.
Default is False.
Returns:
ResourceRegistry: The global EOS Resource Registry instance.
Raises:
RuntimeError: If accessed before initialization with `init=False`.
Usage:
.. code-block:: python
registry = get_resource_registry(init=True) # Initialize and retrieve
registry.register_device(my_device)
"""
global _resource_registry_eos
if _resource_registry_eos is None:
from akkudoktoreos.config.config import ConfigEOS
if not init and not ConfigEOS.documentation_mode():
raise RuntimeError("ResourceRegistry access before init.")
from akkudoktoreos.devices.devices import ResourceRegistry
_resource_registry_eos = ResourceRegistry()
return _resource_registry_eos
class StartMixin(EnergyManagementSystemMixin):
@@ -243,14 +547,7 @@ class StartMixin(EnergyManagementSystemMixin):
Returns:
DateTime: The starting datetime of the current or latest energy management, or None.
"""
# avoid circular dependency at import time
global ems_eos
if ems_eos is None:
from akkudoktoreos.core.ems import get_ems
ems_eos = get_ems()
return ems_eos.start_datetime
return get_ems().start_datetime
class SingletonMixin:
@@ -332,3 +629,43 @@ class SingletonMixin:
if not hasattr(self, "_initialized"):
super().__init__(*args, **kwargs)
self._initialized = True
_singletons_init_running: bool = False
def singletons_init() -> None:
"""Initialize the singletons for adapter, config, measurement, prediction, database, resource registry."""
# Prevent recursive calling
global \
_singletons_init_running, \
_adapter_eos, \
_config_eos, \
_database_eos, \
_measurement_eos, \
_prediction_eos, \
_ems_eos, \
_resource_registry_eos
if _singletons_init_running:
return
_singletons_init_running = True
try:
if _config_eos is None:
get_config(init=True)
if _adapter_eos is None:
get_adapter(init=True)
if _database_eos is None:
get_database(init=True)
if _ems_eos is None:
get_ems(init=True)
if _measurement_eos is None:
get_measurement(init=True)
if _prediction_eos is None:
get_prediction(init=True)
if _resource_registry_eos is None:
get_resource_registry(init=True)
finally:
_singletons_init_running = False

1063
src/akkudoktoreos/core/dataabc.py
View File

File diff suppressed because it is too large Load Diff

1178
src/akkudoktoreos/core/database.py Normal file
View File

File diff suppressed because it is too large Load Diff

2194
src/akkudoktoreos/core/databaseabc.py Normal file
View File

File diff suppressed because it is too large Load Diff

84
src/akkudoktoreos/core/ems.py
View File

@@ -24,7 +24,7 @@ from akkudoktoreos.optimization.genetic.geneticparams import (
)
from akkudoktoreos.optimization.genetic.geneticsolution import GeneticSolution
from akkudoktoreos.optimization.optimization import OptimizationSolution
from akkudoktoreos.utils.datetimeutil import DateTime, compare_datetimes, to_datetime
from akkudoktoreos.utils.datetimeutil import DateTime, to_datetime
# The executor to execute the CPU heavy energy management run
executor = ThreadPoolExecutor(max_workers=1)
@@ -44,6 +44,15 @@ class EnergyManagementStage(Enum):
return self.value
async def ems_manage_energy() -> None:
"""Repeating task for managing energy.
This task should be executed by the server regularly
to ensure proper energy management.
"""
await EnergyManagement().run()
class EnergyManagement(
SingletonMixin, ConfigMixin, PredictionMixin, AdapterMixin, PydanticBaseModel
):
@@ -286,6 +295,9 @@ class EnergyManagement(
error_msg = f"Adapter update failed - phase {cls._stage}: {e}\n{trace}"
logger.error(error_msg)
# Remember energy run datetime.
EnergyManagement._last_run_datetime = to_datetime()
# energy management run finished
cls._stage = EnergyManagementStage.IDLE
@@ -346,73 +358,3 @@ class EnergyManagement(
)
# Run optimization in background thread to avoid blocking event loop
await loop.run_in_executor(executor, func)
async def manage_energy(self) -> None:
"""Repeating task for managing energy.
This task should be executed by the server regularly (e.g., every 10 seconds)
to ensure proper energy management. Configuration changes to the energy management interval
will only take effect if this task is executed.
- Initializes and runs the energy management for the first time if it has never been run
before.
- If the energy management interval is not configured or invalid (NaN), the task will not
trigger any repeated energy management runs.
- Compares the current time with the last run time and runs the energy management if the
interval has elapsed.
- Logs any exceptions that occur during the initialization or execution of the energy
management.
Note: The task maintains the interval even if some intervals are missed.
"""
current_datetime = to_datetime()
interval = self.config.ems.interval # interval maybe changed in between
if EnergyManagement._last_run_datetime is None:
# Never run before
try:
# Remember energy run datetime.
EnergyManagement._last_run_datetime = current_datetime
# Try to run a first energy management. May fail due to config incomplete.
await self.run()
except Exception as e:
trace = "".join(traceback.TracebackException.from_exception(e).format())
message = f"EOS init: {e}\n{trace}"
logger.error(message)
return
if interval is None or interval == float("nan"):
# No Repetition
return
if (
compare_datetimes(current_datetime, EnergyManagement._last_run_datetime).time_diff
< interval
):
# Wait for next run
return
try:
await self.run()
except Exception as e:
trace = "".join(traceback.TracebackException.from_exception(e).format())
message = f"EOS run: {e}\n{trace}"
logger.error(message)
# Remember the energy management run - keep on interval even if we missed some intervals
while (
compare_datetimes(current_datetime, EnergyManagement._last_run_datetime).time_diff
>= interval
):
EnergyManagement._last_run_datetime = EnergyManagement._last_run_datetime.add(
seconds=interval
)
# Initialize the Energy Management System, it is a singleton.
ems = EnergyManagement()
def get_ems() -> EnergyManagement:
"""Gets the EOS Energy Management System."""
return ems

7
src/akkudoktoreos/core/emsettings.py
View File

@@ -29,10 +29,11 @@ class EnergyManagementCommonSettings(SettingsBaseModel):
},
)
interval: Optional[float] = Field(
default=None,
interval: float = Field(
default=300.0,
ge=60.0,
json_schema_extra={
"description": "Intervall in seconds between EOS energy management runs.",
"description": "Intervall between EOS energy management runs [seconds].",
"examples": ["300"],
},
)

65
src/akkudoktoreos/core/pydantic.py
View File

@@ -47,7 +47,12 @@ from pydantic import (
)
from pydantic.fields import ComputedFieldInfo, FieldInfo
from akkudoktoreos.utils.datetimeutil import DateTime, to_datetime, to_duration
from akkudoktoreos.utils.datetimeutil import (
DateTime,
to_datetime,
to_duration,
to_timezone,
)
# Global weakref dictionary to hold external state per model instance
# Used as a workaround for PrivateAttr not working in e.g. Mixin Classes
@@ -683,13 +688,8 @@ class PydanticBaseModel(PydanticModelNestedValueMixin, BaseModel):
self, *args: Any, include_computed_fields: bool = True, **kwargs: Any
) -> dict[str, Any]:
"""Custom dump method to serialize computed fields by default."""
result = super().model_dump(*args, **kwargs)
if not include_computed_fields:
for computed_field_name in self.__class__.model_computed_fields:
result.pop(computed_field_name, None)
return result
kwargs.setdefault("exclude_computed_fields", not include_computed_fields)
return super().model_dump(*args, **kwargs)
def to_dict(self) -> dict:
"""Convert this PredictionRecord instance to a dictionary representation.
@@ -1061,8 +1061,8 @@ class PydanticDateTimeDataFrame(PydanticBaseModel):
valid_base_dtypes = {"int64", "float64", "bool", "object", "string"}
def is_valid_dtype(dtype: str) -> bool:
# Allow timezone-aware or naive datetime64
if dtype.startswith("datetime64[ns"):
# Allow timezone-aware or naive datetime64 - pandas 3.0 also has us
if dtype.startswith("datetime64[ns") or dtype.startswith("datetime64[us"):
return True
return dtype in valid_base_dtypes
@@ -1102,7 +1102,7 @@ class PydanticDateTimeDataFrame(PydanticBaseModel):
# Apply dtypes
for col, dtype in self.dtypes.items():
if dtype.startswith("datetime64[ns"):
if dtype.startswith("datetime64[ns") or dtype.startswith("datetime64[us"):
df[col] = pd.to_datetime(df[col], utc=True)
elif dtype in dtype_mapping.keys():
df[col] = df[col].astype(dtype_mapping[dtype])
@@ -1111,20 +1111,59 @@ class PydanticDateTimeDataFrame(PydanticBaseModel):
return df
@classmethod
def _detect_data_tz(cls, df: pd.DataFrame) -> Optional[str]:
"""Detect timezone of pandas data."""
# Index first (strongest signal)
if isinstance(df.index, pd.DatetimeIndex) and df.index.tz is not None:
return str(df.index.tz)
# Then datetime columns
for col in df.columns:
if is_datetime64_any_dtype(df[col]):
tz = getattr(df[col].dt, "tz", None)
if tz is not None:
return str(tz)
return None
@classmethod
def from_dataframe(
cls, df: pd.DataFrame, tz: Optional[str] = None
) -> "PydanticDateTimeDataFrame":
"""Create a PydanticDateTimeDataFrame instance from a pandas DataFrame."""
index = pd.Index([to_datetime(dt, as_string=True, in_timezone=tz) for dt in df.index])
# resolve timezone
data_tz = cls._detect_data_tz(df)
if tz is not None:
if data_tz and data_tz != tz:
raise ValueError(f"Timezone mismatch: tz='{tz}' but data uses '{data_tz}'")
resolved_tz = tz
else:
if data_tz:
resolved_tz = data_tz
else:
# Use local timezone
resolved_tz = to_timezone(as_string=True)
# normalize index
index = pd.Index(
[to_datetime(dt, as_string=True, in_timezone=resolved_tz) for dt in df.index]
)
df.index = index
# normalize datetime columns
datetime_columns = [col for col in df.columns if is_datetime64_any_dtype(df[col])]
for col in datetime_columns:
if df[col].dt.tz is None:
df[col] = df[col].dt.tz_localize(resolved_tz)
else:
df[col] = df[col].dt.tz_convert(resolved_tz)
return cls(
data=df.to_dict(orient="index"),
dtypes={col: str(dtype) for col, dtype in df.dtypes.items()},
tz=tz,
tz=resolved_tz,
datetime_columns=datetime_columns,
)

228
src/akkudoktoreos/core/version.py
View File

@@ -2,6 +2,7 @@
import hashlib
import re
from dataclasses import dataclass
from fnmatch import fnmatch
from pathlib import Path
from typing import Optional
@@ -16,14 +17,117 @@ HASH_EOS = ""
# Number of digits to append to .dev to identify a development version
VERSION_DEV_PRECISION = 8
# Hashing configuration
DIR_PACKAGE_ROOT = Path(__file__).resolve().parent.parent
ALLOWED_SUFFIXES: set[str] = {".py", ".md", ".json"}
EXCLUDED_DIR_PATTERNS: set[str] = {"*_autosum", "*__pycache__", "*_generated"}
EXCLUDED_FILES: set[Path] = set()
# ------------------------------
# Helpers for version generation
# ------------------------------
def is_excluded_dir(path: Path, excluded_dir_patterns: set[str]) -> bool:
"""Check whether a directory should be excluded based on name patterns."""
return any(fnmatch(path.name, pattern) for pattern in excluded_dir_patterns)
@dataclass
class HashConfig:
"""Configuration for file hashing."""
paths: list[Path]
allowed_suffixes: set[str]
excluded_dir_patterns: set[str]
excluded_files: set[Path]
def __post_init__(self) -> None:
"""Validate configuration."""
for path in self.paths:
if not path.exists():
raise ValueError(f"Path does not exist: {path}")
def is_excluded_dir(path: Path, patterns: set[str]) -> bool:
"""Check if directory matches any exclusion pattern.
Args:
path: Directory path to check
patterns: set of glob-like patterns (e.g., {``*__pycache__``, ``*_test``})
Returns:
True if directory should be excluded
"""
dir_name = path.name
return any(fnmatch(dir_name, pattern) for pattern in patterns)
def collect_files(config: HashConfig) -> list[Path]:
"""Collect all files that should be included in the hash.
This function only collects files - it doesn't hash them.
Makes it easy to inspect what will be hashed.
Args:
config: Hash configuration
Returns:
Sorted list of files to be hashed
Example:
>>> config = HashConfig(
... paths=[Path('src')],
... allowed_suffixes={'.py'},
... excluded_dir_patterns={'*__pycache__'},
... excluded_files=set()
... )
>>> files = collect_files(config)
>>> print(f"Will hash {len(files)} files")
>>> for f in files[:5]:
... print(f" {f}")
"""
collected_files: list[Path] = []
for root in config.paths:
for p in sorted(root.rglob("*")):
# Skip excluded directories
if p.is_dir() and is_excluded_dir(p, config.excluded_dir_patterns):
continue
# Skip files inside excluded directories
if any(is_excluded_dir(parent, config.excluded_dir_patterns) for parent in p.parents):
continue
# Skip excluded files
if p.resolve() in config.excluded_files:
continue
# Collect only allowed file types
if p.is_file() and p.suffix.lower() in config.allowed_suffixes:
collected_files.append(p.resolve())
return sorted(collected_files)
def hash_files(files: list[Path]) -> str:
"""Calculate SHA256 hash of file contents.
Args:
files: list of files to hash (order matters!)
Returns:
SHA256 hex digest
Example:
>>> files = [Path('file1.py'), Path('file2.py')]
>>> hash_value = hash_files(files)
"""
h = hashlib.sha256()
for file_path in files:
if not file_path.exists():
continue
h.update(file_path.read_bytes())
return h.hexdigest()
def hash_tree(
@@ -31,80 +135,93 @@ def hash_tree(
allowed_suffixes: set[str],
excluded_dir_patterns: set[str],
excluded_files: Optional[set[Path]] = None,
) -> str:
"""Return SHA256 hash for files under `paths`.
) -> tuple[str, list[Path]]:
"""Return SHA256 hash for files under `paths` and the list of files hashed.
Restricted by suffix, excluding excluded directory patterns and excluded_files.
Args:
paths: list of root paths to hash
allowed_suffixes: set of file suffixes to include (e.g., {'.py', '.json'})
excluded_dir_patterns: set of directory patterns to exclude
excluded_files: Optional set of specific files to exclude
Returns:
tuple of (hash_digest, list_of_hashed_files)
Example:
>>> hash_digest, files = hash_tree(
... paths=[Path('src')],
... allowed_suffixes={'.py'},
... excluded_dir_patterns={'*__pycache__'},
... )
>>> print(f"Hash: {hash_digest}")
>>> print(f"Based on {len(files)} files")
"""
h = hashlib.sha256()
excluded_files = excluded_files or set()
config = HashConfig(
paths=paths,
allowed_suffixes=allowed_suffixes,
excluded_dir_patterns=excluded_dir_patterns,
excluded_files=excluded_files or set(),
)
for root in paths:
if not root.exists():
raise ValueError(f"Root path does not exist: {root}")
for p in sorted(root.rglob("*")):
# Skip excluded directories
if p.is_dir() and is_excluded_dir(p, excluded_dir_patterns):
continue
files = collect_files(config)
digest = hash_files(files)
# Skip files inside excluded directories
if any(is_excluded_dir(parent, excluded_dir_patterns) for parent in p.parents):
continue
return digest, files
# Skip excluded files
if p.resolve() in excluded_files:
continue
# Hash only allowed file types
if p.is_file() and p.suffix.lower() in allowed_suffixes:
h.update(p.read_bytes())
digest = h.hexdigest()
return digest
# ---------------------
# Version hash function
# ---------------------
def _version_hash() -> str:
"""Calculate project hash.
Only package file ins src/akkudoktoreos can be hashed to make it work also for packages.
Only package files in src/akkudoktoreos can be hashed to make it work also for packages.
Returns:
SHA256 hash of the project files
"""
DIR_PACKAGE_ROOT = Path(__file__).resolve().parent.parent
if not str(DIR_PACKAGE_ROOT).endswith("src/akkudoktoreos"):
error_msg = f"DIR_PACKAGE_ROOT does not end with src/akkudoktoreos: {DIR_PACKAGE_ROOT}"
raise ValueError(error_msg)
# Allowed file suffixes to consider
ALLOWED_SUFFIXES: set[str] = {".py", ".md", ".json"}
# Directory patterns to exclude (glob-like)
EXCLUDED_DIR_PATTERNS: set[str] = {"*_autosum", "*__pycache__", "*_generated"}
# Files to exclude
EXCLUDED_FILES: set[Path] = set()
# Directories whose changes shall be part of the project hash
# Configuration
watched_paths = [DIR_PACKAGE_ROOT]
hash_current = hash_tree(
watched_paths, ALLOWED_SUFFIXES, EXCLUDED_DIR_PATTERNS, excluded_files=EXCLUDED_FILES
# Collect files and calculate hash
hash_digest, hashed_files = hash_tree(
watched_paths,
ALLOWED_SUFFIXES,
EXCLUDED_DIR_PATTERNS,
excluded_files=EXCLUDED_FILES,
)
return hash_current
return hash_digest
def _version_calculate() -> str:
"""Compute version."""
global HASH_EOS
HASH_EOS = _version_hash()
if VERSION_BASE.endswith("dev"):
"""Calculate the full version string.
For release versions: "x.y.z"
For dev versions: "x.y.z.dev<hash>"
Returns:
Full version string
"""
if VERSION_BASE.endswith(".dev"):
# After dev only digits are allowed - convert hexdigest to digits
hash_value = int(HASH_EOS, 16)
hash_value = int(_version_hash(), 16)
hash_digits = str(hash_value % (10**VERSION_DEV_PRECISION)).zfill(VERSION_DEV_PRECISION)
return f"{VERSION_BASE}{hash_digits}"
else:
# Release version - use base as-is
return VERSION_BASE
# ---------------------------
# Project version information
# ----------------------------
# ---------------------------
# The version
__version__ = _version_calculate()
@@ -114,16 +231,13 @@ __version__ = _version_calculate()
# Version info access
# -------------------
# Regular expression to split the version string into pieces
VERSION_RE = re.compile(
r"""
^(?P<base>\d+\.\d+\.\d+) # x.y.z
(?:[\.\+\-] # .dev<hash> starts here
(?:
(?P<dev>dev) # literal 'dev'
(?:(?P<hash>[A-Za-z0-9]+))? # optional <hash>
)
(?:\. # .dev<hash> starts here
(?P<dev>dev) # literal 'dev'
(?P<hash>[a-f0-9]+)? # optional <hash> (hex digits)
)?
$
""",
@@ -143,7 +257,7 @@ def version() -> dict[str, Optional[str]]:
.. code-block:: python
{
"version": "0.2.0+dev.a96a65",
"version": "0.2.0.dev.a96a65",
"base": "x.y.z",
"dev": "dev" or None,
"hash": "<hash>" or None,
@@ -153,7 +267,7 @@ def version() -> dict[str, Optional[str]]:
match = VERSION_RE.match(__version__)
if not match:
raise ValueError(f"Invalid version format: {version}")
raise ValueError(f"Invalid version format: {__version__}") # Fixed: was 'version'
info = match.groupdict()
info["version"] = __version__