Mirrors/EOS
1
0
Fork 0
mirror of https://github.com/Akkudoktor-EOS/EOS.git synced 2025-11-02 07:46:20 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

fix: automatic optimization (#596)

Browse Source 
This fix implements the long term goal to have the EOS server run optimization (or
energy management) on regular intervals automatically. Thus clients can request
the current energy management plan at any time and it is updated on regular
intervals without interaction by the client.

This fix started out to "only" make automatic optimization (or energy management)
runs working. It turned out there are several endpoints that in some way
update predictions or run the optimization. To lock against such concurrent attempts
the code had to be refactored to allow control of execution. During refactoring it
became clear that some classes and files are named without a proper reference
to their usage. Thus not only refactoring but also renaming became necessary.
The names are still not the best, but I hope they are more intuitive.

The fix includes several bug fixes that are not directly related to the automatic optimization
but are necessary to keep EOS running properly to do the automatic optimization and
to test and document the changes.

This is a breaking change as the configuration structure changed once again and
the server API was also enhanced and streamlined. The server API that is used by
Andreas and Jörg in their videos has not changed.

* fix: automatic optimization

  Allow optimization to automatically run on configured intervals gathering all
  optimization parameters from configuration and predictions. The automatic run
  can be configured to only run prediction updates skipping the optimization.
  Extend documentaion to also cover automatic optimization. Lock automatic runs
  against runs initiated by the /optimize or other endpoints. Provide new
  endpoints to retrieve the energy management plan and the genetic solution
  of the latest automatic optimization run. Offload energy management to thread
  pool executor to keep the app more responsive during the CPU heavy optimization
  run.

* fix: EOS servers recognize environment variables on startup

  Force initialisation of EOS configuration on server startup to assure
  all sources of EOS configuration are properly set up and read. Adapt
  server tests and configuration tests to also test for environment
  variable configuration.

* fix: Remove 0.0.0.0 to localhost translation under Windows

  EOS imposed a 0.0.0.0 to localhost translation under Windows for
  convenience. This caused some trouble in user configurations. Now, as the
  default IP address configuration is 127.0.0.1, the user is responsible
  for to set up the correct Windows compliant IP address.

* fix: allow names for hosts additional to IP addresses

* fix: access pydantic model fields by class

  Access by instance is deprecated.

* fix: down sampling key_to_array

* fix: make cache clear endpoint clear all cache files

  Make /v1/admin/cache/clear clear all cache files. Before it only cleared
  expired cache files by default. Add new endpoint /v1/admin/clear-expired
  to only clear expired cache files.

* fix: timezonefinder returns Europe/Paris instead of Europe/Berlin

  timezonefinder 8.10 got more inaccurate for timezones in europe as there is
  a common timezone. Use new package tzfpy instead which is still returning
  Europe/Berlin if you are in Germany. tzfpy also claims to be faster than
  timezonefinder.

* fix: provider settings configuration

  Provider configuration used to be a union holding the settings for several
  providers. Pydantic union handling does not always find the correct type
  for a provider setting. This led to exceptions in specific configurations.
  Now provider settings are explicit comfiguration items for each possible
  provider. This is a breaking change as the configuration structure was
  changed.

* fix: ClearOutside weather prediction irradiance calculation

  Pvlib needs a pandas time index. Convert time index.

* fix: test config file priority

  Do not use config_eos fixture as this fixture already creates a config file.

* fix: optimization sample request documentation

  Provide all data in documentation of optimization sample request.

* fix: gitlint blocking pip dependency resolution

  Replace gitlint by commitizen. Gitlint is not actively maintained anymore.
  Gitlint dependencies blocked pip from dependency resolution.

* fix: sync pre-commit config to actual dependency requirements

  .pre-commit-config.yaml was out of sync, also requirements-dev.txt.

* fix: missing babel in requirements.txt

  Add babel to requirements.txt

* feat: setup default device configuration for automatic optimization

  In case the parameters for automatic optimization are not fully defined a
  default configuration is setup to allow the automatic energy management
  run. The default configuration may help the user to correctly define
  the device configuration.

* feat: allow configuration of genetic algorithm parameters

  The genetic algorithm parameters for number of individuals, number of
  generations, the seed and penalty function parameters are now avaliable
  as configuration options.

* feat: allow configuration of home appliance time windows

  The time windows a home appliance is allowed to run are now configurable
  by the configuration (for /v1 API) and also by the home appliance parameters
  (for the classic /optimize API). If there is no such configuration the
  time window defaults to optimization hours, which was the standard before
  the change. Documentation on how to configure time windows is added.

* feat: standardize mesaurement keys for battery/ ev SoC measurements

  The standardized measurement keys to report battery SoC to the device
  simulations can now be retrieved from the device configuration as a
  read-only config option.

* feat: feed in tariff prediction

  Add feed in tarif predictions needed for automatic optimization. The feed in
  tariff can be retrieved as fixed feed in tarif or can be imported. Also add
  tests for the different feed in tariff providers. Extend documentation to
  cover the feed in tariff providers.

* feat: add energy management plan based on S2 standard instructions

  EOS can generate an energy management plan as a list of simple instructions.
  May be retrieved by the /v1/energy-management/plan endpoint. The instructions
  loosely follow the S2 energy management standard.

* feat: make measurement keys configurable by EOS configuration.

  The fixed measurement keys are replaced by configurable measurement keys.

* feat: make pendulum DateTime, Date, Duration types usable for pydantic models

  Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are
  added to the datetimeutil utility. Remove custom made pendulum adaptations
  from EOS pydantic module. Make EOS modules use the pydantic pendulum types
  managed by the datetimeutil module instead of the core pendulum types.

* feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil.

  The time windows are are added to support home appliance time window
  configuration. All time classes are also pydantic models. Time is the base
  class for time definition derived from pendulum.Time.

* feat: Extend DataRecord by configurable field like data.

  Configurable field like data was added to support the configuration of
  measurement records.

* feat: Add additional information to health information

  Version information is added to the health endpoints of eos and eosDash.
  The start time of the last optimization and the latest run time of the energy
  management is added to the EOS health information.

* feat: add pydantic merge model tests

* feat: add plan tab to EOSdash

  The plan tab displays the current energy management instructions.

* feat: add predictions tab to EOSdash

  The predictions tab displays the current predictions.

* feat: add cache management to EOSdash admin tab

  The admin tab is extended by a section for cache management. It allows to
  clear the cache.

* feat: add about tab to EOSdash

  The about tab resembles the former hello tab and provides extra information.

* feat: Adapt changelog and prepare for release management

  Release management using commitizen is added. The changelog file is adapted and
  teh changelog and a description for release management is added in the
  documentation.

* feat(doc): Improve install and devlopment documentation

  Provide a more concise installation description in Readme.md and add extra
  installation page and development page to documentation.

* chore: Use memory cache for interpolation instead of dict in inverter

  Decorate calculate_self_consumption() with @cachemethod_until_update to cache
  results in memory during an energy management/ optimization run. Replacement
  of dict type caching in inverter is now possible because all optimization
  runs are properly locked and the memory cache CacheUntilUpdateStore is properly
  cleared at the start of any energy management/ optimization operation.

* chore: refactor genetic

  Refactor the genetic algorithm modules for enhanced module structure and better
  readability. Removed unnecessary and overcomplex devices singleton. Also
  split devices configuration from genetic algorithm parameters to allow further
  development independently from genetic algorithm parameter format. Move
  charge rates configuration for electric vehicles from optimization to devices
  configuration to allow to have different charge rates for different cars in
  the future.

* chore: Rename memory cache to CacheEnergyManagementStore

  The name better resembles the task of the cache to chache function and method
  results for an energy management run. Also the decorator functions are renamed
  accordingly: cachemethod_energy_management, cache_energy_management

* chore: use class properties for config/ems/prediction mixin classes

* chore: skip debug logs from mathplotlib

  Mathplotlib is very noisy in debug mode.

* chore: automatically sync bokeh js to bokeh python package

  bokeh was updated to 3.8.0, make JS CDN automatically follow the package version.

* chore: rename hello.py to about.py

  Make hello.py the adapted EOSdash about page.

* chore: remove demo page from EOSdash

  As no the plan and prediction pages are working without configuration, the demo
  page is no longer necessary

* chore: split test_server.py for system test

  Split test_server.py to create explicit test_system.py for system tests.

* chore: move doc utils to generate_config_md.py

  The doc utils are only used in scripts/generate_config_md.py. Move it there to
  attribute for strong cohesion.

* chore: improve pydantic merge model documentation

* chore: remove pendulum warning from readme

* chore: remove GitHub discussions from contributing documentation

  Github discussions is to be replaced by Akkudoktor.net.

* chore(release): bump version to 0.1.0+dev for development

* build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1

  bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break.

* build(deps): bump uvicorn from 0.36.0 to 0.37.0

BREAKING CHANGE: EOS configuration changed. V1 API changed.

  - The available_charge_rates_percent configuration is removed from optimization.
    Use the new charge_rate configuration for the electric vehicle
  - Optimization configuration parameter hours renamed to horizon_hours
  - Device configuration now has to provide the number of devices and device
    properties per device.
  - Specific prediction provider configuration to be provided by explicit
    configuration item (no union for all providers).
  - Measurement keys to be provided as a list.
  - New feed in tariff providers have to be configured.
  - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints.
  - /v1/admin/cache/clear now clears all cache files. Use
    /v1/admin/cache/clear-expired to only clear all expired cache files.

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
This commit is contained in:
Bobby Noelte
2025-10-28 02:50:31 +01:00
committed by GitHub
parent 20a9eb78d8
commit b397b5d43e
146 changed files with 22024 additions and 5339 deletions
Download Patch File Download Diff File Expand all files Collapse all files

76
src/akkudoktoreos/core/cache.py
View File

@@ -28,12 +28,17 @@ from typing import (
import cachebox
from loguru import logger
from pendulum import DateTime, Duration
from pydantic import Field
from akkudoktoreos.core.coreabc import ConfigMixin, SingletonMixin
from akkudoktoreos.core.pydantic import PydanticBaseModel
from akkudoktoreos.utils.datetimeutil import compare_datetimes, to_datetime, to_duration
from akkudoktoreos.utils.datetimeutil import (
DateTime,
Duration,
compare_datetimes,
to_datetime,
to_duration,
)
# ---------------------------------
# In-Memory Caching Functionality
@@ -43,25 +48,28 @@ from akkudoktoreos.utils.datetimeutil import compare_datetimes, to_datetime, to_
TCallable = TypeVar("TCallable", bound=Callable[..., Any])
def cache_until_update_store_callback(event: int, key: Any, value: Any) -> None:
"""Calback function for CacheUntilUpdateStore."""
CacheUntilUpdateStore.last_event = event
CacheUntilUpdateStore.last_key = key
CacheUntilUpdateStore.last_value = value
def cache_energy_management_store_callback(event: int, key: Any, value: Any) -> None:
"""Calback function for CacheEnergyManagementStore."""
CacheEnergyManagementStore.last_event = event
CacheEnergyManagementStore.last_key = key
CacheEnergyManagementStore.last_value = value
if event == cachebox.EVENT_MISS:
CacheUntilUpdateStore.miss_count += 1
CacheEnergyManagementStore.miss_count += 1
elif event == cachebox.EVENT_HIT:
CacheUntilUpdateStore.hit_count += 1
CacheEnergyManagementStore.hit_count += 1
else:
# unreachable code
raise NotImplementedError
class CacheUntilUpdateStore(SingletonMixin):
class CacheEnergyManagementStore(SingletonMixin):
"""Singleton-based in-memory LRU (Least Recently Used) cache.
This cache is shared across the application to store results of decorated
methods or functions until the next EMS (Energy Management System) update.
methods or functions during energy management runs.
Energy management tasks shall clear the cache at the start of the energy management
task.
The cache uses an LRU eviction strategy, storing up to 100 items, with the oldest
items being evicted once the cache reaches its capacity.
@@ -75,14 +83,14 @@ class CacheUntilUpdateStore(SingletonMixin):
miss_count: ClassVar[int] = 0
def __init__(self, *args: Any, **kwargs: Any) -> None:
"""Initializes the `CacheUntilUpdateStore` instance with default parameters.
"""Initializes the `CacheEnergyManagementStore` instance with default parameters.
The cache uses an LRU eviction strategy with a maximum size of 100 items.
This cache is a singleton, meaning only one instance will exist throughout
the application lifecycle.
Example:
>>> cache = CacheUntilUpdateStore()
>>> cache = CacheEnergyManagementStore()
"""
if hasattr(self, "_initialized"):
return
@@ -128,7 +136,7 @@ class CacheUntilUpdateStore(SingletonMixin):
Example:
>>> value = cache["user_data"]
"""
return CacheUntilUpdateStore.cache[key]
return CacheEnergyManagementStore.cache[key]
def __setitem__(self, key: Any, value: Any) -> None:
"""Stores an item in the cache.
@@ -140,15 +148,15 @@ class CacheUntilUpdateStore(SingletonMixin):
Example:
>>> cache["user_data"] = {"name": "Alice", "age": 30}
"""
CacheUntilUpdateStore.cache[key] = value
CacheEnergyManagementStore.cache[key] = value
def __len__(self) -> int:
"""Returns the number of items in the cache."""
return len(CacheUntilUpdateStore.cache)
return len(CacheEnergyManagementStore.cache)
def __repr__(self) -> str:
"""Provides a string representation of the CacheUntilUpdateStore object."""
return repr(CacheUntilUpdateStore.cache)
"""Provides a string representation of the CacheEnergyManagementStore object."""
return repr(CacheEnergyManagementStore.cache)
def clear(self) -> None:
"""Clears the cache, removing all stored items.
@@ -161,22 +169,22 @@ class CacheUntilUpdateStore(SingletonMixin):
>>> cache.clear()
"""
if hasattr(self.cache, "clear") and callable(getattr(self.cache, "clear")):
CacheUntilUpdateStore.cache.clear()
CacheUntilUpdateStore.last_event = None
CacheUntilUpdateStore.last_key = None
CacheUntilUpdateStore.last_value = None
CacheUntilUpdateStore.miss_count = 0
CacheUntilUpdateStore.hit_count = 0
CacheEnergyManagementStore.cache.clear()
CacheEnergyManagementStore.last_event = None
CacheEnergyManagementStore.last_key = None
CacheEnergyManagementStore.last_value = None
CacheEnergyManagementStore.miss_count = 0
CacheEnergyManagementStore.hit_count = 0
else:
raise AttributeError(f"'{self.cache.__class__.__name__}' object has no method 'clear'")
def cachemethod_until_update(method: TCallable) -> TCallable:
def cachemethod_energy_management(method: TCallable) -> TCallable:
"""Decorator for in memory caching the result of an instance method.
This decorator caches the method's result in `CacheUntilUpdateStore`, ensuring
This decorator caches the method's result in `CacheEnergyManagementStore`, ensuring
that subsequent calls with the same arguments return the cached result until the
next EMS update cycle.
next energy management start.
Args:
method (Callable): The instance method to be decorated.
@@ -186,14 +194,14 @@ def cachemethod_until_update(method: TCallable) -> TCallable:
Example:
>>> class MyClass:
>>> @cachemethod_until_update
>>> @cachemethod_energy_management
>>> def expensive_method(self, param: str) -> str:
>>> # Perform expensive computation
>>> return f"Computed {param}"
"""
@cachebox.cachedmethod(
cache=CacheUntilUpdateStore().cache, callback=cache_until_update_store_callback
cache=CacheEnergyManagementStore().cache, callback=cache_energy_management_store_callback
)
@functools.wraps(method)
def wrapper(self: Any, *args: Any, **kwargs: Any) -> Any:
@@ -203,12 +211,12 @@ def cachemethod_until_update(method: TCallable) -> TCallable:
return wrapper
def cache_until_update(func: TCallable) -> TCallable:
def cache_energy_management(func: TCallable) -> TCallable:
"""Decorator for in memory caching the result of a standalone function.
This decorator caches the function's result in `CacheUntilUpdateStore`, ensuring
This decorator caches the function's result in `CacheEnergyManagementStore`, ensuring
that subsequent calls with the same arguments return the cached result until the
next EMS update cycle.
next energy management start.
Args:
func (Callable): The function to be decorated.
@@ -224,7 +232,7 @@ def cache_until_update(func: TCallable) -> TCallable:
"""
@cachebox.cached(
cache=CacheUntilUpdateStore().cache, callback=cache_until_update_store_callback
cache=CacheEnergyManagementStore().cache, callback=cache_energy_management_store_callback
)
@functools.wraps(func)
def wrapper(*args: Any, **kwargs: Any) -> Any:
@@ -435,7 +443,7 @@ class CacheFileStore(ConfigMixin, SingletonMixin):
)
logger.debug(
f"Search: ttl:{ttl_duration}, until:{until_datetime}, at:{at_datetime}, before:{before_datetime} -> hit: {generated_key == cache_file_key}, item: {cache_item.cache_file.seek(0), cache_item.cache_file.read()}"
f"Search: ttl:{ttl_duration}, until:{until_datetime}, at:{at_datetime}, before:{before_datetime} -> hit: {generated_key == cache_file_key}, item: {cache_item.cache_file.seek(0), cache_item.cache_file.read()[:10]}..."
)
if generated_key == cache_file_key:

87
src/akkudoktoreos/core/coreabc.py
View File

@@ -14,13 +14,13 @@ import threading
from typing import Any, ClassVar, Dict, Optional, Type
from loguru import logger
from pendulum import DateTime
from pydantic import computed_field
from akkudoktoreos.core.decorators import classproperty
from akkudoktoreos.utils.datetimeutil import DateTime
config_eos: Any = None
measurement_eos: Any = None
prediction_eos: Any = None
devices_eos: Any = None
ems_eos: Any = None
@@ -46,9 +46,9 @@ class ConfigMixin:
```
"""
@property
def config(self) -> Any:
"""Convenience method/ attribute to retrieve the EOS configuration data.
@classproperty
def config(cls) -> Any:
"""Convenience class method/ attribute to retrieve the EOS configuration data.
Returns:
ConfigEOS: The configuration.
@@ -86,9 +86,9 @@ class MeasurementMixin:
```
"""
@property
def measurement(self) -> Any:
"""Convenience method/ attribute to retrieve the EOS measurement data.
@classproperty
def measurement(cls) -> Any:
"""Convenience class method/ attribute to retrieve the EOS measurement data.
Returns:
Measurement: The measurement.
@@ -126,9 +126,9 @@ class PredictionMixin:
```
"""
@property
def prediction(self) -> Any:
"""Convenience method/ attribute to retrieve the EOS prediction data.
@classproperty
def prediction(cls) -> Any:
"""Convenience class method/ attribute to retrieve the EOS prediction data.
Returns:
Prediction: The prediction.
@@ -143,46 +143,6 @@ class PredictionMixin:
return prediction_eos
class DevicesMixin:
"""Mixin class for managing EOS devices simulation data.
This class serves as a foundational component for EOS-related classes requiring access
to global devices simulation data. It provides a `devices` property that dynamically retrieves
the devices instance, ensuring up-to-date access to devices simulation results.
Usage:
Subclass this base class to gain access to the `devices` attribute, which retrieves the
global devices instance lazily to avoid import-time circular dependencies.
Attributes:
devices (Devices): Property to access the global EOS devices simulation data.
Example:
```python
class MyOptimizationClass(DevicesMixin):
def analyze_mydevicesimulation(self):
device_simulation_data = self.devices.mydevicesresult
# Perform analysis
```
"""
@property
def devices(self) -> Any:
"""Convenience method/ attribute to retrieve the EOS devices simulation data.
Returns:
Devices: The devices simulation.
"""
# avoid circular dependency at import time
global devices_eos
if devices_eos is None:
from akkudoktoreos.devices.devices import get_devices
devices_eos = get_devices()
return devices_eos
class EnergyManagementSystemMixin:
"""Mixin class for managing EOS energy management system.
@@ -207,9 +167,9 @@ class EnergyManagementSystemMixin:
```
"""
@property
def ems(self) -> Any:
"""Convenience method/ attribute to retrieve the EOS energy management system.
@classproperty
def ems(cls) -> Any:
"""Convenience class method/ attribute to retrieve the EOS energy management system.
Returns:
EnergyManagementSystem: The energy management system.
@@ -231,16 +191,21 @@ class StartMixin(EnergyManagementSystemMixin):
- `start_datetime`: The starting datetime of the current or latest energy management.
"""
# Computed field for start_datetime
@computed_field # type: ignore[prop-decorator]
@property
def start_datetime(self) -> Optional[DateTime]:
"""Returns the start datetime of the current or latest energy management.
@classproperty
def ems_start_datetime(cls) -> Optional[DateTime]:
"""Convenience class method/ attribute to retrieve the start datetime of the current or latest energy management.
Returns:
DateTime: The starting datetime of the current or latest energy management, or None.
"""
return self.ems.start_datetime
# avoid circular dependency at import time
global ems_eos
if ems_eos is None:
from akkudoktoreos.core.ems import get_ems
ems_eos = get_ems()
return ems_eos.start_datetime
class SingletonMixin:

275
src/akkudoktoreos/core/dataabc.py
View File

@@ -14,14 +14,23 @@ from abc import abstractmethod
from collections.abc import MutableMapping, MutableSequence
from itertools import chain
from pathlib import Path
from typing import Any, Dict, Iterator, List, Optional, Tuple, Type, Union, overload
from typing import (
Any,
Dict,
Iterator,
List,
Optional,
Tuple,
Type,
Union,
overload,
)
import numpy as np
import pandas as pd
import pendulum
from loguru import logger
from numpydantic import NDArray, Shape
from pendulum import DateTime, Duration
from pydantic import (
AwareDatetime,
ConfigDict,
@@ -29,6 +38,7 @@ from pydantic import (
ValidationError,
computed_field,
field_validator,
model_validator,
)
from akkudoktoreos.core.coreabc import ConfigMixin, SingletonMixin, StartMixin
@@ -37,7 +47,13 @@ from akkudoktoreos.core.pydantic import (
PydanticDateTimeData,
PydanticDateTimeDataFrame,
)
from akkudoktoreos.utils.datetimeutil import compare_datetimes, to_datetime, to_duration
from akkudoktoreos.utils.datetimeutil import (
DateTime,
Duration,
compare_datetimes,
to_datetime,
to_duration,
)
class DataBase(ConfigMixin, StartMixin, PydanticBaseModel):
@@ -55,6 +71,11 @@ class DataRecord(DataBase, MutableMapping):
Fields can be accessed and mutated both using dictionary-style access (`record['field_name']`)
and attribute-style access (`record.field_name`).
The data record also provides configured field like data. Configuration has to be done by the
derived class. Configuration is a list of key strings, which is usually taken from the EOS
configuration. The internal field for these data `configured_data` is mostly hidden from
dictionary-style and attribute-style access.
Attributes:
date_time (Optional[DateTime]): Aware datetime indicating when the data record applies.
@@ -65,9 +86,42 @@ class DataRecord(DataBase, MutableMapping):
date_time: Optional[DateTime] = Field(default=None, description="DateTime")
configured_data: dict[str, Any] = Field(
default_factory=dict,
description="Configured field like data",
examples=[{"load0_mr": 40421}],
)
# Pydantic v2 model configuration
model_config = ConfigDict(arbitrary_types_allowed=True, populate_by_name=True)
@model_validator(mode="before")
@classmethod
def init_configured_field_like_data(cls, data: Any) -> Any:
"""Extracts configured data keys from the input and assigns them to `configured_data`.
This validator is called before the model is initialized. It filters out any keys from the input
dictionary that are listed in the configured data keys, and moves them into
the `configured_data` field of the model. This enables flexible, key-driven population of
dynamic data while keeping the model schema clean.
Args:
data (Any): The raw input data used to initialize the model.
Returns:
Any: The modified input data dictionary, with configured keys moved to `configured_data`.
"""
if not isinstance(data, dict):
return data
configured_keys: Union[list[str], set] = cls.configured_data_keys() or set()
extracted = {k: data.pop(k) for k in list(data.keys()) if k in configured_keys}
if extracted:
data.setdefault("configured_data", {}).update(extracted)
return data
@field_validator("date_time", mode="before")
@classmethod
def transform_to_datetime(cls, value: Any) -> Optional[DateTime]:
@@ -77,18 +131,39 @@ class DataRecord(DataBase, MutableMapping):
return None
return to_datetime(value)
@classmethod
def configured_data_keys(cls) -> Optional[list[str]]:
"""Return the keys for the configured field like data.
Can be overwritten by derived classes to define specific field like data. Usually provided
by configuration data.
"""
return None
@classmethod
def record_keys(cls) -> List[str]:
"""Returns the keys of all fields in the data record."""
key_list = []
key_list.extend(list(cls.model_fields.keys()))
key_list.extend(list(cls.__pydantic_decorators__.computed_fields.keys()))
# Add also keys that may be added by configuration
key_list.remove("configured_data")
configured_keys = cls.configured_data_keys()
if configured_keys is not None:
key_list.extend(configured_keys)
return key_list
@classmethod
def record_keys_writable(cls) -> List[str]:
"""Returns the keys of all fields in the data record that are writable."""
return list(cls.model_fields.keys())
keys_writable = []
keys_writable.extend(list(cls.model_fields.keys()))
# Add also keys that may be added by configuration
keys_writable.remove("configured_data")
configured_keys = cls.configured_data_keys()
if configured_keys is not None:
keys_writable.extend(configured_keys)
return keys_writable
def _validate_key_writable(self, key: str) -> None:
"""Verify that a specified key exists and is writable in the current record keys.
@@ -104,6 +179,40 @@ class DataRecord(DataBase, MutableMapping):
f"Key '{key}' is not in writable record keys: {self.record_keys_writable()}"
)
def __dir__(self) -> list[str]:
"""Extend the default `dir()` output to include configured field like data keys.
This enables editor auto-completion and interactive introspection, while hiding the internal
`configured_data` dictionary.
This ensures the configured field like data values appear like native fields,
in line with the base model's attribute behavior.
"""
base = super().__dir__()
keys = set(base)
# Expose configured data keys as attributes
configured_keys = self.configured_data_keys()
if configured_keys is not None:
keys.update(configured_keys)
# Explicitly hide the 'configured_data' internal dict
keys.discard("configured_data")
return sorted(keys)
def __eq__(self, other: Any) -> bool:
"""Ensure equality comparison includes the contents of the `configured_data` dict.
Contents of the `configured_data` dict are in addition to the base model fields.
"""
if not isinstance(other, self.__class__):
return NotImplemented
# Compare all fields except `configured_data`
if self.model_dump(exclude={"configured_data"}) != other.model_dump(
exclude={"configured_data"}
):
return False
# Compare `configured_data` explicitly
return self.configured_data == other.configured_data
def __getitem__(self, key: str) -> Any:
"""Retrieve the value of a field by key name.
@@ -116,9 +225,11 @@ class DataRecord(DataBase, MutableMapping):
Raises:
KeyError: If the specified key does not exist.
"""
if key in self.model_fields:
return getattr(self, key)
raise KeyError(f"'{key}' not found in the record fields.")
try:
# Let getattr do the work
return self.__getattr__(key)
except:
raise KeyError(f"'{key}' not found in the record fields.")
def __setitem__(self, key: str, value: Any) -> None:
"""Set the value of a field by key name.
@@ -130,9 +241,10 @@ class DataRecord(DataBase, MutableMapping):
Raises:
KeyError: If the specified key does not exist in the fields.
"""
if key in self.model_fields:
setattr(self, key, value)
else:
try:
# Let setattr do the work
self.__setattr__(key, value)
except:
raise KeyError(f"'{key}' is not a recognized field.")
def __delitem__(self, key: str) -> None:
@@ -144,9 +256,9 @@ class DataRecord(DataBase, MutableMapping):
Raises:
KeyError: If the specified key does not exist in the fields.
"""
if key in self.model_fields:
setattr(self, key, None) # Optional: set to None instead of deleting
else:
try:
self.__delattr__(key)
except:
raise KeyError(f"'{key}' is not a recognized field.")
def __iter__(self) -> Iterator[str]:
@@ -155,7 +267,7 @@ class DataRecord(DataBase, MutableMapping):
Returns:
Iterator[str]: An iterator over field names.
"""
return iter(self.model_fields)
return iter(self.record_keys_writable())
def __len__(self) -> int:
"""Return the number of fields in the data record.
@@ -163,7 +275,7 @@ class DataRecord(DataBase, MutableMapping):
Returns:
int: The number of defined fields.
"""
return len(self.model_fields)
return len(self.record_keys_writable())
def __repr__(self) -> str:
"""Provide a string representation of the data record.
@@ -171,7 +283,7 @@ class DataRecord(DataBase, MutableMapping):
Returns:
str: A string representation showing field names and their values.
"""
field_values = {field: getattr(self, field) for field in self.model_fields}
field_values = {field: getattr(self, field) for field in self.__class__.model_fields}
return f"{self.__class__.__name__}({field_values})"
def __getattr__(self, key: str) -> Any:
@@ -186,8 +298,13 @@ class DataRecord(DataBase, MutableMapping):
Raises:
AttributeError: If the field does not exist.
"""
if key in self.model_fields:
if key in self.__class__.model_fields:
return getattr(self, key)
if key in self.configured_data.keys():
return self.configured_data[key]
configured_keys = self.configured_data_keys()
if configured_keys is not None and key in configured_keys:
return None
raise AttributeError(f"'{self.__class__.__name__}' object has no attribute '{key}'")
def __setattr__(self, key: str, value: Any) -> None:
@@ -200,10 +317,14 @@ class DataRecord(DataBase, MutableMapping):
Raises:
AttributeError: If the attribute/field does not exist.
"""
if key in self.model_fields:
if key in self.__class__.model_fields:
super().__setattr__(key, value)
else:
raise AttributeError(f"'{self.__class__.__name__}' object has no attribute '{key}'")
return
configured_keys = self.configured_data_keys()
if configured_keys is not None and key in configured_keys:
self.configured_data[key] = value
return
raise AttributeError(f"'{self.__class__.__name__}' object has no attribute '{key}'")
def __delattr__(self, key: str) -> None:
"""Delete an attribute by setting it to None if it exists as a field.
@@ -214,10 +335,21 @@ class DataRecord(DataBase, MutableMapping):
Raises:
AttributeError: If the attribute/field does not exist.
"""
if key in self.model_fields:
setattr(self, key, None) # Optional: set to None instead of deleting
else:
super().__delattr__(key)
if key in self.__class__.model_fields:
data: Optional[dict]
if key == "configured_data":
data = dict()
else:
data = None
setattr(self, key, data)
return
if key in self.configured_data:
del self.configured_data[key]
return
configured_keys = self.configured_data_keys()
if configured_keys is not None and key in configured_keys:
return
super().__delattr__(key)
@classmethod
def key_from_description(cls, description: str, threshold: float = 0.8) -> Optional[str]:
@@ -352,10 +484,7 @@ class DataSequence(DataBase, MutableSequence):
@property
def record_keys(self) -> List[str]:
"""Returns the keys of all fields in the data records."""
key_list = []
key_list.extend(list(self.record_class().model_fields.keys()))
key_list.extend(list(self.record_class().__pydantic_decorators__.computed_fields.keys()))
return key_list
return self.record_class().record_keys()
@computed_field # type: ignore[prop-decorator]
@property
@@ -369,7 +498,7 @@ class DataSequence(DataBase, MutableSequence):
Returns:
List[str]: A list of field keys that are writable in the data records.
"""
return list(self.record_class().model_fields.keys())
return self.record_class().record_keys_writable()
@classmethod
def record_class(cls) -> Type:
@@ -707,6 +836,38 @@ class DataSequence(DataBase, MutableSequence):
return filtered_data
def key_to_value(self, key: str, target_datetime: DateTime) -> Optional[float]:
"""Returns the value corresponding to the specified key that is nearest to the given datetime.
Args:
key (str): The key of the attribute in DataRecord to extract.
target_datetime (datetime): The datetime to search nearest to.
Returns:
Optional[float]: The value nearest to the given datetime, or None if no valid records are found.
Raises:
KeyError: If the specified key is not found in any of the DataRecords.
"""
self._validate_key(key)
# Filter out records with None or NaN values for the key
valid_records = [
record
for record in self.records
if record.date_time is not None
and getattr(record, key, None) not in (None, float("nan"))
]
if not valid_records:
return None
# Find the record with datetime nearest to target_datetime
target = to_datetime(target_datetime)
nearest_record = min(valid_records, key=lambda r: abs(r.date_time - target))
return getattr(nearest_record, key, None)
def key_to_lists(
self,
key: str,
@@ -868,6 +1029,11 @@ class DataSequence(DataBase, MutableSequence):
KeyError: If the specified key is not found in any of the DataRecords.
"""
self._validate_key(key)
# General check on fill_method
if fill_method not in ("ffill", "bfill", "linear", "none", None):
raise ValueError(f"Unsupported fill method: {fill_method}")
# Ensure datetime objects are normalized
start_datetime = to_datetime(start_datetime, to_maxtime=False) if start_datetime else None
end_datetime = to_datetime(end_datetime, to_maxtime=False) if end_datetime else None
@@ -880,7 +1046,7 @@ class DataSequence(DataBase, MutableSequence):
values_len = len(values)
if values_len < 1:
# No values, assume at at least one value set to None
# No values, assume at least one value set to None
if start_datetime is not None:
dates.append(start_datetime - interval)
else:
@@ -902,6 +1068,11 @@ class DataSequence(DataBase, MutableSequence):
# Truncate all values before latest value before start_datetime
dates = dates[start_index - 1 :]
values = values[start_index - 1 :]
# We have a start_datetime, align to start datetime
resample_origin = start_datetime
else:
# We do not have a start_datetime, align resample buckets to midnight of first day
resample_origin = "start_day"
if end_datetime is not None:
if compare_datetimes(dates[-1], end_datetime).lt:
@@ -922,7 +1093,7 @@ class DataSequence(DataBase, MutableSequence):
if fill_method is None:
fill_method = "linear"
# Resample the series to the specified interval
resampled = series.resample(interval, origin="start").first()
resampled = series.resample(interval, origin=resample_origin).first()
if fill_method == "linear":
resampled = resampled.interpolate(method="linear")
elif fill_method == "ffill":
@@ -936,7 +1107,7 @@ class DataSequence(DataBase, MutableSequence):
if fill_method is None:
fill_method = "ffill"
# Resample the series to the specified interval
resampled = series.resample(interval, origin="start").first()
resampled = series.resample(interval, origin=resample_origin).first()
if fill_method == "ffill":
resampled = resampled.ffill()
elif fill_method == "bfill":
@@ -944,12 +1115,24 @@ class DataSequence(DataBase, MutableSequence):
elif fill_method != "none":
raise ValueError(f"Unsupported fill method for non-numeric data: {fill_method}")
logger.debug(
"Resampled for '{}' with length {}: {}...{}",
key,
len(resampled),
resampled[:10],
resampled[-10:],
)
# Convert the resampled series to a NumPy array
if start_datetime is not None and len(resampled) > 0:
resampled = resampled.truncate(before=start_datetime)
if end_datetime is not None and len(resampled) > 0:
resampled = resampled.truncate(after=end_datetime.subtract(seconds=1))
array = resampled.values
logger.debug(
"Array for '{}' with length {}: {}...{}", key, len(array), array[:10], array[-10:]
)
return array
def to_dataframe(
@@ -1197,7 +1380,7 @@ class DataImportMixin:
the values. `ìnterval` may be used to define the fixed time interval between two values.
On import `self.update_value(datetime, key, value)` is called which has to be provided.
Also `self.start_datetime` may be necessary as a default in case `start_datetime`is not given.
Also `self.ems_start_datetime` may be necessary as a default in case `start_datetime`is not given.
"""
# Attributes required but defined elsehere.
@@ -1315,7 +1498,7 @@ class DataImportMixin:
raise ValueError(f"Invalid start_datetime in import data: {e}")
if start_datetime is None:
start_datetime = self.start_datetime # type: ignore
start_datetime = self.ems_start_datetime # type: ignore
if "interval" in import_data:
try:
@@ -1406,7 +1589,7 @@ class DataImportMixin:
raise ValueError(f"Invalid datetime index in DataFrame: {e}")
else:
if start_datetime is None:
start_datetime = self.start_datetime # type: ignore
start_datetime = self.ems_start_datetime # type: ignore
has_datetime_index = False
# Filter columns based on key_prefix and record_keys_writable
@@ -1463,7 +1646,7 @@ class DataImportMixin:
If start_datetime and or interval is given in the JSON dict it will be used. Otherwise
the given parameters are used. If None is given start_datetime defaults to
'self.start_datetime' and interval defaults to 1 hour.
'self.ems_start_datetime' and interval defaults to 1 hour.
Args:
json_str (str): The JSON string containing the generic data.
@@ -1538,7 +1721,7 @@ class DataImportMixin:
If start_datetime and or interval is given in the JSON dict it will be used. Otherwise
the given parameters are used. If None is given start_datetime defaults to
'self.start_datetime' and interval defaults to 1 hour.
'self.ems_start_datetime' and interval defaults to 1 hour.
Args:
import_file_path (Path): The path to the JSON file containing the generic data.
@@ -1749,7 +1932,12 @@ class DataContainer(SingletonMixin, DataBase, MutableMapping):
force_update (bool, optional): If True, forces the providers to update the data even if still cached.
"""
for provider in self.providers:
provider.update_data(force_enable=force_enable, force_update=force_update)
try:
provider.update_data(force_enable=force_enable, force_update=force_update)
except Exception as ex:
error = f"Provider {provider.provider_id()} fails on update - enabled={provider.enabled()}, force_enable={force_enable}, force_update={force_update}: {ex}"
logger.error(error)
raise RuntimeError(error)
def key_to_series(
self,
@@ -1854,7 +2042,7 @@ class DataContainer(SingletonMixin, DataBase, MutableMapping):
) -> pd.DataFrame:
"""Retrieve a dataframe indexed by fixed time intervals for specified keys from the data in each DataProvider.
Generates a pandas DataFrame using the NumPy arrays for each specified key, ensuring a common time index..
Generates a pandas DataFrame using the NumPy arrays for each specified key, ensuring a common time index.
Args:
keys (list[str]): A list of field names to retrieve.
@@ -1903,8 +2091,15 @@ class DataContainer(SingletonMixin, DataBase, MutableMapping):
end_datetime.add(seconds=1)
# Create a DatetimeIndex based on start, end, and interval
if start_datetime is None or end_datetime is None:
raise ValueError(
f"Can not determine datetime range. Got '{start_datetime}'..'{end_datetime}'."
)
reference_index = pd.date_range(
start=start_datetime, end=end_datetime, freq=interval, inclusive="left"
start=start_datetime,
end=end_datetime,
freq=interval,
inclusive="left",
)
data = {}

1914
src/akkudoktoreos/core/emplan.py Normal file
View File

File diff suppressed because it is too large Load Diff

641
src/akkudoktoreos/core/ems.py
View File

@@ -1,123 +1,50 @@
import traceback
from typing import Any, ClassVar, Optional
from asyncio import Lock, get_running_loop
from concurrent.futures import ThreadPoolExecutor
from functools import partial
from typing import ClassVar, Optional
import numpy as np
from loguru import logger
from numpydantic import NDArray, Shape
from pendulum import DateTime
from pydantic import ConfigDict, Field, computed_field, field_validator, model_validator
from typing_extensions import Self
from pydantic import computed_field
from akkudoktoreos.core.cache import CacheUntilUpdateStore
from akkudoktoreos.core.cache import CacheEnergyManagementStore
from akkudoktoreos.core.coreabc import ConfigMixin, PredictionMixin, SingletonMixin
from akkudoktoreos.core.pydantic import ParametersBaseModel, PydanticBaseModel
from akkudoktoreos.devices.battery import Battery
from akkudoktoreos.devices.generic import HomeAppliance
from akkudoktoreos.devices.inverter import Inverter
from akkudoktoreos.utils.datetimeutil import compare_datetimes, to_datetime
from akkudoktoreos.utils.utils import NumpyEncoder
from akkudoktoreos.core.emplan import EnergyManagementPlan
from akkudoktoreos.core.emsettings import EnergyManagementMode
from akkudoktoreos.core.pydantic import PydanticBaseModel
from akkudoktoreos.optimization.genetic.genetic import GeneticOptimization
from akkudoktoreos.optimization.genetic.geneticparams import (
GeneticOptimizationParameters,
)
from akkudoktoreos.optimization.genetic.geneticsolution import GeneticSolution
from akkudoktoreos.optimization.optimization import OptimizationSolution
from akkudoktoreos.utils.datetimeutil import DateTime, compare_datetimes, to_datetime
class EnergyManagementParameters(ParametersBaseModel):
pv_prognose_wh: list[float] = Field(
description="An array of floats representing the forecasted photovoltaic output in watts for different time intervals."
)
strompreis_euro_pro_wh: list[float] = Field(
description="An array of floats representing the electricity price in euros per watt-hour for different time intervals."
)
einspeiseverguetung_euro_pro_wh: list[float] | float = Field(
description="A float or array of floats representing the feed-in compensation in euros per watt-hour."
)
preis_euro_pro_wh_akku: float = Field(
description="A float representing the cost of battery energy per watt-hour."
)
gesamtlast: list[float] = Field(
description="An array of floats representing the total load (consumption) in watts for different time intervals."
)
@model_validator(mode="after")
def validate_list_length(self) -> Self:
pv_prognose_length = len(self.pv_prognose_wh)
if (
pv_prognose_length != len(self.strompreis_euro_pro_wh)
or pv_prognose_length != len(self.gesamtlast)
or (
isinstance(self.einspeiseverguetung_euro_pro_wh, list)
and pv_prognose_length != len(self.einspeiseverguetung_euro_pro_wh)
)
):
raise ValueError("Input lists have different lengths")
return self
class SimulationResult(ParametersBaseModel):
"""This object contains the results of the simulation and provides insights into various parameters over the entire forecast period."""
Last_Wh_pro_Stunde: list[Optional[float]] = Field(description="TBD")
EAuto_SoC_pro_Stunde: list[Optional[float]] = Field(
description="The state of charge of the EV for each hour."
)
Einnahmen_Euro_pro_Stunde: list[Optional[float]] = Field(
description="The revenue from grid feed-in or other sources in euros per hour."
)
Gesamt_Verluste: float = Field(
description="The total losses in watt-hours over the entire period."
)
Gesamtbilanz_Euro: float = Field(
description="The total balance of revenues minus costs in euros."
)
Gesamteinnahmen_Euro: float = Field(description="The total revenues in euros.")
Gesamtkosten_Euro: float = Field(description="The total costs in euros.")
Home_appliance_wh_per_hour: list[Optional[float]] = Field(
description="The energy consumption of a household appliance in watt-hours per hour."
)
Kosten_Euro_pro_Stunde: list[Optional[float]] = Field(
description="The costs in euros per hour."
)
Netzbezug_Wh_pro_Stunde: list[Optional[float]] = Field(
description="The grid energy drawn in watt-hours per hour."
)
Netzeinspeisung_Wh_pro_Stunde: list[Optional[float]] = Field(
description="The energy fed into the grid in watt-hours per hour."
)
Verluste_Pro_Stunde: list[Optional[float]] = Field(
description="The losses in watt-hours per hour."
)
akku_soc_pro_stunde: list[Optional[float]] = Field(
description="The state of charge of the battery (not the EV) in percentage per hour."
)
Electricity_price: list[Optional[float]] = Field(
description="Used Electricity Price, including predictions"
)
@field_validator(
"Last_Wh_pro_Stunde",
"Netzeinspeisung_Wh_pro_Stunde",
"akku_soc_pro_stunde",
"Netzbezug_Wh_pro_Stunde",
"Kosten_Euro_pro_Stunde",
"Einnahmen_Euro_pro_Stunde",
"EAuto_SoC_pro_Stunde",
"Verluste_Pro_Stunde",
"Home_appliance_wh_per_hour",
"Electricity_price",
mode="before",
)
def convert_numpy(cls, field: Any) -> Any:
return NumpyEncoder.convert_numpy(field)[0]
# The executor to execute the CPU heavy energy management run
executor = ThreadPoolExecutor(max_workers=1)
class EnergyManagement(SingletonMixin, ConfigMixin, PredictionMixin, PydanticBaseModel):
# Disable validation on assignment to speed up simulation runs.
model_config = ConfigDict(
validate_assignment=False,
)
"""Energy management."""
# Start datetime.
_start_datetime: ClassVar[Optional[DateTime]] = None
# last run datetime. Used by energy management task
_last_datetime: ClassVar[Optional[DateTime]] = None
_last_run_datetime: ClassVar[Optional[DateTime]] = None
# energy management plan of latest energy management run with optimization
_plan: ClassVar[Optional[EnergyManagementPlan]] = None
# opimization solution of the latest energy management run
_optimization_solution: ClassVar[Optional[OptimizationSolution]] = None
# Solution of the genetic algorithm of latest energy management run with optimization
# For classic API
_genetic_solution: ClassVar[Optional[GeneticSolution]] = None
# energy management lock (for energy management run)
_run_lock: ClassVar[Lock] = Lock()
@computed_field # type: ignore[prop-decorator]
@property
@@ -127,9 +54,15 @@ class EnergyManagement(SingletonMixin, ConfigMixin, PredictionMixin, PydanticBas
EnergyManagement.set_start_datetime()
return EnergyManagement._start_datetime
@computed_field # type: ignore[prop-decorator]
@property
def last_run_datetime(self) -> Optional[DateTime]:
"""The datetime the last energy management was run."""
return EnergyManagement._last_run_datetime
@classmethod
def set_start_datetime(cls, start_datetime: Optional[DateTime] = None) -> DateTime:
"""Set the start datetime for the next energy management cycle.
"""Set the start datetime for the next energy management run.
If no datetime is provided, the current datetime is used.
@@ -148,142 +81,208 @@ class EnergyManagement(SingletonMixin, ConfigMixin, PredictionMixin, PydanticBas
cls._start_datetime = start_datetime.set(minute=0, second=0, microsecond=0)
return cls._start_datetime
# -------------------------
# TODO: Take from prediction
# -------------------------
@classmethod
def plan(cls) -> Optional[EnergyManagementPlan]:
"""Get the latest energy management plan.
load_energy_array: Optional[NDArray[Shape["*"], float]] = Field(
default=None,
description="An array of floats representing the total load (consumption) in watts for different time intervals.",
)
pv_prediction_wh: Optional[NDArray[Shape["*"], float]] = Field(
default=None,
description="An array of floats representing the forecasted photovoltaic output in watts for different time intervals.",
)
elect_price_hourly: Optional[NDArray[Shape["*"], float]] = Field(
default=None,
description="An array of floats representing the electricity price in euros per watt-hour for different time intervals.",
)
elect_revenue_per_hour_arr: Optional[NDArray[Shape["*"], float]] = Field(
default=None,
description="An array of floats representing the feed-in compensation in euros per watt-hour.",
)
Returns:
Optional[EnergyManagementPlan]: The latest energy management plan or None.
"""
return cls._plan
# -------------------------
# TODO: Move to devices
# -------------------------
@classmethod
def optimization_solution(cls) -> Optional[OptimizationSolution]:
"""Get the latest optimization solution.
battery: Optional[Battery] = Field(default=None, description="TBD.")
ev: Optional[Battery] = Field(default=None, description="TBD.")
home_appliance: Optional[HomeAppliance] = Field(default=None, description="TBD.")
inverter: Optional[Inverter] = Field(default=None, description="TBD.")
Returns:
Optional[OptimizationSolution]: The latest optimization solution.
"""
return cls._optimization_solution
# -------------------------
# TODO: Move to devices
# -------------------------
@classmethod
def genetic_solution(cls) -> Optional[GeneticSolution]:
"""Get the latest solution of the genetic algorithm.
ac_charge_hours: Optional[NDArray[Shape["*"], float]] = Field(default=None, description="TBD")
dc_charge_hours: Optional[NDArray[Shape["*"], float]] = Field(default=None, description="TBD")
ev_charge_hours: Optional[NDArray[Shape["*"], float]] = Field(default=None, description="TBD")
Returns:
Optional[GeneticSolution]: The latest solution of the genetic algorithm.
"""
return cls._genetic_solution
def __init__(self, *args: Any, **kwargs: Any) -> None:
if hasattr(self, "_initialized"):
return
super().__init__(*args, **kwargs)
def set_parameters(
self,
parameters: EnergyManagementParameters,
ev: Optional[Battery] = None,
home_appliance: Optional[HomeAppliance] = None,
inverter: Optional[Inverter] = None,
) -> None:
self.load_energy_array = np.array(parameters.gesamtlast, float)
self.pv_prediction_wh = np.array(parameters.pv_prognose_wh, float)
self.elect_price_hourly = np.array(parameters.strompreis_euro_pro_wh, float)
self.elect_revenue_per_hour_arr = (
parameters.einspeiseverguetung_euro_pro_wh
if isinstance(parameters.einspeiseverguetung_euro_pro_wh, list)
else np.full(
len(self.load_energy_array), parameters.einspeiseverguetung_euro_pro_wh, float
)
)
if inverter:
self.battery = inverter.battery
else:
self.battery = None
self.ev = ev
self.home_appliance = home_appliance
self.inverter = inverter
self.ac_charge_hours = np.full(self.config.prediction.hours, 0.0)
self.dc_charge_hours = np.full(self.config.prediction.hours, 1.0)
self.ev_charge_hours = np.full(self.config.prediction.hours, 0.0)
def set_akku_discharge_hours(self, ds: np.ndarray) -> None:
if self.battery:
self.battery.set_discharge_per_hour(ds)
def set_akku_ac_charge_hours(self, ds: np.ndarray) -> None:
self.ac_charge_hours = ds
def set_akku_dc_charge_hours(self, ds: np.ndarray) -> None:
self.dc_charge_hours = ds
def set_ev_charge_hours(self, ds: np.ndarray) -> None:
self.ev_charge_hours = ds
def set_home_appliance_start(self, ds: int, global_start_hour: int = 0) -> None:
if self.home_appliance:
self.home_appliance.set_starting_time(ds, global_start_hour=global_start_hour)
def reset(self) -> None:
if self.ev:
self.ev.reset()
if self.battery:
self.battery.reset()
def run(
self,
start_hour: Optional[int] = None,
@classmethod
def _run(
cls,
start_datetime: Optional[DateTime] = None,
mode: Optional[EnergyManagementMode] = None,
genetic_parameters: Optional[GeneticOptimizationParameters] = None,
genetic_individuals: Optional[int] = None,
genetic_seed: Optional[int] = None,
force_enable: Optional[bool] = False,
force_update: Optional[bool] = False,
) -> None:
"""Run energy management.
"""Run the energy management.
Sets `start_datetime` to current hour, updates the configuration and the prediction, and
starts simulation at current hour.
This method initializes the energy management run by setting its
start datetime, updating predictions, and optionally starting
optimization depending on the selected mode or configuration.
Args:
start_hour (int, optional): Hour to take as start time for the energy management. Defaults
to now.
force_enable (bool, optional): If True, forces to update even if disabled. This
is mostly relevant to prediction providers.
force_update (bool, optional): If True, forces to update the data even if still cached.
start_datetime (DateTime, optional): The starting timestamp
of the energy management run. Defaults to the current datetime
if not provided.
mode (EnergyManagementMode, optional): The management mode to use. Must be one of:
- "OPTIMIZATION": Runs the optimization process.
- "PREDICTION": Updates the forecast without optimization.
Defaults to the mode defined in the current configuration.
genetic_parameters (GeneticOptimizationParameters, optional): The
parameter set for the genetic algorithm. If not provided, it will
be constructed based on the current configuration and predictions.
genetic_individuals (int, optional): The number of individuals for the
genetic algorithm. Defaults to the algorithm's internal default (400)
if not specified.
genetic_seed (int, optional): The seed for the genetic algorithm. Defaults
to the algorithm's internal random seed if not specified.
force_enable (bool, optional): If True, bypasses any disabled state
to force the update process. This is mostly applicable to
prediction providers.
force_update (bool, optional): If True, forces data to be refreshed
even if a cached version is still valid.
Returns:
None
"""
# Throw away any cached results of the last run.
CacheUntilUpdateStore().clear()
self.set_start_hour(start_hour=start_hour)
# Ensure there is only one optimization/ energy management run at a time
if mode not in (None, "PREDICTION", "OPTIMIZATION"):
raise ValueError(f"Unknown energy management mode {mode}.")
# Check for run definitions
if self.start_datetime is None:
error_msg = "Start datetime unknown."
logger.error(error_msg)
raise ValueError(error_msg)
if self.config.prediction.hours is None:
error_msg = "Prediction hours unknown."
logger.error(error_msg)
raise ValueError(error_msg)
if self.config.optimization.hours is None:
error_msg = "Optimization hours unknown."
logger.error(error_msg)
raise ValueError(error_msg)
logger.info("Starting energy management run.")
self.prediction.update_data(force_enable=force_enable, force_update=force_update)
# TODO: Create optimisation problem that calls into devices.update_data() for simulations.
# Remember/ set the start datetime of this energy management run.
# None leads
cls.set_start_datetime(start_datetime)
logger.info("Energy management run (crippled version - prediction update only)")
# Throw away any memory cached results of the last energy management run.
CacheEnergyManagementStore().clear()
def manage_energy(self) -> None:
if mode is None:
mode = cls.config.ems.mode
if mode is None or mode == "PREDICTION":
# Update the predictions
cls.prediction.update_data(force_enable=force_enable, force_update=force_update)
logger.info("Energy management run done (predictions updated)")
return
# Prepare optimization parameters
# This also creates default configurations for missing values and updates the predictions
logger.info(
"Starting energy management prediction update and optimzation parameter preparation."
)
if genetic_parameters is None:
genetic_parameters = GeneticOptimizationParameters.prepare()
if not genetic_parameters:
logger.error(
"Energy management run canceled. Could not prepare optimisation parameters."
)
return
# Take values from config if not given
if genetic_individuals is None:
genetic_individuals = cls.config.optimization.genetic.individuals
if genetic_seed is None:
genetic_seed = cls.config.optimization.genetic.seed
if cls._start_datetime is None: # Make mypy happy - already set by us
raise RuntimeError("Start datetime not set.")
logger.info("Starting energy management optimization.")
try:
optimization = GeneticOptimization(
verbose=bool(cls.config.server.verbose),
fixed_seed=genetic_seed,
)
solution = optimization.optimierung_ems(
start_hour=cls._start_datetime.hour,
parameters=genetic_parameters,
ngen=genetic_individuals,
)
except:
logger.exception("Energy management optimization failed.")
return
# Make genetic solution public
cls._genetic_solution = solution
# Make optimization solution public
cls._optimization_solution = solution.optimization_solution()
# Make plan public
cls._plan = solution.energy_management_plan()
logger.debug("Energy management genetic solution:\n{}", cls._genetic_solution)
logger.debug("Energy management optimization solution:\n{}", cls._optimization_solution)
logger.debug("Energy management plan:\n{}", cls._plan)
logger.info("Energy management run done (optimization updated)")
async def run(
self,
start_datetime: Optional[DateTime] = None,
mode: Optional[EnergyManagementMode] = None,
genetic_parameters: Optional[GeneticOptimizationParameters] = None,
genetic_individuals: Optional[int] = None,
genetic_seed: Optional[int] = None,
force_enable: Optional[bool] = False,
force_update: Optional[bool] = False,
) -> None:
"""Run the energy management.
This method initializes the energy management run by setting its
start datetime, updating predictions, and optionally starting
optimization depending on the selected mode or configuration.
Args:
start_datetime (DateTime, optional): The starting timestamp
of the energy management run. Defaults to the current datetime
if not provided.
mode (EnergyManagementMode, optional): The management mode to use. Must be one of:
- "OPTIMIZATION": Runs the optimization process.
- "PREDICTION": Updates the forecast without optimization.
Defaults to the mode defined in the current configuration.
genetic_parameters (GeneticOptimizationParameters, optional): The
parameter set for the genetic algorithm. If not provided, it will
be constructed based on the current configuration and predictions.
genetic_individuals (int, optional): The number of individuals for the
genetic algorithm. Defaults to the algorithm's internal default (400)
if not specified.
genetic_seed (int, optional): The seed for the genetic algorithm. Defaults
to the algorithm's internal random seed if not specified.
force_enable (bool, optional): If True, bypasses any disabled state
to force the update process. This is mostly applicable to
prediction providers.
force_update (bool, optional): If True, forces data to be refreshed
even if a cached version is still valid.
Returns:
None
"""
async with self._run_lock:
loop = get_running_loop()
# Create a partial function with parameters "baked in"
func = partial(
EnergyManagement._run,
start_datetime=start_datetime,
mode=mode,
genetic_parameters=genetic_parameters,
genetic_individuals=genetic_individuals,
genetic_seed=genetic_seed,
force_enable=force_enable,
force_update=force_update,
)
# Run optimization in background thread to avoid blocking event loop
await loop.run_in_executor(executor, func)
async def manage_energy(self) -> None:
"""Repeating task for managing energy.
This task should be executed by the server regularly (e.g., every 10 seconds)
@@ -304,13 +303,13 @@ class EnergyManagement(SingletonMixin, ConfigMixin, PredictionMixin, PydanticBas
current_datetime = to_datetime()
interval = self.config.ems.interval # interval maybe changed in between
if EnergyManagement._last_datetime is None:
if EnergyManagement._last_run_datetime is None:
# Never run before
try:
# Remember energy run datetime.
EnergyManagement._last_datetime = current_datetime
EnergyManagement._last_run_datetime = current_datetime
# Try to run a first energy management. May fail due to config incomplete.
self.run()
await self.run()
except Exception as e:
trace = "".join(traceback.TracebackException.from_exception(e).format())
message = f"EOS init: {e}\n{trace}"
@@ -322,14 +321,14 @@ class EnergyManagement(SingletonMixin, ConfigMixin, PredictionMixin, PydanticBas
return
if (
compare_datetimes(current_datetime, EnergyManagement._last_datetime).time_diff
compare_datetimes(current_datetime, EnergyManagement._last_run_datetime).time_diff
< interval
):
# Wait for next run
return
try:
self.run()
await self.run()
except Exception as e:
trace = "".join(traceback.TracebackException.from_exception(e).format())
message = f"EOS run: {e}\n{trace}"
@@ -337,187 +336,13 @@ class EnergyManagement(SingletonMixin, ConfigMixin, PredictionMixin, PydanticBas
# Remember the energy management run - keep on interval even if we missed some intervals
while (
compare_datetimes(current_datetime, EnergyManagement._last_datetime).time_diff
compare_datetimes(current_datetime, EnergyManagement._last_run_datetime).time_diff
>= interval
):
EnergyManagement._last_datetime = EnergyManagement._last_datetime.add(seconds=interval)
def set_start_hour(self, start_hour: Optional[int] = None) -> None:
"""Sets start datetime to given hour.
Args:
start_hour (int, optional): Hour to take as start time for the energy management. Defaults
to now.
"""
if start_hour is None:
self.set_start_datetime()
else:
start_datetime = to_datetime().set(hour=start_hour, minute=0, second=0, microsecond=0)
self.set_start_datetime(start_datetime)
def simulate_start_now(self) -> dict[str, Any]:
start_hour = to_datetime().now().hour
return self.simulate(start_hour)
def simulate(self, start_hour: int) -> dict[str, Any]:
"""Simulate energy usage and costs for the given start hour.
akku_soc_pro_stunde begin of the hour, initial hour state!
last_wh_pro_stunde integral of last hour (end state)
"""
# Check for simulation integrity
required_attrs = [
"load_energy_array",
"pv_prediction_wh",
"elect_price_hourly",
"ev_charge_hours",
"ac_charge_hours",
"dc_charge_hours",
"elect_revenue_per_hour_arr",
]
missing_data = [
attr.replace("_", " ").title() for attr in required_attrs if getattr(self, attr) is None
]
if missing_data:
logger.error("Mandatory data missing - %s", ", ".join(missing_data))
raise ValueError(f"Mandatory data missing: {', '.join(missing_data)}")
# Pre-fetch data
load_energy_array = np.array(self.load_energy_array)
pv_prediction_wh = np.array(self.pv_prediction_wh)
elect_price_hourly = np.array(self.elect_price_hourly)
ev_charge_hours = np.array(self.ev_charge_hours)
ac_charge_hours = np.array(self.ac_charge_hours)
dc_charge_hours = np.array(self.dc_charge_hours)
elect_revenue_per_hour_arr = np.array(self.elect_revenue_per_hour_arr)
# Fetch objects
battery = self.battery
if battery is None:
raise ValueError(f"battery not set: {battery}")
ev = self.ev
home_appliance = self.home_appliance
inverter = self.inverter
if not (len(load_energy_array) == len(pv_prediction_wh) == len(elect_price_hourly)):
error_msg = f"Array sizes do not match: Load Curve = {len(load_energy_array)}, PV Forecast = {len(pv_prediction_wh)}, Electricity Price = {len(elect_price_hourly)}"
logger.error(error_msg)
raise ValueError(error_msg)
end_hour = len(load_energy_array)
total_hours = end_hour - start_hour
# Pre-allocate arrays for the results, optimized for speed
loads_energy_per_hour = np.full((total_hours), np.nan)
feedin_energy_per_hour = np.full((total_hours), np.nan)
consumption_energy_per_hour = np.full((total_hours), np.nan)
costs_per_hour = np.full((total_hours), np.nan)
revenue_per_hour = np.full((total_hours), np.nan)
soc_per_hour = np.full((total_hours), np.nan)
soc_ev_per_hour = np.full((total_hours), np.nan)
losses_wh_per_hour = np.full((total_hours), np.nan)
home_appliance_wh_per_hour = np.full((total_hours), np.nan)
electricity_price_per_hour = np.full((total_hours), np.nan)
# Set initial state
soc_per_hour[0] = battery.current_soc_percentage()
if ev:
soc_ev_per_hour[0] = ev.current_soc_percentage()
for hour in range(start_hour, end_hour):
hour_idx = hour - start_hour
# save begin states
soc_per_hour[hour_idx] = battery.current_soc_percentage()
if ev:
soc_ev_per_hour[hour_idx] = ev.current_soc_percentage()
# Accumulate loads and PV generation
consumption = load_energy_array[hour]
losses_wh_per_hour[hour_idx] = 0.0
# Home appliances
if home_appliance:
ha_load = home_appliance.get_load_for_hour(hour)
consumption += ha_load
home_appliance_wh_per_hour[hour_idx] = ha_load
# E-Auto handling
if ev and ev_charge_hours[hour] > 0:
loaded_energy_ev, verluste_eauto = ev.charge_energy(
None, hour, relative_power=ev_charge_hours[hour]
)
consumption += loaded_energy_ev
losses_wh_per_hour[hour_idx] += verluste_eauto
# Process inverter logic
energy_feedin_grid_actual = energy_consumption_grid_actual = losses = eigenverbrauch = (
0.0
EnergyManagement._last_run_datetime = EnergyManagement._last_run_datetime.add(
seconds=interval
)
hour_ac_charge = ac_charge_hours[hour]
hour_dc_charge = dc_charge_hours[hour]
hourly_electricity_price = elect_price_hourly[hour]
hourly_energy_revenue = elect_revenue_per_hour_arr[hour]
battery.set_charge_allowed_for_hour(hour_dc_charge, hour)
if inverter:
energy_produced = pv_prediction_wh[hour]
(
energy_feedin_grid_actual,
energy_consumption_grid_actual,
losses,
eigenverbrauch,
) = inverter.process_energy(energy_produced, consumption, hour)
# AC PV Battery Charge
if hour_ac_charge > 0.0:
battery.set_charge_allowed_for_hour(1, hour)
battery_charged_energy_actual, battery_losses_actual = battery.charge_energy(
None, hour, relative_power=hour_ac_charge
)
total_battery_energy = battery_charged_energy_actual + battery_losses_actual
consumption += total_battery_energy
energy_consumption_grid_actual += total_battery_energy
losses_wh_per_hour[hour_idx] += battery_losses_actual
# Update hourly arrays
feedin_energy_per_hour[hour_idx] = energy_feedin_grid_actual
consumption_energy_per_hour[hour_idx] = energy_consumption_grid_actual
losses_wh_per_hour[hour_idx] += losses
loads_energy_per_hour[hour_idx] = consumption
electricity_price_per_hour[hour_idx] = hourly_electricity_price
# Financial calculations
costs_per_hour[hour_idx] = energy_consumption_grid_actual * hourly_electricity_price
revenue_per_hour[hour_idx] = energy_feedin_grid_actual * hourly_energy_revenue
total_cost = np.nansum(costs_per_hour)
total_losses = np.nansum(losses_wh_per_hour)
total_revenue = np.nansum(revenue_per_hour)
# Prepare output dictionary
return {
"Last_Wh_pro_Stunde": loads_energy_per_hour,
"Netzeinspeisung_Wh_pro_Stunde": feedin_energy_per_hour,
"Netzbezug_Wh_pro_Stunde": consumption_energy_per_hour,
"Kosten_Euro_pro_Stunde": costs_per_hour,
"akku_soc_pro_stunde": soc_per_hour,
"Einnahmen_Euro_pro_Stunde": revenue_per_hour,
"Gesamtbilanz_Euro": total_cost - total_revenue,
"EAuto_SoC_pro_Stunde": soc_ev_per_hour,
"Gesamteinnahmen_Euro": total_revenue,
"Gesamtkosten_Euro": total_cost,
"Verluste_Pro_Stunde": losses_wh_per_hour,
"Gesamt_Verluste": total_losses,
"Home_appliance_wh_per_hour": home_appliance_wh_per_hour,
"Electricity_price": electricity_price_per_hour,
}
# Initialize the Energy Management System, it is a singleton.
ems = EnergyManagement()

14
src/akkudoktoreos/core/emsettings.py
View File

@@ -3,6 +3,7 @@
Kept in an extra module to avoid cyclic dependencies on package import.
"""
from enum import Enum
from typing import Optional
from pydantic import Field
@@ -10,6 +11,13 @@ from pydantic import Field
from akkudoktoreos.config.configabc import SettingsBaseModel
class EnergyManagementMode(str, Enum):
"""Energy management mode."""
PREDICTION = "PREDICTION"
OPTIMIZATION = "OPTIMIZATION"
class EnergyManagementCommonSettings(SettingsBaseModel):
"""Energy Management Configuration."""
@@ -24,3 +32,9 @@ class EnergyManagementCommonSettings(SettingsBaseModel):
description="Intervall in seconds between EOS energy management runs.",
examples=["300"],
)
mode: Optional[EnergyManagementMode] = Field(
default=None,
description="Energy management mode [OPTIMIZATION | PREDICTION].",
examples=["OPTIMIZATION", "PREDICTION"],
)

4
src/akkudoktoreos/core/logging.py
View File

@@ -42,6 +42,10 @@ class InterceptHandler(pylogging.Handler):
Args:
record (logging.LogRecord): A record object containing log message and metadata.
"""
# Skip DEBUG logs from matplotlib - very noisy
if record.name.startswith("matplotlib") and record.levelno <= pylogging.DEBUG:
return
try:
level = logger.level(record.levelname).name
except AttributeError:

5
src/akkudoktoreos/core/logsettings.py
View File

@@ -15,11 +15,6 @@ from akkudoktoreos.core.logabc import LOGGING_LEVELS
class LoggingCommonSettings(SettingsBaseModel):
"""Logging Configuration."""
level: Optional[str] = Field(
default=None,
deprecated="This is deprecated. Use console_level and file_level instead.",
)
console_level: Optional[str] = Field(
default=None,
description="Logging level when logging to console.",

312
src/akkudoktoreos/core/pydantic.py
View File

@@ -14,7 +14,6 @@ Key Features:
import inspect
import json
import re
import uuid
import weakref
from copy import deepcopy
@@ -32,23 +31,20 @@ from typing import (
from zoneinfo import ZoneInfo
import pandas as pd
import pendulum
from loguru import logger
from pandas.api.types import is_datetime64_any_dtype
from pydantic import (
AwareDatetime,
BaseModel,
ConfigDict,
Field,
PrivateAttr,
RootModel,
TypeAdapter,
ValidationError,
ValidationInfo,
field_validator,
)
from akkudoktoreos.utils.datetimeutil import to_datetime, to_duration
from akkudoktoreos.utils.datetimeutil import DateTime, to_datetime, to_duration
# Global weakref dictionary to hold external state per model instance
# Used as a workaround for PrivateAttr not working in e.g. Mixin Classes
@@ -56,49 +52,40 @@ _model_private_state: "weakref.WeakKeyDictionary[Union[PydanticBaseModel, Pydant
def merge_models(source: BaseModel, update_dict: dict[str, Any]) -> dict[str, Any]:
def deep_update(source_dict: dict[str, Any], update_dict: dict[str, Any]) -> dict[str, Any]:
for key, value in source_dict.items():
if isinstance(value, dict) and isinstance(update_dict.get(key), dict):
update_dict[key] = deep_update(update_dict[key], value)
else:
update_dict[key] = value
return update_dict
"""Merge a Pydantic model instance with an update dictionary.
Values in update_dict (including None) override source values.
Nested dictionaries are merged recursively.
Lists in update_dict replace source lists entirely.
Args:
source (BaseModel): Pydantic model instance serving as the source.
update_dict (dict[str, Any]): Dictionary with updates to apply.
Returns:
dict[str, Any]: Merged dictionary representing combined model data.
"""
def deep_merge(source_data: Any, update_data: Any) -> Any:
if isinstance(source_data, dict) and isinstance(update_data, dict):
merged = dict(source_data)
for key, update_value in update_data.items():
if key in merged:
merged[key] = deep_merge(merged[key], update_value)
else:
merged[key] = update_value
return merged
# If both are lists, replace source list with update list
if isinstance(source_data, list) and isinstance(update_data, list):
return update_data
# For other types or if update_data is None, override source_data
return update_data
source_dict = source.model_dump(exclude_unset=True)
merged_dict = deep_update(source_dict, deepcopy(update_dict))
return merged_dict
class PydanticTypeAdapterDateTime(TypeAdapter[pendulum.DateTime]):
"""Custom type adapter for Pendulum DateTime fields."""
@classmethod
def serialize(cls, value: Any) -> str:
"""Convert pendulum.DateTime to ISO 8601 string."""
if isinstance(value, pendulum.DateTime):
return value.to_iso8601_string()
raise ValueError(f"Expected pendulum.DateTime, got {type(value)}")
@classmethod
def deserialize(cls, value: Any) -> pendulum.DateTime:
"""Convert ISO 8601 string to pendulum.DateTime."""
if isinstance(value, str) and cls.is_iso8601(value):
try:
return pendulum.parse(value)
except pendulum.parsing.exceptions.ParserError as e:
raise ValueError(f"Invalid date format: {value}") from e
elif isinstance(value, pendulum.DateTime):
return value
raise ValueError(f"Expected ISO 8601 string or pendulum.DateTime, got {type(value)}")
@staticmethod
def is_iso8601(value: str) -> bool:
"""Check if the string is a valid ISO 8601 date string."""
iso8601_pattern = (
r"^(\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d{1,3})?(?:Z|[+-]\d{2}:\d{2})?)$"
)
return bool(re.match(iso8601_pattern, value))
merged_result = deep_merge(source_dict, deepcopy(update_dict))
return merged_result
class PydanticModelNestedValueMixin:
@@ -653,67 +640,9 @@ class PydanticBaseModel(PydanticModelNestedValueMixin, BaseModel):
"""
return hash(self._uuid)
@field_validator("*", mode="before")
def validate_and_convert_pendulum(cls, value: Any, info: ValidationInfo) -> Any:
"""Validator to convert fields of type `pendulum.DateTime`.
Converts fields to proper `pendulum.DateTime` objects, ensuring correct input types.
This method is invoked for every field before the field value is set. If the field's type
is `pendulum.DateTime`, it tries to convert string or timestamp values to `pendulum.DateTime`
objects. If the value cannot be converted, a validation error is raised.
Args:
value: The value to be assigned to the field.
info: Validation information for the field.
Returns:
The converted value, if successful.
Raises:
ValidationError: If the value cannot be converted to `pendulum.DateTime`.
"""
# Get the field name and expected type
field_name = info.field_name
expected_type = cls.model_fields[field_name].annotation
# Convert
if expected_type is pendulum.DateTime or expected_type is AwareDatetime:
try:
value = to_datetime(value)
except Exception as e:
raise ValueError(f"Cannot convert {value!r} to datetime: {e}")
return value
# Override Pydantic’s serialization for all DateTime fields
def model_dump(
self, *args: Any, include_computed_fields: bool = True, **kwargs: Any
) -> dict[str, Any]:
"""Custom dump method to handle serialization for DateTime fields."""
result = super().model_dump(*args, **kwargs)
if not include_computed_fields:
for computed_field_name in self.model_computed_fields:
result.pop(computed_field_name, None)
for key, value in result.items():
if isinstance(value, pendulum.DateTime):
result[key] = PydanticTypeAdapterDateTime.serialize(value)
return result
@classmethod
def model_construct(
cls, _fields_set: set[str] | None = None, **values: Any
) -> "PydanticBaseModel":
"""Custom constructor to handle deserialization for DateTime fields."""
for key, value in values.items():
if isinstance(value, str) and PydanticTypeAdapterDateTime.is_iso8601(value):
values[key] = PydanticTypeAdapterDateTime.deserialize(value)
return super().model_construct(_fields_set, **values)
def reset_to_defaults(self) -> "PydanticBaseModel":
"""Resets the fields to their default values."""
for field_name, field_info in self.model_fields.items():
for field_name, field_info in self.__class__.model_fields.items():
if field_info.default_factory is not None: # Handle fields with default_factory
default_value = field_info.default_factory()
else:
@@ -725,6 +654,19 @@ class PydanticBaseModel(PydanticModelNestedValueMixin, BaseModel):
pass
return self
# Override Pydantic’s serialization to include computed fields by default
def model_dump(
self, *args: Any, include_computed_fields: bool = True, **kwargs: Any
) -> dict[str, Any]:
"""Custom dump method to serialize computed fields by default."""
result = super().model_dump(*args, **kwargs)
if not include_computed_fields:
for computed_field_name in self.__class__.model_computed_fields:
result.pop(computed_field_name, None)
return result
def to_dict(self) -> dict:
"""Convert this PredictionRecord instance to a dictionary representation.
@@ -910,16 +852,27 @@ class PydanticDateTimeDataFrame(PydanticBaseModel):
if not v:
return v
valid_dtypes = {"int64", "float64", "bool", "datetime64[ns]", "object", "string"}
invalid_dtypes = set(v.values()) - valid_dtypes
if invalid_dtypes:
raise ValueError(f"Unsupported dtypes: {invalid_dtypes}")
# Allowed exact dtypes
valid_base_dtypes = {"int64", "float64", "bool", "object", "string"}
def is_valid_dtype(dtype: str) -> bool:
# Allow timezone-aware or naive datetime64
if dtype.startswith("datetime64[ns"):
return True
return dtype in valid_base_dtypes
invalid_dtypes = [dtype for dtype in v.values() if not is_valid_dtype(dtype)]
if invalid_dtypes:
raise ValueError(f"Unsupported dtypes: {set(invalid_dtypes)}")
# Cross-check with data column existence
data = info.data.get("data", {})
if data:
columns = set(next(iter(data.values())).keys())
if not all(col in columns for col in v.keys()):
raise ValueError("dtype columns must exist in data columns")
missing_columns = set(v.keys()) - columns
if missing_columns:
raise ValueError(f"dtype columns must exist in data columns: {missing_columns}")
return v
def to_dataframe(self) -> pd.DataFrame:
@@ -927,7 +880,8 @@ class PydanticDateTimeDataFrame(PydanticBaseModel):
df = pd.DataFrame.from_dict(self.data, orient="index")
# Convert index to datetime
index = pd.Index([to_datetime(dt, in_timezone=self.tz) for dt in df.index])
# index = pd.Index([to_datetime(dt, in_timezone=self.tz) for dt in df.index])
index = [to_datetime(dt, in_timezone=self.tz) for dt in df.index]
df.index = index
# Check if 'date_time' column exists, if not, create it
@@ -943,8 +897,8 @@ class PydanticDateTimeDataFrame(PydanticBaseModel):
# Apply dtypes
for col, dtype in self.dtypes.items():
if dtype == "datetime64[ns]":
df[col] = pd.to_datetime(to_datetime(df[col], in_timezone=self.tz))
if dtype.startswith("datetime64[ns"):
df[col] = pd.to_datetime(df[col], utc=True)
elif dtype in dtype_mapping.keys():
df[col] = df[col].astype(dtype_mapping[dtype])
else:
@@ -969,6 +923,132 @@ class PydanticDateTimeDataFrame(PydanticBaseModel):
datetime_columns=datetime_columns,
)
# --- Direct Manipulation Methods ---
def _normalize_index(self, index: str | DateTime) -> str:
"""Normalize index into timezone-aware datetime string.
Args:
index (str | DateTime): A datetime-like value.
Returns:
str: Normalized datetime string based on model timezone.
"""
return to_datetime(index, as_string=True, in_timezone=self.tz)
def add_row(self, index: str | DateTime, row: Dict[str, Any]) -> None:
"""Add a new row to the dataset.
Args:
index (str | DateTime): Timestamp of the new row.
row (Dict[str, Any]): Dictionary of column values. Must match existing columns.
Raises:
ValueError: If row does not contain the exact same columns as existing rows.
"""
idx = self._normalize_index(index)
if self.data:
existing_cols = set(next(iter(self.data.values())).keys())
if set(row.keys()) != existing_cols:
raise ValueError(f"Row must have exactly these columns: {existing_cols}")
self.data[idx] = row
def update_row(self, index: str | DateTime, updates: Dict[str, Any]) -> None:
"""Update values for an existing row.
Args:
index (str | DateTime): Timestamp of the row to modify.
updates (Dict[str, Any]): Key/value pairs of columns to update.
Raises:
KeyError: If row or column does not exist.
"""
idx = self._normalize_index(index)
if idx not in self.data:
raise KeyError(f"Row {idx} not found")
for col, value in updates.items():
if col not in self.data[idx]:
raise KeyError(f"Column {col} does not exist")
self.data[idx][col] = value
def delete_row(self, index: str | DateTime) -> None:
"""Delete a row from the dataset.
Args:
index (str | DateTime): Timestamp of the row to delete.
"""
idx = self._normalize_index(index)
if idx in self.data:
del self.data[idx]
def set_value(self, index: str | DateTime, column: str, value: Any) -> None:
"""Set a single cell value.
Args:
index (str | datetime): Timestamp of the row.
column (str): Column name.
value (Any): New value.
"""
self.update_row(index, {column: value})
def get_value(self, index: str | DateTime, column: str) -> Any:
"""Retrieve a single cell value.
Args:
index (str | DateTime): Timestamp of the row.
column (str): Column name.
Returns:
Any: Value stored at the given location.
"""
idx = self._normalize_index(index)
return self.data[idx][column]
def add_column(self, name: str, default: Any = None, dtype: Optional[str] = None) -> None:
"""Add a new column to all rows.
Args:
name (str): Name of the column to add.
default (Any, optional): Default value for all rows. Defaults to None.
dtype (Optional[str], optional): Declared data type. Defaults to None.
"""
for row in self.data.values():
row[name] = default
if dtype:
self.dtypes[name] = dtype
def rename_column(self, old: str, new: str) -> None:
"""Rename a column across all rows.
Args:
old (str): Existing column name.
new (str): New column name.
Raises:
KeyError: If column does not exist.
"""
for row in self.data.values():
if old not in row:
raise KeyError(f"Column {old} does not exist")
row[new] = row.pop(old)
if old in self.dtypes:
self.dtypes[new] = self.dtypes.pop(old)
if old in self.datetime_columns:
self.datetime_columns = [new if c == old else c for c in self.datetime_columns]
def drop_column(self, name: str) -> None:
"""Remove a column from all rows.
Args:
name (str): Column to remove.
"""
for row in self.data.values():
if name in row:
del row[name]
self.dtypes.pop(name, None)
self.datetime_columns = [c for c in self.datetime_columns if c != name]
class PydanticDateTimeSeries(PydanticBaseModel):
"""Pydantic model for validating pandas Series with datetime index in JSON format.
@@ -1068,10 +1148,6 @@ class PydanticDateTimeSeries(PydanticBaseModel):
)
class ParametersBaseModel(PydanticBaseModel):
model_config = ConfigDict(extra="forbid")
def set_private_attr(
model: Union[PydanticBaseModel, PydanticModelNestedValueMixin], key: str, value: Any
) -> None:

5
src/akkudoktoreos/core/version.py Normal file
View File

@@ -0,0 +1,5 @@
"""Version information for akkudoktoreos."""
# For development add `+dev` to previous release
# For release omit `+dev`.
__version__ = "0.1.0+dev"