fix: automatic optimization (#596)

This fix implements the long term goal to have the EOS server run optimization (or
energy management) on regular intervals automatically. Thus clients can request
the current energy management plan at any time and it is updated on regular
intervals without interaction by the client.

This fix started out to "only" make automatic optimization (or energy management)
runs working. It turned out there are several endpoints that in some way
update predictions or run the optimization. To lock against such concurrent attempts
the code had to be refactored to allow control of execution. During refactoring it
became clear that some classes and files are named without a proper reference
to their usage. Thus not only refactoring but also renaming became necessary.
The names are still not the best, but I hope they are more intuitive.

The fix includes several bug fixes that are not directly related to the automatic optimization
but are necessary to keep EOS running properly to do the automatic optimization and
to test and document the changes.

This is a breaking change as the configuration structure changed once again and
the server API was also enhanced and streamlined. The server API that is used by
Andreas and Jörg in their videos has not changed.

* fix: automatic optimization

  Allow optimization to automatically run on configured intervals gathering all
  optimization parameters from configuration and predictions. The automatic run
  can be configured to only run prediction updates skipping the optimization.
  Extend documentaion to also cover automatic optimization. Lock automatic runs
  against runs initiated by the /optimize or other endpoints. Provide new
  endpoints to retrieve the energy management plan and the genetic solution
  of the latest automatic optimization run. Offload energy management to thread
  pool executor to keep the app more responsive during the CPU heavy optimization
  run.

* fix: EOS servers recognize environment variables on startup

  Force initialisation of EOS configuration on server startup to assure
  all sources of EOS configuration are properly set up and read. Adapt
  server tests and configuration tests to also test for environment
  variable configuration.

* fix: Remove 0.0.0.0 to localhost translation under Windows

  EOS imposed a 0.0.0.0 to localhost translation under Windows for
  convenience. This caused some trouble in user configurations. Now, as the
  default IP address configuration is 127.0.0.1, the user is responsible
  for to set up the correct Windows compliant IP address.

* fix: allow names for hosts additional to IP addresses

* fix: access pydantic model fields by class

  Access by instance is deprecated.

* fix: down sampling key_to_array

* fix: make cache clear endpoint clear all cache files

  Make /v1/admin/cache/clear clear all cache files. Before it only cleared
  expired cache files by default. Add new endpoint /v1/admin/clear-expired
  to only clear expired cache files.

* fix: timezonefinder returns Europe/Paris instead of Europe/Berlin

  timezonefinder 8.10 got more inaccurate for timezones in europe as there is
  a common timezone. Use new package tzfpy instead which is still returning
  Europe/Berlin if you are in Germany. tzfpy also claims to be faster than
  timezonefinder.

* fix: provider settings configuration

  Provider configuration used to be a union holding the settings for several
  providers. Pydantic union handling does not always find the correct type
  for a provider setting. This led to exceptions in specific configurations.
  Now provider settings are explicit comfiguration items for each possible
  provider. This is a breaking change as the configuration structure was
  changed.

* fix: ClearOutside weather prediction irradiance calculation

  Pvlib needs a pandas time index. Convert time index.

* fix: test config file priority

  Do not use config_eos fixture as this fixture already creates a config file.

* fix: optimization sample request documentation

  Provide all data in documentation of optimization sample request.

* fix: gitlint blocking pip dependency resolution

  Replace gitlint by commitizen. Gitlint is not actively maintained anymore.
  Gitlint dependencies blocked pip from dependency resolution.

* fix: sync pre-commit config to actual dependency requirements

  .pre-commit-config.yaml was out of sync, also requirements-dev.txt.

* fix: missing babel in requirements.txt

  Add babel to requirements.txt

* feat: setup default device configuration for automatic optimization

  In case the parameters for automatic optimization are not fully defined a
  default configuration is setup to allow the automatic energy management
  run. The default configuration may help the user to correctly define
  the device configuration.

* feat: allow configuration of genetic algorithm parameters

  The genetic algorithm parameters for number of individuals, number of
  generations, the seed and penalty function parameters are now avaliable
  as configuration options.

* feat: allow configuration of home appliance time windows

  The time windows a home appliance is allowed to run are now configurable
  by the configuration (for /v1 API) and also by the home appliance parameters
  (for the classic /optimize API). If there is no such configuration the
  time window defaults to optimization hours, which was the standard before
  the change. Documentation on how to configure time windows is added.

* feat: standardize mesaurement keys for battery/ ev SoC measurements

  The standardized measurement keys to report battery SoC to the device
  simulations can now be retrieved from the device configuration as a
  read-only config option.

* feat: feed in tariff prediction

  Add feed in tarif predictions needed for automatic optimization. The feed in
  tariff can be retrieved as fixed feed in tarif or can be imported. Also add
  tests for the different feed in tariff providers. Extend documentation to
  cover the feed in tariff providers.

* feat: add energy management plan based on S2 standard instructions

  EOS can generate an energy management plan as a list of simple instructions.
  May be retrieved by the /v1/energy-management/plan endpoint. The instructions
  loosely follow the S2 energy management standard.

* feat: make measurement keys configurable by EOS configuration.

  The fixed measurement keys are replaced by configurable measurement keys.

* feat: make pendulum DateTime, Date, Duration types usable for pydantic models

  Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are
  added to the datetimeutil utility. Remove custom made pendulum adaptations
  from EOS pydantic module. Make EOS modules use the pydantic pendulum types
  managed by the datetimeutil module instead of the core pendulum types.

* feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil.

  The time windows are are added to support home appliance time window
  configuration. All time classes are also pydantic models. Time is the base
  class for time definition derived from pendulum.Time.

* feat: Extend DataRecord by configurable field like data.

  Configurable field like data was added to support the configuration of
  measurement records.

* feat: Add additional information to health information

  Version information is added to the health endpoints of eos and eosDash.
  The start time of the last optimization and the latest run time of the energy
  management is added to the EOS health information.

* feat: add pydantic merge model tests

* feat: add plan tab to EOSdash

  The plan tab displays the current energy management instructions.

* feat: add predictions tab to EOSdash

  The predictions tab displays the current predictions.

* feat: add cache management to EOSdash admin tab

  The admin tab is extended by a section for cache management. It allows to
  clear the cache.

* feat: add about tab to EOSdash

  The about tab resembles the former hello tab and provides extra information.

* feat: Adapt changelog and prepare for release management

  Release management using commitizen is added. The changelog file is adapted and
  teh changelog and a description for release management is added in the
  documentation.

* feat(doc): Improve install and devlopment documentation

  Provide a more concise installation description in Readme.md and add extra
  installation page and development page to documentation.

* chore: Use memory cache for interpolation instead of dict in inverter

  Decorate calculate_self_consumption() with @cachemethod_until_update to cache
  results in memory during an energy management/ optimization run. Replacement
  of dict type caching in inverter is now possible because all optimization
  runs are properly locked and the memory cache CacheUntilUpdateStore is properly
  cleared at the start of any energy management/ optimization operation.

* chore: refactor genetic

  Refactor the genetic algorithm modules for enhanced module structure and better
  readability. Removed unnecessary and overcomplex devices singleton. Also
  split devices configuration from genetic algorithm parameters to allow further
  development independently from genetic algorithm parameter format. Move
  charge rates configuration for electric vehicles from optimization to devices
  configuration to allow to have different charge rates for different cars in
  the future.

* chore: Rename memory cache to CacheEnergyManagementStore

  The name better resembles the task of the cache to chache function and method
  results for an energy management run. Also the decorator functions are renamed
  accordingly: cachemethod_energy_management, cache_energy_management

* chore: use class properties for config/ems/prediction mixin classes

* chore: skip debug logs from mathplotlib

  Mathplotlib is very noisy in debug mode.

* chore: automatically sync bokeh js to bokeh python package

  bokeh was updated to 3.8.0, make JS CDN automatically follow the package version.

* chore: rename hello.py to about.py

  Make hello.py the adapted EOSdash about page.

* chore: remove demo page from EOSdash

  As no the plan and prediction pages are working without configuration, the demo
  page is no longer necessary

* chore: split test_server.py for system test

  Split test_server.py to create explicit test_system.py for system tests.

* chore: move doc utils to generate_config_md.py

  The doc utils are only used in scripts/generate_config_md.py. Move it there to
  attribute for strong cohesion.

* chore: improve pydantic merge model documentation

* chore: remove pendulum warning from readme

* chore: remove GitHub discussions from contributing documentation

  Github discussions is to be replaced by Akkudoktor.net.

* chore(release): bump version to 0.1.0+dev for development

* build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1

  bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break.

* build(deps): bump uvicorn from 0.36.0 to 0.37.0

BREAKING CHANGE: EOS configuration changed. V1 API changed.

  - The available_charge_rates_percent configuration is removed from optimization.
    Use the new charge_rate configuration for the electric vehicle
  - Optimization configuration parameter hours renamed to horizon_hours
  - Device configuration now has to provide the number of devices and device
    properties per device.
  - Specific prediction provider configuration to be provided by explicit
    configuration item (no union for all providers).
  - Measurement keys to be provided as a list.
  - New feed in tariff providers have to be configured.
  - /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints.
  - /v1/admin/cache/clear now clears all cache files. Use
    /v1/admin/cache/clear-expired to only clear all expired cache files.

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>

src/akkudoktoreos/core/dataabc.py

@@ -14,14 +14,23 @@ from abc import abstractmethod
 from collections.abc import MutableMapping, MutableSequence
 from itertools import chain
 from pathlib import Path
-from typing import Any, Dict, Iterator, List, Optional, Tuple, Type, Union, overload
+from typing import (
+    Any,
+    Dict,
+    Iterator,
+    List,
+    Optional,
+    Tuple,
+    Type,
+    Union,
+    overload,
+)
 
 import numpy as np
 import pandas as pd
 import pendulum
 from loguru import logger
 from numpydantic import NDArray, Shape
-from pendulum import DateTime, Duration
 from pydantic import (
     AwareDatetime,
     ConfigDict,
@@ -29,6 +38,7 @@ from pydantic import (
     ValidationError,
     computed_field,
     field_validator,
+    model_validator,
 )
 
 from akkudoktoreos.core.coreabc import ConfigMixin, SingletonMixin, StartMixin
@@ -37,7 +47,13 @@ from akkudoktoreos.core.pydantic import (
     PydanticDateTimeData,
     PydanticDateTimeDataFrame,
 )
-from akkudoktoreos.utils.datetimeutil import compare_datetimes, to_datetime, to_duration
+from akkudoktoreos.utils.datetimeutil import (
+    DateTime,
+    Duration,
+    compare_datetimes,
+    to_datetime,
+    to_duration,
+)
 
 
 class DataBase(ConfigMixin, StartMixin, PydanticBaseModel):
@@ -55,6 +71,11 @@ class DataRecord(DataBase, MutableMapping):
     Fields can be accessed and mutated both using dictionary-style access (`record['field_name']`)
     and attribute-style access (`record.field_name`).
 
+    The data record also provides configured field like data. Configuration has to be done by the
+    derived class. Configuration is a list of key strings, which is usually taken from the EOS
+    configuration. The internal field for these data `configured_data` is mostly hidden from
+    dictionary-style and attribute-style access.
+
     Attributes:
         date_time (Optional[DateTime]): Aware datetime indicating when the data record applies.
 
@@ -65,9 +86,42 @@ class DataRecord(DataBase, MutableMapping):
 
     date_time: Optional[DateTime] = Field(default=None, description="DateTime")
 
+    configured_data: dict[str, Any] = Field(
+        default_factory=dict,
+        description="Configured field like data",
+        examples=[{"load0_mr": 40421}],
+    )
+
     # Pydantic v2 model configuration
     model_config = ConfigDict(arbitrary_types_allowed=True, populate_by_name=True)
 
+    @model_validator(mode="before")
+    @classmethod
+    def init_configured_field_like_data(cls, data: Any) -> Any:
+        """Extracts configured data keys from the input and assigns them to `configured_data`.
+
+        This validator is called before the model is initialized. It filters out any keys from the input
+        dictionary that are listed in the configured data keys, and moves them into
+        the `configured_data` field of the model. This enables flexible, key-driven population of
+        dynamic data while keeping the model schema clean.
+
+        Args:
+            data (Any): The raw input data used to initialize the model.
+
+        Returns:
+            Any: The modified input data dictionary, with configured keys moved to `configured_data`.
+        """
+        if not isinstance(data, dict):
+            return data
+
+        configured_keys: Union[list[str], set] = cls.configured_data_keys() or set()
+        extracted = {k: data.pop(k) for k in list(data.keys()) if k in configured_keys}
+
+        if extracted:
+            data.setdefault("configured_data", {}).update(extracted)
+
+        return data
+
     @field_validator("date_time", mode="before")
     @classmethod
     def transform_to_datetime(cls, value: Any) -> Optional[DateTime]:
@@ -77,18 +131,39 @@ class DataRecord(DataBase, MutableMapping):
             return None
         return to_datetime(value)
 
+    @classmethod
+    def configured_data_keys(cls) -> Optional[list[str]]:
+        """Return the keys for the configured field like data.
+
+        Can be overwritten by derived classes to define specific field like data. Usually provided
+        by configuration data.
+        """
+        return None
+
     @classmethod
     def record_keys(cls) -> List[str]:
         """Returns the keys of all fields in the data record."""
         key_list = []
         key_list.extend(list(cls.model_fields.keys()))
         key_list.extend(list(cls.__pydantic_decorators__.computed_fields.keys()))
+        # Add also keys that may be added by configuration
+        key_list.remove("configured_data")
+        configured_keys = cls.configured_data_keys()
+        if configured_keys is not None:
+            key_list.extend(configured_keys)
         return key_list
 
     @classmethod
     def record_keys_writable(cls) -> List[str]:
         """Returns the keys of all fields in the data record that are writable."""
-        return list(cls.model_fields.keys())
+        keys_writable = []
+        keys_writable.extend(list(cls.model_fields.keys()))
+        # Add also keys that may be added by configuration
+        keys_writable.remove("configured_data")
+        configured_keys = cls.configured_data_keys()
+        if configured_keys is not None:
+            keys_writable.extend(configured_keys)
+        return keys_writable
 
     def _validate_key_writable(self, key: str) -> None:
         """Verify that a specified key exists and is writable in the current record keys.
@@ -104,6 +179,40 @@ class DataRecord(DataBase, MutableMapping):
                 f"Key '{key}' is not in writable record keys: {self.record_keys_writable()}"
             )
 
+    def __dir__(self) -> list[str]:
+        """Extend the default `dir()` output to include configured field like data keys.
+
+        This enables editor auto-completion and interactive introspection, while hiding the internal
+        `configured_data` dictionary.
+
+        This ensures the configured field like data values appear like native fields,
+        in line with the base model's attribute behavior.
+        """
+        base = super().__dir__()
+        keys = set(base)
+        # Expose configured data keys as attributes
+        configured_keys = self.configured_data_keys()
+        if configured_keys is not None:
+            keys.update(configured_keys)
+        # Explicitly hide the 'configured_data' internal dict
+        keys.discard("configured_data")
+        return sorted(keys)
+
+    def __eq__(self, other: Any) -> bool:
+        """Ensure equality comparison includes the contents of the `configured_data` dict.
+
+        Contents of the `configured_data` dict are in addition to the base model fields.
+        """
+        if not isinstance(other, self.__class__):
+            return NotImplemented
+        # Compare all fields except `configured_data`
+        if self.model_dump(exclude={"configured_data"}) != other.model_dump(
+            exclude={"configured_data"}
+        ):
+            return False
+        # Compare `configured_data` explicitly
+        return self.configured_data == other.configured_data
+
     def __getitem__(self, key: str) -> Any:
         """Retrieve the value of a field by key name.
 
@@ -116,9 +225,11 @@ class DataRecord(DataBase, MutableMapping):
         Raises:
             KeyError: If the specified key does not exist.
         """
-        if key in self.model_fields:
-            return getattr(self, key)
-        raise KeyError(f"'{key}' not found in the record fields.")
+        try:
+            # Let getattr do the work
+            return self.__getattr__(key)
+        except:
+            raise KeyError(f"'{key}' not found in the record fields.")
 
     def __setitem__(self, key: str, value: Any) -> None:
         """Set the value of a field by key name.
@@ -130,9 +241,10 @@ class DataRecord(DataBase, MutableMapping):
         Raises:
             KeyError: If the specified key does not exist in the fields.
         """
-        if key in self.model_fields:
-            setattr(self, key, value)
-        else:
+        try:
+            # Let setattr do the work
+            self.__setattr__(key, value)
+        except:
             raise KeyError(f"'{key}' is not a recognized field.")
 
     def __delitem__(self, key: str) -> None:
@@ -144,9 +256,9 @@ class DataRecord(DataBase, MutableMapping):
         Raises:
             KeyError: If the specified key does not exist in the fields.
         """
-        if key in self.model_fields:
-            setattr(self, key, None)  # Optional: set to None instead of deleting
-        else:
+        try:
+            self.__delattr__(key)
+        except:
             raise KeyError(f"'{key}' is not a recognized field.")
 
     def __iter__(self) -> Iterator[str]:
@@ -155,7 +267,7 @@ class DataRecord(DataBase, MutableMapping):
         Returns:
             Iterator[str]: An iterator over field names.
         """
-        return iter(self.model_fields)
+        return iter(self.record_keys_writable())
 
     def __len__(self) -> int:
         """Return the number of fields in the data record.
@@ -163,7 +275,7 @@ class DataRecord(DataBase, MutableMapping):
         Returns:
             int: The number of defined fields.
         """
-        return len(self.model_fields)
+        return len(self.record_keys_writable())
 
     def __repr__(self) -> str:
         """Provide a string representation of the data record.
@@ -171,7 +283,7 @@ class DataRecord(DataBase, MutableMapping):
         Returns:
             str: A string representation showing field names and their values.
         """
-        field_values = {field: getattr(self, field) for field in self.model_fields}
+        field_values = {field: getattr(self, field) for field in self.__class__.model_fields}
         return f"{self.__class__.__name__}({field_values})"
 
     def __getattr__(self, key: str) -> Any:
@@ -186,8 +298,13 @@ class DataRecord(DataBase, MutableMapping):
         Raises:
             AttributeError: If the field does not exist.
         """
-        if key in self.model_fields:
+        if key in self.__class__.model_fields:
             return getattr(self, key)
+        if key in self.configured_data.keys():
+            return self.configured_data[key]
+        configured_keys = self.configured_data_keys()
+        if configured_keys is not None and key in configured_keys:
+            return None
         raise AttributeError(f"'{self.__class__.__name__}' object has no attribute '{key}'")
 
     def __setattr__(self, key: str, value: Any) -> None:
@@ -200,10 +317,14 @@ class DataRecord(DataBase, MutableMapping):
         Raises:
             AttributeError: If the attribute/field does not exist.
         """
-        if key in self.model_fields:
+        if key in self.__class__.model_fields:
             super().__setattr__(key, value)
-        else:
-            raise AttributeError(f"'{self.__class__.__name__}' object has no attribute '{key}'")
+            return
+        configured_keys = self.configured_data_keys()
+        if configured_keys is not None and key in configured_keys:
+            self.configured_data[key] = value
+            return
+        raise AttributeError(f"'{self.__class__.__name__}' object has no attribute '{key}'")
 
     def __delattr__(self, key: str) -> None:
         """Delete an attribute by setting it to None if it exists as a field.
@@ -214,10 +335,21 @@ class DataRecord(DataBase, MutableMapping):
         Raises:
             AttributeError: If the attribute/field does not exist.
         """
-        if key in self.model_fields:
-            setattr(self, key, None)  # Optional: set to None instead of deleting
-        else:
-            super().__delattr__(key)
+        if key in self.__class__.model_fields:
+            data: Optional[dict]
+            if key == "configured_data":
+                data = dict()
+            else:
+                data = None
+            setattr(self, key, data)
+            return
+        if key in self.configured_data:
+            del self.configured_data[key]
+            return
+        configured_keys = self.configured_data_keys()
+        if configured_keys is not None and key in configured_keys:
+            return
+        super().__delattr__(key)
 
     @classmethod
     def key_from_description(cls, description: str, threshold: float = 0.8) -> Optional[str]:
@@ -352,10 +484,7 @@ class DataSequence(DataBase, MutableSequence):
     @property
     def record_keys(self) -> List[str]:
         """Returns the keys of all fields in the data records."""
-        key_list = []
-        key_list.extend(list(self.record_class().model_fields.keys()))
-        key_list.extend(list(self.record_class().__pydantic_decorators__.computed_fields.keys()))
-        return key_list
+        return self.record_class().record_keys()
 
     @computed_field  # type: ignore[prop-decorator]
     @property
@@ -369,7 +498,7 @@ class DataSequence(DataBase, MutableSequence):
         Returns:
             List[str]: A list of field keys that are writable in the data records.
         """
-        return list(self.record_class().model_fields.keys())
+        return self.record_class().record_keys_writable()
 
     @classmethod
     def record_class(cls) -> Type:
@@ -707,6 +836,38 @@ class DataSequence(DataBase, MutableSequence):
 
         return filtered_data
 
+    def key_to_value(self, key: str, target_datetime: DateTime) -> Optional[float]:
+        """Returns the value corresponding to the specified key that is nearest to the given datetime.
+
+        Args:
+            key (str): The key of the attribute in DataRecord to extract.
+            target_datetime (datetime): The datetime to search nearest to.
+
+        Returns:
+            Optional[float]: The value nearest to the given datetime, or None if no valid records are found.
+
+        Raises:
+            KeyError: If the specified key is not found in any of the DataRecords.
+        """
+        self._validate_key(key)
+
+        # Filter out records with None or NaN values for the key
+        valid_records = [
+            record
+            for record in self.records
+            if record.date_time is not None
+            and getattr(record, key, None) not in (None, float("nan"))
+        ]
+
+        if not valid_records:
+            return None
+
+        # Find the record with datetime nearest to target_datetime
+        target = to_datetime(target_datetime)
+        nearest_record = min(valid_records, key=lambda r: abs(r.date_time - target))
+
+        return getattr(nearest_record, key, None)
+
     def key_to_lists(
         self,
         key: str,
@@ -868,6 +1029,11 @@ class DataSequence(DataBase, MutableSequence):
             KeyError: If the specified key is not found in any of the DataRecords.
         """
         self._validate_key(key)
+
+        # General check on fill_method
+        if fill_method not in ("ffill", "bfill", "linear", "none", None):
+            raise ValueError(f"Unsupported fill method: {fill_method}")
+
         # Ensure datetime objects are normalized
         start_datetime = to_datetime(start_datetime, to_maxtime=False) if start_datetime else None
         end_datetime = to_datetime(end_datetime, to_maxtime=False) if end_datetime else None
@@ -880,7 +1046,7 @@ class DataSequence(DataBase, MutableSequence):
         values_len = len(values)
 
         if values_len < 1:
-            # No values, assume at at least one value set to None
+            # No values, assume at least one value set to None
             if start_datetime is not None:
                 dates.append(start_datetime - interval)
             else:
@@ -902,6 +1068,11 @@ class DataSequence(DataBase, MutableSequence):
                 # Truncate all values before latest value before start_datetime
                 dates = dates[start_index - 1 :]
                 values = values[start_index - 1 :]
+            # We have a start_datetime, align to start datetime
+            resample_origin = start_datetime
+        else:
+            # We do not have a start_datetime, align resample buckets to midnight of first day
+            resample_origin = "start_day"
 
         if end_datetime is not None:
             if compare_datetimes(dates[-1], end_datetime).lt:
@@ -922,7 +1093,7 @@ class DataSequence(DataBase, MutableSequence):
             if fill_method is None:
                 fill_method = "linear"
             # Resample the series to the specified interval
-            resampled = series.resample(interval, origin="start").first()
+            resampled = series.resample(interval, origin=resample_origin).first()
             if fill_method == "linear":
                 resampled = resampled.interpolate(method="linear")
             elif fill_method == "ffill":
@@ -936,7 +1107,7 @@ class DataSequence(DataBase, MutableSequence):
             if fill_method is None:
                 fill_method = "ffill"
             # Resample the series to the specified interval
-            resampled = series.resample(interval, origin="start").first()
+            resampled = series.resample(interval, origin=resample_origin).first()
             if fill_method == "ffill":
                 resampled = resampled.ffill()
             elif fill_method == "bfill":
@@ -944,12 +1115,24 @@ class DataSequence(DataBase, MutableSequence):
             elif fill_method != "none":
                 raise ValueError(f"Unsupported fill method for non-numeric data: {fill_method}")
 
+        logger.debug(
+            "Resampled for '{}' with length {}: {}...{}",
+            key,
+            len(resampled),
+            resampled[:10],
+            resampled[-10:],
+        )
+
         # Convert the resampled series to a NumPy array
         if start_datetime is not None and len(resampled) > 0:
             resampled = resampled.truncate(before=start_datetime)
         if end_datetime is not None and len(resampled) > 0:
             resampled = resampled.truncate(after=end_datetime.subtract(seconds=1))
         array = resampled.values
+        logger.debug(
+            "Array for '{}' with length {}: {}...{}", key, len(array), array[:10], array[-10:]
+        )
+
         return array
 
     def to_dataframe(
@@ -1197,7 +1380,7 @@ class DataImportMixin:
     the values. `ìnterval` may be used to define the fixed time interval between two values.
 
     On import `self.update_value(datetime, key, value)` is called which has to be provided.
-    Also `self.start_datetime` may be necessary as a default in case `start_datetime`is not given.
+    Also `self.ems_start_datetime` may be necessary as a default in case `start_datetime`is not given.
     """
 
     # Attributes required but defined elsehere.
@@ -1315,7 +1498,7 @@ class DataImportMixin:
                 raise ValueError(f"Invalid start_datetime in import data: {e}")
 
         if start_datetime is None:
-            start_datetime = self.start_datetime  # type: ignore
+            start_datetime = self.ems_start_datetime  # type: ignore
 
         if "interval" in import_data:
             try:
@@ -1406,7 +1589,7 @@ class DataImportMixin:
                 raise ValueError(f"Invalid datetime index in DataFrame: {e}")
         else:
             if start_datetime is None:
-                start_datetime = self.start_datetime  # type: ignore
+                start_datetime = self.ems_start_datetime  # type: ignore
             has_datetime_index = False
 
         # Filter columns based on key_prefix and record_keys_writable
@@ -1463,7 +1646,7 @@ class DataImportMixin:
 
         If start_datetime and or interval is given in the JSON dict it will be used. Otherwise
         the given parameters are used. If None is given start_datetime defaults to
-        'self.start_datetime' and interval defaults to 1 hour.
+        'self.ems_start_datetime' and interval defaults to 1 hour.
 
         Args:
             json_str (str): The JSON string containing the generic data.
@@ -1538,7 +1721,7 @@ class DataImportMixin:
 
         If start_datetime and or interval is given in the JSON dict it will be used. Otherwise
         the given parameters are used. If None is given start_datetime defaults to
-        'self.start_datetime' and interval defaults to 1 hour.
+        'self.ems_start_datetime' and interval defaults to 1 hour.
 
         Args:
             import_file_path (Path): The path to the JSON file containing the generic data.
@@ -1749,7 +1932,12 @@ class DataContainer(SingletonMixin, DataBase, MutableMapping):
             force_update (bool, optional): If True, forces the providers to update the data even if still cached.
         """
         for provider in self.providers:
-            provider.update_data(force_enable=force_enable, force_update=force_update)
+            try:
+                provider.update_data(force_enable=force_enable, force_update=force_update)
+            except Exception as ex:
+                error = f"Provider {provider.provider_id()} fails on update - enabled={provider.enabled()}, force_enable={force_enable}, force_update={force_update}: {ex}"
+                logger.error(error)
+                raise RuntimeError(error)
 
     def key_to_series(
         self,
@@ -1854,7 +2042,7 @@ class DataContainer(SingletonMixin, DataBase, MutableMapping):
     ) -> pd.DataFrame:
         """Retrieve a dataframe indexed by fixed time intervals for specified keys from the data in each DataProvider.
 
-        Generates a pandas DataFrame using the NumPy arrays for each specified key, ensuring a common time index..
+        Generates a pandas DataFrame using the NumPy arrays for each specified key, ensuring a common time index.
 
         Args:
             keys (list[str]): A list of field names to retrieve.
@@ -1903,8 +2091,15 @@ class DataContainer(SingletonMixin, DataBase, MutableMapping):
                 end_datetime.add(seconds=1)
 
         # Create a DatetimeIndex based on start, end, and interval
+        if start_datetime is None or end_datetime is None:
+            raise ValueError(
+                f"Can not determine datetime range. Got '{start_datetime}'..'{end_datetime}'."
+            )
         reference_index = pd.date_range(
-            start=start_datetime, end=end_datetime, freq=interval, inclusive="left"
+            start=start_datetime,
+            end=end_datetime,
+            freq=interval,
+            inclusive="left",
         )
 
         data = {}