Mirrors/EOS
1
0
Fork 0
mirror of https://github.com/Akkudoktor-EOS/EOS.git synced 2025-10-11 11:56:17 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Fix2 config and predictions revamp. (#281)

Browse Source 
measurement:

- Add new measurement class to hold real world measurements.
- Handles load meter readings, grid import and export meter readings.
- Aggregates load meter readings aka. measurements to total load.
- Can import measurements from files, pandas datetime series,
    pandas datetime dataframes, simple daetime arrays and
    programmatically.
- Maybe expanded to other measurement values.
- Should be used for load prediction adaptions by real world
    measurements.

core/coreabc:

- Add mixin class to access measurements

core/pydantic:

- Add pydantic models for pandas datetime series and dataframes.
- Add pydantic models for simple datetime array

core/dataabc:

- Provide DataImport mixin class for generic import handling.
    Imports from JSON string and files. Imports from pandas datetime dataframes
    and simple datetime arrays. Signature of import method changed to
    allow import datetimes to be given programmatically and by data content.
- Use pydantic models for datetime series, dataframes, arrays
- Validate generic imports by pydantic models
- Provide new attributes min_datetime and max_datetime for DataSequence.
- Add parameter dropna to drop NAN/ None values when creating lists, pandas series
    or numpy array from DataSequence.

config/config:

- Add common settings for the measurement module.

predictions/elecpriceakkudoktor:

- Use mean values of last 7 days to fill prediction values not provided by
    akkudoktor.net (only provides 24 values).

prediction/loadabc:

- Extend the generic prediction keys by 'load_total_adjusted' for load predictions
    that adjust the predicted total load by measured load values.

prediction/loadakkudoktor:

- Extend the Akkudoktor load prediction by load adjustment using measured load
    values.

prediction/load_aggregator:

- Module removed. Load aggregation is now handled by the measurement module.

prediction/load_corrector:

- Module removed. Load correction (aka. adjustment of load prediction by
    measured load energy) is handled by the LoadAkkudoktor prediction and
    the generic 'load_mean_adjusted' prediction key.

prediction/load_forecast:

- Module removed. Functionality now completely handled by the LoadAkkudoktor
    prediction.

utils/cacheutil:

- Use pydantic.
- Fix potential bug in ttl (time to live) duration handling.

utils/datetimeutil:

- Added missing handling of pendulum.DateTime and pendulum.Duration instances
    as input. Handled before as datetime.datetime and datetime.timedelta.

utils/visualize:

- Move main to generate_example_report() for better testing support.

server/server:

- Added new configuration option server_fastapi_startup_server_fasthtml
  to make startup of FastHTML server by FastAPI server conditional.

server/fastapi_server:

- Add APIs for measurements
- Improve APIs to provide or take pandas datetime series and
    datetime dataframes controlled by pydantic model.
- Improve APIs to provide or take simple datetime data arrays
    controlled by pydantic model.
- Move fastAPI server API to v1 for new APIs.
- Update pre v1 endpoints to use new prediction and measurement capabilities.
- Only start FastHTML server if 'server_fastapi_startup_server_fasthtml'
    config option is set.

tests:

- Adapt import tests to changed import method signature
- Adapt server test to use the v1 API
- Extend the dataabc test to test for array generation from data
    with several data interval scenarios.
- Extend the datetimeutil test to also test for correct handling
    of to_datetime() providing now().
- Adapt LoadAkkudoktor test for new adjustment calculation.
- Adapt visualization test to use example report function instead of visualize.py
    run as process.
- Removed test_load_aggregator. Functionality is now tested in test_measurement.
- Added tests for measurement module

docs:

- Remove sphinxcontrib-openapi as it prevents build of documentation.
    "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema
    for t in schema["anyOf"]: KeyError: 'anyOf'"

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
This commit is contained in:
Bobby Noelte
2024-12-29 18:42:49 +01:00
committed by GitHub
parent 2a8e11d7dc
commit 830af85fca
38 changed files with 3671 additions and 948 deletions
Download Patch File Download Diff File Expand all files Collapse all files

574
src/akkudoktoreos/core/dataabc.py
View File

@@ -21,10 +21,21 @@ import pandas as pd
import pendulum
from numpydantic import NDArray, Shape
from pendulum import DateTime, Duration
from pydantic import AwareDatetime, ConfigDict, Field, computed_field, field_validator
from pydantic import (
AwareDatetime,
ConfigDict,
Field,
ValidationError,
computed_field,
field_validator,
)
from akkudoktoreos.core.coreabc import ConfigMixin, SingletonMixin, StartMixin
from akkudoktoreos.core.pydantic import PydanticBaseModel
from akkudoktoreos.core.pydantic import (
PydanticBaseModel,
PydanticDateTimeData,
PydanticDateTimeDataFrame,
)
from akkudoktoreos.utils.datetimeutil import compare_datetimes, to_datetime, to_duration
from akkudoktoreos.utils.logutil import get_logger
@@ -47,22 +58,22 @@ class DataRecord(DataBase, MutableMapping):
and attribute-style access (`record.field_name`).
Attributes:
date_time (Optional[AwareDatetime]): Aware datetime indicating when the data record applies.
date_time (Optional[DateTime]): Aware datetime indicating when the data record applies.
Configurations:
- Allows mutation after creation.
- Supports non-standard data types like `datetime`.
"""
date_time: Optional[AwareDatetime] = Field(default=None, description="DateTime")
date_time: Optional[DateTime] = Field(default=None, description="DateTime")
# Pydantic v2 model configuration
model_config = ConfigDict(arbitrary_types_allowed=True, populate_by_name=True)
@field_validator("date_time", mode="before")
@classmethod
def transform_to_datetime(cls, value: Any) -> DateTime:
"""Converts various datetime formats into AwareDatetime."""
def transform_to_datetime(cls, value: Any) -> Optional[DateTime]:
"""Converts various datetime formats into DateTime."""
if value is None:
# Allow to set to default.
return None
@@ -307,6 +318,38 @@ class DataSequence(DataBase, MutableSequence):
records: List[DataRecord] = Field(default_factory=list, description="List of data records")
# Derived fields (computed)
@computed_field # type: ignore[prop-decorator]
@property
def min_datetime(self) -> Optional[DateTime]:
"""Minimum (earliest) datetime in the sorted sequence of data records.
This property computes the earliest datetime from the sequence of data records.
If no records are present, it returns `None`.
Returns:
Optional[DateTime]: The earliest datetime in the sequence, or `None` if no
data records exist.
"""
if len(self.records) == 0:
return None
return self.records[0].date_time
@computed_field # type: ignore[prop-decorator]
@property
def max_datetime(self) -> DateTime:
"""Maximum (latest) datetime in the sorted sequence of data records.
This property computes the latest datetime from the sequence of data records.
If no records are present, it returns `None`.
Returns:
Optional[DateTime]: The latest datetime in the sequence, or `None` if no
data records exist.
"""
if len(self.records) == 0:
return None
return self.records[-1].date_time
@computed_field # type: ignore[prop-decorator]
@property
def record_keys(self) -> List[str]:
@@ -319,12 +362,31 @@ class DataSequence(DataBase, MutableSequence):
@computed_field # type: ignore[prop-decorator]
@property
def record_keys_writable(self) -> List[str]:
"""Returns the keys of all fields in the data records that are writable."""
"""Get the keys of all writable fields in the data records.
This property retrieves the keys of all fields in the data records that
can be written to. It uses the `record_class` to determine the model's
field structure.
Returns:
List[str]: A list of field keys that are writable in the data records.
"""
return list(self.record_class().model_fields.keys())
@classmethod
def record_class(cls) -> Type:
"""Returns the class of the data record this data sequence handles."""
"""Get the class of the data record handled by this data sequence.
This method determines the class of the data record type associated with
the `records` field of the model. The field is expected to be a list, and
the element type of the list should be a subclass of `DataRecord`.
Raises:
ValueError: If the record type is not a subclass of `DataRecord`.
Returns:
Type: The class of the data record handled by the data sequence.
"""
# Access the model field metadata
field_info = cls.model_fields["records"]
# Get the list element type from the 'type_' attribute
@@ -573,6 +635,7 @@ class DataSequence(DataBase, MutableSequence):
key: str,
start_datetime: Optional[DateTime] = None,
end_datetime: Optional[DateTime] = None,
dropna: Optional[bool] = None,
) -> Dict[DateTime, Any]:
"""Extract a dictionary indexed by the date_time field of the DataRecords.
@@ -583,6 +646,7 @@ class DataSequence(DataBase, MutableSequence):
key (str): The field name in the DataRecord from which to extract values.
start_datetime (datetime, optional): The start date to filter records (inclusive).
end_datetime (datetime, optional): The end date to filter records (exclusive).
dropna: (bool, optional): Whether to drop NAN/ None values before processing. Defaults to True.
Returns:
Dict[datetime, Any]: A dictionary with the date_time of each record as the key
@@ -597,12 +661,22 @@ class DataSequence(DataBase, MutableSequence):
end_datetime = to_datetime(end_datetime, to_maxtime=False) if end_datetime else None
# Create a dictionary to hold date_time and corresponding values
filtered_data = {
to_datetime(record.date_time, as_string=True): getattr(record, key, None)
for record in self.records
if (start_datetime is None or compare_datetimes(record.date_time, start_datetime).ge)
and (end_datetime is None or compare_datetimes(record.date_time, end_datetime).lt)
}
if dropna is None:
dropna = True
filtered_data = {}
for record in self.records:
if (
record.date_time is None
or (dropna and getattr(record, key, None) is None)
or (dropna and getattr(record, key, None) == float("nan"))
):
continue
if (
start_datetime is None or compare_datetimes(record.date_time, start_datetime).ge
) and (end_datetime is None or compare_datetimes(record.date_time, end_datetime).lt):
filtered_data[to_datetime(record.date_time, as_string=True)] = getattr(
record, key, None
)
return filtered_data
@@ -611,6 +685,7 @@ class DataSequence(DataBase, MutableSequence):
key: str,
start_datetime: Optional[DateTime] = None,
end_datetime: Optional[DateTime] = None,
dropna: Optional[bool] = None,
) -> Tuple[List[DateTime], List[Optional[float]]]:
"""Extracts two lists from data records within an optional date range.
@@ -622,6 +697,7 @@ class DataSequence(DataBase, MutableSequence):
key (str): The key of the attribute in DataRecord to extract.
start_datetime (datetime, optional): The start date for filtering the records (inclusive).
end_datetime (datetime, optional): The end date for filtering the records (exclusive).
dropna: (bool, optional): Whether to drop NAN/ None values before processing. Defaults to True.
Returns:
tuple: A tuple containing a list of datetime values and a list of extracted values.
@@ -635,9 +711,15 @@ class DataSequence(DataBase, MutableSequence):
end_datetime = to_datetime(end_datetime, to_maxtime=False) if end_datetime else None
# Create two lists to hold date_time and corresponding values
if dropna is None:
dropna = True
filtered_records = []
for record in self.records:
if record.date_time is None:
if (
record.date_time is None
or (dropna and getattr(record, key, None) is None)
or (dropna and getattr(record, key, None) == float("nan"))
):
continue
if (
start_datetime is None or compare_datetimes(record.date_time, start_datetime).ge
@@ -653,6 +735,7 @@ class DataSequence(DataBase, MutableSequence):
key: str,
start_datetime: Optional[DateTime] = None,
end_datetime: Optional[DateTime] = None,
dropna: Optional[bool] = None,
) -> pd.Series:
"""Extract a series indexed by the date_time field from data records within an optional date range.
@@ -660,6 +743,7 @@ class DataSequence(DataBase, MutableSequence):
key (str): The field name in the DataRecord from which to extract values.
start_datetime (datetime, optional): The start date for filtering the records (inclusive).
end_datetime (datetime, optional): The end date for filtering the records (exclusive).
dropna: (bool, optional): Whether to drop NAN/ None values before processing. Defaults to True.
Returns:
pd.Series: A Pandas Series with the index as the date_time of each record
@@ -668,7 +752,9 @@ class DataSequence(DataBase, MutableSequence):
Raises:
KeyError: If the specified key is not found in any of the DataRecords.
"""
dates, values = self.key_to_lists(key, start_datetime, end_datetime)
dates, values = self.key_to_lists(
key=key, start_datetime=start_datetime, end_datetime=end_datetime, dropna=dropna
)
return pd.Series(data=values, index=pd.DatetimeIndex(dates), name=key)
def key_from_series(self, key: str, series: pd.Series) -> None:
@@ -704,6 +790,7 @@ class DataSequence(DataBase, MutableSequence):
end_datetime: Optional[DateTime] = None,
interval: Optional[Duration] = None,
fill_method: Optional[str] = None,
dropna: Optional[bool] = None,
) -> NDArray[Shape["*"], Any]:
"""Extract an array indexed by fixed time intervals from data records within an optional date range.
@@ -717,6 +804,7 @@ class DataSequence(DataBase, MutableSequence):
- 'ffill': Forward fill missing values.
- 'bfill': Backward fill missing values.
- 'none': Defaults to 'linear' for numeric values, otherwise 'ffill'.
dropna: (bool, optional): Whether to drop NAN/ None values before processing. Defaults to True.
Returns:
np.ndarray: A NumPy Array of the values extracted from the specified key.
@@ -724,10 +812,54 @@ class DataSequence(DataBase, MutableSequence):
Raises:
KeyError: If the specified key is not found in any of the DataRecords.
"""
self._validate_key(key)
# Ensure datetime objects are normalized
start_datetime = to_datetime(start_datetime, to_maxtime=False) if start_datetime else None
end_datetime = to_datetime(end_datetime, to_maxtime=False) if end_datetime else None
resampled = None
if interval is None:
interval = to_duration("1 hour")
series = self.key_to_series(key)
dates, values = self.key_to_lists(key=key, dropna=dropna)
values_len = len(values)
if values_len < 1:
# No values, assume at at least one value set to None
if start_datetime is not None:
dates.append(start_datetime - interval)
else:
dates.append(to_datetime(to_maxtime=False))
values.append(None)
if start_datetime is not None:
start_index = 0
while start_index < values_len:
if compare_datetimes(dates[start_index], start_datetime).ge:
break
start_index += 1
if start_index == 0:
# No value before start
# Add dummy value
dates.insert(0, dates[0] - interval)
values.insert(0, values[0])
elif start_index > 1:
# Truncate all values before latest value before start_datetime
dates = dates[start_index - 1 :]
values = values[start_index - 1 :]
if end_datetime is not None:
if compare_datetimes(dates[-1], end_datetime).lt:
# Add dummy value at end_datetime
dates.append(end_datetime)
values.append(values[-1])
series = pd.Series(data=values, index=pd.DatetimeIndex(dates), name=key)
if not series.index.inferred_type == "datetime64":
raise TypeError(
f"Expected DatetimeIndex, but got {type(series.index)} "
f"infered to {series.index.inferred_type}: {series}"
)
# Handle missing values
if series.dtype in [np.float64, np.float32, np.int64, np.int32]:
@@ -735,7 +867,7 @@ class DataSequence(DataBase, MutableSequence):
if fill_method is None:
fill_method = "linear"
# Resample the series to the specified interval
resampled = series.resample(interval).mean()
resampled = series.resample(interval, origin="start").first()
if fill_method == "linear":
resampled = resampled.interpolate(method="linear")
elif fill_method == "ffill":
@@ -749,7 +881,7 @@ class DataSequence(DataBase, MutableSequence):
if fill_method is None:
fill_method = "ffill"
# Resample the series to the specified interval
resampled = series.resample(interval).first()
resampled = series.resample(interval, origin="start").first()
if fill_method == "ffill":
resampled = resampled.ffill()
elif fill_method == "bfill":
@@ -955,18 +1087,29 @@ class DataProvider(SingletonMixin, DataSequence):
self.sort_by_datetime()
class DataImportProvider(DataProvider):
"""Abstract base class for data providers that import generic data.
class DataImportMixin:
"""Mixin class for import of generic data.
This class is designed to handle generic data provided in the form of a key-value dictionary.
- **Keys**: Represent identifiers from the record keys of a specific data.
- **Values**: Are lists of data values starting at a specified `start_datetime`, where
each value corresponds to a subsequent time interval (e.g., hourly).
Subclasses must implement the logic for managing generic data based on the imported records.
Two special keys are handled. `start_datetime` may be used to defined the starting datetime of
the values. `ìnterval` may be used to define the fixed time interval between two values.
On import `self.update_value(datetime, key, value)` is called which has to be provided.
Also `self.start_datetime` may be necessary as a default in case `start_datetime`is not given.
"""
def import_datetimes(self, value_count: int) -> List[Tuple[DateTime, int]]:
# Attributes required but defined elsehere.
# - start_datetime
# - record_keys_writable
# - update_valu
def import_datetimes(
self, start_datetime: DateTime, value_count: int, interval: Optional[Duration] = None
) -> List[Tuple[DateTime, int]]:
"""Generates a list of tuples containing timestamps and their corresponding value indices.
The function accounts for daylight saving time (DST) transitions:
@@ -975,7 +1118,9 @@ class DataImportProvider(DataProvider):
but they share the same value index.
Args:
start_datetime (DateTime): Start datetime of values
value_count (int): The number of timestamps to generate.
interval (duration, optional): The fixed time interval. Defaults to 1 hour.
Returns:
List[Tuple[DateTime, int]]:
@@ -990,7 +1135,7 @@ class DataImportProvider(DataProvider):
Example:
>>> start_datetime = pendulum.datetime(2024, 11, 3, 0, 0, tz="America/New_York")
>>> import_datetimes(5)
>>> import_datetimes(start_datetime, 5)
[(DateTime(2024, 11, 3, 0, 0, tzinfo=Timezone('America/New_York')), 0),
(DateTime(2024, 11, 3, 1, 0, tzinfo=Timezone('America/New_York')), 1),
(DateTime(2024, 11, 3, 1, 0, tzinfo=Timezone('America/New_York')), 1), # Repeated hour
@@ -998,7 +1143,16 @@ class DataImportProvider(DataProvider):
(DateTime(2024, 11, 3, 3, 0, tzinfo=Timezone('America/New_York')), 3)]
"""
timestamps_with_indices: List[Tuple[DateTime, int]] = []
value_datetime = self.start_datetime
if interval is None:
interval = to_duration("1 hour")
interval_steps_per_hour = int(3600 / interval.total_seconds())
if interval.total_seconds() * interval_steps_per_hour != 3600:
error_msg = f"Interval {interval} does not fit into hour."
logger.error(error_msg)
raise NotImplementedError(error_msg)
value_datetime = start_datetime
value_index = 0
while value_index < value_count:
@@ -1006,37 +1160,219 @@ class DataImportProvider(DataProvider):
logger.debug(f"{i}: Insert at {value_datetime} with index {value_index}")
timestamps_with_indices.append((value_datetime, value_index))
# Check if there is a DST transition
next_time = value_datetime.add(hours=1)
if next_time <= value_datetime:
# Check if there is a DST transition (i.e., ambiguous time during fall back)
# Repeat the hour value (reuse value index)
value_datetime = next_time
logger.debug(f"{i+1}: Repeat at {value_datetime} with index {value_index}")
timestamps_with_indices.append((value_datetime, value_index))
elif next_time.hour != value_datetime.hour + 1 and value_datetime.hour != 23:
# Skip the hour value (spring forward in value index)
value_index += 1
logger.debug(f"{i+1}: Skip at {next_time} with index {value_index}")
next_time = value_datetime.add(seconds=interval.total_seconds())
# Increment value index and value_datetime for new hour
# Check if there is a DST transition
if next_time.dst() != value_datetime.dst():
if next_time.hour == value_datetime.hour:
# We jump back by 1 hour
# Repeat the value(s) (reuse value index)
for i in range(interval_steps_per_hour):
logger.debug(f"{i+1}: Repeat at {next_time} with index {value_index}")
timestamps_with_indices.append((next_time, value_index))
next_time = next_time.add(seconds=interval.total_seconds())
else:
# We jump forward by 1 hour
# Drop the value(s)
logger.debug(
f"{i+1}: Skip {interval_steps_per_hour} at {next_time} with index {value_index}"
)
value_index += interval_steps_per_hour
# Increment value index and value_datetime for new interval
value_index += 1
value_datetime = value_datetime.add(hours=1)
value_datetime = next_time
return timestamps_with_indices
def import_from_json(self, json_str: str, key_prefix: str = "") -> None:
def import_from_dict(
self,
import_data: dict,
key_prefix: str = "",
start_datetime: Optional[DateTime] = None,
interval: Optional[Duration] = None,
) -> None:
"""Updates generic data by importing it from a dictionary.
This method reads generic data from a dictionary, matches keys based on the
record keys and the provided `key_prefix`, and updates the data values sequentially.
All value lists must have the same length.
Args:
import_data (dict): Dictionary containing the generic data with optional
'start_datetime' and 'interval' keys.
key_prefix (str, optional): A prefix to filter relevant keys from the generic data.
Only keys starting with this prefix will be considered. Defaults to an empty string.
start_datetime (DateTime, optional): Start datetime of values if not in dict.
interval (Duration, optional): The fixed time interval if not in dict.
Raises:
ValueError: If value lists have different lengths or if datetime conversion fails.
"""
# Handle datetime and interval from dict or parameters
if "start_datetime" in import_data:
try:
start_datetime = to_datetime(import_data["start_datetime"])
except (ValueError, TypeError) as e:
raise ValueError(f"Invalid start_datetime in import data: {e}")
if start_datetime is None:
start_datetime = self.start_datetime # type: ignore
if "interval" in import_data:
try:
interval = to_duration(import_data["interval"])
except (ValueError, TypeError) as e:
raise ValueError(f"Invalid interval in import data: {e}")
# Filter keys based on key_prefix and record_keys_writable
valid_keys = [
key
for key in import_data.keys()
if key.startswith(key_prefix)
and key in self.record_keys_writable # type: ignore
and key not in ("start_datetime", "interval")
]
if not valid_keys:
return
# Validate all value lists have the same length
value_lengths = []
for key in valid_keys:
value_list = import_data[key]
if not isinstance(value_list, (list, tuple, np.ndarray)):
raise ValueError(f"Value for key '{key}' must be a list, tuple, or array")
value_lengths.append(len(value_list))
if len(set(value_lengths)) > 1:
raise ValueError(
f"All value lists must have the same length. Found lengths: "
f"{dict(zip(valid_keys, value_lengths))}"
)
# Generate datetime mapping once for the common length
values_count = value_lengths[0]
value_datetime_mapping = self.import_datetimes(
start_datetime, values_count, interval=interval
)
# Process each valid key
for key in valid_keys:
try:
value_list = import_data[key]
# Update values, skipping any None/NaN
for value_datetime, value_index in value_datetime_mapping:
value = value_list[value_index]
if value is not None and not pd.isna(value):
self.update_value(value_datetime, key, value) # type: ignore
except (IndexError, TypeError) as e:
raise ValueError(f"Error processing values for key '{key}': {e}")
def import_from_dataframe(
self,
df: pd.DataFrame,
key_prefix: str = "",
start_datetime: Optional[DateTime] = None,
interval: Optional[Duration] = None,
) -> None:
"""Updates generic data by importing it from a pandas DataFrame.
This method reads generic data from a DataFrame, matches columns based on the
record keys and the provided `key_prefix`, and updates the data values using
the DataFrame's index as timestamps.
Args:
df (pd.DataFrame): DataFrame containing the generic data with datetime index
or sequential values.
key_prefix (str, optional): A prefix to filter relevant columns from the DataFrame.
Only columns starting with this prefix will be considered. Defaults to an empty string.
start_datetime (DateTime, optional): Start datetime if DataFrame doesn't have datetime index.
interval (Duration, optional): The fixed time interval if DataFrame doesn't have datetime index.
Raises:
ValueError: If DataFrame structure is invalid or datetime conversion fails.
"""
# Validate DataFrame
if not isinstance(df, pd.DataFrame):
raise ValueError("Input must be a pandas DataFrame")
# Handle datetime index
if isinstance(df.index, pd.DatetimeIndex):
try:
index_datetimes = [to_datetime(dt) for dt in df.index]
has_datetime_index = True
except (ValueError, TypeError) as e:
raise ValueError(f"Invalid datetime index in DataFrame: {e}")
else:
if start_datetime is None:
start_datetime = self.start_datetime # type: ignore
has_datetime_index = False
# Filter columns based on key_prefix and record_keys_writable
valid_columns = [
col
for col in df.columns
if col.startswith(key_prefix) and col in self.record_keys_writable # type: ignore
]
if not valid_columns:
return
# For DataFrame, length validation is implicit since all columns have same length
values_count = len(df)
# Generate value_datetime_mapping once if not using datetime index
if not has_datetime_index:
value_datetime_mapping = self.import_datetimes(
start_datetime, values_count, interval=interval
)
# Process each valid column
for column in valid_columns:
try:
values = df[column].tolist()
if has_datetime_index:
# Use the DataFrame's datetime index
for dt, value in zip(index_datetimes, values):
if value is not None and not pd.isna(value):
self.update_value(dt, column, value) # type: ignore
else:
# Use the pre-generated datetime mapping
for value_datetime, value_index in value_datetime_mapping:
value = values[value_index]
if value is not None and not pd.isna(value):
self.update_value(value_datetime, column, value) # type: ignore
except Exception as e:
raise ValueError(f"Error processing column '{column}': {e}")
def import_from_json(
self,
json_str: str,
key_prefix: str = "",
start_datetime: Optional[DateTime] = None,
interval: Optional[Duration] = None,
) -> None:
"""Updates generic data by importing it from a JSON string.
This method reads generic data from a JSON string, matches keys based on the
record keys and the provided `key_prefix`, and updates the data values sequentially,
starting from the `start_datetime`. Each data value is associated with an hourly
interval.
starting from the `start_datetime`.
If start_datetime and or interval is given in the JSON dict it will be used. Otherwise
the given parameters are used. If None is given start_datetime defaults to
'self.start_datetime' and interval defaults to 1 hour.
Args:
json_str (str): The JSON string containing the generic data.
key_prefix (str, optional): A prefix to filter relevant keys from the generic data.
Only keys starting with this prefix will be considered. Defaults to an empty string.
start_datetime (DateTime, optional): Start datetime of values.
interval (duration, optional): The fixed time interval. Defaults to 1 hour.
Raises:
JSONDecodeError: If the file content is not valid JSON.
@@ -1045,22 +1381,56 @@ class DataImportProvider(DataProvider):
Given a JSON string with the following content:
```json
{
"load0_mean": [20.5, 21.0, 22.1],
"load1_mean": [50, 55, 60]
"start_datetime": "2024-11-10 00:00:00"
"interval": "30 minutes"
"load_mean": [20.5, 21.0, 22.1],
"other_xyz: [10.5, 11.0, 12.1],
}
```
and `key_prefix = "load1"`, only the "load1_mean" key will be processed even though
and `key_prefix = "load"`, only the "load_mean" key will be processed even though
both keys are in the record.
"""
import_data = json.loads(json_str)
for key in self.record_keys_writable:
if key.startswith(key_prefix) and key in import_data:
value_list = import_data[key]
value_datetime_mapping = self.import_datetimes(len(value_list))
for value_datetime, value_index in value_datetime_mapping:
self.update_value(value_datetime, key, value_list[value_index])
# Try pandas dataframe with orient="split"
try:
import_data = PydanticDateTimeDataFrame.model_validate_json(json_str)
self.import_from_dataframe(import_data.to_dataframe())
return
except ValidationError as e:
error_msg = ""
for error in e.errors():
field = " -> ".join(str(x) for x in error["loc"])
message = error["msg"]
error_type = error["type"]
error_msg += f"Field: {field}\nError: {message}\nType: {error_type}\n"
logger.debug(f"PydanticDateTimeDataFrame import: {error_msg}")
def import_from_file(self, import_file_path: Path, key_prefix: str = "") -> None:
# Try dictionary with special keys start_datetime and intervall
try:
import_data = PydanticDateTimeData.model_validate_json(json_str)
self.import_from_dict(import_data.to_dict())
return
except ValidationError as e:
error_msg = ""
for error in e.errors():
field = " -> ".join(str(x) for x in error["loc"])
message = error["msg"]
error_type = error["type"]
error_msg += f"Field: {field}\nError: {message}\nType: {error_type}\n"
logger.debug(f"PydanticDateTimeData import: {error_msg}")
# Use simple dict format
import_data = json.loads(json_str)
self.import_from_dict(
import_data, key_prefix=key_prefix, start_datetime=start_datetime, interval=interval
)
def import_from_file(
self,
import_file_path: Path,
key_prefix: str = "",
start_datetime: Optional[DateTime] = None,
interval: Optional[Duration] = None,
) -> None:
"""Updates generic data by importing it from a file.
This method reads generic data from a JSON file, matches keys based on the
@@ -1068,10 +1438,16 @@ class DataImportProvider(DataProvider):
starting from the `start_datetime`. Each data value is associated with an hourly
interval.
If start_datetime and or interval is given in the JSON dict it will be used. Otherwise
the given parameters are used. If None is given start_datetime defaults to
'self.start_datetime' and interval defaults to 1 hour.
Args:
import_file_path (Path): The path to the JSON file containing the generic data.
key_prefix (str, optional): A prefix to filter relevant keys from the generic data.
Only keys starting with this prefix will be considered. Defaults to an empty string.
start_datetime (DateTime, optional): Start datetime of values.
interval (duration, optional): The fixed time interval. Defaults to 1 hour.
Raises:
FileNotFoundError: If the specified file does not exist.
@@ -1081,16 +1457,32 @@ class DataImportProvider(DataProvider):
Given a JSON file with the following content:
```json
{
"load0_mean": [20.5, 21.0, 22.1],
"load1_mean": [50, 55, 60]
"load_mean": [20.5, 21.0, 22.1],
"other_xyz: [10.5, 11.0, 12.1],
}
```
and `key_prefix = "load1"`, only the "load1_mean" key will be processed even though
and `key_prefix = "load"`, only the "load_mean" key will be processed even though
both keys are in the record.
"""
with import_file_path.open("r") as import_file:
import_str = import_file.read()
self.import_from_json(import_str, key_prefix)
self.import_from_json(
import_str, key_prefix=key_prefix, start_datetime=start_datetime, interval=interval
)
class DataImportProvider(DataImportMixin, DataProvider):
"""Abstract base class for data providers that import generic data.
This class is designed to handle generic data provided in the form of a key-value dictionary.
- **Keys**: Represent identifiers from the record keys of a specific data.
- **Values**: Are lists of data values starting at a specified `start_datetime`, where
each value corresponds to a subsequent time interval (e.g., hourly).
Subclasses must implement the logic for managing generic data based on the imported records.
"""
pass
class DataContainer(SingletonMixin, DataBase, MutableMapping):
@@ -1129,6 +1521,24 @@ class DataContainer(SingletonMixin, DataBase, MutableMapping):
enab.append(provider)
return enab
@property
def record_keys(self) -> list[str]:
"""Returns the keys of all fields in the data records of all enabled providers."""
key_set = set(
chain.from_iterable(provider.record_keys for provider in self.enabled_providers)
)
return list(key_set)
@property
def record_keys_writable(self) -> list[str]:
"""Returns the keys of all fields in the data records that are writable of all enabled providers."""
key_set = set(
chain.from_iterable(
provider.record_keys_writable for provider in self.enabled_providers
)
)
return list(key_set)
def __getitem__(self, key: str) -> pd.Series:
"""Retrieve a Pandas Series for a specified key from the data in each DataProvider.
@@ -1206,9 +1616,7 @@ class DataContainer(SingletonMixin, DataBase, MutableMapping):
Returns:
Iterator[str]: An iterator over the unique keys from all providers.
"""
return iter(
set(chain.from_iterable(provider.record_keys for provider in self.enabled_providers))
)
return iter(self.record_keys)
def __len__(self) -> int:
"""Return the number of keys in the container.
@@ -1216,9 +1624,7 @@ class DataContainer(SingletonMixin, DataBase, MutableMapping):
Returns:
int: The total number of keys in this container.
"""
return len(
list(chain.from_iterable(provider.record_keys for provider in self.enabled_providers))
)
return len(self.record_keys)
def __repr__(self) -> str:
"""Provide a string representation of the DataContainer instance.
@@ -1242,6 +1648,48 @@ class DataContainer(SingletonMixin, DataBase, MutableMapping):
for provider in self.enabled_providers:
provider.update_data(force_enable=force_enable, force_update=force_update)
def key_to_series(
self,
key: str,
start_datetime: Optional[DateTime] = None,
end_datetime: Optional[DateTime] = None,
dropna: Optional[bool] = None,
) -> pd.Series:
"""Extract a series indexed by the date_time field from data records within an optional date range.
Iterates through providers to find and return the first available series for the specified key.
Args:
key (str): The field name in the DataRecord from which to extract values.
start_datetime (datetime, optional): The start date for filtering the records (inclusive).
end_datetime (datetime, optional): The end date for filtering the records (exclusive).
dropna: (bool, optional): Whether to drop NAN/ None values before processing. Defaults to True.
Returns:
pd.Series: A Pandas Series with the index as the date_time of each record
and the values extracted from the specified key.
Raises:
KeyError: If the specified key is not found in any of the DataRecords.
"""
series = None
for provider in self.enabled_providers:
try:
series = provider.key_to_series(
key,
start_datetime=start_datetime,
end_datetime=end_datetime,
dropna=dropna,
)
break
except KeyError:
continue
if series is None:
raise KeyError(f"No data found for key '{key}'.")
return series
def key_to_array(
self,
key: str,