EOS/src/akkudoktoreos/utils/cacheutil.py

743 lines
31 KiB
Python
Raw Normal View History

"""Class for in-memory managing of cache files.
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing
temporary file objects, allowing the creation, retrieval, and management of cache files.
Classes:
--------
- CacheFileStore: A thread-safe, singleton class for in-memory managing of file-like cache objects.
- CacheFileStoreMeta: Metaclass for enforcing the singleton behavior in `CacheFileStore`.
Example usage:
--------------
# CacheFileStore usage
>>> cache_store = CacheFileStore()
>>> cache_store.create('example_key')
>>> cache_file = cache_store.get('example_key')
>>> cache_file.write('Some data')
>>> cache_file.seek(0)
>>> print(cache_file.read()) # Output: 'Some data'
Notes:
------
- Cache files are automatically associated with the current date unless specified.
"""
from __future__ import annotations
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
import hashlib
import inspect
import os
import pickle
import tempfile
import threading
from typing import (
IO,
Any,
Callable,
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
Dict,
Generic,
List,
Literal,
Optional,
ParamSpec,
TypeVar,
)
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
from pendulum import DateTime, Duration
from pydantic import BaseModel, ConfigDict, Field
from akkudoktoreos.utils.datetimeutil import compare_datetimes, to_datetime, to_duration
from akkudoktoreos.utils.logutil import get_logger
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
logger = get_logger(__name__)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
T = TypeVar("T")
Param = ParamSpec("Param")
RetType = TypeVar("RetType")
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
class CacheFileRecord(BaseModel):
# Enable custom serialization globally in config
model_config = ConfigDict(
arbitrary_types_allowed=True,
use_enum_values=True,
validate_assignment=True,
)
cache_file: Any = Field(..., description="File descriptor of the cache file.")
until_datetime: DateTime = Field(..., description="Datetime until the cache file is valid.")
ttl_duration: Optional[Duration] = Field(
default=None, description="Duration the cache file is valid."
)
class CacheFileStoreMeta(type, Generic[T]):
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
"""A thread-safe implementation of CacheFileStore."""
_instances: dict[CacheFileStoreMeta[T], T] = {}
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
_lock: threading.Lock = threading.Lock()
"""Lock object to synchronize threads on first access to CacheFileStore."""
def __call__(cls) -> T:
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
"""Return CacheFileStore instance."""
with cls._lock:
if cls not in cls._instances:
instance = super().__call__()
cls._instances[cls] = instance
return cls._instances[cls]
class CacheFileStore(metaclass=CacheFileStoreMeta):
"""A key-value store that manages file-like tempfile objects to be used as cache files.
Cache files are associated with a date. If no date is specified, the cache files are
associated with the current date by default. The class provides methods to create
new cache files, retrieve existing ones, delete specific files, and clear all cache
entries.
CacheFileStore is a thread-safe singleton. Only one store instance will ever be created.
Attributes:
store (dict): A dictionary that holds the in-memory cache file objects
with their associated keys and dates.
Example usage:
>>> cache_store = CacheFileStore()
>>> cache_store.create('example_file')
>>> cache_file = cache_store.get('example_file')
>>> cache_file.write('Some data')
>>> cache_file.seek(0)
>>> print(cache_file.read()) # Output: 'Some data'
"""
def __init__(self) -> None:
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
"""Initializes the CacheFileStore instance.
This constructor sets up an empty key-value store (a dictionary) where each key
corresponds to a cache file that is associated with a given key and an optional date.
"""
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
self._store: Dict[str, CacheFileRecord] = {}
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
self._store_lock = threading.Lock()
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
def _until_datetime_by_options(
self,
until_date: Optional[Any] = None,
until_datetime: Optional[Any] = None,
with_ttl: Optional[Any] = None,
) -> tuple[DateTime, Optional[Duration]]:
"""Get until_datetime and ttl_duration from the given options."""
ttl_duration = None
if until_datetime:
until_datetime = to_datetime(until_datetime)
elif with_ttl:
ttl_duration = to_duration(with_ttl)
until_datetime = to_datetime() + ttl_duration
elif until_date:
until_datetime = to_datetime(until_date).end_of("day")
else:
# end of today
until_datetime = to_datetime().end_of("day")
return (until_datetime, ttl_duration)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
def _generate_cache_file_key(
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
self,
key: str,
until_date: Optional[Any] = None,
until_datetime: Optional[Any] = None,
with_ttl: Optional[Any] = None,
) -> tuple[str, DateTime, Optional[Duration]]:
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
"""Generates a unique cache file key based on the key and date.
The cache file key is a combination of the input key and the date (if provided),
hashed using SHA-256 to ensure uniqueness.
Args:
key (str): The key that identifies the cache file.
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
until_datetime (Optional[DateTime]): The datetime
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
until the cache file is valid. The default is the current date at maximum time
(23:59:59).
Returns:
A tuple of:
str: A hashed string that serves as the unique identifier for the cache file.
datetime: The datetime until the the cache file is valid.
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
Optional[ttl_duration]: Duration for ttl control.
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
"""
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
until_datetime_dt, ttl_duration = self._until_datetime_by_options(
until_date, until_datetime, with_ttl
)
if ttl_duration:
# We need a special key for with_ttl, only encoding the with_ttl
key_datetime = ttl_duration.in_words()
else:
key_datetime = to_datetime(until_datetime_dt, as_string="UTC")
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
cache_key = hashlib.sha256(f"{key}{key_datetime}".encode("utf-8")).hexdigest()
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
return (f"{cache_key}", until_datetime_dt, ttl_duration)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
def _get_file_path(self, file_obj: IO[bytes]) -> Optional[str]:
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
"""Retrieve the file path from a file-like object.
Args:
file_obj: A file-like object (e.g., an instance of
NamedTemporaryFile, BytesIO, StringIO) from which to retrieve the
file path.
Returns:
str or None: The file path if available, or None if the file-like
object does not provide a file path.
"""
file_path = None
if hasattr(file_obj, "name"):
file_path = file_obj.name # Get the file path from the cache file object
return file_path
def _is_valid_cache_item(
self,
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
cache_item: CacheFileRecord,
until_datetime: Optional[DateTime] = None,
at_datetime: Optional[DateTime] = None,
before_datetime: Optional[DateTime] = None,
) -> bool:
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
if (
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
(until_datetime and until_datetime == cache_item.until_datetime)
or (at_datetime and at_datetime <= cache_item.until_datetime)
or (before_datetime and cache_item.until_datetime < before_datetime)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
):
return True
return False
def _search(
self,
key: str,
until_datetime: Optional[Any] = None,
at_datetime: Optional[Any] = None,
before_datetime: Optional[Any] = None,
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
ttl_duration: Optional[Any] = None,
) -> tuple[str, Optional[CacheFileRecord]]:
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
"""Searches for a cached item that matches the key and falls within the datetime range.
This method looks for a cache item with a key that matches the given `key`, and whose associated
datetime (`cache_file_datetime`) falls on or after the `at_datetime`. If both conditions are met,
it returns the cache item. Otherwise, it returns `None`.
Args:
key (str): The key to identify the cache item.
until_date (Optional[Any]): The date
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
until the cache file is valid. Time of day is set to maximum time (23:59:59).
at_datetime (Optional[Any]): The datetime to compare with the cache item's datetime.
before_datetime (Optional[Any]): The datetime to compare the cache item's datetime to be before.
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
Returns:
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
tuple[str, Optional[CacheFileRecord]]: Returns the cache_file_key, cache file record if found, otherwise returns `None`.
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
"""
# Convert input to datetime if they are not None
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
if ttl_duration is not None:
# TTL duration - use current datetime
if until_datetime or at_datetime or before_datetime:
raise NotImplementedError(
f"Search with ttl_duration and datetime filter until:{until_datetime}, at:{at_datetime}, before:{before_datetime} is not implemented"
)
at_datetime = to_datetime()
else:
if until_datetime is not None:
until_datetime = to_datetime(until_datetime)
if at_datetime is not None:
at_datetime = to_datetime(at_datetime)
if before_datetime is not None:
before_datetime = to_datetime(before_datetime)
if until_datetime is None and at_datetime is None and before_datetime is None:
at_datetime = to_datetime().end_of("day")
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
for cache_file_key, cache_item in self._store.items():
# Check if the cache file datetime matches the given criteria
if self._is_valid_cache_item(
cache_item,
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
until_datetime=until_datetime,
at_datetime=at_datetime,
before_datetime=before_datetime,
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
):
# This cache file is within the given datetime range
# Generate a cache file key based on the given key and the cache file datetime
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
if cache_item.ttl_duration:
generated_key, _until_dt, _ttl_duration = self._generate_cache_file_key(
key, with_ttl=cache_item.ttl_duration
)
else:
generated_key, _until_dt, _ttl_duration = self._generate_cache_file_key(
key, until_datetime=cache_item.until_datetime
)
logger.debug(
f"Search: ttl:{ttl_duration}, until:{until_datetime}, at:{at_datetime}, before:{before_datetime} -> hit: {generated_key == cache_file_key}, item: {cache_item.cache_file.seek(0), cache_item.cache_file.read()}"
)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
if generated_key == cache_file_key:
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
# The key matches, return the cache item
return (cache_file_key, cache_item)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
# Return None if no matching cache item is found
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
return ("<not found>", None)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
def create(
self,
key: str,
until_date: Optional[Any] = None,
until_datetime: Optional[Any] = None,
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
with_ttl: Optional[Any] = None,
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
mode: str = "wb+",
delete: bool = False,
suffix: Optional[str] = None,
) -> IO[bytes]:
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
"""Creates a new file-like tempfile object associated with the given key.
If a cache file with the given key and valid timedate already exists, the existing file is
returned. Otherwise, a new tempfile object is created and stored in the key-value store.
Args:
key (str): The key to store the cache file under.
until_date (Optional[Any]): The date
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
until the cache file is valid. Time of day is set to maximum time (23:59:59).
until_datetime (Optional[Any]): The datetime
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
until the cache file is valid. Time of day is set to maximum time (23:59:59) if not
provided.
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
with_ttl (Optional[Any]): The time to live that the cache file is valid. Time starts now.
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
mode (str, optional): The mode in which the tempfile is opened
(e.g., 'w+', 'r+', 'wb+'). Defaults to 'wb+'.
delete (bool, optional): Whether to delete the file after it is closed.
Defaults to False (keeps the file).
suffix (str, optional): The suffix for the cache file (e.g., '.txt', '.log').
Defaults to None.
Returns:
file_obj: A file-like object representing the cache file.
Example:
>>> cache_file = cache_store.create('example_file', suffix='.txt')
>>> cache_file.write('Some cached data')
>>> cache_file.seek(0)
>>> print(cache_file.read()) # Output: 'Some cached data'
"""
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
cache_file_key, until_datetime_dt, ttl_duration = self._generate_cache_file_key(
key, until_datetime=until_datetime, until_date=until_date, with_ttl=with_ttl
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
)
with self._store_lock: # Synchronize access to _store
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
if (cache_item := self._store.get(cache_file_key)) is not None:
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
# File already available
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
cache_file_obj = cache_item.cache_file
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
else:
cache_file_obj = tempfile.NamedTemporaryFile(
mode=mode, delete=delete, suffix=suffix
)
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
self._store[cache_file_key] = CacheFileRecord(
cache_file=cache_file_obj,
until_datetime=until_datetime_dt,
ttl_duration=ttl_duration,
)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
cache_file_obj.seek(0)
return cache_file_obj
def set(
self,
key: str,
file_obj: IO[bytes],
until_date: Optional[Any] = None,
until_datetime: Optional[Any] = None,
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
with_ttl: Optional[Any] = None,
) -> None:
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
"""Stores a file-like object in the cache under the specified key and date.
This method allows you to manually set a file-like object into the cache with a specific key
and optional date.
Args:
key (str): The key to store the file object under.
file_obj: The file-like object.
until_date (Optional[Any]): The date
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
until the cache file is valid. Time of day is set to maximum time (23:59:59).
until_datetime (Optional[Any]): The datetime
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
until the cache file is valid. Time of day is set to maximum time (23:59:59) if not
provided.
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
with_ttl (Optional[Any]): The time to live that the cache file is valid. Time starts now.
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
Raises:
ValueError: If the key is already in store.
Example:
>>> cache_store.set('example_file', io.BytesIO(b'Some binary data'))
"""
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
cache_file_key, until_datetime_dt, ttl_duration = self._generate_cache_file_key(
key, until_datetime=until_datetime, until_date=until_date, with_ttl=with_ttl
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
)
with self._store_lock: # Synchronize access to _store
if cache_file_key in self._store:
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
if ttl_duration:
# Special with_ttl case
if compare_datetimes(
self._store[cache_file_key].until_datetime, to_datetime()
).lt:
# File is outdated - replace by new file
self.delete(key=cache_file_key)
else:
raise ValueError(f"Key already in store: `{key}`.")
else:
raise ValueError(f"Key already in store: `{key}`.")
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
self._store[cache_file_key] = CacheFileRecord(
cache_file=file_obj, until_datetime=until_datetime_dt, ttl_duration=ttl_duration
)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
def get(
self,
key: str,
until_date: Optional[Any] = None,
until_datetime: Optional[Any] = None,
at_datetime: Optional[Any] = None,
before_datetime: Optional[Any] = None,
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
ttl_duration: Optional[Any] = None,
) -> Optional[IO[bytes]]:
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
"""Retrieves the cache file associated with the given key and validity datetime.
If no cache file is found for the provided key and datetime, the method returns None.
The retrieved file is a file-like object that can be read from or written to.
Args:
key (str): The key to retrieve the cache file for.
until_date (Optional[Any]): The date
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
until the cache file is valid. Time of day is set to maximum time (23:59:59).
until_datetime (Optional[Any]): The datetime
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
until the cache file is valid. Time of day is set to maximum time (23:59:59) if not
provided.
at_datetime (Optional[Any]): The datetime the
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
cache file shall be valid at. Time of day is set to maximum time (23:59:59) if not
provided. Defaults to the current datetime if None is provided.
before_datetime (Optional[Any]): The datetime
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
to compare the cache files datetime to be before.
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
ttl_duration (Optional[Any]): The time to live to compare the cache files time to live
to be equal.
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
Returns:
file_obj: The file-like cache object, or None if no file is found.
Example:
>>> cache_file = cache_store.get('example_file')
>>> if cache_file:
>>> cache_file.seek(0)
>>> print(cache_file.read()) # Output: Cached data (if exists)
"""
if until_datetime or until_date:
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
until_datetime, _ttl_duration = self._until_datetime_by_options(
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
until_datetime=until_datetime, until_date=until_date
)
with self._store_lock: # Synchronize access to _store
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
_cache_file_key, search_item = self._search(
key,
until_datetime=until_datetime,
at_datetime=at_datetime,
before_datetime=before_datetime,
ttl_duration=ttl_duration,
)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
if search_item is None:
return None
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
return search_item.cache_file
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
def delete(
self,
key: str,
until_date: Optional[Any] = None,
until_datetime: Optional[Any] = None,
before_datetime: Optional[Any] = None,
) -> None:
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
"""Deletes the cache file associated with the given key and datetime.
This method removes the cache file from the store.
Args:
key (str): The key of the cache file to delete.
until_date (Optional[Any]): The date
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
until the cache file is valid. Time of day is set to maximum time (23:59:59).
until_datetime (Optional[Any]): The datetime
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
until the cache file is valid. Time of day is set to maximum time (23:59:59) if not
provided.
before_datetime (Optional[Any]): The datetime
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
the cache file shall become or be invalid at. Time of day is set to maximum time
(23:59:59) if not provided. Defaults to tommorow start of day.
"""
if until_datetime or until_date:
until_datetime = self._until_datetime_by_options(
until_datetime=until_datetime, until_date=until_date
)
elif before_datetime:
before_datetime = to_datetime(before_datetime)
else:
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
# Make before_datetime tommorow at start of day
before_datetime = to_datetime().add(days=1).start_of("day")
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
with self._store_lock: # Synchronize access to _store
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
cache_file_key, search_item = self._search(
key, until_datetime=until_datetime, before_datetime=before_datetime
)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
if search_item:
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
file_path = self._get_file_path(search_item.cache_file)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
if file_path is None:
logger.warning(
f"The cache file with key '{cache_file_key}' is an in memory "
f"file object. Will only delete store entry but not file."
)
self._store.pop(cache_file_key)
return
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
# Get the file path from the cache file object
file_path = search_item.cache_file.name
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
del self._store[cache_file_key]
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
if file_path and os.path.exists(file_path):
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
try:
os.remove(file_path)
logger.debug(f"Deleted cache file: {file_path}")
except OSError as e:
logger.error(f"Error deleting cache file {file_path}: {e}")
def clear(
self,
clear_all: bool = False,
before_datetime: Optional[Any] = None,
) -> None:
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
"""Deletes all cache files or those expiring before `before_datetime`.
Args:
clear_all (bool, optional): Delete all cache files. Default is False.
before_datetime (Optional[Any]): The
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
threshold date. Cache files that are only valid before this date will be deleted.
The default datetime is beginning of today.
Raises:
OSError: If there's an error during file deletion.
"""
delete_keys = [] # List of keys to delete, prevent deleting when traversing the store
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
# Some weired logic to prevent calling to_datetime on clear_all.
# Clear_all may be set on __del__. At this time some info for to_datetime will
# not be available anymore.
if not clear_all:
if before_datetime is None:
before_datetime = to_datetime().start_of("day")
else:
before_datetime = to_datetime(before_datetime)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
with self._store_lock: # Synchronize access to _store
for cache_file_key, cache_item in self._store.items():
# Some weired logic to prevent calling to_datetime on clear_all.
# Clear_all may be set on __del__. At this time some info for to_datetime will
# not be available anymore.
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
if clear_all:
clear_file = True
else:
clear_file = compare_datetimes(cache_item.until_datetime, before_datetime).lt
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
if clear_file:
# We have to clear this cache file
delete_keys.append(cache_file_key)
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
file_path = self._get_file_path(cache_item.cache_file)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
if file_path is None:
# In memory file like object
logger.warning(
f"The cache file with key '{cache_file_key}' is an in memory "
f"file object. Will only delete store entry but not file."
)
continue
if not os.path.exists(file_path):
# Already deleted
logger.warning(f"The cache file '{file_path}' was already deleted.")
continue
# Finally remove the file
try:
os.remove(file_path)
logger.debug(f"Deleted cache file: {file_path}")
except OSError as e:
logger.error(f"Error deleting cache file {file_path}: {e}")
for delete_key in delete_keys:
del self._store[delete_key]
def cache_in_file(
ignore_params: List[str] = [],
force_update: Optional[bool] = None,
until_date: Optional[Any] = None,
until_datetime: Optional[Any] = None,
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
with_ttl: Optional[Any] = None,
mode: Literal["w", "w+", "wb", "wb+", "r", "r+", "rb", "rb+"] = "wb+",
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
delete: bool = False,
suffix: Optional[str] = None,
) -> Callable[[Callable[Param, RetType]], Callable[Param, RetType]]:
"""Cache the output of a function into a temporary file.
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
This decorator caches the result of a function call in a temporary file. The cache is
identified by a key derived from the function's input arguments, excluding those specified
in `ignore_params`. This is useful for caching results of expensive computations while
avoiding redundant recalculations.
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
The cache file is created using `CacheFileStore` and stored with the generated key. If a valid
cache file exists, it is returned instead of recomputing the result. The cache expiration is
controlled by the `until_date`, `until_datetime`, `with_ttl`, or `force_update` arguments.
If these arguments are present in the function call, their values override those specified in
the decorator.
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
By default, cache files are pickled to save storage space unless a `suffix` is provided. The
`mode` parameter allows specifying file modes for reading and writing, and the `delete`
parameter controls whether the cache file is deleted after use.
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
Args:
ignore_params (List[str], optional):
List of parameter names to ignore when generating the cache key. Useful for excluding
non-deterministic or irrelevant inputs, such as timestamps or large constant objects.
force_update (bool, optional):
Forces the cache to update, bypassing any existing cached results. If not provided,
the function will check for a `force_update` argument in the decorated function call.
until_date (Optional[Any], optional):
Date until which the cache file is valid. If a date is provided, the time is set to
the end of the day (23:59:59). If not specified, the function call arguments are checked.
until_datetime (Optional[Any], optional):
Datetime until which the cache file is valid. Time of day is set to maximum time
(23:59:59) if not provided.
with_ttl (Union[timedelta, str, int, float, None], optional):
Time-to-live (TTL) for the cache file, starting from the time of caching. Can be
specified as a `timedelta`, a numeric value (in seconds), or a string.
mode (Literal["w", "w+", "wb", "wb+", "r", "r+", "rb", "rb+"], optional):
File mode for opening the cache file. Defaults to "wb+" (write-binary with updates).
delete (bool, optional):
If True, deletes the cache file after it is closed. Defaults to False.
suffix (Optional[str], optional):
A file suffix (e.g., ".txt" or ".json") for the cache file. Defaults to None. If not
provided, files are pickled by default.
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
Returns:
Callable[[Callable[Param, RetType]], Callable[Param, RetType]]:
A decorated function that caches its result in a temporary file.
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
Example:
>>> from datetime import date
>>> @cache_in_file(suffix='.txt')
>>> def expensive_computation(until_date=None):
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
>>> # Perform some expensive computation
>>> return 'Some large result'
>>>
>>> result = expensive_computation(until_date=date.today())
Notes:
- The cache key is based on the function arguments after excluding those in `ignore_params`.
- If conflicting expiration parameters are provided (`until_date`, `until_datetime`,
`with_ttl`), the one in the function call takes precedence.
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
"""
def decorator(func: Callable[Param, RetType]) -> Callable[Param, RetType]:
nonlocal \
ignore_params, \
force_update, \
until_date, \
until_datetime, \
with_ttl, \
mode, \
delete, \
suffix
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
func_source_code = inspect.getsource(func)
def wrapper(*args: Param.args, **kwargs: Param.kwargs) -> RetType:
nonlocal \
ignore_params, \
force_update, \
until_date, \
until_datetime, \
with_ttl, \
mode, \
delete, \
suffix
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
# Convert args to a dictionary based on the function's signature
args_names = func.__code__.co_varnames[: func.__code__.co_argcount]
args_dict = dict(zip(args_names, args))
# Search for caching parameters of function and remove
for param in ["force_update", "until_datetime", "with_ttl", "until_date"]:
if param in kwargs:
if param == "force_update":
force_update = kwargs[param] # type: ignore[assignment]
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
kwargs.pop("force_update")
if param == "until_datetime":
until_datetime = kwargs[param]
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
until_date = None
with_ttl = None
elif param == "with_ttl":
until_datetime = None
until_date = None
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
with_ttl = kwargs[param]
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
elif param == "until_date":
until_datetime = None
until_date = kwargs[param]
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
with_ttl = None
kwargs.pop("force_update", None)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
kwargs.pop("until_datetime", None)
kwargs.pop("until_date", None)
kwargs.pop("with_ttl", None)
break
# Remove ignored params
kwargs_clone = kwargs.copy()
for param in ignore_params:
args_dict.pop(param, None)
kwargs_clone.pop(param, None)
# Create key based on argument names, argument values, and function source code
key = str(args_dict) + str(kwargs_clone) + str(func_source_code)
result: Optional[RetType | bytes] = None
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
# Get cache file that is currently valid
Fix2 config and predictions revamp. (#281) measurement: - Add new measurement class to hold real world measurements. - Handles load meter readings, grid import and export meter readings. - Aggregates load meter readings aka. measurements to total load. - Can import measurements from files, pandas datetime series, pandas datetime dataframes, simple daetime arrays and programmatically. - Maybe expanded to other measurement values. - Should be used for load prediction adaptions by real world measurements. core/coreabc: - Add mixin class to access measurements core/pydantic: - Add pydantic models for pandas datetime series and dataframes. - Add pydantic models for simple datetime array core/dataabc: - Provide DataImport mixin class for generic import handling. Imports from JSON string and files. Imports from pandas datetime dataframes and simple datetime arrays. Signature of import method changed to allow import datetimes to be given programmatically and by data content. - Use pydantic models for datetime series, dataframes, arrays - Validate generic imports by pydantic models - Provide new attributes min_datetime and max_datetime for DataSequence. - Add parameter dropna to drop NAN/ None values when creating lists, pandas series or numpy array from DataSequence. config/config: - Add common settings for the measurement module. predictions/elecpriceakkudoktor: - Use mean values of last 7 days to fill prediction values not provided by akkudoktor.net (only provides 24 values). prediction/loadabc: - Extend the generic prediction keys by 'load_total_adjusted' for load predictions that adjust the predicted total load by measured load values. prediction/loadakkudoktor: - Extend the Akkudoktor load prediction by load adjustment using measured load values. prediction/load_aggregator: - Module removed. Load aggregation is now handled by the measurement module. prediction/load_corrector: - Module removed. Load correction (aka. adjustment of load prediction by measured load energy) is handled by the LoadAkkudoktor prediction and the generic 'load_mean_adjusted' prediction key. prediction/load_forecast: - Module removed. Functionality now completely handled by the LoadAkkudoktor prediction. utils/cacheutil: - Use pydantic. - Fix potential bug in ttl (time to live) duration handling. utils/datetimeutil: - Added missing handling of pendulum.DateTime and pendulum.Duration instances as input. Handled before as datetime.datetime and datetime.timedelta. utils/visualize: - Move main to generate_example_report() for better testing support. server/server: - Added new configuration option server_fastapi_startup_server_fasthtml to make startup of FastHTML server by FastAPI server conditional. server/fastapi_server: - Add APIs for measurements - Improve APIs to provide or take pandas datetime series and datetime dataframes controlled by pydantic model. - Improve APIs to provide or take simple datetime data arrays controlled by pydantic model. - Move fastAPI server API to v1 for new APIs. - Update pre v1 endpoints to use new prediction and measurement capabilities. - Only start FastHTML server if 'server_fastapi_startup_server_fasthtml' config option is set. tests: - Adapt import tests to changed import method signature - Adapt server test to use the v1 API - Extend the dataabc test to test for array generation from data with several data interval scenarios. - Extend the datetimeutil test to also test for correct handling of to_datetime() providing now(). - Adapt LoadAkkudoktor test for new adjustment calculation. - Adapt visualization test to use example report function instead of visualize.py run as process. - Removed test_load_aggregator. Functionality is now tested in test_measurement. - Added tests for measurement module docs: - Remove sphinxcontrib-openapi as it prevents build of documentation. "site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema for t in schema["anyOf"]: KeyError: 'anyOf'" Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2024-12-29 18:42:49 +01:00
cache_file = CacheFileStore().get(
key, until_date=until_date, until_datetime=until_datetime, ttl_duration=with_ttl
)
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
if not force_update and cache_file is not None:
# cache file is available
try:
logger.debug("Used cache file for function: " + func.__name__)
cache_file.seek(0)
if "b" in mode:
result = pickle.load(cache_file)
else:
result = cache_file.read()
except Exception as e:
logger.info(f"Read failed: {e}")
# Fail gracefully - force creation
force_update = True
if force_update or cache_file is None:
# Otherwise, call the function and save its result to the cache
logger.debug("Created cache file for function: " + func.__name__)
cache_file = CacheFileStore().create(
key,
mode=mode,
delete=delete,
suffix=suffix,
until_datetime=until_datetime,
until_date=until_date,
with_ttl=with_ttl,
)
result = func(*args, **kwargs)
try:
# Assure we have an empty file
cache_file.truncate(0)
if "b" in mode:
pickle.dump(result, cache_file)
else:
cache_file.write(result) # type: ignore[call-overload]
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
except Exception as e:
logger.info(f"Write failed: {e}")
CacheFileStore().delete(key)
return result # type: ignore[return-value]
Add test to PVForecast (#174) * Add documentation to class_pv_forecast.py. Added documentation. Beware mostly generated by ChatGPT. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add CacheFileStore, datetime and logger utilities. The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing temporary file objects, allowing the creation, retrieval, and management of cache files. The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection (`to_timezone). - Cache files are automatically valid for the the current date unless specified otherwise. This is to mimic the current behaviour used in several classes. - The logger supports rotating log files to prevent excessive log file size. - The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats. They provide the time conversion that is e.g. used in PVForecast. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Improve testability of PVForecast Improvements for testing of PVForecast - Use common utility functions to allow for general testing at one spot. - to_datetime - CacheFileStore - Use logging instead of print to easily capture in testing. - Add validation of the json schema for Akkudoktor PV forecast data. - Allow to create an empty PVForecast instance as base instance for testing. - Make process_data() complete for filling a PVForecast instance for testing. - Normalize forecast datetime to timezone of system given in loaded data. - Do not print report but provide report for test checks. - Get rid of cache file path using the CachFileStore to automate cache file usage. - Improved module documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * Add test for PVForecast and newly extracted utility modules. - Add test for PVForecast - Add test for CacheFileStore in the new cachefilestore module - Add test for to_datetime, to_timestamp, to_timezone in the new datetimeutil module - Add test for get_logger in the new logutil module Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> --------- Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com>
2024-11-10 23:49:10 +01:00
return wrapper
return decorator