Mirrors/EOS
1
0
Fork 0
mirror of https://github.com/Akkudoktor-EOS/EOS.git synced 2026-07-20 00:38:12 +00:00
Code Issues Actions 6 Packages Projects Releases Wiki Activity

fix: move data management to async (#1015)

Browse Source 
FAstAPI is an async framework. Data may be imported and exported, load and save, set and get
asynchronously. Prevent interleaving data operations to corrupt the data. In the previous design
sync and async data access was intermixed leading to data corruption.

The basic data classes DataSequence and DataContainer and the derived classes like Provider and
Measurement now are async. Data access is protected by several async locks.

To support the async design of the data classes the database interface became async.

The energy management is also adapted to the new async design. Optimization is still off-loaded
to another thread, but the prepration for the optimization and the post optimization actions now
follow the async design.

Adapter operations are now also protected by async locks.

Tests were adapted to the async design and new tests were created.

Besides this major fix several other improvements and fixes are included in this PR.

* fix: key_to_dict/list/array only regard data records with key value set.

  Before the exclusion of no value data records was only done if the dropna flag was set.

* fix: test for visual result pdf generation

  Due to updates in the library the generated charts text was a little bit different.
  Adapt the test to create the comaprison pdf in the test data durectory and
  update the reference pdf.

* chore: Remove MutableMapping from DataSequence and DataContainer.

  Mutable Mapping does not fit to the now async design.

* chore: Add NoDB database backend

  This backend implements the full database backend interface but performs
  no actual persistence. It is intended for configurations where database
  persistence is disabled (`provider=None`).

* chore: Improve measurement data import testing with real world scenarios.

  Added two new endpoints to support testing.

* chore: Add mermaid to supported documentation tools

* chore: Add documentation about async design

* chore: Add documentation about generic data handling

  Covers the basics of measurement and prediction time series data handling.

* chore: Add empty lines around markdown lists.

* chore: sync pre-commit config to updated package versions

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
This commit is contained in:
Bobby Noelte
2026-07-15 16:38:53 +02:00
committed by GitHub
parent 38011780c5
commit eb9e966de9
99 changed files with 11971 additions and 5981 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.env
View File

@@ -11,7 +11,7 @@ DOCKER_COMPOSE_DATA_DIR=${HOME}/.local/share/net.akkudoktor.eos
# -----------------------------------------------------------------------------
# Image / build
# -----------------------------------------------------------------------------
VERSION=0.3.0.dev2604141105859917
VERSION=0.3.0.dev2607150960221650
PYTHON_VERSION=3.13.9
# -----------------------------------------------------------------------------

16
.pre-commit-config.yaml
View File

@@ -15,13 +15,13 @@ repos:
# --- Import sorting ---
- repo: https://github.com/PyCQA/isort
rev: 7.0.0
rev: 8.0.1
hooks:
- id: isort
# --- Linting + Formatting via Ruff ---
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.7
rev: v0.15.21
hooks:
# Run the linter and fix simple isssues automatically
- id: ruff
@@ -31,20 +31,20 @@ repos:
# --- Static type checking ---
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.19.1
rev: v2.3.0
hooks:
- id: mypy
additional_dependencies:
- types-requests==2.33.0.20260408
- pandas-stubs==3.0.0.260204
- types-requests==2.33.0.20260712
- pandas-stubs==3.0.3.260530
- tokenize-rt==6.2.0
- types-docutils==0.22.3.20260322
- types-PyYaml==6.0.12.20260408
- types-docutils==0.22.3.20260518
- types-PyYaml==6.0.12.20260518
pass_filenames: false
# --- Markdown linter ---
- repo: https://github.com/jackdewinter/pymarkdown
rev: v0.9.36
rev: v0.9.39
hooks:
- id: pymarkdown
files: ^docs/

2
Makefile
View File

@@ -60,7 +60,7 @@ install: version-txt
# Target to rebuild the virtual environment.
update-env:
@echo "Rebuilding virtual environment to match pyproject.toml..."
uv rebuild
$(UV) sync --upgrade --extra dev
@echo "Environment rebuilt."
# Target to create a distribution.

2
config.yaml
View File

@@ -6,7 +6,7 @@
# the root directory (no add-on folder as usual).
name: "Akkudoktor-EOS"
version: "0.3.0.dev2604141105859917"
version: "0.3.0.dev2607150960221650"
slug: "eos"
description: "Akkudoktor-EOS add-on"
url: "https://github.com/Akkudoktor-EOS/EOS"

3
docs/_generated/configdatabase.md
View File

@@ -64,7 +64,8 @@ None indicates forever. Database namespaces may have diverging definitions. |
"batch_size": 100,
"providers": [
"LMDB",
"SQLite"
"SQLite",
"NoDB"
]
}
}

59
docs/_generated/openapi.md
View File

@@ -1,6 +1,6 @@
# Akkudoktor-EOS
**Version**: `v0.3.0.dev2604141105859917`
**Version**: `v0.3.0.dev2607150960221650`
<!-- pyml disable line-length -->
**Description**: This project provides a comprehensive solution for simulating and optimizing an energy system based on renewable energy sources. With a focus on photovoltaic (PV) systems, battery storage (batteries), load management (consumer requirements), heat pumps, electric vehicles, and consideration of electricity price data, this system enables forecasting and optimization of energy flow and costs over a specified period.
@@ -338,6 +338,31 @@ Returns:
---
## POST /v1/admin/database/save
<!-- pyml disable line-length -->
**Links**: [local](http://localhost:8503/docs#/default/fastapi_admin_database_save_post_v1_admin_database_save_post), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_admin_database_save_post_v1_admin_database_save_post)
<!-- pyml enable line-length -->
Fastapi Admin Database Save Post
<!-- pyml disable line-length -->
```python
"""
Save in memory data to database.
Returns:
data (dict): The database stats after saving the records.
"""
```
<!-- pyml enable line-length -->
**Responses**:
- **200**: Successful Response
---
## GET /v1/admin/database/stats
<!-- pyml disable line-length -->
@@ -882,6 +907,38 @@ Fastapi Measurement Keys Get
---
## DELETE /v1/measurement/range
<!-- pyml disable line-length -->
**Links**: [local](http://localhost:8503/docs#/default/fastapi_measurement_range_delete_v1_measurement_range_delete), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_measurement_range_delete_v1_measurement_range_delete)
<!-- pyml enable line-length -->
Fastapi Measurement Range Delete
<!-- pyml disable line-length -->
```python
"""
Delete measurement values for a key within a datetime range.
"""
```
<!-- pyml enable line-length -->
**Parameters**:
- `key` (query, required): Measurement key.
- `start_datetime` (query, optional): Start datetime.
- `end_datetime` (query, optional): End datetime.
**Responses**:
- **200**: Successful Response
- **422**: Validation Error
---
## GET /v1/measurement/series
<!-- pyml disable line-length -->

3
docs/akkudoktoreos/adapter/adapternodered.md
View File

@@ -38,6 +38,7 @@ Typical use cases with Node-RED:
#### 1. Enable and configure the Node-RED adapter
EOS must be configured with access to the Node-RED instance in Config->adapter.
* prerequisite is an already installed and running Node-RED instance
* adapter.nodered.host: 192.168.1.109 (example IP of your Node-RED instance)
* adapter.nodered.port: 1880 (default)
@@ -46,10 +47,12 @@ EOS must be configured with access to the Node-RED instance in Config->adapter.
#### 2. Run energy optimisation
Before the run, EOS receives:
* EOS receives measurement values via HTTP-IN "eos_data_acquisition" before optimisation.
(The HTTP-IN Node "eos_data_acquisition" is NOT yet functional)
After the run, EOS provides:
* The device instruction and solution entities for the current time slot via HTTP-IN "Control Dispatch".
### Configuration steps in NodeRED

4
docs/akkudoktoreos/database.md
View File

@@ -284,6 +284,7 @@ DatabaseTimestamp.from_datetime(dt: DateTime) -> "20241027T123456[Z]"
```
**Properties:**
- Always stored in UTC (timezone-aware required)
- Lexicographically sortable
- Bijective conversion to/from `pendulum.DateTime`
@@ -330,6 +331,7 @@ The system uses a progressive loading model to minimize memory footprint:
### Boundary Extension
When loading a range `[start, end)`, the system automatically extends boundaries to include:
- **First record before** `start` (for interpolation/context)
- **First record at or after** `end` (for closing boundary)
@@ -348,6 +350,7 @@ SELECT * FROM records WHERE namespace='measurement'
```
**Default Namespace:**
- Can be set during `open(namespace="default")`
- Operations with `namespace=None` use the default
- Each record class typically defines its own namespace via `db_namespace()`
@@ -537,6 +540,7 @@ db_vacuum(keep_timestamp=cutoff) # Keep from cutoff onward
```
**Strategy:**
- Computes cutoff relative to `max_timestamp - keep_hours`
- Deletes all records before cutoff
- Immediately persists changes via `db_save_records()`

1
docs/akkudoktoreos/optimpost.md
View File

@@ -223,6 +223,7 @@ penalty = ac_wh_charged × (break_even_price − best_uncovered_price) × factor
```
where:
- `break_even_price = charge_price / η_round_trip`
- `best_uncovered_price` = highest future price not already covered by free PV battery energy
- `factor` = `optimization.genetic.penalties.ac_charge_break_even` (default `1.0`)

4
docs/conf.py
View File

@@ -33,6 +33,7 @@ extensions = [
"sphinx_rtd_theme",
"myst_parser",
"sphinx_tabs.tabs",
"sphinxcontrib.mermaid",
]
templates_path = ["_templates"]
@@ -141,3 +142,6 @@ napoleon_use_rtype = True
napoleon_preprocess_types = False
napoleon_type_aliases = None
napoleon_attr_annotations = True
# -- Options for mermaid -------------------------------------------------
mermaid_output_format = "html" # no mmdc to be installed

184
docs/develop/async.md Normal file
View File

@@ -0,0 +1,184 @@
# Asynchronous Design of Akkudoktor‑EOS
The Akkudoktor‑EOS server is built on **FastAPI** and is transitioning to a fully asynchronous
design to improve scalability, responsiveness, and resource utilisation, especially for
long‑running or I/O‑bound operations.
The server manages a variety of background tasks through a central **Retention Manager**, which
schedules and supervises all periodic and asynchronous work.
## Core Asynchronous Components
- **FastAPI REST Interface**
All HTTP endpoints are defined as `async def` handlers (or synchronous where no I/O waiting is
required), allowing the server to handle many concurrent connections without blocking the event
loop.
- **Retention Manager**
A dedicated component responsible for orchestrating all background tasks. It runs an asynchronous
tick loop that executes registered functions at configurable intervals. The manager also provides
graceful shutdown handling, waiting for in‑flight jobs to finish.
<!-- pyml disable line-length -->
- **Managed Asynchronous Tasks**
The following tasks are registered with the Retention Manager and run periodically:
| Task Name | Function | Interval Configuration | Description |
|-----------|----------|------------------------|-------------|
| `supervise_eosdash` | `supervise_eosdash` | `server/eosdash_supervise_interval_sec` | Monitors and restarts the EOSdash UI process if needed. |
| `autosave_config` | `autosave_config` | `general/config_save_interval_sec` | Saves the current configuration to disk automatically. |
| `cache_clear` | `cache_clear` | `cache/cleanup_interval` | Removes expired entries from the cache. |
| `save_eos_database` | `save_eos_database` | `database/autosave_interval_sec` | Persists in‑memory measurement and prediction data to the database. |
| `compact_eos_database` | `compact_eos_database` | `database/compaction_interval_sec` | Compacts and vacuums the database to reclaim space and improve performance. |
| `manage_energy` | `ems_manage_energy` | `ems/interval` | Core energy management loop: triggers predictions, optimisation, and device control. |
The `manage_energy` task is the central orchestrator that itself calls asynchronous prediction
updates, adapter scheduling and optimisation runs. It uses the `EnergyManagementSystem`
(`get_ems()`) which internally manages concurrency to ensure only one energy management run
happens at a time.
<!-- pyml enable line-length -->
- **On‑Demand Asynchronous Endpoints**
Several REST endpoints are asynchronous and delegate heavy work to the `EnergyManagementSystem`.
Examples include:
- `POST /v1/prediction/update` – updates all prediction providers asynchronously.
- `POST /optimize` – runs a genetic optimisation (deprecated, but still async).
- `POST /v1/admin/server/restart` – spawns a new process and schedules a shutdown task.
## Asynchronous Workflow
The diagram below shows how periodic tasks are registered and executed by the Retention Manager,
and how a client request can trigger an asynchronous update.
```mermaid
sequenceDiagram
participant Client
participant FastAPI as FastAPI (Async)
participant Retention as Retention Manager (Async)
participant EMS as EnergyManagementSystem (Async)
participant Adapter as Adapter (Async)
participant Prediction as Prediction (Async)
participant Measurement as Measurement (Async)
participant DatabaseRecords as DatabaseRecords (Async)
participant Database as Database (Sync)
Note over Retention: On startup (lifespan)
FastAPI->>Retention: register tasks with intervals
Retention-->>FastAPI: tasks registered
Retention->>Retention: start tick loop
loop every EMS interval
Retention->>EMS: run(mode=PREDICTION+OPTIMIZATION)
EMS->>Adapter: update_data, DATA_AQUISITION
Adapter->>Measurement: update_value
Measurement-->>Adapter: done
Adapter-->>EMS: done
EMS->>Prediction: update_data, FORECAST_RETRIEVAL
Prediction-->>EMS: done
EMS->>Adapter: update_data, CONTROL_DISPATCH
Adapter-->>EMS: done
EMS-->>Retention: done
end
loop every DatabaseRecords autosave interval
Retention->>Prediction: save
Prediction->>DatabaseRecords: db_save_records
DatabaseRecords-->>Prediction: done
Prediction-->>Retention: done
Retention->>Measurement: save
Measurement->>DatabaseRecords: db_save_records
DatabaseRecords-->>Measurement: done
Measurement-->>Retention: done
end
loop every DatabaseRecords compaction interval
Retention->>Prediction: db_compact
Prediction->>DatabaseRecords: db_delete_records
DatabaseRecords-->>Prediction: done
Prediction->>DatabaseRecords: db_save_records
DatabaseRecords-->>Prediction: done
Prediction-->>Retention: done
Retention->>Measurement: db_compact
Measurement->>DatabaseRecords: db_delete_records
DatabaseRecords-->>Measurement: done
Measurement->>DatabaseRecords: db_save_records
DatabaseRecords-->>Measurement: done
Measurement-->>Retention: done
end
Client->>FastAPI: POST /v1/prediction/update
FastAPI->>EMS: await run(mode=PREDICTION)
EMS-->>FastAPI: predictions updated
FastAPI-->>Client: 200 OK
```
For server shutdown or restart, the Retention Manager’s task is cancelled, and the server waits
for in‑flight jobs to finish (shutdown timeout = 10 seconds). The state is saved via
`save_eos_state()`.
## Asynchronous vs. Synchronous
- **REST endpoints** that perform I/O or heavy computation are `async def`.
- **Retention Manager** uses `asyncio.create_task()` to run the tick loop and individual task
executions.
- **Database** that synchronizes database access is `async def`, but the database backends
**DataBaseBackendABC** are synchronous.
- **DatabaseRecordProtocolMixin** provides asynchronous access to in memory data records and
database storage.
- **Prediction** and **Measurement** provide asynchronous access to the undelying database using
the **DatabaseRecordProtocolMixin**.
- **Energy management runs** are serialised using an internal lock (via `EMS.run()`) to avoid
overlapping optimisation cycles.
- **Process management** for shutdown/restart uses `asyncio.create_task(server_shutdown_task())`
to gracefully terminate after a delay.
## Benefits of Full Asynchrony
- **Higher throughput** – FastAPI’s event loop can handle thousands of idle keep‑alive connections.
- **Lower latency** – Long‑running tasks (database compaction, prediction updates) do not block HTTP
responses.
- **Easier maintenance** – Uniform async patterns replace mixed sync/async code.
- **Better resource usage** – The Retention Manager can throttle, skip, or prioritise tasks based on
configuration and system load.
- **Graceful shutdown** – All background tasks are cancelled cooperatively, and state is saved
before exit.
## Additional Asynchronous Patterns
Beyond the Retention Manager, the server uses:
- **Asynchronous shutdown/restart** – When a restart is requested, a new process is spawned, and the
current process schedules a delayed termination (`server_shutdown_task`). This ensures zero
downtime if the new process starts before the old one exits.
- **Concurrent request handling** – Multiple clients can call prediction update endpoints
simultaneously; the `EMS.run()` method serialises them internally, preventing race conditions.
- **Non‑blocking logging** – Log entries are written asynchronously (via loguru’s async sinks when
configured).
The combination of FastAPI, the Retention Manager, and asynchronous I/O enables Akkudoktor‑EOS to
run efficiently on resource‑constrained devices (e.g., Raspberry Pi) while maintaining responsive
REST APIs and reliable background data maintenance.
## Asynchronous Detailed Design
### Database
The database backends are synchronous. The selected backend is wrapped in the asynchronous
thread-safe database singleton defined by the Database class.
### DatabaseRecordProtocol, DataRecordProtocol
The DatabaseRecordProtocol completely manages in memory records and database storage. It acesses
the database backend by the database singleton and is therefor asynchronous. The
DatabaseRecordProtocol has a minimum expectation for data records defined by the DataRecordProtocol.
Data records are expected to be synchronous.
### Data Records
Data records are synchronous.
### Data Sequence, Data Provide, Data Container
Data sequences, the derived data provider, and the data provider aggregation data containers are
asynchronous. Data sequences' data can be backed by the database using the
DatabaseRecordProtocol to access the data. That is the reason why all the classes are
asynchronous.

568
docs/develop/dataabc.md Normal file
View File

@@ -0,0 +1,568 @@
# `dataabc` — Generic Data Handling
## Overview
The `dataabc` module provides the foundational abstractions for managing time-series data
in EOS. It defines a layered class hierarchy that covers individual data records, ordered
sequences of records, singleton data providers, and multi-provider containers.
All classes in this module are designed for use in predictive modelling workflows and share
three cross-cutting concerns:
- **Configuration access** via `ConfigMixin`, exposing the global EOS configuration as
`self.config`.
- **Database persistence** via `DatabaseRecordProtocolMixin`, providing optional storage
in a time-series database alongside in-memory records.
- **Async safety** via a three-level locking scheme described in detail in the
[Concurrency and Locking](#concurrency-and-locking) section.
## Class Hierarchy
```text
PydanticBaseModel
└── DataABC (ConfigMixin, StartMixin)
├── DataRecord (MutableMapping)
└── DataSequence (DatabaseRecordProtocolMixin)
└── DataProvider (SingletonMixin)
└── DataImportProvider (DataImportMixin)
DataContainer (SingletonMixin, MutableMapping)
DataImportMixin (StartMixin)
```
## Classes
### `DataABC`
Base class for all data-handling objects. Inherits from `ConfigMixin` and `StartMixin`,
making the global EOS configuration available as `self.config` on every derived instance.
Not intended to be instantiated directly.
### `DataRecord`
A single measurement or forecast point at a specific datetime, implemented as a
`MutableMapping` so that field values can be accessed and mutated both as dictionary
entries (`record["field"]`) and as attributes (`record.field`).
#### Key concepts
**Static fields** are declared as Pydantic model fields in the class body. They are always
present and validated on assignment.
**Configured fields** are dynamic: their names are returned by the classmethod
`configured_data_keys()`, which derived classes override to pull key names from the EOS
configuration. These keys are stored in the internal `configured_data` dict but appear
transparent to callers — they show up in `dir()`, `iter()`, and attribute access just like
static fields.
```python
class MeasurementDataRecord(DataRecord):
@classmethod
def configured_data_keys(cls) -> Optional[list[str]]:
return cls.config.measurement.keys
```
#### Important methods
| Method | Description |
|---|---|
| `record_keys()` | All field names, including configured keys. |
| `record_keys_writable()` | Subset of `record_keys()` that can be written. |
| `key_from_description(desc)` | Fuzzy-matches a description string to a field name. |
| `keys_from_descriptions(descs)` | Batch version of `key_from_description`. |
---
### `DataSequence`
An ordered, mutable collection of `DataRecord` instances with time-series behaviour.
Records are always kept sorted in ascending `date_time` order. `DataSequence` is also the
level at which **async safety is enforced** — see
[Concurrency and Locking](#concurrency-and-locking).
Derived classes must redeclare the `records` field with the concrete record type:
```python
class Measurement(DataSequence):
records: list[MeasurementDataRecord] = Field(default_factory=list)
```
#### Public async interface of `DataSequence`
All methods that mutate sequence state are `async`. Callers in an async context (e.g.
FastAPI endpoint handlers) must `await` them.
| Method | Description |
|---|---|
| `await insert_by_datetime(record)` | Insert or merge a record by its datetime. |
| `await update_value(date, key, value)` | Insert or update a single field value at a datetime. |
| `await key_from_lists(key, dates, values)` | Populate a field from parallel date/value lists. |
| `await key_from_series(key, series)` | Populate a field from a `pd.Series`. |
| `await save()` | Persist all records to the configured storage backend. |
| `await load()` | Load records from the configured storage backend. |
| `await import_from_dict(data)` | Import records from a key-value dictionary. |
| `await import_from_dataframe(df)` | Import records from a `pd.DataFrame`. |
| `await import_from_json(json_str)` | Import records from a JSON string. |
| `await import_from_file(path)` | Import records from a JSON file. |
#### Internal sync interface of `DataSequence`
Each public async method has a private sync counterpart prefixed with `_`. These are
intended **only** for callers that already hold the appropriate locks, or that are running
in a purely sequential context (startup, `_load()` internals, tests). See
[Choosing the right call site](#choosing-the-right-call-site).
| Internal method | Corresponding public method |
|---|---|
| `_insert_by_datetime(record)` | `.insert_by_datetime()` |
| `_update_value(date, ...)` | `.update_value()` |
| `_key_from_lists(key, dates, values)` | `.key_from_lists()` |
| `_key_from_series(key, series)` | `.key_from_series()` |
| `_save()` | `.save()` |
| `_load()` | `.load()` |
| `_import_from_dict(...)` | `.import_from_dict()` |
| `_import_from_dataframe(...)` | `.import_from_dataframe()` |
| `_import_from_json(...)` | `.import_from_json()` |
| `_import_from_file(...)` | `.import_from_file()` |
#### Read-only methods (always sync) of `DataSequence`
These methods do not modify sequence state and require no locking:
| Method | Description |
|---|---|
| `get_by_datetime(dt)` | Exact or nearest record lookup. |
| `get_nearest_by_datetime(dt)` | Nearest record within an optional time window. |
| `key_to_dict(key, ...)` | Extract a `{datetime: value}` dict for a key. |
| `key_to_lists(key, ...)` | Extract parallel date and value lists. |
| `key_to_series(key, ...)` | Extract a `pd.Series` indexed by datetime. |
| `key_to_array(key, ...)` | Extract a resampled `np.ndarray` at a fixed interval. |
| `key_to_value(key, dt)` | Scalar lookup nearest to a datetime. |
| `to_dataframe(...)` | Convert all records to a `pd.DataFrame`. |
| `delete_by_datetime(...)` | Delete records within a datetime range. |
| `key_delete_by_datetime(key, ...)` | Set a field to `None` across a datetime range. |
#### Computed properties of `DataSequence`
| Property | Description |
|---|---|
| `min_datetime` | Earliest datetime in the sequence. |
| `max_datetime` | Latest datetime in the sequence. |
| `record_keys` | All field names for this sequence's record type. |
| `record_keys_writable` | Writable subset of `record_keys`. |
### `DataProvider`
Abstract singleton base class for objects that own and update a `DataSequence`. Each
concrete provider represents one data source (e.g. weather forecast, load measurement,
grid price). `DataProvider` is a specialisation of `DataSequence` and inherits its full
async interface and locking infrastructure — `_record_lock` and `_sequence_lock` — without
adding any new locks of its own.
Derived classes must implement:
| Abstract method | Description |
|---|---|
| `provider_id() -> str` | Unique string identifier for this provider. |
| `enabled() -> bool` | Whether this provider is active per configuration. |
| `_update_data(force_update)` | Custom data fetch/update logic. |
#### `_update_data` contract of `DataProvider`
`_update_data` is always called while the provider's own `_sequence_lock` and
`_record_lock` are both held by `update_data`. Implementations must therefore observe
the following constraints to avoid deadlock:
- Use only the internal sync methods (`_insert_by_datetime`, `_update_value`,
`_key_from_lists`, `_key_from_series`). Never call their public `async` counterparts,
which would attempt to re-acquire `_record_lock`.
- Do not call `save()`, `load()`, `_save()`, or `_load()`. Both locks are already held;
attempting to re-acquire either will deadlock.
- Network or I/O calls are permitted but should be kept short. Offload long-running I/O
to a thread via `asyncio.to_thread` in the caller before entering the lock scope.
```python
# Correct — both _sequence_lock and _record_lock are already held by the caller
def _update_data(self, force_update=False):
for dt, value in self._fetch_from_api():
self._update_value(dt, "temperature_c", value)
# Wrong — would deadlock: _record_lock is not reentrant
def _update_data(self, force_update=False):
for dt, value in self._fetch_from_api():
await self.update_value(dt, "temperature_c", value)
```
#### Public async interface of `DataProvider`
<!-- pyml disable line-length -->
| Method | Description |
|---|---|
| `await update_data(force_enable, force_update)` | Call `_update_data` if enabled or forced, holding both `_sequence_lock` and `_record_lock` for the duration. |
<!-- pyml enable line-length -->
### `DataImportMixin`
Mixin that adds bulk import capability to any class that also provides `update_value` and
`record_keys_writable`. Provides `import_from_dict`, `import_from_dataframe`,
`import_from_json`, and `import_from_file`.
The mixin expects values to be lists aligned to a fixed time interval starting from a
`start_datetime`. Two special dictionary keys are handled automatically:
- `start_datetime` — overrides the start of the import window.
- `interval` — overrides the fixed time step between values.
### `DataImportProvider`
Convenience base class combining `DataImportMixin` and `DataProvider`. Derive from this
when a provider's data arrives via JSON, file, or dict import rather than a live API.
### `DataContainer`
A singleton `MutableMapping` that aggregates multiple `DataProvider` instances and
presents their combined data through a single interface. Providers are tried in order;
the first one that contains the requested key wins.
`DataContainer` carries its own pair of locks (`_record_lock` and `_container_lock`)
that are independent of the locks on each provider. See
[Concurrency and Locking](#concurrency-and-locking) for the full acquisition matrix.
#### Public async interface of `DataContainer`
| Method | Description |
|---|---|
| `await __setitem__(key, series)` | Write a `pd.Series` into the appropriate provider. |
| `await __delitem__(key)` | Clear a field across all providers. |
| `await update_data(force_enable, force_update)` | Update all providers. |
| `await save()` | Save all providers to persistent storage. |
| `await load()` | Load all providers from persistent storage. |
| `await db_vacuum()` | Remove old records from all provider databases. |
| `await db_compact()` | Apply tiered compaction to all provider databases. |
| `await keys_to_dataframe(keys, ...)` | Consistent cross-provider snapshot as a `pd.DataFrame`. |
#### Read-only methods (always sync) of `DataContainer`
| Method | Description |
|---|---|
| `__getitem__(key)` | Return a `pd.Series` for a key from the first matching provider. |
| `__iter__()` | Iterate over all unique keys across enabled providers. |
| `__len__()` | Total number of unique keys. |
| `key_to_series(key, ...)` | Extract a series from the first matching provider. |
| `key_to_array(key, ...)` | Extract a resampled array from the first matching provider. |
| `provider_by_id(provider_id)` | Look up a provider by its string identifier. |
| `enabled_providers` | List of currently active providers. |
| `record_keys` | Union of all record keys across enabled providers. |
| `record_keys_writable` | Union of all writable record keys across enabled providers. |
## Concurrency and Locking
### Problem
EOS runs under FastAPI with an async event loop. Multiple HTTP requests are handled
concurrently as coroutines on a single thread. Because coroutines yield at `await` points,
two coroutines can interleave between a read and a subsequent write, producing a
**check-then-act race condition**:
```text
Coroutine A: db_get_record(ts) → None # record does not exist yet
Coroutine B: db_get_record(ts) → None # same — A has not inserted yet
Coroutine A: db_insert_record(new_rec) # inserts successfully
Coroutine B: db_insert_record(new_rec) # raises: duplicate timestamp
```
This is the root cause of the `ValueError: Duplicate timestamp` errors seen in production.
### Solution: three-level locking
The solution uses three distinct `asyncio.Lock` objects, each protecting a different scope
of state. Each name answers the question "what is being protected":
<!-- pyml disable line-length -->
| Lock | Defined on | Attribute | Protects |
|---|---|---|---|
| Record lock | `DataSequence` | `_record_lock` | A single check-then-act on one record. |
| Sequence lock | `DataSequence` | `_sequence_lock` | The full sequence state during operations that touch many records at once. |
| Container lock | `DataContainer` | `_container_lock` | Cross-provider consistency during container-level bulk operations. |
<!-- pyml enable line-length -->
`DataProvider` inherits `_record_lock` and `_sequence_lock` from `DataSequence` without
adding any new locks of its own. It is a specialisation of `DataSequence`, not a new
locking scope.
The fixed acquisition order across all levels is:
```text
_container_lock → _record_lock → provider _sequence_lock → provider _record_lock
```
No code path may acquire a finer-grained lock and then wait for a coarser one at the
same level. This invariant prevents deadlock.
### Lock creation
All locks are created lazily on first access via `_get_or_create_lock()`, which bypasses
Pydantic's `__setattr__` using `object.__setattr__` directly. This is necessary because
Pydantic v2 rejects attributes that are not declared model fields, and `asyncio.Lock`
cannot be safely created outside a running event loop, making `__init__`-time creation
unsafe. `cached_property` is not used for the same reason — Pydantic v2's tight
`__dict__` control makes it unreliable on model instances.
```python
def _get_or_create_lock(self, attr: str) -> asyncio.Lock:
try:
return object.__getattribute__(self, attr)
except AttributeError:
lock = asyncio.Lock()
object.__setattr__(self, attr, lock)
return lock
@property
def _record_lock(self) -> asyncio.Lock:
return self._get_or_create_lock("_record_lock_instance")
@property
def _sequence_lock(self) -> asyncio.Lock: # DataSequence / DataProvider
return self._get_or_create_lock("_sequence_lock_instance")
@property
def _container_lock(self) -> asyncio.Lock: # DataContainer only
return self._get_or_create_lock("_container_lock_instance")
```
### Lock acquisition rules
**Individual record writes** acquire only `_record_lock`, held for the minimum time
needed to make the check-then-act atomic:
```python
async def update_value(self, date, *args, **kwargs):
async with self._record_lock:
self._update_value(date, *args, **kwargs)
```
**Sequence-level bulk operations** (`save`, `load`, `import_from_*`) acquire
`_sequence_lock` first, then `_record_lock`. This blocks both concurrent bulk operations
and concurrent individual writes for the full duration:
```python
async def load(self):
async with self._sequence_lock:
async with self._record_lock:
self._load()
```
**`update_data`** on a provider acquires both `_sequence_lock` and `_record_lock`,
protecting the provider's full sequence state against concurrent saves, loads, imports,
or individual record writes for the entire duration of `_update_data`:
```python
async def update_data(self, force_enable=False, force_update=False):
if not force_enable and not self.enabled():
return
async with self._sequence_lock:
async with self._record_lock:
self._update_data(force_update=force_update)
```
**Container-level bulk operations** acquire `_container_lock` and `_record_lock` on the
container, then delegate to each provider (which independently acquires its own
`_sequence_lock` and `_record_lock`):
```python
async def save(self): # DataContainer
async with self._container_lock:
async with self._record_lock:
for provider in self.providers:
await provider.save() # provider acquires _sequence_lock + _record_lock
```
**Cross-provider consistent reads** acquire only the container's `_record_lock`, which
prevents concurrent writes from producing an inconsistent snapshot without blocking
other read operations:
```python
async def keys_to_dataframe(self, keys, ...):
async with self._record_lock:
... # read from multiple providers atomically
```
### Lock acquisition matrix
#### `DataSequence` / `DataProvider` lock
| Operation | `_sequence_lock` | `_record_lock` |
|---|---|---|
| `insert_by_datetime` | | ✓ |
| `update_value` | | ✓ |
| `key_from_lists` | | ✓ |
| `key_from_series` | | ✓ |
| `update_data` | ✓ | ✓ |
| `save` | ✓ | ✓ |
| `load` | ✓ | ✓ |
| `import_from_dict` | ✓ | ✓ |
| `import_from_dataframe` | ✓ | ✓ |
| `import_from_json` | ✓ | ✓ |
| `import_from_file` | ✓ | ✓ |
| Read-only methods | | |
#### `DataContainer` lock
<!-- pyml disable line-length -->
| Operation | `_container_lock` | `_record_lock` | Provider `_sequence_lock` | Provider `_record_lock` |
|---|---|---|---|---|
| `update_data` | ✓ | ✓ | ✓ | ✓ |
| `__setitem__` | | ✓ | | |
| `__delitem__` | | ✓ | | |
| `keys_to_dataframe` | | ✓ | | |
| `save` | ✓ | ✓ | ✓ | ✓ |
| `load` | ✓ | ✓ | ✓ | ✓ |
| `db_vacuum` | ✓ | ✓ | | |
| `db_compact` | ✓ | ✓ | | |
| Read-only methods | | | | |
<!-- pyml enable line-length -->
### Why `asyncio.Lock` and not `threading.Lock`
FastAPI's default async workers run all coroutines on a **single OS thread**. Coroutines
interleave at `await` points, not at thread boundaries. `threading.Lock` would not protect
against this interleaving and would risk deadlock if the same coroutine attempts to
re-acquire it. `asyncio.Lock` yields control correctly at `await` points while keeping
other coroutines blocked.
### Why `asyncio.Lock` is not reentrant
Python's `asyncio.Lock` is intentionally non-reentrant. If a coroutine that already holds
`_record_lock` calls a public async method (which also tries to acquire `_record_lock`),
it will deadlock. This is why `_update_data()` implementations and all other internal
callers must use the private `_method()` variants rather than `await self.method()`.
### Choosing the right call site
<!-- pyml disable line-length -->
| Caller context | Correct call |
|---|---|
| FastAPI endpoint or any `async def` | `await sequence.insert_by_datetime(record)` |
| `_update_data()` implementation | `self._insert_by_datetime(record)` |
| Startup, `_load()` internals, tests | `sequence._insert_by_datetime(record)` |
| `DataContainer` delegating to a provider | `await provider.save()` — container holds its own locks; provider independently holds its own |
<!-- pyml enable line-length -->
## Usage Examples
### Reading data (sync, no locking required)
```python
measurement = get_measurement()
# Scalar lookup
value = measurement.key_to_value("grid_import_w", target_datetime=now)
# Resampled array for the next 24 hours
array = measurement.key_to_array(
key="grid_import_w",
start_datetime=now,
end_datetime=now.add(hours=24),
interval=to_duration("1 hour"),
)
# Pandas Series
series = measurement.key_to_series("grid_import_w", start_datetime=now)
```
### Writing a single value from a FastAPI endpoint
```python
@router.put("/measurement/value")
async def put_measurement_value(datetime: str, key: str, value: float):
dt = to_datetime(datetime)
await get_measurement().update_value(dt, key, value)
```
### Bulk import from a FastAPI endpoint
```python
@router.put("/measurement/import")
async def put_measurement_import(data: dict):
await get_measurement().import_from_dict(data)
```
### Writing from a sync context (startup / tests)
```python
def load_initial_data(measurement: Measurement, records: list[MeasurementDataRecord]):
# Sequential — no concurrency, use internal sync methods directly
for record in records:
measurement._insert_by_datetime(record)
```
### Implementing a custom `DataProvider`
```python
class MyProvider(DataProvider):
records: list[MyRecord] = Field(default_factory=list)
def provider_id(self) -> str:
return "MyProvider"
def enabled(self) -> bool:
return self.config.my_provider.enabled
def _update_data(self, force_update: Optional[bool] = False) -> None:
# Both _sequence_lock and _record_lock are already held by the caller.
# Use internal sync methods only — never await public async counterparts.
for dt, reading in fetch_from_hardware():
self._update_value(dt, "sensor_w", reading)
def db_namespace(self) -> str:
return "MyProvider"
def db_keep_datetime(self) -> Optional[DateTime]:
return to_datetime().subtract(hours=48)
```
---
## Design Notes
### Singleton providers and containers
`DataProvider` and `DataContainer` both inherit from `SingletonMixin`. A single instance
is shared across all coroutines handling concurrent requests. This is precisely why
locking is necessary — every concurrent request operates on the same in-memory object.
### Lock naming rationale
The three lock names were chosen to reflect *what is being protected*, not *how coarse
the operation is*:
- `_record_lock` — makes it immediately clear that a single record's check-then-act is
being made atomic. The same name is used at both `DataSequence` and `DataContainer`
level because it plays the same role in both: serialising individual write operations.
- `_sequence_lock` — defined on `DataSequence` and inherited unchanged by `DataProvider`.
The name reflects that the entire sequence state is held exclusively, not just one
record. `DataProvider` is a specialisation of `DataSequence`, so the name remains
accurate at both levels without needing a separate `_provider_lock`.
- `_container_lock` — exclusive to `DataContainer`, reflecting that cross-provider
container-level state is held exclusively during bulk operations.
This naming also makes lock misuse visible in code review: a method that acquires
`_sequence_lock` without also acquiring `_record_lock` (or acquires them in the wrong
order) stands out immediately against the documented fixed acquisition order.
### Database vs in-memory storage
`DataSequence` uses `DatabaseRecordProtocolMixin` to optionally back its records with a
persistent time-series database. When a database is configured, `save()` and `load()`
delegate to it; otherwise they fall back to a JSON file. The in-memory `records` list acts
as a write-through cache. The locking scheme covers both paths — the database operations
are included inside the lock scope so that in-memory state and database state remain
consistent.
### Pydantic v2 compatibility
The `asyncio.Lock` instances are stored on the instance using `object.__setattr__`,
bypassing Pydantic's field validation entirely. This is intentional: `asyncio.Lock` is not
serialisable and must not appear in `model_dump()` or `model_dump_json()` output. The
locks are therefore invisible to Pydantic serialisation and are reconstructed fresh on
each process start, which is correct behaviour for a lock.

1
docs/index.md
View File

@@ -63,6 +63,7 @@ akkudoktoreos/api.rst
develop/develop.md
develop/release.md
develop/async.md
develop/CHANGELOG.md
```

107
openapi.json
View File

@@ -8,7 +8,7 @@
"name": "Apache 2.0",
"url": "https://www.apache.org/licenses/LICENSE-2.0.html"
},
"version": "v0.3.0.dev2604141105859917"
"version": "v0.3.0.dev2607150960221650"
},
"paths": {
"/v1/admin/cache/clear": {
@@ -155,6 +155,30 @@
}
}
},
"/v1/admin/database/save": {
"post": {
"tags": [
"admin"
],
"summary": "Fastapi Admin Database Save Post",
"description": "Save in memory data to database.\n\nReturns:\n data (dict): The database stats after saving the records.",
"operationId": "fastapi_admin_database_save_post_v1_admin_database_save_post",
"responses": {
"200": {
"description": "Successful Response",
"content": {
"application/json": {
"schema": {
"additionalProperties": true,
"type": "object",
"title": "Response Fastapi Admin Database Save Post V1 Admin Database Save Post"
}
}
}
}
}
}
},
"/v1/admin/database/vacuum": {
"post": {
"tags": [
@@ -1198,6 +1222,87 @@
}
}
},
"/v1/measurement/range": {
"delete": {
"tags": [
"measurement"
],
"summary": "Fastapi Measurement Range Delete",
"description": "Delete measurement values for a key within a datetime range.",
"operationId": "fastapi_measurement_range_delete_v1_measurement_range_delete",
"parameters": [
{
"name": "key",
"in": "query",
"required": true,
"schema": {
"type": "string",
"description": "Measurement key.",
"title": "Key"
},
"description": "Measurement key."
},
{
"name": "start_datetime",
"in": "query",
"required": false,
"schema": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "Start datetime.",
"title": "Start Datetime"
},
"description": "Start datetime."
},
{
"name": "end_datetime",
"in": "query",
"required": false,
"schema": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "End datetime.",
"title": "End Datetime"
},
"description": "End datetime."
}
],
"responses": {
"200": {
"description": "Successful Response",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/PydanticDateTimeSeries"
}
}
}
},
"422": {
"description": "Validation Error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
}
}
}
}
},
"/v1/prediction/providers": {
"get": {
"tags": [

1
pyproject.toml
View File

@@ -76,6 +76,7 @@ dev = [
"GitPython==3.1.51",
"myst-parser==5.1.0",
"docutils==0.21.2",
"sphinxcontrib-mermaid==2.0.3 ",
# Pytest
"pytest==9.1.1",

47
src/akkudoktoreos/adapter/adapterabc.py
View File

@@ -1,5 +1,6 @@
"""Abstract and base classes for adapters."""
import asyncio
from abc import abstractmethod
from typing import Any, Optional
@@ -53,12 +54,27 @@ class AdapterProvider(SingletonMixin, ConfigMixin, MeasurementMixin, StartMixin,
return self.provider_id() in self.config.adapter.provider
return False
@property
def _adapter_lock(self) -> asyncio.Lock:
"""Per-instance asyncio lock guarding adapter-level bulk operations.
The lock guards the full adapter state during bulk operations.
"""
try:
return object.__getattribute__(self, "_adapter_lock_instance")
except AttributeError:
lock = asyncio.Lock()
object.__setattr__(self, "_adapter_lock_instance", lock)
return lock
@abstractmethod
def _update_data(self) -> None:
async def _update_data(self) -> None:
"""Abstract method for custom adapter data update logic, to be implemented by derived classes.
Data update may be requested at different stages of energy management. The stage can be
detected by self.ems.stage().
This method is always called while `_adapter_lock` is held by the caller.
"""
pass
@@ -67,7 +83,7 @@ class AdapterProvider(SingletonMixin, ConfigMixin, MeasurementMixin, StartMixin,
return
super().__init__(*args, **kwargs)
def update_data(
async def update_data(
self,
force_enable: Optional[bool] = False,
) -> None:
@@ -81,8 +97,9 @@ class AdapterProvider(SingletonMixin, ConfigMixin, MeasurementMixin, StartMixin,
return
# Call the custom update logic
logger.debug(f"Update adapter provider: {self.provider_id()}")
self._update_data()
async with self._adapter_lock:
logger.debug(f"Update adapter provider: {self.provider_id()}")
await self._update_data()
class AdapterContainer(SingletonMixin, ConfigMixin, PydanticBaseModel):
@@ -105,6 +122,19 @@ class AdapterContainer(SingletonMixin, ConfigMixin, PydanticBaseModel):
)
return value
@property
def _container_lock(self) -> asyncio.Lock:
"""Coarse-grained lock for bulk operations across providers.
The lock guards cross-provider consistency during container operations.
"""
try:
return object.__getattribute__(self, "_container_lock_instance")
except AttributeError:
lock = asyncio.Lock()
object.__setattr__(self, "_container_lock_instance", lock)
return lock
@property
def enabled_providers(self) -> list[Any]:
"""List of providers that are currently enabled."""
@@ -145,7 +175,7 @@ class AdapterContainer(SingletonMixin, ConfigMixin, PydanticBaseModel):
raise ValueError(error_msg)
return providers[provider_id]
def update_data(
async def update_data(
self,
force_enable: Optional[bool] = False,
) -> None:
@@ -154,7 +184,10 @@ class AdapterContainer(SingletonMixin, ConfigMixin, PydanticBaseModel):
Args:
force_enable (bool, optional): If True, forces the update even if the provider is disabled.
"""
if len(self.providers) <= 0:
return
# Call the custom update logic
if len(self.providers) > 0:
async with self._container_lock:
for provider in self.providers:
provider.update_data(force_enable=force_enable)
await provider.update_data(force_enable=force_enable)

12
src/akkudoktoreos/adapter/homeassistant.py
View File

@@ -419,7 +419,7 @@ class HomeAssistantAdapter(AdapterProvider):
# Preserve original state for enums and free-text states
return raw_state
def _update_data(self) -> None:
async def _update_data(self) -> None:
stage = self.ems.stage()
if stage == EnergyManagementStage.DATA_ACQUISITION:
# Sync configuration
@@ -451,7 +451,7 @@ class HomeAssistantAdapter(AdapterProvider):
logger.debug(f"Entity {entity_id}: {state}")
if state:
measurement_value = float(state)
self.measurement.update_value(
await self.measurement.update_value(
self.ems_start_datetime, measurement_key, measurement_value
)
except Exception as e:
@@ -473,7 +473,7 @@ class HomeAssistantAdapter(AdapterProvider):
logger.debug(f"Entity {entity_id}: {state}")
if state:
measurement_value = float(state)
self.measurement.update_value(
await self.measurement.update_value(
self.ems_start_datetime, measurement_key, measurement_value
)
except Exception as e:
@@ -495,7 +495,7 @@ class HomeAssistantAdapter(AdapterProvider):
logger.debug(f"Entity {entity_id}: {state}")
if state:
measurement_value = float(state)
self.measurement.update_value(
await self.measurement.update_value(
self.ems_start_datetime, measurement_key, measurement_value
)
except Exception as e:
@@ -517,7 +517,7 @@ class HomeAssistantAdapter(AdapterProvider):
logger.debug(f"Entity {entity_id}: {state}")
if state:
measurement_value = float(state)
self.measurement.update_value(
await self.measurement.update_value(
self.ems_start_datetime, measurement_key, measurement_value
)
except Exception as e:
@@ -539,7 +539,7 @@ class HomeAssistantAdapter(AdapterProvider):
logger.debug(f"Entity {entity_id}: {state}")
if state:
measurement_value = float(state)
self.measurement.update_value(
await self.measurement.update_value(
self.ems_start_datetime, measurement_key, measurement_value
)
except Exception as e:

2
src/akkudoktoreos/adapter/nodered.py
View File

@@ -66,7 +66,7 @@ class NodeREDAdapter(AdapterProvider):
"""Return the unique identifier for the adapter provider."""
return "NodeRED"
def _update_data(self) -> None:
async def _update_data(self) -> None:
"""Custom adapter data update logic.
Data update may be requested at different stages of energy management. The stage can be

1036
src/akkudoktoreos/core/dataabc.py
View File

File diff suppressed because it is too large Load Diff

662
src/akkudoktoreos/core/database.py
View File

@@ -9,25 +9,34 @@ namespaces with a `namespace` column.
from __future__ import annotations
import asyncio
import shutil
import sqlite3
from pathlib import Path
from typing import Any, Dict, Iterable, Iterator, List, Optional, Tuple
from typing import (
Any,
AsyncIterator,
Dict,
Iterable,
Iterator,
List,
Optional,
Tuple,
)
import lmdb
from loguru import logger
from pydantic import Field, computed_field, field_validator
from akkudoktoreos.config.configabc import SettingsBaseModel
from akkudoktoreos.core.coreabc import SingletonMixin
from akkudoktoreos.core.coreabc import ConfigMixin, SingletonMixin
from akkudoktoreos.core.databaseabc import (
DATABASE_METADATA_KEY,
DatabaseABC,
DatabaseBackendABC,
)
# Valid database providers
database_providers: List[str] = ["LMDB", "SQLite"]
database_providers: List[str] = ["LMDB", "SQLite", "NoDB"]
class DatabaseCommonSettings(SettingsBaseModel):
@@ -40,6 +49,8 @@ class DatabaseCommonSettings(SettingsBaseModel):
batch_size: Batch size for batch operations.
"""
model_config = {"validate_assignment": True} # keeps field validation on assignment
provider: Optional[str] = Field(
default=None,
json_schema_extra={
@@ -177,12 +188,17 @@ class LMDBDatabase(DatabaseBackendABC):
"""Return the unique identifier for the database provider."""
return "LMDB"
def open(self, namespace: Optional[str] = None) -> None:
def open(self, *, namespace: Optional[str] = None) -> None:
"""Open LMDB environment and optionally ensure a namespace DBI.
Args:
namespace: Optional default namespace to open (DBI created on demand).
"""
if self.is_open:
if namespace is not None:
self._ensure_dbi(namespace=namespace)
return
self.storage_path.mkdir(parents=True, exist_ok=True)
self.env = lmdb.open(
@@ -201,7 +217,7 @@ class LMDBDatabase(DatabaseBackendABC):
self.default_namespace = namespace
if namespace is not None:
self._ensure_dbi(namespace)
self._ensure_dbi(namespace=namespace)
def close(self) -> None:
"""Close the LMDB environment and clear cached DBIs."""
@@ -214,7 +230,7 @@ class LMDBDatabase(DatabaseBackendABC):
self._dbis.clear()
logger.debug("Closed LMDB at %s", self.storage_path)
def flush(self, namespace: Optional[str] = None) -> None:
def flush(self, *, namespace: Optional[str] = None) -> None:
"""Sync LMDB environment (writes to disk)."""
if not isinstance(self.env, lmdb.Environment):
raise ValueError(f"LMDB Environment is of wrong tpe `{type(self.env)}`.")
@@ -230,7 +246,7 @@ class LMDBDatabase(DatabaseBackendABC):
"""Return explicit namespace or default if None."""
return namespace if namespace is not None else self.default_namespace
def _ensure_dbi(self, namespace: Optional[str]) -> Optional[Any]:
def _ensure_dbi(self, *, namespace: Optional[str]) -> Optional[Any]:
"""Open and cache a DBI for the given namespace.
Args:
@@ -271,15 +287,15 @@ class LMDBDatabase(DatabaseBackendABC):
if not isinstance(self.env, lmdb.Environment):
raise RuntimeError(f"LMDB Environment is of wrong tpe `{type(self.env)}`.")
dbi = self._ensure_dbi(namespace)
dbi = self._ensure_dbi(namespace=namespace)
with self.env.begin(write=True) as txn:
if metadata is None:
txn.delete(DATABASE_METADATA_KEY)
txn.delete(DATABASE_METADATA_KEY, db=dbi)
else:
txn.put(DATABASE_METADATA_KEY, metadata)
txn.put(DATABASE_METADATA_KEY, metadata, db=dbi)
def get_metadata(self, namespace: Optional[str] = None) -> Optional[bytes]:
def get_metadata(self, *, namespace: Optional[str] = None) -> Optional[bytes]:
"""Load metadata for a given namespace.
Returns None if no metadata exists.
@@ -293,10 +309,10 @@ class LMDBDatabase(DatabaseBackendABC):
if not isinstance(self.env, lmdb.Environment):
raise RuntimeError(f"LMDB Environment is of wrong tpe `{type(self.env)}`.")
dbi = self._ensure_dbi(namespace)
dbi = self._ensure_dbi(namespace=namespace)
with self.env.begin(write=False) as txn:
return txn.get(DATABASE_METADATA_KEY)
return txn.get(DATABASE_METADATA_KEY, db=dbi)
# ------------------------------------------------------------------
# Bulk Write Operations
@@ -305,6 +321,7 @@ class LMDBDatabase(DatabaseBackendABC):
def save_records(
self,
records: Iterable[tuple[bytes, bytes]],
*,
namespace: Optional[str] = None,
) -> int:
"""Save multiple records into the specified namespace (or default).
@@ -324,7 +341,7 @@ class LMDBDatabase(DatabaseBackendABC):
if not isinstance(self.env, lmdb.Environment):
raise RuntimeError(f"LMDB Environment is of wrong tpe `{type(self.env)}`.")
dbi = self._ensure_dbi(namespace)
dbi = self._ensure_dbi(namespace=namespace)
saved = 0
with self.lock:
@@ -338,6 +355,7 @@ class LMDBDatabase(DatabaseBackendABC):
def delete_records(
self,
keys: Iterable[bytes],
*,
namespace: Optional[str] = None,
) -> int:
"""Delete multiple records by key from the specified namespace.
@@ -352,7 +370,7 @@ class LMDBDatabase(DatabaseBackendABC):
if not isinstance(self.env, lmdb.Environment):
raise RuntimeError("Database not open")
dbi = self._ensure_dbi(namespace)
dbi = self._ensure_dbi(namespace=namespace)
deleted = 0
with self.lock:
@@ -371,6 +389,7 @@ class LMDBDatabase(DatabaseBackendABC):
self,
start_key: Optional[bytes] = None,
end_key: Optional[bytes] = None,
*,
namespace: Optional[str] = None,
reverse: bool = False,
) -> Iterator[tuple[bytes, bytes]]:
@@ -391,11 +410,12 @@ class LMDBDatabase(DatabaseBackendABC):
if not isinstance(self.env, lmdb.Environment):
raise RuntimeError(f"LMDB Environment is of wrong type `{type(self.env)}`.")
dbi = self._ensure_dbi(namespace)
dbi = self._ensure_dbi(namespace=namespace)
META = DATABASE_METADATA_KEY
results: list[tuple[bytes, bytes]] = []
cursor = None
txn = self.env.begin(write=False)
try:
cursor = txn.cursor(dbi)
@@ -454,7 +474,8 @@ class LMDBDatabase(DatabaseBackendABC):
finally:
# Ensure reader slot is always released
cursor.close()
if cursor is not None:
cursor.close()
txn.abort()
# Transaction is closed here — safe to yield
@@ -478,7 +499,7 @@ class LMDBDatabase(DatabaseBackendABC):
if not isinstance(self.env, lmdb.Environment):
raise RuntimeError(f"LMDB Environment is of wrong tpe `{type(self.env)}`.")
dbi = self._ensure_dbi(namespace)
dbi = self._ensure_dbi(namespace=namespace)
META = DATABASE_METADATA_KEY
count = 0
@@ -510,13 +531,14 @@ class LMDBDatabase(DatabaseBackendABC):
def get_key_range(
self,
*,
namespace: Optional[str] = None,
) -> tuple[Optional[bytes], Optional[bytes]]:
"""Return (min_key, max_key) in the given namespace or (None, None) if empty."""
if not isinstance(self.env, lmdb.Environment):
raise RuntimeError(f"LMDB Environment is of wrong tpe `{type(self.env)}`.")
dbi = self._ensure_dbi(namespace)
dbi = self._ensure_dbi(namespace=namespace)
with self.env.begin(write=False) as txn:
cursor = txn.cursor(db=dbi)
@@ -541,12 +563,12 @@ class LMDBDatabase(DatabaseBackendABC):
return min_key, max_key
def get_backend_stats(self, namespace: Optional[str] = None) -> dict[str, Any]:
def get_backend_stats(self, *, namespace: Optional[str] = None) -> dict[str, Any]:
"""Get LMDB backend-specific statistics."""
if not self.env:
return {}
dbi = self._ensure_dbi(namespace)
dbi = self._ensure_dbi(namespace=namespace)
with self.env.begin(write=False) as txn:
stat = txn.stat(db=dbi)
@@ -657,12 +679,15 @@ class SQLiteDatabase(DatabaseBackendABC):
"""Return the unique identifier for the database provider."""
return "SQLite"
def open(self, namespace: Optional[str] = None) -> None:
def open(self, *, namespace: Optional[str] = None) -> None:
"""Open SQLite connection and optionally set default namespace.
Args:
namespace: Optional default namespace to use when operations omit namespace.
"""
if self.is_open:
return
self.storage_path.mkdir(parents=True, exist_ok=True)
self.conn = sqlite3.connect(
@@ -700,7 +725,7 @@ class SQLiteDatabase(DatabaseBackendABC):
self._is_open = False
logger.debug("Closed SQLite at %s", self.db_file)
def flush(self, namespace: Optional[str] = None) -> None:
def flush(self, *, namespace: Optional[str] = None) -> None:
"""Commit any pending transactions to disk (no-op if autocommit)."""
if not isinstance(self.conn, sqlite3.Connection):
raise RuntimeError(f"SQLite connection is of wrong tpe `{type(self.conn)}`.")
@@ -745,7 +770,7 @@ class SQLiteDatabase(DatabaseBackendABC):
(ns, metadata),
)
def get_metadata(self, namespace: Optional[str] = None) -> Optional[bytes]:
def get_metadata(self, *, namespace: Optional[str] = None) -> Optional[bytes]:
"""Load metadata for a given namespace.
Returns None if no metadata exists.
@@ -777,6 +802,7 @@ class SQLiteDatabase(DatabaseBackendABC):
def save_records(
self,
records: Iterable[tuple[bytes, bytes]],
*,
namespace: Optional[str] = None,
) -> int:
"""Bulk insert or replace records.
@@ -794,18 +820,18 @@ class SQLiteDatabase(DatabaseBackendABC):
return 0
with self.lock:
self.conn.execute("BEGIN")
self.conn.executemany(
"INSERT OR REPLACE INTO records (namespace, key, value) VALUES (?, ?, ?)",
rows,
)
self.conn.execute("COMMIT")
with self.conn:
self.conn.executemany(
"INSERT OR REPLACE INTO records (namespace, key, value) VALUES (?, ?, ?)",
rows,
)
return len(rows)
def delete_records(
self,
keys: Iterable[bytes],
*,
namespace: Optional[str] = None,
) -> int:
"""Delete multiple records by key.
@@ -817,21 +843,25 @@ class SQLiteDatabase(DatabaseBackendABC):
ns = self._ns(namespace)
deleted: int = 0
with self.lock:
for key in keys:
cursor = self.conn.execute(
"DELETE FROM records WHERE namespace = ? AND key = ?",
(ns, key),
)
deleted += cursor.rowcount
rows = [(ns, key) for key in keys]
return deleted
if not rows:
return 0
with self.lock:
with self.conn:
cursor = self.conn.executemany(
"DELETE FROM records WHERE namespace = ? AND key = ?",
rows,
)
return cursor.rowcount
def iterate_records(
self,
start_key: Optional[bytes] = None,
end_key: Optional[bytes] = None,
*,
namespace: Optional[str] = None,
reverse: bool = False,
) -> Iterator[Tuple[bytes, bytes]]:
@@ -913,7 +943,7 @@ class SQLiteDatabase(DatabaseBackendABC):
return int(cursor.fetchone()[0])
def get_key_range(
self, namespace: Optional[str] = None
self, *, namespace: Optional[str] = None
) -> Tuple[Optional[bytes], Optional[bytes]]:
"""Return (min_key, max_key) for the namespace or (None, None) if empty."""
if not isinstance(self.conn, sqlite3.Connection):
@@ -928,7 +958,7 @@ class SQLiteDatabase(DatabaseBackendABC):
result = cursor.fetchone()
return result[0], result[1]
def get_backend_stats(self, namespace: Optional[str] = None) -> Dict[str, Any]:
def get_backend_stats(self, *, namespace: Optional[str] = None) -> Dict[str, Any]:
"""Return SQLite-specific stats and namespace metrics."""
if not self.conn:
return {}
@@ -959,10 +989,216 @@ class SQLiteDatabase(DatabaseBackendABC):
logger.info("SQLite vacuum completed")
# ==================== Generic Database Implementation ====================
# ==================== NoDB Implementation ====================
class Database(DatabaseABC, SingletonMixin):
class NoDB(DatabaseBackendABC):
"""No-op database backend.
This backend implements the full database backend interface but performs
no actual persistence. It is intended for configurations where database
persistence is disabled (`provider=None`).
Characteristics:
- All write operations are ignored.
- All read operations return empty results.
- No files or external resources are created.
- Lifecycle operations are no-ops.
- Compression is disabled.
This backend allows the database wrapper to avoid special-case handling
for a missing database provider by always providing a valid backend
implementation.
"""
def __init__(self, **kwargs: Any) -> None:
"""Initialize the no-op backend."""
super().__init__()
self._is_open = True # Stateless
# ------------------------------------------------------------------
# Lifecycle
# ------------------------------------------------------------------
def provider_id(self) -> str:
"""Return the unique identifier for the database provider."""
return "NoDB"
def open(self, *, namespace: Optional[str] = None) -> None:
"""Mark backend as open.
Args:
namespace: Ignored.
"""
self.default_namespace = namespace
def close(self) -> None:
"""Mark backend as closed."""
return
def flush(self, *, namespace: Optional[str] = None) -> None:
"""Flush pending writes (no-op).
Args:
namespace: Ignored.
"""
return
# ------------------------------------------------------------------
# Effective backend configuration
# ------------------------------------------------------------------
@property
def is_open(self) -> bool:
"""Return dummy not open."""
return False
@property
def storage_path(self) -> Path:
"""Return dummy storage path."""
return Path("/dev/null")
@property
def compression_level(self) -> int:
"""Compression is disabled."""
return 0
@property
def compression(self) -> bool:
"""Compression is disabled."""
return False
# ------------------------------------------------------------------
# Metadata operations
# ------------------------------------------------------------------
def set_metadata(
self,
metadata: Optional[bytes],
*,
namespace: Optional[str] = None,
) -> None:
"""Store metadata (ignored).
Args:
metadata: Ignored.
namespace: Ignored.
"""
return
def get_metadata(
self,
*,
namespace: Optional[str] = None,
) -> Optional[bytes]:
"""Load metadata.
Always returns:
None
"""
return None
# ------------------------------------------------------------------
# Record operations
# ------------------------------------------------------------------
def save_records(
self,
records: Iterable[tuple[bytes, bytes]],
*,
namespace: Optional[str] = None,
) -> int:
"""Pretend to save records.
Args:
records: Ignored.
namespace: Ignored.
Returns:
Always 0.
"""
return 0
def delete_records(
self,
keys: Iterable[bytes],
*,
namespace: Optional[str] = None,
) -> int:
"""Pretend to delete records.
Args:
keys: Ignored.
namespace: Ignored.
Returns:
Always 0.
"""
return 0
def iterate_records(
self,
start_key: Optional[bytes] = None,
end_key: Optional[bytes] = None,
*,
namespace: Optional[str] = None,
reverse: bool = False,
) -> Iterator[tuple[bytes, bytes]]:
"""Iterate records.
Always yields:
Nothing
"""
return iter(())
def count_records(
self,
start_key: Optional[bytes] = None,
end_key: Optional[bytes] = None,
*,
namespace: Optional[str] = None,
) -> int:
"""Count records.
Returns:
Always 0.
"""
return 0
def get_key_range(
self,
*,
namespace: Optional[str] = None,
) -> tuple[Optional[bytes], Optional[bytes]]:
"""Return key range.
Returns:
Always (None, None).
"""
return None, None
def get_backend_stats(
self,
*,
namespace: Optional[str] = None,
) -> dict[str, Any]:
"""Return backend statistics.
Returns:
Minimal backend information.
"""
return {
"backend": "none",
"namespace": namespace,
"persistent": False,
"records": 0,
}
# ==================== Generic Database ====================
class Database(ConfigMixin, SingletonMixin):
"""Generic database.
All operations accept an optional `namespace` argument. Implementations should
@@ -971,16 +1207,15 @@ class Database(DatabaseABC, SingletonMixin):
a namespace column).
"""
_db: Optional[DatabaseBackendABC] = None
_db: DatabaseBackendABC = NoDB()
@classmethod
def reset_instance(cls) -> None:
"""Resets the singleton instance, forcing it to be recreated on next access."""
with cls._lock:
# Close current database backend
if cls._db:
cls._db.close()
cls._db = None
cls._db.close()
cls._db = NoDB()
# Remove current database instance
if cls in cls._instances:
del cls._instances[cls]
@@ -989,95 +1224,228 @@ class Database(DatabaseABC, SingletonMixin):
def __init__(self) -> None:
"""Initialize database."""
super().__init__()
self._db = None
self._db = NoDB()
def _setup_db(self) -> None:
"""Setup database."""
# Database helpers
@property
def _database_lock(self) -> asyncio.Lock:
"""Per-instance asyncio lock guarding database operations.
The lock guards the database state during async operations.
"""
# Lock must be a loop-local asyncio lock to provide singleton + pytest async compatibility.
loop = asyncio.get_running_loop()
try:
locks = object.__getattribute__(self, "_database_locks")
except AttributeError:
locks = {}
object.__setattr__(self, "_database_locks", locks)
lock = locks.get(loop)
if lock is None:
lock = asyncio.Lock()
locks[loop] = lock
return lock
async def _ensure_open(
self,
*,
namespace: Optional[str] = None,
) -> DatabaseBackendABC:
"""Ensure the configured backend exists, is open, and namespace is prepared.
Expects the database lock to already be held when called.
Args:
namespace: Optional namespace to prepare/open.
Returns:
The active and opened backend instance.
Raises:
RuntimeError: If no database provider is configured.
"""
provider_id = self.config.database.provider
database: Optional[DatabaseBackendABC] = None
if provider_id is None:
database = None
elif provider_id == "LMDB":
database = LMDBDatabase()
elif provider_id == "SQLite":
database = SQLiteDatabase()
else:
raise RuntimeError("Invalid database provider '{provider_id}'")
if self._db is not None:
self._db.close()
self._db = database
old_provider_id = self._db.provider_id()
def _database(self) -> DatabaseBackendABC:
"""Get database."""
provider_id = self.config.database.provider
if provider_id is None:
raise RuntimeError("Database not configured")
if self._db is None or self._db.provider_id() != provider_id:
if old_provider_id != provider_id:
# No database or configuration does not match
self._setup_db()
if self._db is None:
raise RuntimeError("Database not configured")
logger.debug(
f"Switching database provider from '{old_provider_id}' to '{provider_id}'."
)
old_db = self._db
database: DatabaseBackendABC
if provider_id is None or provider_id == "NoDB":
database = NoDB()
elif provider_id == "LMDB":
database = LMDBDatabase()
elif provider_id == "SQLite":
database = SQLiteDatabase()
else:
raise RuntimeError(f"Invalid database provider '{provider_id}'")
# Auto-close database that was used before
if old_db.is_open:
old_db.close()
self._db = database
if not self._db.is_open:
self._db.open()
await asyncio.to_thread(self._db.open, namespace=namespace)
elif namespace is not None:
# Allow backend to lazily prepare namespace resources
await asyncio.to_thread(self._db.open, namespace=namespace)
return self._db
async def _run_db(
self,
method_name: str,
*args: Any,
**kwargs: Any,
) -> Any:
"""Execute a synchronous database backend operation in a thread-safe, non-blocking way.
The backend is automatically initialized/opened before execution.
If a ``namespace`` keyword argument is present, the namespace is
prepared during backend initialization.
This helper ensures that all interactions with the underlying synchronous
database backend are:
- **Serialized** using an instance-level ``asyncio.Lock`` to protect backend state.
- **Non-blocking** by offloading execution to a worker thread via
``asyncio.to_thread``.
It should be used for all backend calls that may perform blocking I/O or
CPU-bound work.
Args:
method_name: The synchronous callable to execute (a backend method).
*args: Positional arguments forwarded to ``method``.
**kwargs: Keyword arguments forwarded to ``method``.
Returns:
The return value of ``method``.
Raises:
Any exception raised by ``func`` is propagated unchanged.
Notes:
- The callable is executed in a separate thread, so it must be thread-safe.
- The database lock is held for the duration of the call, ensuring that
no concurrent operations interfere with backend state.
- Avoid passing coroutines or async functions to this method; it is
intended strictly for synchronous callables.
"""
namespace = kwargs.get("namespace")
async with self._database_lock:
db = await self._ensure_open(namespace=namespace)
# Get actual method. _ensure_open may have changed the backend
method = getattr(db, method_name)
def backend_call() -> Any:
result = method(
*args,
**kwargs,
)
# Materialize iterators inside worker thread
if isinstance(result, Iterator):
return list(result)
return result
return await asyncio.to_thread(backend_call)
def provider_id(self) -> str:
"""Return the unique identifier for the database provider."""
try:
return self._database().provider_id()
except:
return "None"
return self._db.provider_id()
@property
def is_open(self) -> bool:
"""Return whether the database connection is open."""
try:
return self._database().is_open
except:
return False
return self._db.is_open
@property
def storage_path(self) -> Path:
"""Storage path for the database."""
return self._database().storage_path
"""Return effective storage path of active backend."""
return self._db.storage_path
@property
def compression_level(self) -> int:
"""Compression level for database record data."""
return self._database().compression_level
"""Return effective compression level of active backend."""
return self._db.compression_level
@property
def compression(self) -> bool:
"""Whether to compress stored values."""
return self._database().compression_level > 0
"""Return whether the active backend compresses stored values.
Returns:
True if compression is enabled for the active backend,
False if disabled,
or None if no backend is currently initialized.
"""
return self._db.compression_level > 0
# Lifecycle
def open(self, namespace: Optional[str] = None) -> None:
"""Open database connection and optionally set default namespace.
async def ensure_open(
self,
*,
namespace: Optional[str] = None,
) -> DatabaseBackendABC:
"""Ensure the configured backend exists, is open, and namespace is prepared.
Args:
namespace: Optional default namespace to prepare.
namespace: Optional namespace to prepare/open.
Returns:
The active and opened backend instance.
Raises:
RuntimeError: If the database cannot be opened.
RuntimeError: If no database provider is configured.
"""
self._database().open(namespace)
async with self._database_lock:
return await self._ensure_open(namespace=namespace)
def close(self) -> None:
async def open(self, *, namespace: Optional[str] = None) -> None:
"""Ensure the database backend is initialized and open.
If the backend is already open, this operation is a no-op except
that the backend may prepare the specified namespace.
Args:
namespace: Optional namespace to prepare/open.
Raises:
RuntimeError: If no database provider is configured or opening fails.
"""
async with self._database_lock:
await self._ensure_open(namespace=namespace)
async def close(self) -> None:
"""Close the database connection and cleanup resources."""
self._database().close()
async with self._database_lock:
if self._db is not None and self._db.is_open:
await asyncio.to_thread(self._db.close)
def flush(self, namespace: Optional[str] = None) -> None:
async def flush(self, *, namespace: Optional[str] = None) -> None:
"""Force synchronization of pending writes to storage (optional per-namespace)."""
return self._database().flush(namespace)
await self._run_db("flush", namespace=namespace)
# Metadata operations
def set_metadata(self, metadata: Optional[bytes], *, namespace: Optional[str] = None) -> None:
async def set_metadata(
self, metadata: Optional[bytes], *, namespace: Optional[str] = None
) -> None:
"""Save metadata for a given namespace.
Metadata is treated separately from data records and stored as a single object.
@@ -1086,9 +1454,13 @@ class Database(DatabaseABC, SingletonMixin):
metadata (bytes): Arbitrary metadata to save or None to delete metadata.
namespace (Optional[str]): Optional namespace under which to store metadata.
"""
self._database().set_metadata(metadata, namespace=namespace)
await self._run_db(
"set_metadata",
metadata,
namespace=namespace,
)
def get_metadata(self, namespace: Optional[str] = None) -> Optional[bytes]:
async def get_metadata(self, *, namespace: Optional[str] = None) -> Optional[bytes]:
"""Load metadata for a given namespace.
Returns None if no metadata exists.
@@ -1099,12 +1471,15 @@ class Database(DatabaseABC, SingletonMixin):
Returns:
Optional[bytes]: The loaded metadata, or None if not found.
"""
return self._database().get_metadata(namespace=namespace)
return await self._run_db(
"get_metadata",
namespace=namespace,
)
# Basic record operations
def save_records(
self, records: Iterable[tuple[bytes, bytes]], namespace: Optional[str] = None
async def save_records(
self, records: Iterable[tuple[bytes, bytes]], *, namespace: Optional[str] = None
) -> int:
"""Save multiple records into the specified namespace (or default).
@@ -1120,9 +1495,15 @@ class Database(DatabaseABC, SingletonMixin):
Raises:
RuntimeError: If DB not open or write failed.
"""
return self._database().save_records(records, namespace)
return await self._run_db(
"save_records",
records,
namespace=namespace,
)
def delete_records(self, keys: Iterable[bytes], namespace: Optional[str] = None) -> int:
async def delete_records(
self, keys: Iterable[bytes], *, namespace: Optional[str] = None
) -> int:
"""Delete multiple records by key from the specified namespace.
Args:
@@ -1132,15 +1513,20 @@ class Database(DatabaseABC, SingletonMixin):
Returns:
Number of records actually deleted.
"""
return self._database().delete_records(keys, namespace)
return await self._run_db(
"delete_records",
keys,
namespace=namespace,
)
def iterate_records(
async def iterate_records(
self,
start_key: Optional[bytes] = None,
end_key: Optional[bytes] = None,
*,
namespace: Optional[str] = None,
reverse: bool = False,
) -> Iterator[tuple[bytes, bytes]]:
) -> AsyncIterator[tuple[bytes, bytes]]:
"""Iterate over records for a namespace with optional bounds.
Args:
@@ -1152,9 +1538,18 @@ class Database(DatabaseABC, SingletonMixin):
Yields:
Tuples of (key, record).
"""
return self._database().iterate_records(start_key, end_key, namespace, reverse)
records: list[tuple[bytes, bytes]] = await self._run_db(
"iterate_records",
start_key,
end_key,
namespace=namespace,
reverse=reverse,
)
def count_records(
for item in records:
yield item
async def count_records(
self,
start_key: Optional[bytes] = None,
end_key: Optional[bytes] = None,
@@ -1165,14 +1560,49 @@ class Database(DatabaseABC, SingletonMixin):
Excludes metadata records.
"""
return self._database().count_records(start_key, end_key, namespace=namespace)
return await self._run_db(
"count_records",
start_key,
end_key,
namespace=namespace,
)
def get_key_range(
self, namespace: Optional[str] = None
async def get_key_range(
self, *, namespace: Optional[str] = None
) -> Tuple[Optional[bytes], Optional[bytes]]:
"""Return (min_key, max_key) in the given namespace or (None, None) if empty."""
return self._database().get_key_range(namespace)
return await self._run_db(
"get_key_range",
namespace=namespace,
)
def get_backend_stats(self, namespace: Optional[str] = None) -> Dict[str, Any]:
async def get_backend_stats(self, *, namespace: Optional[str] = None) -> Dict[str, Any]:
"""Get backend-specific statistics; implementations may return namespace-specific data."""
return self._database().get_backend_stats(namespace)
return await self._run_db(
"get_backend_stats",
namespace=namespace,
)
# Compression helpers
def serialize_data(self, data: bytes) -> bytes:
"""Optionally compress raw pickled data before storage.
Args:
data: Raw pickled bytes.
Returns:
Possibly compressed bytes.
"""
return self._db.serialize_data(data)
def deserialize_data(self, data: bytes) -> bytes:
"""Optionally decompress stored data.
Args:
data: Stored bytes.
Returns:
Raw pickled bytes (decompressed if needed).
"""
return self._db.deserialize_data(data)

343
src/akkudoktoreos/core/databaseabc.py
View File

@@ -12,6 +12,7 @@ from threading import Lock
from typing import (
TYPE_CHECKING,
Any,
AsyncIterator,
Final,
Generic,
Iterable,
@@ -46,20 +47,38 @@ DATABASE_METADATA_KEY: bytes = b"__metadata__"
# ==================== Abstract Database Interface ====================
class DatabaseABC(ABC, ConfigMixin):
"""Abstract base class for database.
class DatabaseBackendABC(ABC, ConfigMixin, SingletonMixin):
"""Abstract base class for database backends.
All operations accept an optional `namespace` argument. Implementations should
treat None as the default/root namespace. Concrete implementations can map
namespace -> native namespace (LMDB DBI) or emulate namespaces (SQLite uses
a namespace column).
The database backend provides a synchronous interface. Asynchrounous access is
handled by the generic database class.
"""
connection: Any
lock: Lock
_is_open: bool
default_namespace: Optional[str]
def __init__(self, **kwargs: Any) -> None:
"""Initialize the DatabaseBackendABC base.
Args:
**kwargs: Backend-specific options (ignored by base).
"""
self.connection = None
self.lock = Lock()
self._is_open = False
self.default_namespace = None
@property
@abstractmethod
def is_open(self) -> bool:
"""Return whether the database connection is open."""
raise NotImplementedError
return self._is_open
@property
def storage_path(self) -> Path:
@@ -87,7 +106,7 @@ class DatabaseABC(ABC, ConfigMixin):
raise NotImplementedError
@abstractmethod
def open(self, namespace: Optional[str] = None) -> None:
def open(self, *, namespace: Optional[str] = None) -> None:
"""Open database connection and optionally set default namespace.
Args:
@@ -104,7 +123,7 @@ class DatabaseABC(ABC, ConfigMixin):
raise NotImplementedError
@abstractmethod
def flush(self, namespace: Optional[str] = None) -> None:
def flush(self, *, namespace: Optional[str] = None) -> None:
"""Force synchronization of pending writes to storage (optional per-namespace)."""
raise NotImplementedError
@@ -123,7 +142,7 @@ class DatabaseABC(ABC, ConfigMixin):
raise NotImplementedError
@abstractmethod
def get_metadata(self, namespace: Optional[str] = None) -> Optional[bytes]:
def get_metadata(self, *, namespace: Optional[str] = None) -> Optional[bytes]:
"""Load metadata for a given namespace.
Returns None if no metadata exists.
@@ -140,7 +159,7 @@ class DatabaseABC(ABC, ConfigMixin):
@abstractmethod
def save_records(
self, records: Iterable[tuple[bytes, bytes]], namespace: Optional[str] = None
self, records: Iterable[tuple[bytes, bytes]], *, namespace: Optional[str] = None
) -> int:
"""Save multiple records into the specified namespace (or default).
@@ -159,7 +178,7 @@ class DatabaseABC(ABC, ConfigMixin):
raise NotImplementedError
@abstractmethod
def delete_records(self, keys: Iterable[bytes], namespace: Optional[str] = None) -> int:
def delete_records(self, keys: Iterable[bytes], *, namespace: Optional[str] = None) -> int:
"""Delete multiple records by key from the specified namespace.
Args:
@@ -176,6 +195,7 @@ class DatabaseABC(ABC, ConfigMixin):
self,
start_key: Optional[bytes] = None,
end_key: Optional[bytes] = None,
*,
namespace: Optional[str] = None,
reverse: bool = False,
) -> Iterator[tuple[bytes, bytes]]:
@@ -208,13 +228,13 @@ class DatabaseABC(ABC, ConfigMixin):
@abstractmethod
def get_key_range(
self, namespace: Optional[str] = None
self, *, namespace: Optional[str] = None
) -> tuple[Optional[bytes], Optional[bytes]]:
"""Return (min_key, max_key) in the given namespace or (None, None) if empty."""
raise NotImplementedError
@abstractmethod
def get_backend_stats(self, namespace: Optional[str] = None) -> dict[str, Any]:
def get_backend_stats(self, *, namespace: Optional[str] = None) -> dict[str, Any]:
"""Get backend-specific statistics; implementations may return namespace-specific data."""
raise NotImplementedError
@@ -250,37 +270,6 @@ class DatabaseABC(ABC, ConfigMixin):
return data
class DatabaseBackendABC(DatabaseABC, SingletonMixin):
"""Abstract base class for database backends.
All operations accept an optional `namespace` argument. Implementations should
treat None as the default/root namespace. Concrete implementations can map
namespace -> native namespace (LMDB DBI) or emulate namespaces (SQLite uses
a namespace column).
"""
connection: Any
lock: Lock
_is_open: bool
default_namespace: Optional[str]
def __init__(self, **kwargs: Any) -> None:
"""Initialize the DatabaseBackendABC base.
Args:
**kwargs: Backend-specific options (ignored by base).
"""
self.connection = None
self.lock = Lock()
self._is_open = False
self.default_namespace = None
@property
def is_open(self) -> bool:
"""Return whether the database connection is open."""
return self._is_open
# ==================== Database Record Protocol Mixin ====================
@@ -442,7 +431,7 @@ class DatabaseRecordProtocol(Protocol, Generic[T_Record]):
@property
def db_enabled(self) -> bool: ...
def db_timestamp_range(self) -> tuple[DatabaseTimestampType, DatabaseTimestampType]: ...
async def db_timestamp_range(self) -> tuple[DatabaseTimestampType, DatabaseTimestampType]: ...
def db_generate_timestamps(
self,
@@ -451,52 +440,49 @@ class DatabaseRecordProtocol(Protocol, Generic[T_Record]):
interval: Optional[Duration] = None,
) -> Iterator[DatabaseTimestamp]: ...
def db_get_record(self, target_timestamp: DatabaseTimestamp) -> Optional[T_Record]: ...
async def db_get_record(self, target_timestamp: DatabaseTimestamp) -> Optional[T_Record]: ...
def db_insert_record(
async def db_insert_record(
self,
record: T_Record,
*,
mark_dirty: bool = True,
) -> None: ...
def db_iterate_records(
async def db_iterate_records(
self,
start_timestamp: Optional[DatabaseTimestampType] = None,
end_timestamp: Optional[DatabaseTimestampType] = None,
) -> Iterator[T_Record]: ...
) -> AsyncIterator[T_Record]: ...
def db_load_records(
async def db_load_records(
self,
start_timestamp: Optional[DatabaseTimestampType] = None,
end_timestamp: Optional[DatabaseTimestampType] = None,
) -> int: ...
def db_delete_records(
async def db_delete_records(
self,
start_timestamp: Optional[DatabaseTimestampType] = None,
end_timestamp: Optional[DatabaseTimestampType] = None,
) -> int: ...
# ---- dirty tracking ----
def db_mark_dirty_record(self, record: T_Record) -> None: ...
async def db_mark_dirty_record(self, record: T_Record) -> None: ...
def db_save_records(self) -> int: ...
# ---- autosave ----
def db_autosave(self) -> int: ...
async def db_save_records(self) -> int: ...
# ---- Remove old records from database to free space ----
def db_vacuum(
async def db_vacuum(
self,
keep_hours: Optional[int] = None,
keep_datetime: Optional[DatabaseTimestampType] = None,
) -> int: ...
# ---- statistics about database storage ----
def db_count_records(self) -> int: ...
async def db_count_records(self) -> int: ...
def db_get_stats(self) -> dict: ...
async def db_get_stats(self) -> dict: ...
T_DatabaseRecordProtocol = TypeVar("T_DatabaseRecordProtocol", bound="DatabaseRecordProtocol")
@@ -533,10 +519,12 @@ class DatabaseRecordProtocolMixin(
Completely manages in memory records and database storage.
Expects records with date_time (DatabaseTimestamp) property and the a record list
Expects records with date_time (DatabaseTimestamp) property and a record list
in self.records of the derived class.
DatabaseRecordProtocolMixin expects the derived classes to be singletons.
DatabaseRecordProtocolMixin expects the derived classes to be singletons and to have
the sequence guarded against asynchronous sequence state (_sequence_lock) and
asynchronous record state (_record_lock) changes.
"""
# Tell mypy these attributes exist (will be provided by subclasses)
@@ -549,7 +537,7 @@ class DatabaseRecordProtocolMixin(
@property
def record_keys_writable(self) -> list[str]: ...
def key_to_array(
async def key_to_array(
self,
key: str,
start_datetime: Optional[DateTime] = None,
@@ -583,7 +571,18 @@ class DatabaseRecordProtocolMixin(
# Initialization
# -----------------------------------------------------
def _db_ensure_initialized(self) -> None:
async def _db_init_metadata(self) -> None:
"""Initialize DB metadata."""
self._db_metadata: Optional[dict] = {
"version": self._db_version,
"created": to_datetime(as_string=True),
"provider_id": getattr(self, "provider_id", lambda: "unknown")(),
"compression": self.database.compression,
"backend": self.database.__class__.__name__,
}
await self._db_save_metadata(self._db_metadata)
async def _db_ensure_initialized(self) -> None:
"""Initialize DB runtime state.
Idempotent — safe to call multiple times.
@@ -613,25 +612,18 @@ class DatabaseRecordProtocolMixin(
self._db_version: int = 1
# Storage
self._db_metadata: Optional[dict] = None
self._db_metadata = None
self._db_storage_initialized: bool = False
self._db_initialized: bool = True
if not self._db_storage_initialized and self.db_enabled:
# Metadata
existing_metadata = self._db_load_metadata()
existing_metadata = await self._db_load_metadata()
if existing_metadata:
self._db_metadata = existing_metadata
else:
self._db_metadata = {
"version": self._db_version,
"created": to_datetime(as_string=True),
"provider_id": getattr(self, "provider_id", lambda: "unknown")(),
"compression": self.database.compression,
"backend": self.database.__class__.__name__,
}
self._db_save_metadata(self._db_metadata)
await self._db_init_metadata()
logger.info(
f"Initialized {self.database.__class__.__name__}:{self.db_namespace()} storage at "
@@ -641,12 +633,6 @@ class DatabaseRecordProtocolMixin(
self._db_storage_initialized = True
def model_post_init(self, __context: Any) -> None:
"""Initialize DB state attributes immediately after Pydantic construction."""
# Always call super() first — other mixins may also define model_post_init
super().model_post_init(__context) # type: ignore[misc]
self._db_ensure_initialized()
# -----------------------------------------------------
# Helpers
# -----------------------------------------------------
@@ -669,7 +655,7 @@ class DatabaseRecordProtocolMixin(
db_datetime_after = DatabaseTimestamp.from_datetime(target.add(seconds=1))
return db_datetime_after
def db_previous_timestamp(
async def db_previous_timestamp(
self,
timestamp: DatabaseTimestamp,
) -> Optional[DatabaseTimestamp]:
@@ -677,7 +663,7 @@ class DatabaseRecordProtocolMixin(
Search memory-first, then fallback to database if necessary.
"""
self._db_ensure_initialized()
await self._db_ensure_initialized()
# Step 1: Memory-first search
if self._db_sorted_timestamps:
@@ -689,7 +675,7 @@ class DatabaseRecordProtocolMixin(
if not self.db_enabled:
return None
db_min_key, _ = self.database.get_key_range(self.db_namespace())
db_min_key, _ = await self.database.get_key_range(namespace=self.db_namespace())
if db_min_key is None:
return None
@@ -710,7 +696,7 @@ class DatabaseRecordProtocolMixin(
start_key = self._db_key_from_timestamp(loaded_start)
previous_ts: Optional[DatabaseTimestamp] = None
for key, _ in self.database.iterate_records(
async for key, _ in self.database.iterate_records(
start_key=start_key,
end_key=end_key,
namespace=self.db_namespace(),
@@ -722,7 +708,7 @@ class DatabaseRecordProtocolMixin(
return previous_ts
def db_next_timestamp(
async def db_next_timestamp(
self,
timestamp: DatabaseTimestamp,
) -> Optional[DatabaseTimestamp]:
@@ -730,7 +716,7 @@ class DatabaseRecordProtocolMixin(
Search memory-first, then fallback to database if necessary.
"""
self._db_ensure_initialized()
await self._db_ensure_initialized()
# Step 1: Memory-first search
if self._db_sorted_timestamps:
@@ -742,7 +728,7 @@ class DatabaseRecordProtocolMixin(
if not self.db_enabled:
return None
_, db_max_key = self.database.get_key_range(self.db_namespace())
_, db_max_key = await self.database.get_key_range(namespace=self.db_namespace())
if db_max_key is None:
return None
@@ -762,7 +748,7 @@ class DatabaseRecordProtocolMixin(
if isinstance(loaded_end, DatabaseTimestamp) and timestamp < loaded_end:
start_key = self._db_key_from_timestamp(max(timestamp, loaded_end))
for key, _ in self.database.iterate_records(
async for key, _ in self.database.iterate_records(
start_key=start_key,
end_key=end_key,
namespace=self.db_namespace(),
@@ -796,22 +782,22 @@ class DatabaseRecordProtocolMixin(
record_data = pickle.loads(data) # noqa: S301
return self.record_class()(**record_data)
def _db_save_metadata(self, metadata: dict) -> None:
async def _db_save_metadata(self, metadata: dict) -> None:
"""Save metadata to database."""
if not self.db_enabled:
return
key = DATABASE_METADATA_KEY
value = pickle.dumps(metadata)
self.database.set_metadata(value, namespace=self.db_namespace())
await self.database.set_metadata(value, namespace=self.db_namespace())
def _db_load_metadata(self) -> Optional[dict]:
async def _db_load_metadata(self) -> Optional[dict]:
"""Load metadata from database."""
if not self.db_enabled:
return None
try:
value = self.database.get_metadata(namespace=self.db_namespace())
value = await self.database.get_metadata(namespace=self.db_namespace())
return pickle.loads(value) # noqa: S301
except Exception:
logger.debug("Can not load metadata.")
@@ -942,7 +928,7 @@ class DatabaseRecordProtocolMixin(
return loaded_start <= start_timestamp and end_timestamp <= loaded_end
def _db_load_initial_window(
async def _db_load_initial_window(
self,
center_timestamp: Optional[DatabaseTimestampType] = None,
) -> None:
@@ -1001,11 +987,11 @@ class DatabaseRecordProtocolMixin(
window = to_duration(window_h * 3600)
start, end = self._search_window(center_timestamp, window)
self.db_load_records(start, end)
await self.db_load_records(start, end)
self._db_load_phase = DatabaseRecordProtocolLoadPhase.INITIAL
def _db_load_full(self) -> int:
async def _db_load_full(self) -> int:
"""Load all remaining records from the database into memory.
This method performs a **full load** of the database, ensuring that all
@@ -1038,14 +1024,14 @@ class DatabaseRecordProtocolMixin(
# Perform full database load (memory is authoritative; skips duplicates)
# This also sets _db_loaded_range
loaded_count = self.db_load_records()
loaded_count = await self.db_load_records()
# Update state
self._db_load_phase = DatabaseRecordProtocolLoadPhase.FULL
return loaded_count
def _extend_boundaries(
async def _extend_boundaries(
self,
start_timestamp: DatabaseTimestampType,
end_timestamp: DatabaseTimestampType,
@@ -1069,7 +1055,7 @@ class DatabaseRecordProtocolMixin(
):
# There may be earlier DB records
# Reverse iterate to get nearest smaller key
for key, _ in self.database.iterate_records(
async for key, _ in self.database.iterate_records(
start_key=UNBOUND_START,
end_key=self._db_key_from_timestamp(start_timestamp),
namespace=self.db_namespace(),
@@ -1091,7 +1077,7 @@ class DatabaseRecordProtocolMixin(
and end_timestamp > self._db_sorted_timestamps[-1]
):
# There may be later DB records
for key, _ in self.database.iterate_records(
async for key, _ in self.database.iterate_records(
start_key=self._db_key_from_timestamp(end_timestamp),
end_key=UNBOUND_END,
namespace=self.db_namespace(),
@@ -1107,7 +1093,7 @@ class DatabaseRecordProtocolMixin(
return new_start, new_end
def _db_ensure_loaded(
async def _db_ensure_loaded(
self,
start_timestamp: Optional[DatabaseTimestampType] = None,
end_timestamp: Optional[DatabaseTimestampType] = None,
@@ -1172,11 +1158,11 @@ class DatabaseRecordProtocolMixin(
# Phase 0: NOTHING LOADED
if self._db_load_phase is DatabaseRecordProtocolLoadPhase.NONE:
if start_timestamp is UNBOUND_START and end_timestamp is UNBOUND_END:
self._db_load_initial_window(center_timestamp)
await self._db_load_initial_window(center_timestamp)
# _db_load_initial_window sets _db_loaded_range and _db_load_phase
else:
# Load the records
loaded = self.db_load_records(start_timestamp, end_timestamp)
loaded = await self.db_load_records(start_timestamp, end_timestamp)
self._db_load_phase = DatabaseRecordProtocolLoadPhase.INITIAL
return
@@ -1196,7 +1182,7 @@ class DatabaseRecordProtocolMixin(
return # already have it
if start_timestamp == UNBOUND_START and end_timestamp == UNBOUND_END:
self._db_load_full()
await self._db_load_full()
return
current_start, current_end = self._db_loaded_range
@@ -1207,11 +1193,11 @@ class DatabaseRecordProtocolMixin(
# Left expansion
if start_timestamp < current_start:
self.db_load_records(start_timestamp, current_start)
await self.db_load_records(start_timestamp, current_start)
# Right expansion
if end_timestamp > current_end:
self.db_load_records(current_end, end_timestamp)
await self.db_load_records(current_end, end_timestamp)
return
@@ -1251,15 +1237,15 @@ class DatabaseRecordProtocolMixin(
def db_enabled(self) -> bool:
return self.database.is_open
def db_timestamp_range(
async def db_timestamp_range(
self,
) -> tuple[Optional[DatabaseTimestamp], Optional[DatabaseTimestamp]]:
"""Get the timestamp range of records in database.
Regards records in storage plus extra records in memory.
"""
# Defensive call - model_post_init() may not have initialized metadata
self._db_ensure_initialized()
# Ensure db in memory data and metadata is initialized
await self._db_ensure_initialized()
if self._db_sorted_timestamps:
memory_min_timestamp: Optional[DatabaseTimestamp] = self._db_sorted_timestamps[0]
@@ -1271,7 +1257,7 @@ class DatabaseRecordProtocolMixin(
if not self.db_enabled:
return memory_min_timestamp, memory_max_timestamp
db_min_key, db_max_key = self.database.get_key_range(self.db_namespace())
db_min_key, db_max_key = await self.database.get_key_range(namespace=self.db_namespace())
if db_min_key is None or db_max_key is None:
return memory_min_timestamp, memory_max_timestamp
@@ -1331,7 +1317,7 @@ class DatabaseRecordProtocolMixin(
yield DatabaseTimestamp.from_datetime(current_utc)
current_utc = current_utc.add(seconds=step_seconds)
def db_get_record(
async def db_get_record(
self,
target_timestamp: DatabaseTimestamp,
*,
@@ -1353,11 +1339,11 @@ class DatabaseRecordProtocolMixin(
Returns:
Exact match, nearest record within the window, or None.
"""
self._db_ensure_initialized()
await self._db_ensure_initialized()
if time_window is None:
# Exact match only — load the minimal range containing this point
self._db_ensure_loaded(
await self._db_ensure_loaded(
target_timestamp,
self._db_timestamp_after(target_timestamp),
center_timestamp=target_timestamp,
@@ -1367,7 +1353,7 @@ class DatabaseRecordProtocolMixin(
# load the relevant range
# in case of unbounded escalates to FULL
search_start, search_end = self._search_window(target_timestamp, time_window)
self._db_ensure_loaded(search_start, search_end, center_timestamp=target_timestamp)
await self._db_ensure_loaded(search_start, search_end, center_timestamp=target_timestamp)
# Exact match first (works for all three cases once loaded)
record = self._db_record_index.get(target_timestamp, None)
@@ -1406,19 +1392,19 @@ class DatabaseRecordProtocolMixin(
return record
def db_insert_record(
async def db_insert_record(
self,
record: T_Record,
*,
mark_dirty: bool = True,
) -> None:
# Defensive call - model_post_init() may not have initialized metadata
self._db_ensure_initialized()
# Ensure db in memory data and metadata is initialized
await self._db_ensure_initialized()
# Ensure normalized to UTC
db_record_date_time = DatabaseTimestamp.from_datetime(record.date_time)
self._db_ensure_loaded(
await self._db_ensure_loaded(
start_timestamp=db_record_date_time,
end_timestamp=db_record_date_time,
)
@@ -1446,7 +1432,7 @@ class DatabaseRecordProtocolMixin(
# Load (range)
# -----------------------------------------------------
def db_load_records(
async def db_load_records(
self,
start_timestamp: Optional[DatabaseTimestampType] = None,
end_timestamp: Optional[DatabaseTimestampType] = None,
@@ -1475,8 +1461,8 @@ class DatabaseRecordProtocolMixin(
Note:
record.date_time shall be DateTime or None
"""
# Defensive call - model_post_init() may not have initialized metadata
self._db_ensure_initialized()
# Ensure db in memory data and metadata is initialized
await self._db_ensure_initialized()
if not self.db_enabled:
return 0
@@ -1488,7 +1474,7 @@ class DatabaseRecordProtocolMixin(
end_timestamp = UNBOUND_END
# Extend boundaries to include first record < start and first record >= end
query_start, query_end = self._extend_boundaries(start_timestamp, end_timestamp)
query_start, query_end = await self._extend_boundaries(start_timestamp, end_timestamp)
if isinstance(query_start, _DatabaseTimestampUnbound):
start_key = None
@@ -1504,7 +1490,7 @@ class DatabaseRecordProtocolMixin(
loaded_count = 0
# Iterate DB records (already sorted by key)
for db_key, value in self.database.iterate_records(
async for db_key, value in self.database.iterate_records(
start_key=start_key,
end_key=end_key,
namespace=namespace,
@@ -1551,16 +1537,16 @@ class DatabaseRecordProtocolMixin(
# Delete (range)
# -----------------------------------------------------
def db_delete_records(
async def db_delete_records(
self,
start_timestamp: Optional[DatabaseTimestampType] = None,
end_timestamp: Optional[DatabaseTimestampType] = None,
) -> int:
# Defensive call - model_post_init() may not have initialized metadata
self._db_ensure_initialized()
# Ensure db in memory data and metadata is initialized
await self._db_ensure_initialized()
# Deletion is global — ensure we see everything
self._db_ensure_loaded(
await self._db_ensure_loaded(
start_timestamp=start_timestamp,
end_timestamp=end_timestamp,
)
@@ -1598,21 +1584,21 @@ class DatabaseRecordProtocolMixin(
# Iteration from DB (no duplicates)
# -----------------------------------------------------
def db_iterate_records(
async def db_iterate_records(
self,
start_timestamp: Optional[DatabaseTimestampType] = None,
end_timestamp: Optional[DatabaseTimestampType] = None,
) -> Iterator[T_Record]:
) -> AsyncIterator[T_Record]:
"""Iterate records in requested range.
Ensures storage is loaded into memory first,
then iterates over in-memory records only.
"""
# Defensive call - model_post_init() may not have initialized metadata
self._db_ensure_initialized()
# Ensure db in memory data and metadata is initialized
await self._db_ensure_initialized()
# Ensure memory contains required range
self._db_ensure_loaded(
await self._db_ensure_loaded(
start_timestamp=start_timestamp,
end_timestamp=end_timestamp,
)
@@ -1635,9 +1621,9 @@ class DatabaseRecordProtocolMixin(
# Dirty tracking
# -----------------------------------------------------
def db_mark_dirty_record(self, record: T_Record) -> None:
# Defensive call - model_post_init() may not have initialized metadata
self._db_ensure_initialized()
async def db_mark_dirty_record(self, record: T_Record) -> None:
# Ensure db in memory data and metadata is initialized
await self._db_ensure_initialized()
record_date_time_timestamp = DatabaseTimestamp.from_datetime(record.date_time)
self._db_dirty_timestamps.add(record_date_time_timestamp)
@@ -1646,9 +1632,9 @@ class DatabaseRecordProtocolMixin(
# Bulk save (flush dirty only)
# -----------------------------------------------------
def db_save_records(self) -> int:
# Defensive call - model_post_init() may not have initialized metadata
self._db_ensure_initialized()
async def db_save_records(self) -> int:
# Ensure db in memory data and metadata is initialized
await self._db_ensure_initialized()
if not self.db_enabled:
return 0
@@ -1670,23 +1656,20 @@ class DatabaseRecordProtocolMixin(
save_items.append((key, value))
saved_count = len(save_items)
if saved_count:
self.database.save_records(save_items, namespace=namespace)
await self.database.save_records(save_items, namespace=namespace)
self._db_dirty_timestamps.clear()
self._db_new_timestamps.clear()
# --- handle deletions ---
if self._db_deleted_timestamps:
delete_keys = [self._db_key_from_timestamp(dt) for dt in self._db_deleted_timestamps]
self.database.delete_records(delete_keys, namespace=namespace)
await self.database.delete_records(delete_keys, namespace=namespace)
deleted_count = len(self._db_deleted_timestamps)
self._db_deleted_timestamps.clear()
return saved_count + deleted_count
def db_autosave(self) -> int:
return self.db_save_records()
def db_vacuum(
async def db_vacuum(
self,
keep_hours: Optional[int] = None,
keep_timestamp: Optional[DatabaseTimestampType] = None,
@@ -1708,8 +1691,8 @@ class DatabaseRecordProtocolMixin(
Returns:
Number of records deleted
"""
# Defensive call - model_post_init() may not have initialized metadata
self._db_ensure_initialized()
# Ensure db in memory data and metadata is initialized
await self._db_ensure_initialized()
if keep_hours is None and keep_timestamp is None:
keep_duration = self.db_keep_duration()
@@ -1722,7 +1705,7 @@ class DatabaseRecordProtocolMixin(
keep_hours = keep_duration.hours
if keep_hours is not None:
_, db_max = self.db_timestamp_range()
_, db_max = await self.db_timestamp_range()
if db_max is None or isinstance(db_max, _DatabaseTimestampUnbound):
# No records
return 0 # nothing to delete
@@ -1740,9 +1723,9 @@ class DatabaseRecordProtocolMixin(
raise ValueError("Must specify either keep_hours or keep_timestamp")
# Delete records
deleted_count = self.db_delete_records(end_timestamp=db_cutoff_timestamp)
deleted_count = await self.db_delete_records(end_timestamp=db_cutoff_timestamp)
self.db_save_records()
await self.db_save_records()
logger.info(
f"Vacuumed {deleted_count} old records from database '{self.db_namespace()}' "
@@ -1750,14 +1733,14 @@ class DatabaseRecordProtocolMixin(
)
return deleted_count
def db_count_records(self) -> int:
async def db_count_records(self) -> int:
"""Return total logical number of records.
Memory is authoritative. If DB is enabled but not fully loaded,
we conservatively include storage-only records.
"""
# Defensive call - model_post_init() may not have initialized metadata
self._db_ensure_initialized()
# Ensure db in memory data and metadata is initialized
await self._db_ensure_initialized()
if not self.db_enabled:
return len(self.records)
@@ -1766,13 +1749,13 @@ class DatabaseRecordProtocolMixin(
if self._db_load_phase is DatabaseRecordProtocolLoadPhase.FULL:
return len(self.records)
storage_count = self.database.count_records(namespace=self.db_namespace())
storage_count = await self.database.count_records(namespace=self.db_namespace())
pending_deletes = len(self._db_deleted_timestamps)
new_count = len(self._db_new_timestamps)
return storage_count + new_count - pending_deletes
def db_get_stats(self) -> dict:
async def db_get_stats(self) -> dict:
"""Get comprehensive statistics about database storage.
Returns:
@@ -1783,6 +1766,8 @@ class DatabaseRecordProtocolMixin(
ns = self.db_namespace()
total_records = await self.database.count_records(namespace=ns)
stats = {
"enabled": True,
"backend": self.database.__class__.__name__,
@@ -1791,13 +1776,14 @@ class DatabaseRecordProtocolMixin(
"compression_enabled": self.database.compression,
"keep_duration_h": self.config.database.keep_duration_h,
"autosave_interval_sec": self.config.database.autosave_interval_sec,
"total_records": self.database.count_records(namespace=ns),
"total_records": total_records,
}
# Add backend-specific stats
stats.update(self.database.get_backend_stats(namespace=ns))
backend_stats = await self.database.get_backend_stats(namespace=ns)
stats.update(backend_stats)
min_timestamp, max_timestamp = self.db_timestamp_range()
min_timestamp, max_timestamp = await self.db_timestamp_range()
stats["timestamp_range"] = {
"min": str(min_timestamp),
"max": str(max_timestamp),
@@ -1866,7 +1852,7 @@ class DatabaseRecordProtocolMixin(
cutoff_str = self._db_metadata.get(key)
return DatabaseTimestamp(cutoff_str) if cutoff_str else None
def _db_set_compact_state(
async def _db_set_compact_state(
self,
tier_interval: Duration,
cutoff_ts: DatabaseTimestamp,
@@ -1881,13 +1867,13 @@ class DatabaseRecordProtocolMixin(
self._db_metadata = {}
key = f"last_compact_cutoff_{int(tier_interval.total_seconds())}"
self._db_metadata[key] = str(cutoff_ts)
self._db_save_metadata(self._db_metadata)
await self._db_save_metadata(self._db_metadata)
# ------------------------------------------------------------------
# Single-tier worker
# ------------------------------------------------------------------
def _db_compact_tier(
async def _db_compact_tier(
self,
age_threshold: Duration,
target_interval: Duration,
@@ -1923,14 +1909,14 @@ class DatabaseRecordProtocolMixin(
Number of original records deleted (before re-insertion of downsampled
records). Returns 0 if skipped.
"""
self._db_ensure_initialized()
await self._db_ensure_initialized()
interval_sec = int(target_interval.total_seconds())
if interval_sec <= 0:
return 0
# ---- Determine raw new cutoff ------------------------------------
_, db_max = self.db_timestamp_range()
_, db_max = await self.db_timestamp_range()
if db_max is None or isinstance(db_max, _DatabaseTimestampUnbound):
return 0
@@ -1955,7 +1941,7 @@ class DatabaseRecordProtocolMixin(
)
return 0
db_min, _ = self.db_timestamp_range()
db_min, _ = await self.db_timestamp_range()
if db_min is None or isinstance(db_min, _DatabaseTimestampUnbound):
return 0
@@ -1981,7 +1967,11 @@ class DatabaseRecordProtocolMixin(
window_end_ts = new_cutoff_ts
# ---- Sparse-data guard -------------------------------------------
existing_count = self.database.count_records(
# Ensure the window is loaded into memory so the sparse guard's
# records_in_window list comprehension sees actual data.
await self._db_ensure_loaded(window_start_ts, window_end_ts)
existing_count = await self.database.count_records(
start_key=self._db_key_from_timestamp(window_start_ts),
end_key=self._db_key_from_timestamp(window_end_ts),
namespace=self.db_namespace(),
@@ -1993,7 +1983,7 @@ class DatabaseRecordProtocolMixin(
if existing_count == 0:
# Nothing in window — just advance the cutoff
self._db_set_compact_state(target_interval, new_cutoff_ts)
await self._db_set_compact_state(target_interval, new_cutoff_ts)
return 0
if existing_count <= resampled_count:
@@ -2016,7 +2006,7 @@ class DatabaseRecordProtocolMixin(
f"and all timestamps already aligned "
f"(window={window_start_dt}..{window_end_dt})"
)
self._db_set_compact_state(target_interval, new_cutoff_ts)
await self._db_set_compact_state(target_interval, new_cutoff_ts)
return 0
# ---- Sparse but misaligned: full window rewrite -----------------
@@ -2049,7 +2039,7 @@ class DatabaseRecordProtocolMixin(
bucket[key] = val
# Delete entire window (aligned + misaligned)
deleted = self.db_delete_records(
deleted = await self.db_delete_records(
start_timestamp=window_start_ts,
end_timestamp=window_end_ts,
)
@@ -2060,10 +2050,10 @@ class DatabaseRecordProtocolMixin(
continue
snapped_dt = DateTime.fromtimestamp(snapped_epoch, tz="UTC")
record = self.record_class()(date_time=snapped_dt, **values)
self.db_insert_record(record, mark_dirty=True)
await self.db_insert_record(record, mark_dirty=True)
self.db_save_records()
self._db_set_compact_state(target_interval, new_cutoff_ts)
await self.db_save_records()
await self._db_set_compact_state(target_interval, new_cutoff_ts)
logger.info(
f"Rewrote sparse window in namespace '{self.db_namespace()}' "
f"tier {target_interval}: deleted={deleted}, "
@@ -2086,7 +2076,7 @@ class DatabaseRecordProtocolMixin(
if key == "date_time":
continue
try:
array = self.key_to_array(
array = await self.key_to_array(
key,
start_datetime=window_start_dt,
end_datetime=window_end_dt,
@@ -2095,7 +2085,9 @@ class DatabaseRecordProtocolMixin(
boundary="context",
align_to_interval=True,
)
except (KeyError, TypeError, ValueError):
logger.debug(f"key={key}, array_len={len(array)}")
except (KeyError, TypeError, ValueError) as e:
logger.error(f"key_to_array failed for {key}: {e}")
continue # non-numeric or missing key — skip silently
if len(array) == 0:
@@ -2125,11 +2117,11 @@ class DatabaseRecordProtocolMixin(
if not compacted_data or not compacted_timestamps:
# Nothing to write back — still advance cutoff
self._db_set_compact_state(target_interval, new_cutoff_ts)
await self._db_set_compact_state(target_interval, new_cutoff_ts)
return 0
# ---- Delete originals, re-insert downsampled records -------------
deleted = self.db_delete_records(
deleted = await self.db_delete_records(
start_timestamp=window_start_ts,
end_timestamp=window_end_ts,
)
@@ -2142,12 +2134,12 @@ class DatabaseRecordProtocolMixin(
}
if values:
record = self.record_class()(date_time=dt, **values)
self.db_insert_record(record, mark_dirty=True)
await self.db_insert_record(record, mark_dirty=True)
self.db_save_records()
await self.db_save_records()
# Persist the aligned new cutoff for this tier
self._db_set_compact_state(target_interval, new_cutoff_ts)
await self._db_set_compact_state(target_interval, new_cutoff_ts)
logger.info(
f"Compacted tier {target_interval}: deleted {deleted} records in "
@@ -2161,7 +2153,7 @@ class DatabaseRecordProtocolMixin(
# Public entry point
# ------------------------------------------------------------------
def db_compact(
async def db_compact(
self,
compact_tiers: Optional[list[tuple[Duration, Duration]]] = None,
) -> int:
@@ -2183,12 +2175,13 @@ class DatabaseRecordProtocolMixin(
compact_tiers = self.db_compact_tiers()
if not compact_tiers:
logger.debug(f"Compaction called but no compact_tiers '{compact_tiers}' given.")
return 0
total_deleted = 0
# Coarsest tier first (reversed) to avoid redundant work
for age_threshold, target_interval in reversed(compact_tiers):
total_deleted += self._db_compact_tier(age_threshold, target_interval)
total_deleted += await self._db_compact_tier(age_threshold, target_interval)
return total_deleted

386
src/akkudoktoreos/core/ems.py
View File

@@ -2,8 +2,7 @@ import traceback
from asyncio import Lock, get_running_loop
from concurrent.futures import ThreadPoolExecutor
from enum import StrEnum
from functools import partial
from typing import ClassVar, Optional
from typing import ClassVar, Optional, cast
from loguru import logger
from pydantic import computed_field
@@ -147,11 +146,11 @@ class EnergyManagement(
"""
return cls._genetic_solution
@classmethod
def _run(
cls,
start_datetime: DateTime,
mode: EnergyManagementMode,
async def run(
self,
start_datetime: Optional[DateTime] = None,
mode: Optional[EnergyManagementMode] = None,
algorithm: Optional[str] = None,
genetic_parameters: Optional[GeneticOptimizationParameters] = None,
genetic_individuals: Optional[int] = None,
genetic_seed: Optional[int] = None,
@@ -161,7 +160,6 @@ class EnergyManagement(
"""Run the energy management.
This method initializes the energy management run by setting its
start datetime, updating predictions, and optionally starting
optimization depending on the selected mode or configuration.
@@ -171,161 +169,20 @@ class EnergyManagement(
- "OPTIMIZATION": Runs the optimization process.
- "PREDICTION": Updates the forecast without optimization.
- "DISABLED": Does not run.
genetic_parameters (GeneticOptimizationParameters, optional): The
parameter set for the genetic algorithm. If not provided, it will
be constructed based on the current configuration and predictions.
genetic_individuals (int, optional): The number of individuals for the
genetic algorithm. Defaults to the algorithm's internal default (400)
if not specified.
genetic_seed (int, optional): The seed for the genetic algorithm. Defaults
to the algorithm's internal random seed if not specified.
force_enable (bool, optional): If True, bypasses any disabled state
to force the update process. This is mostly applicable to
prediction providers.
force_update (bool, optional): If True, forces data to be refreshed
even if a cached version is still valid.
Returns:
None
"""
# Ensure there is only one optimization/ energy management run at a time
if not mode in EnergyManagementMode._value2member_map_:
raise ValueError(f"Unknown energy management mode {mode}.")
if mode == EnergyManagementMode.DISABLED:
return
logger.info("Starting energy management run.")
cls._stage = EnergyManagementStage.DATA_ACQUISITION
# Remember/ set the start datetime of this energy management run.
# None leads
cls.set_start_datetime(start_datetime)
# Throw away any memory cached results of the last energy management run.
CacheEnergyManagementStore().clear()
# Do data aquisition by adapters
try:
cls.adapter.update_data(force_enable)
except Exception as e:
trace = "".join(traceback.TracebackException.from_exception(e).format())
error_msg = f"Adapter update failed - phase {cls._stage}:\n{e}\n{trace}"
logger.error(error_msg)
cls._stage = EnergyManagementStage.FORECAST_RETRIEVAL
if mode == EnergyManagementMode.PREDICTION:
# Update the predictions
cls.prediction.update_data(force_enable=force_enable, force_update=force_update)
logger.info("Energy management run done (predictions updated)")
cls._stage = EnergyManagementStage.IDLE
return
# Prepare optimization parameters
# This also creates default configurations for missing values and updates the predictions
logger.info(
"Starting energy management prediction update and optimzation parameter preparation."
)
if genetic_parameters is None:
genetic_parameters = GeneticOptimizationParameters.prepare()
if not genetic_parameters:
logger.error(
"Energy management run canceled. Could not prepare optimisation parameters."
)
cls._stage = EnergyManagementStage.IDLE
return
cls._stage = EnergyManagementStage.OPTIMIZATION
logger.info("Starting energy management optimization.")
# Take values from config if not given
if genetic_individuals is None:
genetic_individuals = cls.config.optimization.genetic.individuals
if genetic_seed is None:
genetic_seed = cls.config.optimization.genetic.seed
if cls._start_datetime is None: # Make mypy happy - already set by us
raise RuntimeError("Start datetime not set.")
try:
optimization = GeneticOptimization(
verbose=bool(cls.config.server.verbose),
fixed_seed=genetic_seed,
)
solution = optimization.optimierung_ems(
start_hour=cls._start_datetime.hour,
parameters=genetic_parameters,
ngen=genetic_individuals,
)
except:
logger.exception("Energy management optimization failed.")
cls._stage = EnergyManagementStage.IDLE
return
cls._stage = EnergyManagementStage.CONTROL_DISPATCH
# Make genetic solution public
cls._genetic_solution = solution
# Make optimization solution public
cls._optimization_solution = solution.optimization_solution()
# Make plan public
cls._plan = solution.energy_management_plan()
logger.debug("Energy management genetic solution:\n{}", cls._genetic_solution)
logger.debug("Energy management optimization solution:\n{}", cls._optimization_solution)
logger.debug("Energy management plan:\n{}", cls._plan)
logger.info("Energy management run done (optimization updated)")
# Do control dispatch by adapters
try:
cls.adapter.update_data(force_enable)
except Exception as e:
trace = "".join(traceback.TracebackException.from_exception(e).format())
error_msg = f"Adapter update failed - phase {cls._stage}:\n{e}\n{trace}"
logger.error(error_msg)
# Remember energy run datetime.
EnergyManagement._last_run_datetime = to_datetime()
# energy management run finished
cls._stage = EnergyManagementStage.IDLE
async def run(
self,
start_datetime: Optional[DateTime] = None,
mode: Optional[EnergyManagementMode] = None,
genetic_parameters: Optional[GeneticOptimizationParameters] = None,
genetic_individuals: Optional[int] = None,
genetic_seed: Optional[int] = None,
force_enable: Optional[bool] = False,
force_update: Optional[bool] = False,
) -> None:
"""Run the energy management.
This method initializes the energy management run by setting its
start datetime, updating predictions, and optionally starting
optimization depending on the selected mode or configuration.
Args:
start_datetime (DateTime, optional): The starting timestamp
of the energy management run. Defaults to the current datetime
if not provided.
mode (EnergyManagementMode, optional): The management mode to use. Must be one of:
- "OPTIMIZATION": Runs the optimization process.
- "PREDICTION": Updates the forecast without optimization.
Defaults to the mode defined in the current configuration.
algorithm (str, optional):
The algorithm to use. Must be one of:
- "GENETIC": Optimization uses the `GENETIC` optimization algorithm.
Defaults to the algorithm defined in the current configuration.
genetic_parameters (GeneticOptimizationParameters, optional): The
parameter set for the genetic algorithm. If not provided, it will
parameter set for the `GENETIC` algorithm. If not provided, it will
be constructed based on the current configuration and predictions.
genetic_individuals (int, optional): The number of individuals for the
genetic algorithm. Defaults to the algorithm's internal default (400)
`GENETIC` algorithm. Defaults to the algorithm's internal default (400)
if not specified.
genetic_seed (int, optional): The seed for the genetic algorithm. Defaults
genetic_seed (int, optional): The seed for the `GENETIC` algorithm. Defaults
to the algorithm's internal random seed if not specified.
force_enable (bool, optional): If True, bypasses any disabled state
to force the update process. This is mostly applicable to
@@ -336,22 +193,205 @@ class EnergyManagement(
Returns:
None
"""
async with self._run_lock:
loop = get_running_loop()
# Create a partial function with parameters "baked in"
if start_datetime is None:
start_datetime = to_datetime()
async with EnergyManagement._run_lock:
if mode is None:
mode = self.config.ems.mode
func = partial(
EnergyManagement._run,
start_datetime=start_datetime,
mode=mode,
genetic_parameters=genetic_parameters,
genetic_individuals=genetic_individuals,
genetic_seed=genetic_seed,
force_enable=force_enable,
force_update=force_update,
if mode not in EnergyManagementMode._value2member_map_:
raise ValueError(f"Unknown energy management mode {mode}.")
if mode == EnergyManagementMode.DISABLED:
logger.info("Energy management run disabled.")
return
logger.info("Starting energy management run.")
# --- Data Aquisition ---
EnergyManagement._stage = EnergyManagementStage.DATA_ACQUISITION
# Remember/ set the start datetime of this energy management run.
# None leads to current time as start datetime
self.set_start_datetime(start_datetime)
# Throw away any memory cached results of the last energy management run.
CacheEnergyManagementStore().clear()
# --- Adapter update ---
try:
await self.adapter.update_data(force_enable)
except Exception as e:
trace = "".join(traceback.TracebackException.from_exception(e).format())
error_msg = (
f"Adapter update failed - phase {EnergyManagement._stage}:\n{e}\n{trace}"
)
logger.error(error_msg)
# --- Prediction ---
EnergyManagement._stage = EnergyManagementStage.FORECAST_RETRIEVAL
# Update the predictions
logger.info("Starting energy management prediction update.")
await self.prediction.update_data(force_enable=force_enable, force_update=force_update)
if mode == EnergyManagementMode.PREDICTION:
logger.info("Energy management run done (predictions updated)")
EnergyManagement._stage = EnergyManagementStage.IDLE
return
# --- Optimization ---
EnergyManagement._stage = EnergyManagementStage.OPTIMIZATION
optimization_start = to_datetime()
logger.info("Starting energy management optimization.")
if algorithm is None:
algorithm = self.config.optimization.algorithm
if algorithm == "GENETIC":
# Prepare optimization parameters
# This also creates default configurations for missing values and updates the predictions
logger.info("Starting optimzation parameter preparation.")
if genetic_parameters is None:
genetic_parameters = await GeneticOptimizationParameters.prepare()
if genetic_parameters is None:
logger.error(
"Energy management run canceled. Could not prepare optimisation parameters."
)
EnergyManagement._stage = EnergyManagementStage.IDLE
return
# Take values from config if not given
if genetic_individuals is None:
genetic_individuals = self.config.optimization.genetic.individuals
if genetic_seed is None:
genetic_seed = self.config.optimization.genetic.seed
if EnergyManagement._start_datetime is None: # Make mypy happy - already set by us
raise RuntimeError("Start datetime not set.")
# --- Optimization (CPU-bound → MUST offload) ---
try:
optimization = GeneticOptimization(
verbose=bool(self.config.server.verbose),
fixed_seed=genetic_seed,
)
loop = get_running_loop()
start_hour = EnergyManagement._start_datetime.hour
solution = await loop.run_in_executor(
None,
lambda: optimization.optimierung_ems(
start_hour=start_hour,
parameters=cast(
GeneticOptimizationParameters, genetic_parameters
), # cast for mypy
ngen=genetic_individuals,
),
)
except Exception:
logger.exception("Energy management optimization failed.")
EnergyManagement._stage = EnergyManagementStage.IDLE
return
else:
logger.error(f"Unknown optimization algorithm: '{algorithm}'. Skipping.")
EnergyManagement._stage = EnergyManagementStage.IDLE
return
optimization_duration = to_datetime() - optimization_start
logger.info(
"Energy management optimization ({}) completed in {:.1f} seconds.",
algorithm,
optimization_duration.total_seconds(),
)
# Run optimization in background thread to avoid blocking event loop
await loop.run_in_executor(executor, func)
logger.debug(
"Energy management optimization solution:\n{}",
EnergyManagement._optimization_solution,
)
logger.debug("Energy management plan:\n{}", EnergyManagement._plan)
# --- Control dispatch by adapters ---
EnergyManagement._stage = EnergyManagementStage.CONTROL_DISPATCH
# Make genetic solution public
EnergyManagement._genetic_solution = solution
# Make optimization solution public
EnergyManagement._optimization_solution = await solution.optimization_solution()
# Make plan public
EnergyManagement._plan = solution.energy_management_plan()
logger.debug(
"Energy management genetic solution:\n{}", EnergyManagement._genetic_solution
)
if genetic_parameters is None:
genetic_parameters = await GeneticOptimizationParameters.prepare()
if not genetic_parameters:
logger.error("Energy management run canceled. Could not prepare parameters.")
EnergyManagement._stage = EnergyManagementStage.IDLE
return
EnergyManagement._stage = EnergyManagementStage.OPTIMIZATION
if genetic_individuals is None:
genetic_individuals = self.config.optimization.genetic.individuals
if genetic_seed is None:
genetic_seed = self.config.optimization.genetic.seed
if EnergyManagement._start_datetime is None:
raise RuntimeError("Start datetime not set.")
# --- Optimization (CPU-bound → MUST offload) ---
try:
optimization = GeneticOptimization(
verbose=bool(self.config.server.verbose),
fixed_seed=genetic_seed,
)
loop = get_running_loop()
start_hour = EnergyManagement._start_datetime.hour
solution = await loop.run_in_executor(
None,
lambda: optimization.optimierung_ems(
start_hour=start_hour,
parameters=genetic_parameters,
ngen=genetic_individuals,
),
)
except Exception:
logger.exception("Energy management optimization failed.")
EnergyManagement._stage = EnergyManagementStage.IDLE
return
EnergyManagement._genetic_solution = solution
EnergyManagement._optimization_solution = await solution.optimization_solution()
EnergyManagement._plan = solution.energy_management_plan()
logger.debug("Genetic solution:\n{}", EnergyManagement._genetic_solution)
logger.debug("Optimization solution:\n{}", EnergyManagement._optimization_solution)
logger.debug("Plan:\n{}", EnergyManagement._plan)
logger.info("Energy management run done (optimization updated)")
# --- Dispatch control by adapters ---
EnergyManagement._stage = EnergyManagementStage.CONTROL_DISPATCH
# Dispatch (sync → optionally offload)
try:
await self.adapter.update_data(force_enable)
except Exception as e:
trace = "".join(traceback.TracebackException.from_exception(e).format())
error_msg = (
f"Adapter update failed - phase {EnergyManagement._stage}:\n{e}\n{trace}"
)
logger.error(error_msg)
# --- Idle ---
# Remember energy run datetime.
EnergyManagement._last_run_datetime = to_datetime()
# energy management run finished
EnergyManagement._stage = EnergyManagementStage.IDLE

6
src/akkudoktoreos/core/pydantic.py
View File

@@ -543,10 +543,10 @@ class PydanticModelNestedValueMixin:
if not inspect.isclass(model):
raise TypeError(f"Model '{model}' is not of class type.")
if key not in model.model_fields:
if key not in model.model_fields: # type: ignore[attr-defined]
raise TypeError(f"Field '{key}' does not exist in model '{model.__name__}'.")
field_annotation = model.model_fields[key].annotation
field_annotation = model.model_fields[key].annotation # type: ignore[attr-defined]
if not field_annotation:
raise TypeError(
f"Missing type annotation for field '{key}' in model '{model.__name__}'."
@@ -692,7 +692,7 @@ class PydanticBaseModel(PydanticModelNestedValueMixin, BaseModel):
return super().model_dump(*args, **kwargs)
def to_dict(self) -> dict:
"""Convert this PredictionRecord instance to a dictionary representation.
"""Convert this pydantic model instance to a dictionary representation.
Returns:
dict: A dictionary where the keys are the field names of the PydanticBaseModel,

24
src/akkudoktoreos/measurement/measurement.py
View File

@@ -149,7 +149,7 @@ class Measurement(SingletonMixin, DataImportMixin, DataSequence):
# Return ceiling of division to include partial intervals
return int(np.ceil(diff_seconds / interval_seconds))
def _energy_from_meter_readings(
async def _energy_from_meter_readings(
self,
key: str,
start_datetime: DateTime,
@@ -170,7 +170,7 @@ class Measurement(SingletonMixin, DataImportMixin, DataSequence):
"""
size = self._interval_count(start_datetime, end_datetime, interval)
energy_mr_array = self.key_to_array(
energy_mr_array = await self.key_to_array(
key=key,
start_datetime=start_datetime,
end_datetime=end_datetime + interval,
@@ -203,7 +203,7 @@ class Measurement(SingletonMixin, DataImportMixin, DataSequence):
logger.debug(debug_msg)
return energy_array
def load_total_kwh(
async def load_total_kwh(
self,
start_datetime: Optional[DateTime] = None,
end_datetime: Optional[DateTime] = None,
@@ -232,9 +232,11 @@ class Measurement(SingletonMixin, DataImportMixin, DataSequence):
return np.zeros(size)
if start_datetime is None:
start_datetime = self.min_datetime
start_datetime = await self.min_datetime()
if end_datetime is None:
end_datetime = self.max_datetime.add(seconds=1)
end_datetime = await self.max_datetime()
if end_datetime:
end_datetime = end_datetime.add(seconds=1)
size = self._interval_count(start_datetime, end_datetime, interval)
load_total_kwh_array = np.zeros(size)
@@ -242,7 +244,7 @@ class Measurement(SingletonMixin, DataImportMixin, DataSequence):
if isinstance(self.config.measurement.load_emr_keys, list):
for key in self.config.measurement.load_emr_keys:
# Calculate load per interval
load_array = self._energy_from_meter_readings(
load_array = await self._energy_from_meter_readings(
key=key,
start_datetime=start_datetime,
end_datetime=end_datetime,
@@ -270,14 +272,14 @@ class Measurement(SingletonMixin, DataImportMixin, DataSequence):
"""
return to_datetime().subtract(hours=self.config.measurement.historic_hours)
def save(self) -> bool:
async def save(self) -> bool:
"""Save the measurements to persistent storage.
Returns:
True in case the measurements were saved, False otherwise.
"""
# Use db storage if available
saved_to_db = DataSequence.save(self)
saved_to_db = await DataSequence.save(self)
if not saved_to_db:
measurement_file_path = self._measurement_file_path()
if measurement_file_path is None:
@@ -292,14 +294,14 @@ class Measurement(SingletonMixin, DataImportMixin, DataSequence):
logger.exception("Cannot save measurements")
return True
def load(self) -> bool:
async def load(self) -> bool:
"""Load measurements from persistent storage.
Returns:
True in case the measurements were loaded, False otherwise.
"""
# Use db storage if available
loaded_from_db = DataSequence.load(self)
loaded_from_db = await DataSequence.load(self)
if not loaded_from_db:
measurement_file_path = self._measurement_file_path()
if measurement_file_path is None:
@@ -314,7 +316,7 @@ class Measurement(SingletonMixin, DataImportMixin, DataSequence):
# Explicitly add data records to the existing singleton
for record in loaded.records:
self.insert_by_datetime(record)
await self.insert_by_datetime(record)
except Exception as e:
logger.exception("Cannot load measurements")
return True

18
src/akkudoktoreos/optimization/genetic/geneticparams.py
View File

@@ -147,7 +147,7 @@ class GeneticOptimizationParameters(
return start_solution
@classmethod
def prepare(cls) -> "Optional[GeneticOptimizationParameters]":
async def prepare(cls) -> "Optional[GeneticOptimizationParameters]":
"""Prepare optimization parameters from config, forecast and measurement data.
Fills in values needed for optimization from available configuration, predictions and
@@ -231,11 +231,11 @@ class GeneticOptimizationParameters(
raise ValueError(error_msg)
# Assure predictions are uptodate
cls.prediction.update_data()
await cls.prediction.update_data()
try:
pvforecast_ac_power = (
cls.prediction.key_to_array(
await cls.prediction.key_to_array(
key="pvforecast_ac_power",
start_datetime=parameter_start_datetime,
end_datetime=parameter_end_datetime,
@@ -290,7 +290,7 @@ class GeneticOptimizationParameters(
# Retry
continue
try:
elecprice_marketprice_wh = cls.prediction.key_to_array(
elecprice_marketprice_wh = await cls.prediction.key_to_array(
key="elecprice_marketprice_wh",
start_datetime=parameter_start_datetime,
end_datetime=parameter_end_datetime,
@@ -306,7 +306,7 @@ class GeneticOptimizationParameters(
# Retry
continue
try:
loadforecast_power_w = cls.prediction.key_to_array(
loadforecast_power_w = await cls.prediction.key_to_array(
key="loadforecast_power_w",
start_datetime=parameter_start_datetime,
end_datetime=parameter_end_datetime,
@@ -331,7 +331,7 @@ class GeneticOptimizationParameters(
# Retry
continue
try:
feed_in_tariff_wh = cls.prediction.key_to_array(
feed_in_tariff_wh = await cls.prediction.key_to_array(
key="feed_in_tariff_wh",
start_datetime=parameter_start_datetime,
end_datetime=parameter_end_datetime,
@@ -358,7 +358,7 @@ class GeneticOptimizationParameters(
# Retry
continue
try:
weather_temp_air = cls.prediction.key_to_array(
weather_temp_air = await cls.prediction.key_to_array(
key="weather_temp_air",
start_datetime=parameter_start_datetime,
end_datetime=parameter_end_datetime,
@@ -418,7 +418,7 @@ class GeneticOptimizationParameters(
battery_lcos_kwh = battery_config.levelized_cost_of_storage_kwh
# Initial SOC
try:
initial_soc_factor = cls.measurement.key_to_value(
initial_soc_factor = await cls.measurement.key_to_value(
key=battery_config.measurement_key_soc_factor,
target_datetime=ems.start_datetime,
time_window=to_duration(to_duration("48 hours")),
@@ -490,7 +490,7 @@ class GeneticOptimizationParameters(
continue
# Initial SOC
try:
initial_soc_factor = cls.measurement.key_to_value(
initial_soc_factor = await cls.measurement.key_to_value(
key=electric_vehicle_config.measurement_key_soc_factor,
target_datetime=ems.start_datetime,
time_window=to_duration(to_duration("48 hours")),

4
src/akkudoktoreos/optimization/genetic/geneticsolution.py
View File

@@ -344,7 +344,7 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel):
return effective_ac, effective_dc, effective_dis
def optimization_solution(self) -> OptimizationSolution:
async def optimization_solution(self) -> OptimizationSolution:
"""Provide the genetic solution as a general optimization solution.
The battery modes are controlled by the grid control triggers:
@@ -612,7 +612,7 @@ class GeneticSolution(ConfigMixin, GeneticParametersBaseModel):
),
]:
if pred_key in pred.record_keys:
array = pred.key_to_array(
array = await pred.key_to_array(
key=pred_key,
start_datetime=start_datetime,
end_datetime=end_datetime,

10
src/akkudoktoreos/prediction/elecpriceakkudoktor.py
View File

@@ -135,7 +135,7 @@ class ElecPriceAkkudoktor(ElecPriceProvider):
clean_history = self._cap_outliers(history)
return np.full(hours, np.median(clean_history))
def _update_data(
async def _update_data(
self, force_update: Optional[bool] = False
) -> None: # tuple[np.ndarray, np.ndarray, np.ndarray]:
"""Update forecast data in the ElecPriceDataRecord format.
@@ -170,10 +170,10 @@ class ElecPriceAkkudoktor(ElecPriceProvider):
series_data.at[orig_datetime] = price_wh
# Update values using key_from_series
self.key_from_series("elecprice_marketprice_wh", series_data)
await self.key_from_series("elecprice_marketprice_wh", series_data)
# Generate history array for prediction
history = self.key_to_array(
history = await self.key_to_array(
key="elecprice_marketprice_wh", end_datetime=highest_orig_datetime, fill_method="linear"
)
@@ -213,9 +213,9 @@ class ElecPriceAkkudoktor(ElecPriceProvider):
for i in range(len(prediction))
],
)
self.key_from_series("elecprice_marketprice_wh", prediction_series)
await self.key_from_series("elecprice_marketprice_wh", prediction_series)
# history2 = self.key_to_array(key="elecprice_marketprice_wh", fill_method="linear") + 0.0002
# history2 = await self.key_to_array(key="elecprice_marketprice_wh", fill_method="linear") + 0.0002
# return history, history2, prediction # for debug main

10
src/akkudoktoreos/prediction/elecpriceenergycharts.py
View File

@@ -195,7 +195,7 @@ class ElecPriceEnergyCharts(ElecPriceProvider):
clean_history = self._cap_outliers(history)
return np.full(hours, np.median(clean_history))
def _update_data(
async def _update_data(
self, force_update: Optional[bool] = False
) -> None: # tuple[np.ndarray, np.ndarray, np.ndarray]:
"""Update forecast data in the ElecPriceDataRecord format.
@@ -217,7 +217,7 @@ class ElecPriceEnergyCharts(ElecPriceProvider):
# Determine if update is needed and how many days
past_days = 35
if self.highest_orig_datetime:
history_series = self.key_to_series(
history_series = await self.key_to_series(
key="elecprice_marketprice_wh", start_datetime=self.ems_start_datetime
)
# If history lower, then start_datetime
@@ -244,14 +244,14 @@ class ElecPriceEnergyCharts(ElecPriceProvider):
# Parse and store data
series_data = self._parse_data(energy_charts_data)
self.highest_orig_datetime = series_data.index.max()
self.key_from_series("elecprice_marketprice_wh", series_data)
await self.key_from_series("elecprice_marketprice_wh", series_data)
else:
logger.info(
f"No Update ElecPriceEnergyCharts is needed, last in history: {self.highest_orig_datetime}"
)
# Generate history array for prediction
history = self.key_to_array(
history = await self.key_to_array(
key="elecprice_marketprice_wh",
end_datetime=self.highest_orig_datetime,
fill_method="linear",
@@ -293,4 +293,4 @@ class ElecPriceEnergyCharts(ElecPriceProvider):
for i in range(len(prediction))
],
)
self.key_from_series("elecprice_marketprice_wh", prediction_series)
await self.key_from_series("elecprice_marketprice_wh", prediction_series)

4
src/akkudoktoreos/prediction/elecpricefixed.py
View File

@@ -59,7 +59,7 @@ class ElecPriceFixed(ElecPriceProvider):
"""Return the unique identifier for the ElecPriceFixed provider."""
return "ElecPriceFixed"
def _update_data(self, force_update: Optional[bool] = False) -> None:
async def _update_data(self, force_update: Optional[bool] = False) -> None:
"""Update electricity price data from fixed schedule.
Generates electricity prices based on the configured time windows
@@ -106,6 +106,6 @@ class ElecPriceFixed(ElecPriceProvider):
# Convert kWh → Wh and store one entry per interval step.
for idx, price_kwh in enumerate(prices_kwh):
current_dt = start_datetime.add(seconds=idx * interval_seconds)
self.update_value(current_dt, "elecprice_marketprice_wh", price_kwh / 1000.0)
await self.update_value(current_dt, "elecprice_marketprice_wh", price_kwh / 1000.0)
logger.debug(f"Successfully generated {len(prices_kwh)} fixed electricity price entries")

8
src/akkudoktoreos/prediction/elecpriceimport.py
View File

@@ -63,14 +63,16 @@ class ElecPriceImport(ElecPriceProvider, PredictionImportProvider):
"""Return the unique identifier for the ElecPriceImport provider."""
return "ElecPriceImport"
def _update_data(self, force_update: Optional[bool] = False) -> None:
async def _update_data(self, force_update: Optional[bool] = False) -> None:
# Both _sequence_lock and _record_lock are already held by the caller.
# Use internal sync methods only — never await public async counterparts.
if self.config.elecprice.elecpriceimport.import_file_path:
self.import_from_file(
await self._import_from_file(
self.config.elecprice.elecpriceimport.import_file_path,
key_prefix="elecprice",
)
if self.config.elecprice.elecpriceimport.import_json:
self.import_from_json(
await self._import_from_json(
self.config.elecprice.elecpriceimport.import_json,
key_prefix="elecprice",
)

4
src/akkudoktoreos/prediction/feedintarifffixed.py
View File

@@ -34,7 +34,7 @@ class FeedInTariffFixed(FeedInTariffProvider):
"""Return the unique identifier for the FeedInTariffFixed provider."""
return "FeedInTariffFixed"
def _update_data(self, force_update: Optional[bool] = False) -> None:
async def _update_data(self, force_update: Optional[bool] = False) -> None:
error_msg = "Feed in tariff not provided"
try:
feed_in_tariff = (
@@ -47,4 +47,4 @@ class FeedInTariffFixed(FeedInTariffProvider):
logger.error(error_msg)
raise ValueError(error_msg)
feed_in_tariff_wh = feed_in_tariff / 1000
self.update_value(to_datetime(), "feed_in_tariff_wh", feed_in_tariff_wh)
await self.update_value(to_datetime(), "feed_in_tariff_wh", feed_in_tariff_wh)

8
src/akkudoktoreos/prediction/feedintariffimport.py
View File

@@ -64,17 +64,19 @@ class FeedInTariffImport(FeedInTariffProvider, PredictionImportProvider):
"""Return the unique identifier for the FeedInTariffImport provider."""
return "FeedInTariffImport"
def _update_data(self, force_update: Optional[bool] = False) -> None:
async def _update_data(self, force_update: Optional[bool] = False) -> None:
# Both _sequence_lock and _record_lock are already held by the caller.
# Use internal sync methods only — never await public async counterparts.
if self.config.feedintariff.provider_settings.FeedInTariffImport is None:
logger.debug(f"{self.provider_id()} data update without provider settings.")
return
if self.config.feedintariff.provider_settings.FeedInTariffImport.import_file_path:
self.import_from_file(
await self._import_from_file(
self.config.provider_settings.FeedInTariffImport.import_file_path,
key_prefix="feedintariff",
)
if self.config.feedintariff.provider_settings.FeedInTariffImport.import_json:
self.import_from_json(
await self._import_from_json(
self.config.feedintariff.provider_settings.FeedInTariffImport.import_json,
key_prefix="feedintariff",
)

29
src/akkudoktoreos/prediction/loadakkudoktor.py
View File

@@ -68,7 +68,7 @@ class LoadAkkudoktor(LoadProvider):
raise ValueError(error_msg)
return data_year_energy
def _update_data(self, force_update: Optional[bool] = False) -> None:
async def _update_data(self, force_update: Optional[bool] = False) -> None:
"""Adds the load means and standard deviations."""
data_year_energy = self.load_data()
# We provide prediction starting at start of day, to be compatible to old system.
@@ -84,7 +84,7 @@ class LoadAkkudoktor(LoadProvider):
"loadakkudoktor_mean_power_w": hourly_stats[0],
"loadakkudoktor_std_power_w": hourly_stats[1],
}
self.update_value(date, values)
await self.update_value(date, values)
date += to_duration("1 hour")
# We are working on fresh data (no cache), report update time
self.update_datetime = to_datetime(in_timezone=self.config.general.timezone)
@@ -98,7 +98,9 @@ class LoadAkkudoktorAdjusted(LoadAkkudoktor):
"""Return the unique identifier for the LoadAkkudoktor provider."""
return "LoadAkkudoktorAdjusted"
def _calculate_adjustment(self, data_year_energy: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
async def _calculate_adjustment(
self, data_year_energy: np.ndarray
) -> tuple[np.ndarray, np.ndarray]:
"""Calculate weekday and week end adjustment from total load measurement data.
Returns:
@@ -110,19 +112,22 @@ class LoadAkkudoktorAdjusted(LoadAkkudoktor):
weekend_adjust = np.zeros(24)
weekend_adjust_weight = np.zeros(24)
if self.measurement.max_datetime is None:
max_dt = await self.measurement.max_datetime()
if max_dt is None:
# No measurements - return 0 adjustment
return (weekday_adjust, weekday_adjust)
min_dt = await self.measurement.min_datetime()
# compare predictions with real measurement - try to use last 7 days
compare_start = self.measurement.max_datetime - to_duration("7 days")
if compare_datetimes(compare_start, self.measurement.min_datetime).lt:
compare_start = max_dt - to_duration("7 days")
if compare_datetimes(compare_start, min_dt).lt:
# Not enough measurements for 7 days - use what is available
compare_start = self.measurement.min_datetime
compare_end = self.measurement.max_datetime
compare_start = min_dt
compare_end = max_dt
compare_interval = to_duration("1 hour")
load_total_kwh_array = self.measurement.load_total_kwh(
load_total_kwh_array = await self.measurement.load_total_kwh(
start_datetime=compare_start,
end_datetime=compare_end,
interval=compare_interval,
@@ -159,10 +164,10 @@ class LoadAkkudoktorAdjusted(LoadAkkudoktor):
return (weekday_adjust, weekend_adjust)
def _update_data(self, force_update: Optional[bool] = False) -> None:
async def _update_data(self, force_update: Optional[bool] = False) -> None:
"""Adds the load means and standard deviations."""
data_year_energy = self.load_data()
weekday_adjust, weekend_adjust = self._calculate_adjustment(data_year_energy)
weekday_adjust, weekend_adjust = await self._calculate_adjustment(data_year_energy)
# We provide prediction starting at start of day, to be compatible to old system.
# End date for prediction is prediction hours from now.
date = self.ems_start_datetime.start_of("day")
@@ -182,7 +187,7 @@ class LoadAkkudoktorAdjusted(LoadAkkudoktor):
# Saturday, Sunday (5, 6)
value_adjusted = hourly_stats[0] + weekend_adjust[date.hour]
values["loadforecast_power_w"] = max(0, value_adjusted)
self.update_value(date, values)
await self.update_value(date, values)
date += to_duration("1 hour")
# We are working on fresh data (no cache), report update time
self.update_datetime = to_datetime(in_timezone=self.config.general.timezone)

10
src/akkudoktoreos/prediction/loadimport.py
View File

@@ -62,8 +62,12 @@ class LoadImport(LoadProvider, PredictionImportProvider):
"""Return the unique identifier for the LoadImport provider."""
return "LoadImport"
def _update_data(self, force_update: Optional[bool] = False) -> None:
async def _update_data(self, force_update: Optional[bool] = False) -> None:
# Both _sequence_lock and _record_lock are already held by the caller.
# Use internal sync methods only — never await public async counterparts.
if self.config.load.loadimport.import_file_path:
self.import_from_file(self.config.load.loadimport.import_file_path, key_prefix="load")
await self._import_from_file(
self.config.load.loadimport.import_file_path, key_prefix="load"
)
if self.config.load.loadimport.import_json:
self.import_from_json(self.config.load.loadimport.import_json, key_prefix="load")
await self._import_from_json(self.config.load.loadimport.import_json, key_prefix="load")

9
src/akkudoktoreos/prediction/loadvrm.py
View File

@@ -84,7 +84,7 @@ class LoadVrm(LoadProvider):
"""Convert UNIX ms timestamp to timezone-aware datetime."""
return to_datetime(timestamp / 1000, in_timezone=self.config.general.timezone)
def _update_data(self, force_update: Optional[bool] = False) -> None:
async def _update_data(self, force_update: Optional[bool] = False) -> None:
"""Fetch and store VRM load forecast as loadforecast_power_w and related values."""
if self.enabled is False:
logger.info("LoadVrm is disabled, skipping update.")
@@ -102,7 +102,7 @@ class LoadVrm(LoadProvider):
date = self._ts_to_datetime(timestamp)
rounded_value = round(value, 2)
self.update_value(
await self.update_value(
date,
{"loadforecast_power_w": rounded_value},
)
@@ -111,8 +111,3 @@ class LoadVrm(LoadProvider):
logger.debug(f"Updated loadforecast_power_w with {len(loadforecast_power_w_data)} entries.")
self.update_datetime = to_datetime(in_timezone=self.config.general.timezone)
if __name__ == "__main__":
lv = LoadVrm()
lv._update_data()

2
src/akkudoktoreos/prediction/prediction.py
View File

@@ -14,7 +14,7 @@ Example:
# Create singleton prediction instance with prediction providers
from akkudoktoreos.prediction.prediction import prediction
prediction.update_data()
await prediction.update_data()
print("Prediction:", prediction)
Classes:

6
src/akkudoktoreos/prediction/predictionabc.py
View File

@@ -225,7 +225,7 @@ class PredictionProvider(PredictionStartEndKeepMixin, DataProvider):
hours = max(self.config.prediction.hours, self.config.prediction_historic_hours, 24)
return to_duration(hours * 3600)
def update_data(
async def update_data(
self,
force_enable: Optional[bool] = False,
force_update: Optional[bool] = False,
@@ -243,10 +243,10 @@ class PredictionProvider(PredictionStartEndKeepMixin, DataProvider):
return
# Delete outdated records before updating
self.delete_by_datetime(end_datetime=self.keep_datetime)
await self.delete_by_datetime(end_datetime=self.keep_datetime)
# Call the custom update logic
self._update_data(force_update=force_update)
await self._update_data(force_update=force_update)
class PredictionImportProvider(PredictionProvider, DataImportProvider):

20
src/akkudoktoreos/prediction/pvforecastakkudoktor.py
View File

@@ -52,10 +52,10 @@ Example:
forecast = PVForecastAkkudoktor(settings=config)
# Get an actual forecast
forecast.update_data()
await forecast.update_data()
# Update the AC power measurement for a specific date and time
forecast.update_value(to_datetime(None, to_maxtime=False), "pvforecastakkudoktor_ac_power_measured", 1000.0)
await forecast.update_value(to_datetime(None, to_maxtime=False), "pvforecastakkudoktor_ac_power_measured", 1000.0)
# Report the DC and AC power forecast along with AC measurements
print(forecast.report_ac_power_and_measurement())
@@ -286,7 +286,7 @@ class PVForecastAkkudoktor(PVForecastProvider):
return akkudoktor_data
def _update_data(self, force_update: Optional[bool] = False) -> None:
async def _update_data(self, force_update: Optional[bool] = False) -> None:
"""Update forecast data in the PVForecastAkkudoktorDataRecord format.
Retrieves data from Akkudoktor. The processed data is inserted into the sequence as
@@ -341,7 +341,7 @@ class PVForecastAkkudoktor(PVForecastProvider):
"pvforecastakkudoktor_temp_air": forecast_values[0].temperature,
}
self.update_value(dt, data)
await self.update_value(dt, data)
if len(self) < self.config.prediction.hours:
raise ValueError(
@@ -385,7 +385,7 @@ class PVForecastAkkudoktor(PVForecastProvider):
# Example of how to use the PVForecastAkkudoktor class
if __name__ == "__main__":
async def main() -> None:
"""Main execution block to demonstrate the use of the PVForecastAkkudoktor class.
Sets up the forecast configuration fields, fetches PV power forecast data,
@@ -441,12 +441,18 @@ if __name__ == "__main__":
forecast = PVForecastAkkudoktor()
# Get an actual forecast
forecast.update_data()
await forecast.update_data()
# Update the AC power measurement for a specific date and time
forecast.update_value(
await forecast.update_value(
to_datetime(None, to_maxtime=False), "pvforecastakkudoktor_ac_power_measured", 1000.0
)
# Report the DC and AC power forecast along with AC measurements
print(forecast.report_ac_power_and_measurement())
if __name__ == "__main__":
import asyncio
asyncio.run(main())

8
src/akkudoktoreos/prediction/pvforecastimport.py
View File

@@ -64,17 +64,19 @@ class PVForecastImport(PVForecastProvider, PredictionImportProvider):
"""Return the unique identifier for the PVForecastImport provider."""
return "PVForecastImport"
def _update_data(self, force_update: Optional[bool] = False) -> None:
async def _update_data(self, force_update: Optional[bool] = False) -> None:
# Both _sequence_lock and _record_lock are already held by the caller.
# Use internal sync methods only — never await public async counterparts.
if self.config.pvforecast.provider_settings.PVForecastImport is None:
logger.debug(f"{self.provider_id()} data update without provider settings.")
return
if self.config.pvforecast.provider_settings.PVForecastImport.import_file_path is not None:
self.import_from_file(
await self._import_from_file(
self.config.pvforecast.provider_settings.PVForecastImport.import_file_path,
key_prefix="pvforecast",
)
if self.config.pvforecast.provider_settings.PVForecastImport.import_json is not None:
self.import_from_json(
await self._import_from_json(
self.config.pvforecast.provider_settings.PVForecastImport.import_json,
key_prefix="pvforecast",
)

10
src/akkudoktoreos/prediction/pvforecastvrm.py
View File

@@ -83,7 +83,7 @@ class PVForecastVrm(PVForecastProvider):
"""Convert UNIX ms timestamp to timezone-aware datetime."""
return to_datetime(timestamp / 1000, in_timezone=self.config.general.timezone)
def _update_data(self, force_update: Optional[bool] = False) -> None:
async def _update_data(self, force_update: Optional[bool] = False) -> None:
"""Update forecast data in the PVForecastDataRecord format."""
if self.enabled is False:
logger.info("PVForecastVrm is disabled, skipping update.")
@@ -101,16 +101,10 @@ class PVForecastVrm(PVForecastProvider):
date = self._ts_to_datetime(timestamp)
dc_power = round(value, 2)
ac_power = round(dc_power * 0.96, 2)
self.update_value(
await self.update_value(
date, {"pvforecast_dc_power": dc_power, "pvforecast_ac_power": ac_power}
)
pv_forecast.append((date, dc_power))
logger.debug(f"Updated pvforecast_dc_power with {len(pv_forecast)} entries.")
self.update_datetime = to_datetime(in_timezone=self.config.general.timezone)
# Example usage
if __name__ == "__main__":
pv = PVForecastVrm()
pv._update_data()

26
src/akkudoktoreos/prediction/weatherbrightsky.py
View File

@@ -111,7 +111,7 @@ class WeatherBrightSky(WeatherProvider):
self.update_datetime = to_datetime(in_timezone=self.config.general.timezone)
return brightsky_data
def _description_to_series(self, description: str) -> pd.Series:
async def _description_to_series(self, description: str) -> pd.Series:
"""Retrieve a pandas Series corresponding to a weather data description.
This method fetches the key associated with the provided description
@@ -132,10 +132,10 @@ class WeatherBrightSky(WeatherProvider):
error_msg = f"No WeatherDataRecord key for '{description}'"
logger.error(error_msg)
raise ValueError(error_msg)
series = self.key_to_series(key)
series = await self.key_to_series(key)
return series
def _description_from_series(self, description: str, data: pd.Series) -> None:
async def _description_from_series(self, description: str, data: pd.Series) -> None:
"""Update a weather data with a pandas Series based on its description.
This method fetches the key associated with the provided description
@@ -154,9 +154,9 @@ class WeatherBrightSky(WeatherProvider):
error_msg = f"No WeatherDataRecord key for '{description}'"
logger.error(error_msg)
raise ValueError(error_msg)
self.key_from_series(key, data)
await self.key_from_series(key, data)
def _update_data(self, force_update: Optional[bool] = False) -> None:
async def _update_data(self, force_update: Optional[bool] = False) -> None:
"""Update forecast data in the WeatherDataRecord format.
Retrieves data from BrightSky, maps each BrightSky field to the corresponding
@@ -197,31 +197,31 @@ class WeatherBrightSky(WeatherProvider):
else:
value = value * corr_factor
setattr(weather_record, key, value)
self.insert_by_datetime(weather_record)
await self.insert_by_datetime(weather_record)
# Converting the cloud cover into Irradiance (GHI, DNI, DHI)
description = "Total Clouds (% Sky Obscured)"
cloud_cover = self._description_to_series(description)
cloud_cover = await self._description_to_series(description)
ghi, dni, dhi = self.estimate_irradiance_from_cloud_cover(
self.config.general.latitude, self.config.general.longitude, cloud_cover
)
description = "Global Horizontal Irradiance (W/m2)"
ghi = pd.Series(data=ghi, index=cloud_cover.index)
self._description_from_series(description, ghi)
await self._description_from_series(description, ghi)
description = "Direct Normal Irradiance (W/m2)"
dni = pd.Series(data=dni, index=cloud_cover.index)
self._description_from_series(description, dni)
await self._description_from_series(description, dni)
description = "Diffuse Horizontal Irradiance (W/m2)"
dhi = pd.Series(data=dhi, index=cloud_cover.index)
self._description_from_series(description, dhi)
await self._description_from_series(description, dhi)
# Add Preciptable Water (PWAT) with a PVLib method.
key = WeatherDataRecord.key_from_description("Temperature (°C)")
assert key # noqa: S101
temperature = self.key_to_array(
temperature = await self.key_to_array(
key=key,
start_datetime=self.ems_start_datetime,
end_datetime=self.end_datetime,
@@ -234,7 +234,7 @@ class WeatherBrightSky(WeatherProvider):
return
key = WeatherDataRecord.key_from_description("Relative Humidity (%)")
assert key # noqa: S101
humidity = self.key_to_array(
humidity = await self.key_to_array(
key=key,
start_datetime=self.ems_start_datetime,
end_datetime=self.end_datetime,
@@ -258,4 +258,4 @@ class WeatherBrightSky(WeatherProvider):
),
)
description = "Preciptable Water (cm)"
self._description_from_series(description, pwat)
await self._description_from_series(description, pwat)

6
src/akkudoktoreos/prediction/weatherclearoutside.py
View File

@@ -97,7 +97,7 @@ class WeatherClearOutside(WeatherProvider):
self.update_datetime = to_datetime(in_timezone=self.config.general.timezone)
return response
def _update_data(self, force_update: Optional[bool] = None) -> None:
async def _update_data(self, force_update: Optional[bool] = None) -> None:
"""Scrape weather forecast data from ClearOutside's website.
This method requests weather forecast data from ClearOutside based on latitude
@@ -202,7 +202,7 @@ class WeatherClearOutside(WeatherProvider):
raise ValueError(error_msg)
# Delete all records that will be newly added
self.delete_by_datetime(start_datetime=forecast_start_datetime)
await self.delete_by_datetime(start_datetime=forecast_start_datetime)
# Collect weather data, loop over all days
for day, p_day in enumerate(p_days):
@@ -341,4 +341,4 @@ class WeatherClearOutside(WeatherProvider):
if corr_factor:
value = value * corr_factor
setattr(weather_record, key, value)
self.insert_by_datetime(weather_record)
await self.insert_by_datetime(weather_record)

8
src/akkudoktoreos/prediction/weatherimport.py
View File

@@ -64,17 +64,19 @@ class WeatherImport(WeatherProvider, PredictionImportProvider):
"""Return the unique identifier for the WeatherImport provider."""
return "WeatherImport"
def _update_data(self, force_update: Optional[bool] = False) -> None:
async def _update_data(self, force_update: Optional[bool] = False) -> None:
# Both _sequence_lock and _record_lock are already held by the caller.
# Use internal sync methods only — never await public async counterparts.
if self.config.weather.provider_settings.WeatherImport is None:
logger.debug(f"{self.provider_id()} data update without provider settings.")
return
if self.config.weather.provider_settings.WeatherImport.import_file_path:
self.import_from_file(
await self._import_from_file(
self.config.weather.provider_settings.WeatherImport.import_file_path,
key_prefix="weather",
)
if self.config.weather.provider_settings.WeatherImport.import_json:
self.import_from_json(
await self._import_from_json(
self.config.weather.provider_settings.WeatherImport.import_json,
key_prefix="weather",
)

20
src/akkudoktoreos/prediction/weatheropenmeteo.py
View File

@@ -197,7 +197,7 @@ class WeatherOpenMeteo(WeatherProvider):
self.update_datetime = to_datetime(in_timezone=self.config.general.timezone)
return openmeteo_data
def _description_to_series(self, description: str) -> pd.Series:
async def _description_to_series(self, description: str) -> pd.Series:
"""Retrieve a pandas Series corresponding to a weather data description.
This method fetches the key associated with the provided description
@@ -218,10 +218,10 @@ class WeatherOpenMeteo(WeatherProvider):
error_msg = f"No WeatherDataRecord key for '{description}'"
logger.error(error_msg)
raise ValueError(error_msg)
series = self.key_to_series(key)
series = await self.key_to_series(key)
return series
def _description_from_series(self, description: str, data: pd.Series) -> None:
async def _description_from_series(self, description: str, data: pd.Series) -> None:
"""Update a weather data with a pandas Series based on its description.
This method fetches the key associated with the provided description
@@ -240,9 +240,9 @@ class WeatherOpenMeteo(WeatherProvider):
error_msg = f"No WeatherDataRecord key for '{description}'"
logger.error(error_msg)
raise ValueError(error_msg)
self.key_from_series(key, data)
await self.key_from_series(key, data)
def _update_data(self, force_update: Optional[bool] = False) -> None:
async def _update_data(self, force_update: Optional[bool] = False) -> None:
"""Update forecast data in the WeatherDataRecord format.
Retrieves data from Open-Meteo, maps each Open-Meteo field to the corresponding
@@ -299,11 +299,11 @@ class WeatherOpenMeteo(WeatherProvider):
setattr(weather_record, key, value)
self.insert_by_datetime(weather_record)
await self.insert_by_datetime(weather_record)
# Check whether radiation values exist (for logging)
description_ghi = "Global Horizontal Irradiance (W/m2)"
ghi_series = self._description_to_series(description_ghi)
ghi_series = await self._description_to_series(description_ghi)
if ghi_series.isnull().all():
logger.warning("No GHI data received from Open-Meteo")
@@ -315,7 +315,7 @@ class WeatherOpenMeteo(WeatherProvider):
# Add Precipitable Water (PWAT) using PVLib method
key = WeatherDataRecord.key_from_description("Temperature (°C)")
assert key # noqa: S101
temperature = self.key_to_array(
temperature = await self.key_to_array(
key=key,
start_datetime=self.ems_start_datetime,
end_datetime=self.end_datetime,
@@ -329,7 +329,7 @@ class WeatherOpenMeteo(WeatherProvider):
key = WeatherDataRecord.key_from_description("Relative Humidity (%)")
assert key # noqa: S101
humidity = self.key_to_array(
humidity = await self.key_to_array(
key=key,
start_datetime=self.ems_start_datetime,
end_datetime=self.end_datetime,
@@ -354,4 +354,4 @@ class WeatherOpenMeteo(WeatherProvider):
),
)
description = "Precipitable Water (cm)"
self._description_from_series(description, pwat)
await self._description_from_series(description, pwat)

247
src/akkudoktoreos/server/eos.py
View File

@@ -73,20 +73,18 @@ from akkudoktoreos.utils.datetimeutil import to_datetime, to_duration
# ----------------------
def save_eos_state() -> None:
async def save_eos_state() -> None:
"""Save EOS state."""
get_resource_registry().save()
get_prediction().save()
get_measurement().save()
await get_prediction().save()
await get_measurement().save()
cache_save() # keep last
def load_eos_state() -> None:
async def load_eos_state() -> None:
"""Load EOS state."""
cache_load() # keep first
get_measurement().load()
get_prediction().load()
get_resource_registry().load()
await get_measurement().load()
await get_prediction().load()
def terminate_eos() -> None:
@@ -100,18 +98,18 @@ def terminate_eos() -> None:
logger.info(f"🚀 EOS terminated, PID {pid}")
def save_eos_database() -> None:
async def save_eos_database() -> None:
"""Save EOS database."""
get_prediction().save()
get_measurement().save()
await get_prediction().save()
await get_measurement().save()
def compact_eos_database() -> None:
async def compact_eos_database() -> None:
"""Compact EOS database."""
get_prediction().db_compact()
get_measurement().db_compact()
get_prediction().db_vacuum()
get_measurement().db_vacuum()
await get_prediction().db_compact()
await get_measurement().db_compact()
await get_prediction().db_vacuum()
await get_measurement().db_vacuum()
def autosave_config() -> None:
@@ -134,7 +132,7 @@ async def server_shutdown_task() -> None:
Finally, logs a message indicating that the EOS server has been terminated.
"""
save_eos_state()
await save_eos_state()
# Give EOS time to finish some work
await asyncio.sleep(5)
@@ -162,7 +160,7 @@ def config_eos_ready() -> bool:
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
"""Lifespan manager for the app."""
# On startup
load_eos_state()
await load_eos_state()
# Prepare the Manager and all task that are handled by the manager
manager = RetentionManager(
@@ -200,7 +198,7 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
await asyncio.gather(retention_manager_task, return_exceptions=True)
# On shutdown
save_eos_state()
await save_eos_state()
app = FastAPI(
@@ -297,7 +295,7 @@ def fastapi_admin_cache_get() -> dict:
@app.get("/v1/admin/database/stats", tags=["admin"])
def fastapi_admin_database_stats_get() -> dict:
async def fastapi_admin_database_stats_get() -> dict:
"""Get statistics from database.
Returns:
@@ -306,8 +304,8 @@ def fastapi_admin_database_stats_get() -> dict:
data = {}
try:
# Get the stats
data[get_measurement().db_namespace()] = get_measurement().db_get_stats()
data[get_prediction().__class__.__name__] = get_prediction().db_get_stats()
data[get_measurement().db_namespace()] = await get_measurement().db_get_stats()
data[get_prediction().__class__.__name__] = await get_prediction().db_get_stats()
except Exception as e:
trace = "".join(traceback.TracebackException.from_exception(e).format())
raise HTTPException(
@@ -316,8 +314,28 @@ def fastapi_admin_database_stats_get() -> dict:
return data
@app.post("/v1/admin/database/save", tags=["admin"])
async def fastapi_admin_database_save_post() -> dict:
"""Save in memory data to database.
Returns:
data (dict): The database stats after saving the records.
"""
data = {}
try:
await get_measurement().save()
await get_prediction().save()
# Get the stats
data[get_measurement().db_namespace()] = await get_measurement().db_get_stats()
data[get_prediction().__class__.__name__] = await get_prediction().db_get_stats()
except Exception as e:
trace = "".join(traceback.TracebackException.from_exception(e).format())
raise HTTPException(status_code=400, detail=f"Error on database save: {e}\n{trace}")
return data
@app.post("/v1/admin/database/vacuum", tags=["admin"])
def fastapi_admin_database_vacuum_post() -> dict:
async def fastapi_admin_database_vacuum_post() -> dict:
"""Remove old records from database.
Returns:
@@ -325,11 +343,13 @@ def fastapi_admin_database_vacuum_post() -> dict:
"""
data = {}
try:
get_measurement().db_vacuum()
get_prediction().db_vacuum()
await get_measurement().db_vacuum()
await get_prediction().db_vacuum()
# Get the stats
data[get_measurement().db_namespace()] = get_measurement().db_get_stats()
data[get_prediction().__class__.__name__] = get_prediction().db_get_stats()
measuremet_stats = await get_measurement().db_get_stats()
data[get_measurement().db_namespace()] = measuremet_stats
prediction_stats = await get_prediction().db_get_stats()
data[get_prediction().__class__.__name__] = prediction_stats
except Exception as e:
trace = "".join(traceback.TracebackException.from_exception(e).format())
raise HTTPException(status_code=400, detail=f"Error on database vacuum: {e}\n{trace}")
@@ -342,7 +362,7 @@ async def fastapi_admin_server_restart_post() -> dict:
Restart EOS properly by starting a new instance before exiting the old one.
"""
save_eos_state()
await save_eos_state()
# Start a new EOS (Uvicorn) process
logger.info("🔄 Restarting EOS...")
@@ -525,7 +545,11 @@ def fastapi_config_put(settings: SettingsEOS) -> ConfigEOS:
get_config().merge_settings(settings)
return get_config()
except Exception as e:
raise HTTPException(status_code=400, detail=f"Error on update of configuration: {e}")
trace = "".join(traceback.TracebackException.from_exception(e).format())
raise HTTPException(
status_code=400,
detail=f"Error on update of configuration '{settings}':\n{e}\n{trace}",
)
@app.put("/v1/config/{path:path}", tags=["config"])
@@ -687,14 +711,14 @@ def fastapi_measurement_keys_get() -> list[str]:
@app.get("/v1/measurement/series", tags=["measurement"])
def fastapi_measurement_series_get(
async def fastapi_measurement_series_get(
key: Annotated[str, Query(description="Measurement key.")],
) -> PydanticDateTimeSeries:
"""Get the measurements of given key as series."""
try:
if key not in get_measurement().record_keys:
raise HTTPException(status_code=404, detail=f"Key '{key}' is not available.")
pdseries = get_measurement().key_to_series(key=key)
pdseries = await get_measurement().key_to_series(key=key)
return PydanticDateTimeSeries.from_series(pdseries)
except HTTPException:
# Re-raise HTTP exceptions
@@ -710,7 +734,7 @@ def fastapi_measurement_series_get(
@app.put("/v1/measurement/value", tags=["measurement"])
def fastapi_measurement_value_put(
async def fastapi_measurement_value_put(
datetime: Annotated[str, Query(description="Datetime.")],
key: Annotated[str, Query(description="Measurement key.")],
value: Union[float | str],
@@ -740,8 +764,8 @@ def fastapi_measurement_value_put(
detail=f"Invalid datetime '{datetime}': {e}",
)
get_measurement().update_value(dt, key, value)
pdseries = get_measurement().key_to_series(key=key)
await get_measurement().update_value(dt, key, value)
pdseries = await get_measurement().key_to_series(key=key)
return PydanticDateTimeSeries.from_series(pdseries)
except HTTPException:
raise
@@ -755,7 +779,7 @@ def fastapi_measurement_value_put(
@app.put("/v1/measurement/series", tags=["measurement"])
def fastapi_measurement_series_put(
async def fastapi_measurement_series_put(
key: Annotated[str, Query(description="Measurement key.")], series: PydanticDateTimeSeries
) -> PydanticDateTimeSeries:
"""Merge measurement given as series into given key."""
@@ -763,8 +787,8 @@ def fastapi_measurement_series_put(
if key not in get_measurement().record_keys:
raise HTTPException(status_code=404, detail=f"Key '{key}' is not available.")
pdseries = series.to_series() # make pandas series from PydanticDateTimeSeries
get_measurement().key_from_series(key=key, series=pdseries)
pdseries = get_measurement().key_to_series(key=key)
await get_measurement().key_from_series(key=key, series=pdseries)
pdseries = await get_measurement().key_to_series(key=key)
return PydanticDateTimeSeries.from_series(pdseries)
except HTTPException:
# Re-raise HTTP exceptions
@@ -780,11 +804,11 @@ def fastapi_measurement_series_put(
@app.put("/v1/measurement/dataframe", tags=["measurement"])
def fastapi_measurement_dataframe_put(data: PydanticDateTimeDataFrame) -> None:
async def fastapi_measurement_dataframe_put(data: PydanticDateTimeDataFrame) -> None:
"""Merge the measurement data given as dataframe into EOS measurements."""
try:
dataframe = data.to_dataframe()
get_measurement().import_from_dataframe(dataframe)
await get_measurement().import_from_dataframe(dataframe)
except Exception as e:
# Log unexpected errors
trace = "".join(traceback.TracebackException.from_exception(e).format())
@@ -796,11 +820,11 @@ def fastapi_measurement_dataframe_put(data: PydanticDateTimeDataFrame) -> None:
@app.put("/v1/measurement/data", tags=["measurement"])
def fastapi_measurement_data_put(data: PydanticDateTimeData) -> None:
async def fastapi_measurement_data_put(data: PydanticDateTimeData) -> None:
"""Merge the measurement data given as datetime data into EOS measurements."""
try:
datetimedata = data.to_dict()
get_measurement().import_from_dict(datetimedata)
await get_measurement().import_from_dict(datetimedata)
except Exception as e:
# Log unexpected errors
trace = "".join(traceback.TracebackException.from_exception(e).format())
@@ -811,6 +835,49 @@ def fastapi_measurement_data_put(data: PydanticDateTimeData) -> None:
)
@app.delete("/v1/measurement/range", tags=["measurement"])
async def fastapi_measurement_range_delete(
key: Annotated[str, Query(description="Measurement key.")],
start_datetime: Annotated[Optional[str], Query(description="Start datetime.")] = None,
end_datetime: Annotated[Optional[str], Query(description="End datetime.")] = None,
) -> PydanticDateTimeSeries:
"""Delete measurement values for a key within a datetime range."""
try:
if key not in get_measurement().record_keys:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Key '{key}' not found in measurements",
)
try:
start_dt = to_datetime(start_datetime) if start_datetime else None
end_dt = to_datetime(end_datetime) if end_datetime else None
except Exception as e:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Invalid datetime: {e}",
)
await get_measurement().key_delete_by_datetime(
key=key,
start_datetime=start_dt,
end_datetime=end_dt,
)
pdseries = await get_measurement().key_to_series(key=key)
return PydanticDateTimeSeries.from_series(pdseries)
except HTTPException:
raise
except Exception as e:
trace = "".join(traceback.TracebackException.from_exception(e).format())
logger.exception(f"Unexpected error deleting measurement range: {key}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Internal server error:\n{e}\n{trace}",
)
@app.get("/v1/prediction/providers", tags=["prediction"])
def fastapi_prediction_providers_get(enabled: Optional[bool] = None) -> list[str]:
"""Get a list of available prediction providers.
@@ -838,7 +905,7 @@ def fastapi_prediction_keys_get() -> list[str]:
@app.get("/v1/prediction/series", tags=["prediction"])
def fastapi_prediction_series_get(
async def fastapi_prediction_series_get(
key: Annotated[str, Query(description="Prediction key.")],
start_datetime: Annotated[
Optional[str],
@@ -868,14 +935,14 @@ def fastapi_prediction_series_get(
end_datetime = get_prediction().end_datetime
else:
end_datetime = to_datetime(end_datetime)
pdseries = get_prediction().key_to_series(
pdseries = await get_prediction().key_to_series(
key=key, start_datetime=start_datetime, end_datetime=end_datetime
)
return PydanticDateTimeSeries.from_series(pdseries)
@app.get("/v1/prediction/dataframe", tags=["prediction"])
def fastapi_prediction_dataframe_get(
async def fastapi_prediction_dataframe_get(
keys: Annotated[list[str], Query(description="Prediction keys.")],
start_datetime: Annotated[
Optional[str],
@@ -911,14 +978,14 @@ def fastapi_prediction_dataframe_get(
end_datetime = get_prediction().end_datetime
else:
end_datetime = to_datetime(end_datetime)
df = get_prediction().keys_to_dataframe(
df = await get_prediction().keys_to_dataframe(
keys=keys, start_datetime=start_datetime, end_datetime=end_datetime, interval=interval
)
return PydanticDateTimeDataFrame.from_dataframe(df, tz=get_config().general.timezone)
@app.get("/v1/prediction/list", tags=["prediction"])
def fastapi_prediction_list_get(
async def fastapi_prediction_list_get(
key: Annotated[str, Query(description="Prediction key.")],
start_datetime: Annotated[
Optional[str],
@@ -958,16 +1025,13 @@ def fastapi_prediction_list_get(
interval = to_duration("1 hour")
else:
interval = to_duration(interval)
prediction_list = (
get_prediction()
.key_to_array(
key=key,
start_datetime=start_datetime,
end_datetime=end_datetime,
interval=interval,
)
.tolist()
prediction_array = await get_prediction().key_to_array(
key=key,
start_datetime=start_datetime,
end_datetime=end_datetime,
interval=interval,
)
prediction_list = prediction_array.tolist()
return prediction_list
@@ -1130,22 +1194,19 @@ async def fastapi_strompreis() -> list[float]:
start_datetime = to_datetime().start_of("day")
end_datetime = start_datetime.add(days=2)
try:
elecprice = (
get_prediction()
.key_to_array(
key="elecprice_marketprice_wh",
start_datetime=start_datetime,
end_datetime=end_datetime,
)
.tolist()
elecprice_array = await get_prediction().key_to_array(
key="elecprice_marketprice_wh",
start_datetime=start_datetime,
end_datetime=end_datetime,
)
elecprice_list = elecprice_array.tolist()
except Exception as e:
raise HTTPException(
status_code=404,
detail=f"Can not get the electricity price forecast: {e}.\nDid you configure the electricity price forecast provider?",
)
return elecprice
return elecprice_list
class GesamtlastRequest(PydanticBaseModel):
@@ -1191,7 +1252,7 @@ async def fastapi_gesamtlast(request: GesamtlastRequest) -> list[float]:
# Insert measured data into EOS measurement
# Convert from energy per interval to dummy energy meter readings
measurement_key = "gesamtlast_emr"
get_measurement().key_delete_by_datetime(
await get_measurement().key_delete_by_datetime(
key=measurement_key
) # delete all gesamtlast_emr measurements
energy = {}
@@ -1218,7 +1279,7 @@ async def fastapi_gesamtlast(request: GesamtlastRequest) -> list[float]:
energy_mr_values.append(0.0)
energy_mr_dates.append(dt)
energy_mr_values.append(energy_mr)
get_measurement().key_from_lists(measurement_key, energy_mr_dates, energy_mr_values)
await get_measurement().key_from_lists(measurement_key, energy_mr_dates, energy_mr_values)
# Ensure there is only one optimization/ energy management run at a time
try:
@@ -1236,15 +1297,12 @@ async def fastapi_gesamtlast(request: GesamtlastRequest) -> list[float]:
start_datetime = to_datetime().start_of("day")
end_datetime = start_datetime.add(days=2)
try:
prediction_list = (
get_prediction()
.key_to_array(
key="loadforecast_power_w",
start_datetime=start_datetime,
end_datetime=end_datetime,
)
.tolist()
prediction_array = await get_prediction().key_to_array(
key="loadforecast_power_w",
start_datetime=start_datetime,
end_datetime=end_datetime,
)
prediction_list = prediction_array.tolist()
except Exception as e:
raise HTTPException(
status_code=404,
@@ -1299,15 +1357,12 @@ async def fastapi_gesamtlast_simple(year_energy: float) -> list[float]:
start_datetime = to_datetime().start_of("day")
end_datetime = start_datetime.add(days=2)
try:
prediction_list = (
get_prediction()
.key_to_array(
key="loadforecast_power_w",
start_datetime=start_datetime,
end_datetime=end_datetime,
)
.tolist()
prediction_array = await get_prediction().key_to_array(
key="loadforecast_power_w",
start_datetime=start_datetime,
end_datetime=end_datetime,
)
prediction_list = prediction_array.tolist()
except Exception as e:
raise HTTPException(
status_code=404,
@@ -1358,24 +1413,18 @@ async def fastapi_pvforecast() -> ForecastResponse:
start_datetime = to_datetime().start_of("day")
end_datetime = start_datetime.add(days=2)
try:
ac_power = (
get_prediction()
.key_to_array(
key="pvforecast_ac_power",
start_datetime=start_datetime,
end_datetime=end_datetime,
)
.tolist()
ac_power_array = await get_prediction().key_to_array(
key="pvforecast_ac_power",
start_datetime=start_datetime,
end_datetime=end_datetime,
)
temp_air = (
get_prediction()
.key_to_array(
key="pvforecastakkudoktor_temp_air",
start_datetime=start_datetime,
end_datetime=end_datetime,
)
.tolist()
ac_power_list = ac_power_array.tolist()
temp_air_array = await get_prediction().key_to_array(
key="pvforecastakkudoktor_temp_air",
start_datetime=start_datetime,
end_datetime=end_datetime,
)
temp_air_list = temp_air_array.tolist()
except Exception as e:
raise HTTPException(
status_code=404,
@@ -1383,7 +1432,7 @@ async def fastapi_pvforecast() -> ForecastResponse:
)
# Return both forecasts as a JSON response
return ForecastResponse(temperature=temp_air, pvpower=ac_power)
return ForecastResponse(temperature=temp_air_list, pvpower=ac_power_list)
@app.post("/optimize", tags=["optimize"])

16
tests/single_test_optimization.py
View File

@@ -24,7 +24,7 @@ prediction_eos = get_prediction()
ems_eos = get_ems()
def prepare_optimization_real_parameters() -> GeneticOptimizationParameters:
async def prepare_optimization_real_parameters() -> GeneticOptimizationParameters:
"""Prepare and return optimization parameters with real world data.
Returns:
@@ -43,6 +43,7 @@ def prepare_optimization_real_parameters() -> GeneticOptimizationParameters:
"optimization": {
"horizon_hours": 24,
"interval": 3600,
"algorithm": "GENETIC",
"genetic": {
"individuals": 300,
"generations": 400,
@@ -129,10 +130,10 @@ def prepare_optimization_real_parameters() -> GeneticOptimizationParameters:
print(
f"Real data prediction from {prediction_eos.ems_start_datetime} to {prediction_eos.end_datetime}"
)
prediction_eos.update_data()
await prediction_eos.update_data()
# PV Forecast (in W)
pv_forecast = prediction_eos.key_to_array(
pv_forecast = await prediction_eos.key_to_array(
key="pvforecast_ac_power",
start_datetime=prediction_eos.ems_start_datetime,
end_datetime=prediction_eos.end_datetime,
@@ -140,7 +141,7 @@ def prepare_optimization_real_parameters() -> GeneticOptimizationParameters:
print(f"pv_forecast: {pv_forecast}")
# Temperature Forecast (in degree C)
temperature_forecast = prediction_eos.key_to_array(
temperature_forecast = await prediction_eos.key_to_array(
key="weather_temp_air",
start_datetime=prediction_eos.ems_start_datetime,
end_datetime=prediction_eos.end_datetime,
@@ -148,7 +149,7 @@ def prepare_optimization_real_parameters() -> GeneticOptimizationParameters:
print(f"temperature_forecast: {temperature_forecast}")
# Electricity Price (in Euro per Wh)
strompreis_euro_pro_wh = prediction_eos.key_to_array(
strompreis_euro_pro_wh = await prediction_eos.key_to_array(
key="elecprice_marketprice_wh",
start_datetime=prediction_eos.ems_start_datetime,
end_datetime=prediction_eos.end_datetime,
@@ -156,7 +157,7 @@ def prepare_optimization_real_parameters() -> GeneticOptimizationParameters:
print(f"strompreis_euro_pro_wh: {strompreis_euro_pro_wh}")
# Overall System Load (in W)
gesamtlast = prediction_eos.key_to_array(
gesamtlast = await prediction_eos.key_to_array(
key="load_mean",
start_datetime=prediction_eos.ems_start_datetime,
end_datetime=prediction_eos.end_datetime,
@@ -215,6 +216,7 @@ def prepare_optimization_parameters() -> GeneticOptimizationParameters:
"optimization": {
"horizon_hours": 48,
"interval": 3600,
"algorithm": "GENETIC",
"genetic": {
"individuals": 300,
"generations": 400,
@@ -414,7 +416,7 @@ def run_optimization(
with open(parameters_file, "r") as f:
parameters = GeneticOptimizationParameters(**json.load(f))
elif real_world:
parameters = prepare_optimization_real_parameters()
parameters = asyncio.run(prepare_optimization_real_parameters())
else:
parameters = prepare_optimization_parameters()
logger.info("Optimization Parameters:")

7
tests/single_test_prediction.py
View File

@@ -1,6 +1,7 @@
#!/usr/bin/env python3
import argparse
import asyncio
import cProfile
import pstats
import sys
@@ -159,7 +160,7 @@ def run_prediction(provider_id: str, verbose: bool = False) -> str:
provider = prediction_eos.provider_by_id(provider_id)
prediction_eos.update_data()
asyncio.run(prediction_eos.update_data())
# Return result of prediction
if verbose:
@@ -176,7 +177,7 @@ def run_prediction(provider_id: str, verbose: bool = False) -> str:
print(f"enabled: {provider.enabled()}")
for key in provider.record_keys:
print(f"\n{key}\n----------")
print(f"Array: {provider.key_to_array(key)}")
print(f"Array: {asyncio.run(provider.key_to_array(key))}")
return provider.model_dump_json(indent=4)
@@ -225,4 +226,4 @@ def main():
if __name__ == "__main__":
main()
asyncio.run(main())

25
tests/test_adapternodered.py
View File

@@ -38,6 +38,7 @@ def adapter(config_eos, mock_ems: MagicMock) -> NodeREDAdapter:
return ad
@pytest.mark.asyncio
class TestNodeREDAdapter:
def test_provider_id(self, adapter: NodeREDAdapter):
@@ -52,37 +53,37 @@ class TestNodeREDAdapter:
assert adapter.enabled() is True
@patch("requests.get")
def test_update_datetime(self, mock_get, adapter: NodeREDAdapter):
async def test_update_datetime(self, mock_get, adapter: NodeREDAdapter):
adapter.ems.stage.return_value = EnergyManagementStage.DATA_ACQUISITION
mock_get.return_value.status_code = 200
mock_get.return_value.json.return_value = {"foo": "bar"}
now = to_datetime()
adapter.update_data(force_enable=True)
await adapter.update_data(force_enable=True)
mock_get.assert_called_once()
assert compare_datetimes(adapter.update_datetime, now).approximately_equal
@patch("requests.get")
def test_update_data_data_acquisition_success(self, mock_get , adapter: NodeREDAdapter):
async def test_update_data_data_acquisition_success(self, mock_get , adapter: NodeREDAdapter):
adapter.ems.stage.return_value = EnergyManagementStage.DATA_ACQUISITION
mock_get.return_value.status_code = 200
mock_get.return_value.json.return_value = {"foo": "bar"}
adapter.update_data(force_enable=True)
await adapter.update_data(force_enable=True)
mock_get.assert_called_once()
url, = mock_get.call_args[0]
assert "/eos/data_aquisition" in url
@patch("requests.get", side_effect=Exception("boom"))
def test_update_data_data_acquisition_failure(self, mock_get, adapter: NodeREDAdapter):
async def test_update_data_data_acquisition_failure(self, mock_get, adapter: NodeREDAdapter):
adapter.ems.stage.return_value = EnergyManagementStage.DATA_ACQUISITION
with pytest.raises(RuntimeError):
adapter.update_data(force_enable=True)
await adapter.update_data(force_enable=True)
@patch("requests.post")
def test_update_data_control_dispatch_instructions(self, mock_post, adapter: NodeREDAdapter):
async def test_update_data_control_dispatch_instructions(self, mock_post, adapter: NodeREDAdapter):
adapter.ems.stage.return_value = EnergyManagementStage.CONTROL_DISPATCH
instr1 = DDBCInstruction(
@@ -98,7 +99,7 @@ class TestNodeREDAdapter:
mock_post.return_value.status_code = 200
mock_post.return_value.json.return_value = {}
adapter.update_data(force_enable=True)
await adapter.update_data(force_enable=True)
_, kwargs = mock_post.call_args
payload = kwargs["json"]
@@ -110,18 +111,18 @@ class TestNodeREDAdapter:
assert "/eos/control_dispatch" in url
@patch("requests.post")
def test_update_data_disabled_provider(self, mock_post, adapter: NodeREDAdapter):
async def test_update_data_disabled_provider(self, mock_post, adapter: NodeREDAdapter):
adapter.config.adapter.provider = ["HomeAssistant"] # NodeRED disabled
adapter.update_data(force_enable=False)
await adapter.update_data(force_enable=False)
mock_post.assert_not_called()
@patch("requests.post")
def test_update_data_force_enable_overrides_disabled(self, mock_post, adapter: NodeREDAdapter):
async def test_update_data_force_enable_overrides_disabled(self, mock_post, adapter: NodeREDAdapter):
adapter.config.adapter.provider = ["HomeAssistant"]
adapter.ems.stage.return_value = EnergyManagementStage.CONTROL_DISPATCH
mock_post.return_value.status_code = 200
mock_post.return_value.json.return_value = {}
adapter.update_data(force_enable=True)
await adapter.update_data(force_enable=True)
mock_post.assert_called_once()

1627
tests/test_dataabc.py
View File

File diff suppressed because it is too large Load Diff

544
tests/test_dataabccompact.py
View File

File diff suppressed because it is too large Load Diff

308
tests/test_dataabccontainer.py Normal file
View File

@@ -0,0 +1,308 @@
import asyncio
import json
from datetime import datetime, timezone
from typing import Any, ClassVar, List, Optional, Union
import numpy as np
import pandas as pd
import pendulum
import pytest
import pytest_asyncio
from pydantic import Field, PrivateAttr
from akkudoktoreos.config.configabc import SettingsBaseModel
from akkudoktoreos.core.coreabc import get_ems
from akkudoktoreos.core.dataabc import (
DataABC,
DataContainer,
DataImportProvider,
DataProvider,
DataRecord,
DataSequence,
)
from akkudoktoreos.core.databaseabc import DatabaseTimestamp
from akkudoktoreos.utils.datetimeutil import compare_datetimes, to_datetime, to_duration
# ---------------------------------------------------------------------------
# Derived classes for testing
# ---------------------------------------------------------------------------
class DerivedRecord(DataRecord):
"""DataRecord with a numeric field and configured field-like data."""
data_value: Optional[float] = Field(default=None, description="Data Value")
@classmethod
def configured_data_keys(cls) -> Optional[list[str]]:
return ["dish_washer_emr", "solar_power", "temp"]
class DerivedDataProvider(DataProvider):
"""Concrete DataProvider for testing."""
records: List[DerivedRecord] = Field(
default_factory=list, description="List of DerivedRecord records"
)
provider_enabled: ClassVar[bool] = True
provider_updated: ClassVar[bool] = False
@classmethod
def record_class(cls) -> Any:
return DerivedRecord
def db_namespace(self) -> str:
return "DerivedDataProvider"
def provider_id(self) -> str:
return "DerivedDataProvider"
def enabled(self) -> bool:
return self.provider_enabled
async def _update_data(self, force_update: Optional[bool] = False) -> None:
DerivedDataProvider.provider_updated = True
class DerivedDataContainer(DataContainer):
providers: List[Union[DerivedDataProvider, DataProvider]] = Field(
default_factory=list, description="List of data providers"
)
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def make_record(date, value: float) -> DerivedRecord:
return DerivedRecord(date_time=to_datetime(date), data_value=value)
async def make_provider_with_records() -> DerivedDataProvider:
"""Return a fresh provider with three hourly records."""
provider = DerivedDataProvider()
await provider.delete_by_datetime() # wipe singleton state
await provider.insert_by_datetime(make_record(datetime(2024, 1, 1, 0), 1.0))
await provider.insert_by_datetime(make_record(datetime(2024, 1, 1, 1), 2.0))
await provider.insert_by_datetime(make_record(datetime(2024, 1, 1, 2), 3.0))
return provider
async def make_container() -> DerivedDataContainer:
"""Return a container with one populated provider."""
provider = await make_provider_with_records()
container = DerivedDataContainer()
container.providers.clear()
container.providers.append(provider)
return container
# ---------------------------------------------------------------------------
# Tests
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
class TestDataContainer:
# -----------------------------------------------------------------------
# Fixtures
# -----------------------------------------------------------------------
@pytest_asyncio.fixture
async def container(self):
"""Empty container (no providers)."""
c = DerivedDataContainer()
c.providers.clear()
return c
@pytest_asyncio.fixture
async def populated(self):
"""Container with one provider holding three records."""
return await make_container()
# -----------------------------------------------------------------------
# Provider management
# -----------------------------------------------------------------------
async def test_append_provider(self, container):
assert len(container.providers) == 0
provider = DerivedDataProvider()
container.providers.append(provider)
assert len(container.providers) == 1
assert isinstance(container.providers[0], DerivedDataProvider)
async def test_enabled_providers_reflects_enabled_flag(self, populated):
DerivedDataProvider.provider_enabled = True
assert len(populated.enabled_providers) == 1
DerivedDataProvider.provider_enabled = False
assert len(populated.enabled_providers) == 0
DerivedDataProvider.provider_enabled = True # restore
async def test_provider_by_id_found(self, populated):
provider = populated.provider_by_id("DerivedDataProvider")
assert isinstance(provider, DerivedDataProvider)
async def test_provider_by_id_unknown_raises(self, populated):
with pytest.raises(ValueError, match="Unknown provider id"):
populated.provider_by_id("NonExistentProvider")
# -----------------------------------------------------------------------
# record_keys / record_keys_writable
# -----------------------------------------------------------------------
async def test_record_keys_contains_expected_fields(self, populated):
keys = populated.record_keys
assert "data_value" in keys
assert "date_time" in keys
# configured keys
for k in ("dish_washer_emr", "solar_power", "temp"):
assert k in keys
async def test_record_keys_writable_contains_expected_fields(self, populated):
keys = populated.record_keys_writable
assert "data_value" in keys
for k in ("dish_washer_emr", "solar_power", "temp"):
assert k in keys
async def test_record_keys_empty_when_no_providers(self, container):
assert container.record_keys == []
assert container.record_keys_writable == []
# -----------------------------------------------------------------------
# iter / len / repr / keys()
# -----------------------------------------------------------------------
async def test_iter_yields_record_keys(self, populated):
keys = list(populated)
assert "data_value" in keys
async def test_len_equals_number_of_record_keys(self, populated):
assert len(populated) == len(populated.record_keys)
async def test_repr_contains_class_and_provider(self, populated):
r = repr(populated)
assert r.startswith("DerivedDataContainer(")
assert "DerivedDataProvider" in r
async def test_keys_view(self, populated):
kv = populated.keys()
assert "data_value" in kv
# -----------------------------------------------------------------------
# key_to_series
# -----------------------------------------------------------------------
async def test_key_to_series_returns_series(self, populated):
series = await populated.key_to_series("data_value")
assert isinstance(series, pd.Series)
assert series.name == "data_value"
async def test_key_to_series_values(self, populated):
series = await populated.key_to_series("data_value")
assert sorted(series.tolist()) == [1.0, 2.0, 3.0]
async def test_key_to_series_with_datetime_range(self, populated):
start = to_datetime(datetime(2024, 1, 1, 1))
end = to_datetime(datetime(2024, 1, 1, 3))
series = await populated.key_to_series("data_value", start_datetime=start, end_datetime=end)
assert len(series) == 2
assert sorted(series.tolist()) == [2.0, 3.0]
async def test_key_to_series_unknown_key_raises(self, populated):
with pytest.raises(KeyError, match="No data found for key"):
await populated.key_to_series("non_existent_key")
async def test_key_to_series_no_enabled_providers_raises(self, populated):
DerivedDataProvider.provider_enabled = False
try:
with pytest.raises(KeyError, match="No data found for key"):
await populated.key_to_series("data_value")
finally:
DerivedDataProvider.provider_enabled = True
# -----------------------------------------------------------------------
# key_to_array
# -----------------------------------------------------------------------
async def test_key_to_array_returns_ndarray(self, populated):
start = to_datetime(datetime(2024, 1, 1, 0))
end = to_datetime(datetime(2024, 1, 1, 3))
array = await populated.key_to_array("data_value", start_datetime=start, end_datetime=end)
assert isinstance(array, np.ndarray)
assert len(array) == 3
async def test_key_to_array_unknown_key_raises(self, populated):
with pytest.raises(KeyError, match="No data found for key"):
await populated.key_to_array("non_existent_key")
# -----------------------------------------------------------------------
# update_data
# -----------------------------------------------------------------------
async def test_update_data_calls_provider(self, populated):
DerivedDataProvider.provider_updated = False
DerivedDataProvider.provider_enabled = True
await populated.update_data(force_enable=True)
assert DerivedDataProvider.provider_updated is True
async def test_update_data_skips_disabled_provider(self, populated):
DerivedDataProvider.provider_enabled = False
DerivedDataProvider.provider_updated = False
await populated.update_data()
assert DerivedDataProvider.provider_updated is False
DerivedDataProvider.provider_enabled = True # restore
async def test_update_data_force_enable_runs_disabled_provider(self, populated):
DerivedDataProvider.provider_enabled = False
DerivedDataProvider.provider_updated = False
await populated.update_data(force_enable=True)
assert DerivedDataProvider.provider_updated is True
DerivedDataProvider.provider_enabled = True # restore
# -----------------------------------------------------------------------
# save / load
# -----------------------------------------------------------------------
async def test_save_and_load_roundtrip(self, populated):
"""Save then wipe in-memory records and verify load restores data if db is available."""
start = to_datetime(datetime(2024, 1, 1, 0))
end = to_datetime(datetime(2024, 1, 1, 3))
# Confirm data is present before save
series_before = await populated.key_to_series(
"data_value", start_datetime=start, end_datetime=end
)
assert sorted(series_before.tolist()) == [1.0, 2.0, 3.0]
for provider in populated.providers:
assert provider.db_enabled == False
saved = await populated.save()
if not saved:
# No database configured — verify save correctly reported nothing was persisted
pytest.skip("No database configured, skipping roundtrip persistence check")
# Wipe in-memory state only (not the database)
for provider in populated.providers:
provider.records.clear()
assert all(len(p.records) == 0 for p in populated.providers)
loaded = await populated.load()
assert loaded is True
# Verify data is restored via the public async API
series_after = await populated.key_to_series(
"data_value", start_datetime=start, end_datetime=end
)
assert sorted(series_after.tolist()) == [1.0, 2.0, 3.0]
# -----------------------------------------------------------------------
# db_get_stats
# -----------------------------------------------------------------------
async def test_db_get_stats_returns_dict(self, populated):
stats = await populated.db_get_stats()
assert isinstance(stats, dict)
assert "DerivedDataProvider" in stats

332
tests/test_dataabcprovider.py Normal file
View File

@@ -0,0 +1,332 @@
import asyncio
import json
from datetime import datetime, timezone
from typing import Any, ClassVar, List, Optional, Union
import numpy as np
import pandas as pd
import pendulum
import pytest
from pydantic import Field, PrivateAttr, ValidationError
from akkudoktoreos.config.configabc import SettingsBaseModel
from akkudoktoreos.core.coreabc import get_ems
from akkudoktoreos.core.dataabc import (
DataABC,
DataContainer,
DataImportProvider,
DataProvider,
DataRecord,
DataSequence,
)
from akkudoktoreos.core.databaseabc import DatabaseTimestamp
from akkudoktoreos.utils.datetimeutil import compare_datetimes, to_datetime, to_duration
# Derived classes for testing
# ---------------------------
class DerivedConfig(SettingsBaseModel):
env_var: Optional[int] = Field(default=None, description="Test config by environment var")
instance_field: Optional[str] = Field(default=None, description="Test config by instance field")
class_constant: Optional[int] = Field(default=None, description="Test config by class constant")
class DerivedBase(DataABC):
instance_field: Optional[str] = Field(default=None, description="Field Value")
class_constant: ClassVar[int] = 30
class DerivedRecord(DataRecord):
"""Date Record derived from base class DataRecord.
The derived data record got the
- `data_value` field and the
- `dish_washer_emr`, `solar_power`, `temp` configurable field like data.
"""
data_value: Optional[float] = Field(default=None, description="Data Value")
@classmethod
def configured_data_keys(cls) -> Optional[list[str]]:
return ["dish_washer_emr", "solar_power", "temp"]
class DerivedSequence(DataSequence):
# overload
records: List[DerivedRecord] = Field(
default_factory=list, description="List of DerivedRecord records"
)
@classmethod
def record_class(cls) -> Any:
return DerivedRecord
def db_namespace(self) -> str:
return "DerivedSequence"
class DerivedSequence2(DataSequence):
# overload
records: List[DerivedRecord] = Field(
default_factory=list, description="List of DerivedRecord records"
)
@classmethod
def record_class(cls) -> Any:
return DerivedRecord
def db_namespace(self) -> str:
return "DerivedSequence2"
class DerivedDataProvider(DataProvider):
"""A concrete subclass of DataProvider for testing purposes."""
# overload
records: List[DerivedRecord] = Field(
default_factory=list, description="List of DerivedRecord records"
)
provider_enabled: ClassVar[bool] = False
provider_updated: ClassVar[bool] = False
@classmethod
def record_class(cls) -> Any:
return DerivedRecord
def db_namespace(self) -> str:
return "DerivedDataProvider"
# Implement abstract methods for test purposes
def provider_id(self) -> str:
return "DerivedDataProvider"
def enabled(self) -> bool:
return self.provider_enabled
async def _update_data(self, force_update: Optional[bool] = False) -> None:
# Simulate update logic
DerivedDataProvider.provider_updated = True
class DerivedDataImportProvider(DataImportProvider):
"""A concrete subclass of DataImportProvider for testing purposes."""
# overload
records: List[DerivedRecord] = Field(
default_factory=list, description="List of DerivedRecord records"
)
provider_enabled: ClassVar[bool] = False
provider_updated: ClassVar[bool] = False
_updates: list = PrivateAttr(default_factory=list)
@classmethod
def record_class(cls) -> Any:
return DerivedRecord
# Implement abstract methods for test purposes
def provider_id(self) -> str:
return "DerivedDataImportProvider"
def enabled(self) -> bool:
return self.provider_enabled
async def _update_data(self, force_update: Optional[bool] = False) -> None:
# Simulate update logic
DerivedDataProvider.provider_updated = True
async def _update_value(self, date, *args, **kwargs) -> None:
# Simulate update logic
self._updates.append((date, args, kwargs))
await super()._update_value(date, *args, **kwargs)
class DerivedDataContainer(DataContainer):
providers: List[Union[DerivedDataProvider, DataProvider]] = Field(
default_factory=list, description="List of data providers"
)
# Tests
# ----------
@pytest.mark.asyncio
class TestDataProvider:
# Fixtures and helper functions
@pytest.fixture
def provider(self):
"""Fixture to provide an instance of TestDataProvider for testing."""
DerivedDataProvider.provider_enabled = True
DerivedDataProvider.provider_updated = False
return DerivedDataProvider()
@pytest.fixture
def sample_start_datetime(self):
"""Fixture for a sample start datetime."""
return to_datetime(datetime(2024, 11, 1, 12, 0))
def create_test_record(self, date, value):
"""Helper function to create a test DataRecord."""
return DerivedRecord(date_time=date, data_value=value)
# Tests
async def test_singleton_behavior(self, provider):
"""Test that DataProvider enforces singleton behavior."""
instance1 = provider
instance2 = DerivedDataProvider()
assert instance1 is instance2, (
"Singleton pattern is not enforced; instances are not the same."
)
async def test_update_method_with_defaults(self, provider, sample_start_datetime, monkeypatch):
"""Test the `update` method with default parameters."""
ems_eos = get_ems()
ems_eos.set_start_datetime(sample_start_datetime)
await provider.update_data()
assert provider.ems_start_datetime == sample_start_datetime
async def test_update_method_force_enable(self, provider, monkeypatch):
"""Test that `update` executes when `force_enable` is True, even if `enabled` is False."""
# Override enabled to return False for this test
DerivedDataProvider.provider_enabled = False
DerivedDataProvider.provider_updated = False
await provider.update_data(force_enable=True)
assert provider.enabled() is False, "Provider should be disabled, but enabled() is True."
assert DerivedDataProvider.provider_updated is True, (
"Provider should have been executed, but was not."
)
async def test_delete_by_datetime(self, provider, sample_start_datetime):
"""Test `delete_by_datetime` method for removing records by datetime range."""
# Add records to the provider for deletion testing
records = [
self.create_test_record(sample_start_datetime - to_duration("3 hours"), 1),
self.create_test_record(sample_start_datetime - to_duration("1 hour"), 2),
self.create_test_record(sample_start_datetime + to_duration("1 hour"), 3),
]
for record in records:
await provider.insert_by_datetime(record)
await provider.delete_by_datetime(
start_datetime=sample_start_datetime - to_duration("2 hours"),
end_datetime=sample_start_datetime + to_duration("2 hours"),
)
assert len(provider.records) == 1, (
"Only one record should remain after deletion by datetime."
)
assert provider.records[0].date_time == sample_start_datetime - to_duration("3 hours"), (
"Unexpected record remains."
)
@pytest.mark.asyncio
class TestDataImportProvider:
@pytest.fixture
def provider(self):
DerivedDataImportProvider.provider_enabled = True
DerivedDataImportProvider.provider_updated = True
p = DerivedDataImportProvider()
p._updates.clear()
p.records.clear()
return p
async def test_import_from_dict_basic(self, provider):
data = {
"start_datetime": "2024-01-01 00:00:00",
"interval": "1 hour",
"solar_power": [1, 2, 3],
}
await provider.import_from_dict(data)
assert provider.records is not None
assert provider.records[0]["solar_power"] == 1
assert provider.records[1]["solar_power"] == 2
async def test_import_from_dict_default_start_and_interval(self, provider):
data = {"solar_power": [10, 20]}
await provider.import_from_dict(data)
assert len(provider._updates) == 2
async def test_import_from_dict_with_prefix(self, provider):
data = {
"dish_washer_emr": [1, 2],
"data_value": [5, 6],
}
await provider.import_from_dict(data, key_prefix="dish")
assert len(provider._updates) == 2
assert all(update[1][0] == "dish_washer_emr" for update in provider._updates)
async def test_import_from_dict_mismatching_lengths(self, provider):
data = {
"solar_power": [1, 2],
"temp": [1],
}
with pytest.raises(ValueError):
await provider.import_from_dict(data)
async def test_import_from_dict_invalid_interval(self, provider):
data = {
"interval": "17 minutes",
"solar_power": [1, 2, 3],
}
with pytest.raises(NotImplementedError):
await provider.import_from_dict(data)
async def test_import_from_dict_skips_none_and_nan(self, provider):
data = {"solar_power": [1, None, np.nan, 4]}
await provider.import_from_dict(data)
assert len(provider._updates) == 2
assert provider._updates[0][1][1] == 1
assert provider._updates[1][1][1] == 4
async def test_import_from_dict_invalid_value_type(self, provider):
data = {"solar_power": "not a list"}
with pytest.raises(ValueError):
await provider.import_from_dict(data)
async def test_import_from_dataframe_with_datetime_index(self, provider):
index = pd.date_range("2024-01-01", periods=3, freq="h")
df = pd.DataFrame({"solar_power": [1, 2, 3]}, index=index)
await provider.import_from_dataframe(df)
assert len(provider._updates) == 3
assert provider._updates[0][1][1] == 1
async def test_import_from_dataframe_without_datetime_index(self, provider):
df = pd.DataFrame({"solar_power": [5, 6, 7]})
await provider.import_from_dataframe(
df,
start_datetime=to_datetime(datetime(2024, 1, 1)),
interval=to_duration("1 hour"),
)
assert len(provider._updates) == 3
async def test_import_from_dataframe_prefix_filter(self, provider):
df = pd.DataFrame({
"dish_washer_emr": [1, 2],
"data_value": [3, 4],
})
await provider.import_from_dataframe(df, key_prefix="dish")
assert len(provider._updates) == 2
assert all(update[1][0] == "dish_washer_emr" for update in provider._updates)
async def test_import_from_dataframe_invalid_input(self, provider):
with pytest.raises(ValueError):
await provider.import_from_dataframe("not a dataframe")
async def test_import_from_json_simple_dict(self, provider):
json_str = json.dumps({"solar_power": [1, 2, 3]})
await provider.import_from_json(json_str)
assert len(provider._updates) == 3
async def test_import_from_json_invalid(self, provider):
with pytest.raises(ValueError):
await provider.import_from_json("this is not json")
async def test_import_from_file(self, provider, tmp_path):
file_path = tmp_path / "data.json"
file_path.write_text(json.dumps({"solar_power": [1, 2]}))
await provider.import_from_file(file_path)
assert len(provider._updates) == 2

312
tests/test_dataabcrecord.py Normal file
View File

@@ -0,0 +1,312 @@
"""Pytest test for data records fro dataabc module."""
import asyncio
import json
from datetime import datetime, timezone
from typing import Any, ClassVar, List, Optional, Union
import numpy as np
import pandas as pd
import pendulum
import pytest
from pydantic import Field, ValidationError
from akkudoktoreos.config.configabc import SettingsBaseModel
from akkudoktoreos.core.dataabc import (
DataABC,
DataRecord,
)
from akkudoktoreos.core.databaseabc import DatabaseTimestamp
from akkudoktoreos.utils.datetimeutil import compare_datetimes, to_datetime, to_duration
# Derived classes for testing
# ---------------------------
class DerivedConfig(SettingsBaseModel):
env_var: Optional[int] = Field(default=None, description="Test config by environment var")
instance_field: Optional[str] = Field(default=None, description="Test config by instance field")
class_constant: Optional[int] = Field(default=None, description="Test config by class constant")
class DerivedBase(DataABC):
instance_field: Optional[str] = Field(default=None, description="Field Value")
class_constant: ClassVar[int] = 30
class DerivedRecord(DataRecord):
"""Date Record derived from base class DataRecord.
The derived data record got the
- `data_value` field and the
- `dish_washer_emr`, `solar_power`, `temp` configurable field like data.
"""
data_value: Optional[float] = Field(default=None, description="Data Value")
@classmethod
def configured_data_keys(cls) -> Optional[list[str]]:
return ["dish_washer_emr", "solar_power", "temp"]
# Tests
# ----------
class TestDataABC:
@pytest.fixture
def base(self):
# Provide default values for configuration
derived = DerivedBase()
return derived
def test_get_config_value_key_error(self, base):
with pytest.raises(AttributeError):
base.config.non_existent_key
class TestDataRecord:
def create_test_record(self, date, value):
"""Helper function to create a test DataRecord."""
return DerivedRecord(date_time=date, data_value=value)
@pytest.fixture
def record(self):
"""Fixture to create a sample DerivedDataRecord with some data set."""
rec = DerivedRecord(date_time=to_datetime("1967-01-11"), data_value=10.0)
rec.configured_data = {"dish_washer_emr": 123.0, "solar_power": 456.0}
return rec
def test_getitem(self):
record = self.create_test_record(datetime(2024, 1, 3, tzinfo=timezone.utc), 10.0)
assert record["data_value"] == 10.0
def test_setitem(self):
record = self.create_test_record(datetime(2024, 1, 3, tzinfo=timezone.utc), 10.0)
record["data_value"] = 20.0
assert record.data_value == 20.0
def test_delitem(self):
record = self.create_test_record(datetime(2024, 1, 3, tzinfo=timezone.utc), 10.0)
record.data_value = 20.0
del record["data_value"]
assert record.data_value is None
def test_len(self):
record = self.create_test_record(datetime(2024, 1, 3, tzinfo=timezone.utc), 10.0)
record.date_time = None
record.data_value = 20.0
assert len(record) == 5 # 2 regular fields + 3 configured data "fields"
def test_to_dict(self):
record = self.create_test_record(datetime(2024, 1, 3, tzinfo=timezone.utc), 10.0)
record.data_value = 20.0
record_dict = record.to_dict()
assert "data_value" in record_dict
assert record_dict["data_value"] == 20.0
record2 = DerivedRecord.from_dict(record_dict)
assert record2.model_dump() == record.model_dump()
def test_to_json(self):
record = self.create_test_record(datetime(2024, 1, 3, tzinfo=timezone.utc), 10.0)
record.data_value = 20.0
json_str = record.to_json()
assert "data_value" in json_str
assert "20.0" in json_str
record2 = DerivedRecord.from_json(json_str)
assert record2.model_dump() == record.model_dump()
def test_record_keys_includes_configured_data_keys(self, record):
"""Ensure record_keys includes all configured configured data keys."""
assert set(record.record_keys()) >= set(record.configured_data_keys())
def test_record_keys_writable_includes_configured_data_keys(self, record):
"""Ensure record_keys_writable includes all configured configured data keys."""
assert set(record.record_keys_writable()) >= set(record.configured_data_keys())
def test_getitem_existing_field(self, record):
"""Test that __getitem__ returns correct value for existing native field."""
record.date_time = "2024-01-01T00:00:00+00:00"
assert record["date_time"] is not None
def test_getitem_existing_configured_data(self, record):
"""Test that __getitem__ retrieves existing configured data values."""
assert record["dish_washer_emr"] == 123.0
assert record["solar_power"] == 456.0
def test_getitem_missing_configured_data_returns_none(self, record):
"""Test that __getitem__ returns None for missing but known configured data keys."""
assert record["temp"] is None
def test_getitem_raises_keyerror(self, record):
"""Test that __getitem__ raises KeyError for completely unknown keys."""
with pytest.raises(KeyError):
_ = record["nonexistent"]
def test_setitem_field(self, record):
"""Test setting a native field using __setitem__."""
record["date_time"] = "2025-01-01T12:00:00+00:00"
assert str(record.date_time).startswith("2025-01-01")
def test_setitem_configured_data(self, record):
"""Test setting a known configured data key using __setitem__."""
record["temp"] = 25.5
assert record.configured_data["temp"] == 25.5
def test_setitem_invalid_key_raises(self, record):
"""Test that __setitem__ raises KeyError for unknown keys."""
with pytest.raises(KeyError):
record["unknown_key"] = 123
def test_delitem_field(self, record):
"""Test deleting a native field using __delitem__."""
record["date_time"] = "2025-01-01T12:00:00+00:00"
del record["date_time"]
assert record.date_time is None
def test_delitem_configured_data(self, record):
"""Test deleting a known configured data key using __delitem__."""
del record["solar_power"]
assert "solar_power" not in record.configured_data
def test_delitem_unknown_raises(self, record):
"""Test that __delitem__ raises KeyError for unknown keys."""
with pytest.raises(KeyError):
del record["nonexistent"]
def test_attribute_get_existing_field(self, record):
"""Test accessing a native field via attribute."""
record.date_time = "2025-01-01T12:00:00+00:00"
assert record.date_time is not None
def test_attribute_get_existing_configured_data(self, record):
"""Test accessing an existing configured data via attribute."""
assert record.dish_washer_emr == 123.0
def test_attribute_get_missing_configured_data(self, record):
"""Test accessing a missing but known configured data returns None."""
assert record.temp is None
def test_attribute_get_invalid_raises(self, record):
"""Test accessing an unknown attribute raises AttributeError."""
with pytest.raises(AttributeError):
_ = record.nonexistent
def test_attribute_set_existing_field(self, record):
"""Test setting a native field via attribute."""
record.date_time = "2025-06-25T12:00:00+00:00"
assert record.date_time is not None
def test_attribute_set_existing_configured_data(self, record):
"""Test setting a known configured data key via attribute."""
record.temp = 99.9
assert record.configured_data["temp"] == 99.9
def test_attribute_set_invalid_raises(self, record):
"""Test setting an unknown attribute raises AttributeError."""
with pytest.raises(AttributeError):
record.invalid = 123
def test_delattr_field(self, record):
"""Test deleting a native field via attribute."""
record.date_time = "2025-06-25T12:00:00+00:00"
del record.date_time
assert record.date_time is None
def test_delattr_configured_data(self, record):
"""Test deleting a known configured data key via attribute."""
record.temp = 88.0
del record.temp
assert "temp" not in record.configured_data
def test_delattr_ignored_missing_configured_data_key(self, record):
"""Test deleting a known configured data key that was never set is a no-op."""
del record.temp
assert "temp" not in record.configured_data
def test_len_and_iter(self, record):
"""Test that __len__ and __iter__ behave as expected."""
keys = list(iter(record))
assert set(record.record_keys_writable()) == set(keys)
assert len(record) == len(keys)
def test_in_operator_includes_configured_data(self, record):
"""Test that 'in' operator includes configured data keys."""
assert "dish_washer_emr" in record
assert "temp" in record # known key, even if not yet set
assert "nonexistent" not in record
def test_hasattr_behavior(self, record):
"""Test that hasattr returns True for fields and known configured dataWs."""
assert hasattr(record, "date_time")
assert hasattr(record, "dish_washer_emr")
assert hasattr(record, "temp") # allowed, even if not yet set
assert not hasattr(record, "nonexistent")
def test_model_validate_roundtrip(self, record):
"""Test that MeasurementDataRecord can be serialized and revalidated."""
dumped = record.model_dump()
restored = DerivedRecord.model_validate(dumped)
assert restored.dish_washer_emr == 123.0
assert restored.solar_power == 456.0
assert restored.temp is None # not set
def test_copy_preserves_configured_data(self, record):
"""Test that copying preserves configured data values."""
record.temp = 22.2
copied = record.model_copy()
assert copied.dish_washer_emr == 123.0
assert copied.temp == 22.2
assert copied is not record
def test_equality_includes_configured_data(self, record):
"""Test that equality includes the `configured data` content."""
other = record.model_copy()
assert record == other
def test_inequality_differs_with_configured_data(self, record):
"""Test that records with different configured datas are not equal."""
other = record.model_copy(deep=True)
# Modify one configured data value in the copy
other.configured_data["dish_washer_emr"] = 999.9
assert record != other
def test_in_operator_for_configured_data_and_fields(self, record):
"""Ensure 'in' works for both fields and configured configured data keys."""
assert "dish_washer_emr" in record
assert "solar_power" in record
assert "date_time" in record # standard field
assert "temp" in record # allowed but not yet set
assert "unknown" not in record
def test_hasattr_equivalence_to_getattr(self, record):
"""hasattr should return True for all valid keys/configured datas."""
assert hasattr(record, "dish_washer_emr")
assert hasattr(record, "temp")
assert hasattr(record, "date_time")
assert not hasattr(record, "nonexistent")
def test_dir_includes_configured_data_keys(self, record):
"""`dir(record)` should include configured data keys for introspection.
It shall not include the internal 'configured datas' attribute.
"""
keys = dir(record)
assert "configured datas" not in keys
for key in record.configured_data_keys():
assert key in keys
def test_init_configured_field_like_data_applies_before_model_init(self):
"""Test that keys listed in `_configured_data_keys` are moved to `configured_data` at init time."""
record = DerivedRecord(
date_time="2024-01-03T00:00:00+00:00",
data_value=42.0,
dish_washer_emr=111.1,
solar_power=222.2,
temp=333.3 # assume `temp` is also a valid configured key
)
assert record.data_value == 42.0
assert record.configured_data == {
"dish_washer_emr": 111.1,
"solar_power": 222.2,
"temp": 333.3,
}

979
tests/test_dataabcsequence.py Normal file
View File

@@ -0,0 +1,979 @@
import asyncio
import json
from datetime import datetime, timezone
from typing import Any, ClassVar, List, Optional, Union
import numpy as np
import pandas as pd
import pendulum
import pytest
import pytest_asyncio
from pydantic import Field, ValidationError
from akkudoktoreos.config.configabc import SettingsBaseModel
from akkudoktoreos.core.coreabc import get_ems
from akkudoktoreos.core.dataabc import (
DataABC,
DataContainer,
DataImportProvider,
DataProvider,
DataRecord,
DataSequence,
)
from akkudoktoreos.core.databaseabc import DatabaseTimestamp
from akkudoktoreos.utils.datetimeutil import compare_datetimes, to_datetime, to_duration
# Derived classes for testing
# ---------------------------
class DerivedConfig(SettingsBaseModel):
env_var: Optional[int] = Field(default=None, description="Test config by environment var")
instance_field: Optional[str] = Field(default=None, description="Test config by instance field")
class_constant: Optional[int] = Field(default=None, description="Test config by class constant")
class DerivedBase(DataABC):
instance_field: Optional[str] = Field(default=None, description="Field Value")
class_constant: ClassVar[int] = 30
class DerivedRecord(DataRecord):
"""Date Record derived from base class DataRecord.
The derived data record got the
- `data_value` field and the
- `dish_washer_emr`, `solar_power`, `temp` configurable field like data.
"""
data_value: Optional[float] = Field(default=None, description="Data Value")
@classmethod
def configured_data_keys(cls) -> Optional[list[str]]:
return ["dish_washer_emr", "solar_power", "temp"]
class DerivedSequence(DataSequence):
# overload
records: List[DerivedRecord] = Field(
default_factory=list, description="List of DerivedRecord records"
)
@classmethod
def record_class(cls) -> Any:
return DerivedRecord
def db_namespace(self) -> str:
return "DerivedSequence"
class DerivedSequence2(DataSequence):
# overload
records: List[DerivedRecord] = Field(
default_factory=list, description="List of DerivedRecord records"
)
@classmethod
def record_class(cls) -> Any:
return DerivedRecord
def db_namespace(self) -> str:
return "DerivedSequence2"
# Tests
# ----------
@pytest.mark.asyncio
class TestDataSequence:
@pytest_asyncio.fixture
async def sequence(self):
sequence0 = DerivedSequence()
mem_len = len(sequence0)
db_len = await sequence0.db_count_records()
assert mem_len == 0
assert db_len == 0
return sequence0
@pytest_asyncio.fixture
async def sequence2(self):
sequence = DerivedSequence()
record1 = self.create_test_record(datetime(1970, 1, 1), 1970)
record2 = self.create_test_record(datetime(1971, 1, 1), 1971)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
mem_len = len(sequence)
db_len = await sequence.db_count_records()
assert mem_len == 2
assert db_len == 2
return sequence
def create_test_record(self, date, value):
"""Helper function to create a test DataRecord."""
return DerivedRecord(date_time=date, data_value=value)
# Test cases
@pytest.mark.parametrize("tz_name", ["UTC", "Europe/Berlin", "Atlantic/Canary"])
async def test_min_max_datetime_timezone_and_order(self, sequence, tz_name, monkeypatch, config_eos):
# Monkeypatch the read-only timezone property
monkeypatch.setattr(config_eos.general.__class__, "timezone", property(lambda self: tz_name))
# Create timezone-aware datetimes using the patched config
dt_early = to_datetime("2024-01-01T00:00:00", in_timezone=config_eos.general.timezone)
dt_late = to_datetime("2024-01-02T00:00:00", in_timezone=config_eos.general.timezone)
# Insert in reverse order to verify sorting
record1 = self.create_test_record(dt_late, 1)
record2 = self.create_test_record(dt_early, 2)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
min_dt = await sequence.min_datetime()
max_dt = await sequence.max_datetime()
# --- Basic correctness ---
assert min_dt == dt_early
assert max_dt == dt_late
# --- Must be timezone aware ---
assert min_dt.tzinfo is not None
assert max_dt.tzinfo is not None
# --- Must preserve timezone ---
assert min_dt.tzinfo.name == tz_name
assert max_dt.tzinfo.name == tz_name
async def test_get_by_datetime(self, sequence):
assert len(sequence) == 0
dt = to_datetime("2024-01-01 00:00:00")
record = self.create_test_record(dt, 0)
await sequence.insert_by_datetime(record)
item = await sequence.get_by_datetime(dt)
assert isinstance(item, DerivedRecord)
async def test_insert_by_datetime(self, sequence2):
dt = to_datetime("2024-01-03", in_timezone="UTC")
record = self.create_test_record(dt, 1)
await sequence2.insert_by_datetime(record)
assert sequence2.records[2].date_time == dt
async def test_insert_reversed_date_record(self, sequence2):
dt1 = to_datetime("2023-11-05", in_timezone="UTC")
dt2 = to_datetime("2024-01-03", in_timezone="UTC")
record1 = self.create_test_record(dt2, 0.8)
record2 = self.create_test_record(dt1, 0.9) # reversed date
await sequence2.insert_by_datetime(record1)
assert sequence2.records[2].date_time == dt2
await sequence2.insert_by_datetime(record2)
assert len(sequence2) == 4
assert sequence2.records[2] == record2
async def test_insert_duplicate_date_record(self, sequence):
dt1 = to_datetime("2023-11-05")
record1 = self.create_test_record(dt1, 0.8)
record2 = self.create_test_record(dt1, 0.9) # Duplicate date
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
assert len(sequence) == 1
retrieved_record = await sequence.get_by_datetime(dt1)
assert retrieved_record.data_value == 0.9 # Record should have merged with new value
async def test_key_to_series(self, sequence):
dt = to_datetime(datetime(2023, 11, 6))
record = self.create_test_record(dt, 0.8)
await sequence.insert_by_datetime(record)
series = await sequence.key_to_series("data_value")
assert isinstance(series, pd.Series)
retrieved_record = await sequence.get_by_datetime(dt)
assert retrieved_record is not None
assert retrieved_record.data_value == 0.8
async def test_key_from_series(self, sequence):
dt1 = to_datetime(datetime(2023, 11, 5))
dt2 = to_datetime(datetime(2023, 11, 6))
series = pd.Series(
data=[0.8, 0.9], index=pd.to_datetime([dt1, dt2])
)
await sequence.key_from_series("data_value", series)
assert len(sequence) == 2
record1 = await sequence.get_by_datetime(dt1)
assert record1 is not None
assert record1.data_value == 0.8
record2 = await sequence.get_by_datetime(dt2)
assert record2 is not None
assert record2.data_value == 0.9
async def test_key_to_array(self, sequence):
interval = to_duration("1 day")
start_datetime = to_datetime("2023-11-6")
last_datetime = to_datetime("2023-11-8")
end_datetime = to_datetime("2023-11-9")
record1 = self.create_test_record(start_datetime, float(start_datetime.day))
await sequence.insert_by_datetime(record1)
record2 = self.create_test_record(last_datetime, float(last_datetime.day))
await sequence.insert_by_datetime(record2)
retrieved_record1 = await sequence.get_by_datetime(start_datetime)
assert retrieved_record1 is not None
assert retrieved_record1.data_value == 6.0
retrieved_record2 = await sequence.get_by_datetime(last_datetime)
assert retrieved_record2 is not None
assert retrieved_record2.data_value == 8.0
series = await sequence.key_to_series(
key="data_value", start_datetime=start_datetime, end_datetime=end_datetime
)
assert len(series) == 2
assert series[to_datetime("2023-11-6")] == 6
assert series[to_datetime("2023-11-8")] == 8
array = await sequence.key_to_array(
key="data_value",
start_datetime=start_datetime,
end_datetime=end_datetime,
interval=interval,
)
assert isinstance(array, np.ndarray)
np.testing.assert_equal(array, [6.0, 7.0, 8.0])
async def test_key_to_array_linear_interpolation(self, sequence):
"""Test key_to_array with linear interpolation for numeric data."""
interval = to_duration("1 hour")
record1 = self.create_test_record(pendulum.datetime(2023, 11, 6, 0), 0.8)
record2 = self.create_test_record(pendulum.datetime(2023, 11, 6, 2), 1.0) # Gap of 2 hours
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
array = await sequence.key_to_array(
key="data_value",
start_datetime=pendulum.datetime(2023, 11, 6),
end_datetime=pendulum.datetime(2023, 11, 6, 3),
interval=interval,
fill_method="linear",
)
assert len(array) == 3
assert array[0] == 0.8
assert array[1] == 0.9 # Interpolated value
assert array[2] == 1.0
async def test_key_to_array_linear_interpolation_out_of_grid(self, sequence):
"""Test key_to_array with linear interpolation out of grid."""
interval = to_duration("1 hour")
start_datetime= to_datetime("2023-11-06T00:30:00") # out of grid
end_datetime=to_datetime("2023-11-06T01:30:00") # out of grid
record1_datetime = to_datetime("2023-11-06T00:00:00")
record1 = self.create_test_record(record1_datetime, 1.0)
record2_datetime = to_datetime("2023-11-06T02:00:00")
record2 = self.create_test_record(record2_datetime, 2.0) # Gap of 2 hours
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
# Check test setup
record1_timestamp = DatabaseTimestamp.from_datetime(record1_datetime)
record2_timestamp = DatabaseTimestamp.from_datetime(record2_datetime)
start_timestamp = DatabaseTimestamp.from_datetime(start_datetime)
end_timestamp = DatabaseTimestamp.from_datetime(end_datetime)
start_previous_timestamp = await sequence.db_previous_timestamp(start_timestamp)
assert start_previous_timestamp == record1_timestamp
end_next_timestamp = await sequence.db_next_timestamp(end_timestamp)
assert end_next_timestamp == record2_timestamp
# Test
array = await sequence.key_to_array(
key="data_value",
start_datetime=start_datetime,
end_datetime=end_datetime,
interval=interval,
fill_method="linear",
boundary="context",
)
np.testing.assert_equal(array, [1.5])
async def test_key_to_array_ffill(self, sequence):
"""Test key_to_array with forward filling for missing values."""
interval = to_duration("1 hour")
record1 = self.create_test_record(pendulum.datetime(2023, 11, 6, 0), 0.8)
record2 = self.create_test_record(pendulum.datetime(2023, 11, 6, 2), 1.0)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
array = await sequence.key_to_array(
key="data_value",
start_datetime=pendulum.datetime(2023, 11, 6),
end_datetime=pendulum.datetime(2023, 11, 6, 3),
interval=interval,
fill_method="ffill",
)
assert len(array) == 3
assert array[0] == 0.8
assert array[1] == 0.8 # Forward-filled value
assert array[2] == 1.0
async def test_key_to_array_ffill_one_value(self, sequence):
"""Test key_to_array with forward filling for missing values and only one value at end available."""
interval = to_duration("1 hour")
record1 = self.create_test_record(pendulum.datetime(2023, 11, 6, 2), 1.0)
await sequence.insert_by_datetime(record1)
array = await sequence.key_to_array(
key="data_value",
start_datetime=pendulum.datetime(2023, 11, 6),
end_datetime=pendulum.datetime(2023, 11, 6, 4),
interval=interval,
fill_method="ffill",
)
assert len(array) == 4
assert array[0] == 1.0 # Backward-filled value
assert array[1] == 1.0 # Backward-filled value
assert array[2] == 1.0
assert array[2] == 1.0 # Forward-filled value
async def test_key_to_array_bfill(self, sequence):
"""Test key_to_array with backward filling for missing values."""
interval = to_duration("1 hour")
record1 = self.create_test_record(pendulum.datetime(2023, 11, 6, 0), 0.8)
record2 = self.create_test_record(pendulum.datetime(2023, 11, 6, 2), 1.0)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
array = await sequence.key_to_array(
key="data_value",
start_datetime=pendulum.datetime(2023, 11, 6),
end_datetime=pendulum.datetime(2023, 11, 6, 3),
interval=interval,
fill_method="bfill",
)
assert len(array) == 3
assert array[0] == 0.8
assert array[1] == 1.0 # Backward-filled value
assert array[2] == 1.0
async def test_key_to_array_with_truncation(self, sequence):
"""Test truncation behavior in key_to_array."""
interval = to_duration("1 hour")
record1 = self.create_test_record(pendulum.datetime(2023, 11, 5, 23), 0.8)
record2 = self.create_test_record(pendulum.datetime(2023, 11, 6, 1), 1.0)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
#assert sequence is None
array = await sequence.key_to_array(
key="data_value",
start_datetime=pendulum.datetime(2023, 11, 5, 23),
end_datetime=pendulum.datetime(2023, 11, 6, 2),
interval=interval,
)
assert len(array) == 3
assert array[0] == 0.8
assert array[1] == 0.9 # Interpolated from previous day
assert array[2] == 1.0
async def test_key_to_array_with_none(self, sequence):
"""Test handling of empty series in key_to_array."""
interval = to_duration("1 hour")
array = await sequence.key_to_array(
key="data_value",
start_datetime=pendulum.datetime(2023, 11, 6),
end_datetime=pendulum.datetime(2023, 11, 6, 3),
interval=interval,
)
assert isinstance(array, np.ndarray)
assert np.all(array == None)
async def test_key_to_array_with_one(self, sequence):
"""Test handling of one element series in key_to_array."""
interval = to_duration("1 hour")
record1 = self.create_test_record(pendulum.datetime(2023, 11, 5, 23), 0.8)
await sequence.insert_by_datetime(record1)
array = await sequence.key_to_array(
key="data_value",
start_datetime=pendulum.datetime(2023, 11, 5, 23),
end_datetime=pendulum.datetime(2023, 11, 6, 2),
interval=interval,
)
assert len(array) == 3
assert array[0] == 0.8
assert array[1] == 0.8 # Interpolated from previous day
assert array[2] == 0.8 # Interpolated from previous day
async def test_key_to_array_invalid_fill_method(self, sequence):
"""Test invalid fill_method raises an error."""
interval = to_duration("1 hour")
record1 = self.create_test_record(pendulum.datetime(2023, 11, 6, 0), 0.8)
await sequence.insert_by_datetime(record1)
with pytest.raises(ValueError, match="Unsupported fill method: invalid"):
await sequence.key_to_array(
key="data_value",
start_datetime=pendulum.datetime(2023, 11, 6),
end_datetime=pendulum.datetime(2023, 11, 6, 1),
interval=interval,
fill_method="invalid",
)
async def test_key_to_array_resample_mean(self, sequence):
"""Test that numeric resampling uses mean when multiple values fall into one interval."""
interval = to_duration("1 hour")
# Insert values every 15 minutes within the same hour
record1 = self.create_test_record(pendulum.datetime(2023, 11, 6, 0, 0), 1.0)
record2 = self.create_test_record(pendulum.datetime(2023, 11, 6, 0, 15), 2.0)
record3 = self.create_test_record(pendulum.datetime(2023, 11, 6, 0, 30), 3.0)
record4 = self.create_test_record(pendulum.datetime(2023, 11, 6, 0, 45), 4.0)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
await sequence.insert_by_datetime(record3)
await sequence.insert_by_datetime(record4)
# Resample to hourly interval, expecting the mean of the 4 values
array = await sequence.key_to_array(
key="data_value",
start_datetime=pendulum.datetime(2023, 11, 6, 0),
end_datetime=pendulum.datetime(2023, 11, 6, 1),
interval=interval,
)
assert isinstance(array, np.ndarray)
assert len(array) == 1 # one interval: 0:00-1:00
# The first interval mean = (1+2+3+4)/4 = 2.5
assert array[0] == pytest.approx(2.5)
# ------------------------------------------------------------------
# key_to_array — align_to_interval parameter
# ------------------------------------------------------------------
#
# The existing tests above use start_datetime values that already sit on
# clean hour/day boundaries, so the default alignment (origin=query_start)
# and clock alignment (origin=epoch-floor) produce identical results.
# The tests below specifically use off-boundary start times to expose
# the difference and verify the new parameter.
async def test_key_to_array_align_false_origin_is_query_start(self, sequence):
"""Without align_to_interval the first bucket sits at query_start, not a clock boundary.
With start_datetime at 10:07:00 and 15-min interval the first resampled
bucket must be at 10:07:00 (origin = query_start), NOT at 10:00:00 or 10:15:00.
"""
# Off-boundary start: 10:07
start_dt = pendulum.datetime(2024, 6, 1, 10, 7, tz="UTC")
end_dt = pendulum.datetime(2024, 6, 1, 12, 7, tz="UTC")
# Records every 15 min so the resampled mean equals the input values
for m in range(0, 120, 15):
dt = pendulum.datetime(2024, 6, 1, 10, 7, tz="UTC").add(minutes=m)
await sequence.insert_by_datetime(self.create_test_record(dt, float(m)))
array = await sequence.key_to_array(
key="data_value",
start_datetime=start_dt,
end_datetime=end_dt,
interval=to_duration("15 minutes"),
fill_method="time",
boundary="strict",
align_to_interval=False,
)
assert len(array) > 0
# Reconstruct the pandas index that key_to_array used: origin=start_dt
idx = pd.date_range(start=start_dt, periods=len(array), freq="900s")
# First bucket must be exactly at start_dt (10:07)
assert idx[0].minute == 7
assert idx[0].second == 0
async def test_key_to_array_align_true_15min_buckets_on_quarter_hours(self, sequence):
"""align_to_interval=True produces timestamps on :00/:15/:30/:45 boundaries."""
# Off-boundary start: 10:07
start_dt = pendulum.datetime(2024, 6, 1, 10, 7, tz="UTC")
end_dt = pendulum.datetime(2024, 6, 1, 12, 7, tz="UTC")
# 1-min records across the window so resampling has data to work with
for m in range(0, 121):
dt = pendulum.datetime(2024, 6, 1, 10, 7, tz="UTC").add(minutes=m)
await sequence.insert_by_datetime(self.create_test_record(dt, float(m)))
array = await sequence.key_to_array(
key="data_value",
start_datetime=start_dt,
end_datetime=end_dt,
interval=to_duration("15 minutes"),
fill_method="time",
boundary="strict",
align_to_interval=True,
)
assert len(array) > 0
# Reconstruct the epoch-aligned index that key_to_array must have used
import math
epoch = int(start_dt.timestamp())
floored_epoch = (epoch // 900) * 900 # floor to nearest 15-min boundary
idx = pd.date_range(
start=pd.Timestamp(floored_epoch, unit="s", tz="UTC"),
periods=len(array),
freq="900s",
)
# Every bucket must land on a :00/:15/:30/:45 minute mark with zero seconds
for ts in idx:
assert ts.minute % 15 == 0, (
f"Bucket at {ts} is not on a 15-min boundary (minute={ts.minute})"
)
assert ts.second == 0, (
f"Bucket at {ts} has non-zero seconds ({ts.second})"
)
async def test_key_to_array_align_true_1hour_buckets_on_the_hour(self, sequence):
"""align_to_interval=True with 1-hour interval produces on-the-hour timestamps."""
# Off-boundary start: 10:23
start_dt = pendulum.datetime(2024, 6, 1, 10, 23, tz="UTC")
end_dt = pendulum.datetime(2024, 6, 1, 15, 23, tz="UTC")
for m in range(0, 301, 15):
dt = pendulum.datetime(2024, 6, 1, 10, 23, tz="UTC").add(minutes=m)
await sequence.insert_by_datetime(self.create_test_record(dt, float(m)))
array = await sequence.key_to_array(
key="data_value",
start_datetime=start_dt,
end_datetime=end_dt,
interval=to_duration("1 hour"),
fill_method="time",
boundary="strict",
align_to_interval=True,
)
assert len(array) > 0
epoch = int(start_dt.timestamp())
floored_epoch = (epoch // 3600) * 3600 # floor to nearest hour
idx = pd.date_range(
start=pd.Timestamp(floored_epoch, unit="s", tz="UTC"),
periods=len(array),
freq="1h",
)
for ts in idx:
assert ts.minute == 0, (
f"Bucket at {ts} should be on the hour (minute={ts.minute})"
)
assert ts.second == 0, (
f"Bucket at {ts} has non-zero seconds ({ts.second})"
)
async def test_key_to_array_align_true_when_start_already_on_boundary(self, sequence):
"""align_to_interval=True is a no-op when start_datetime is exactly on a boundary.
With start at a clean 15-min mark both modes must produce identical arrays.
"""
# Exactly on boundary: 10:00:00
start_dt = pendulum.datetime(2024, 6, 1, 10, 0, tz="UTC")
end_dt = pendulum.datetime(2024, 6, 1, 12, 0, tz="UTC")
for m in range(0, 121, 15):
dt = pendulum.datetime(2024, 6, 1, 10, 0, tz="UTC").add(minutes=m)
await sequence.insert_by_datetime(self.create_test_record(dt, float(m)))
arr_aligned = await sequence.key_to_array(
key="data_value",
start_datetime=start_dt,
end_datetime=end_dt,
interval=to_duration("15 minutes"),
fill_method="time",
boundary="strict",
align_to_interval=True,
)
arr_default = await sequence.key_to_array(
key="data_value",
start_datetime=start_dt,
end_datetime=end_dt,
interval=to_duration("15 minutes"),
fill_method="time",
boundary="strict",
align_to_interval=False,
)
assert len(arr_aligned) == len(arr_default)
np.testing.assert_array_almost_equal(arr_aligned, arr_default, decimal=6)
async def test_key_to_array_align_true_without_start_datetime(self, sequence):
"""align_to_interval=True with no start_datetime must not raise.
Without a query_start there is no origin to snap; behaviour falls back
to 'start_day' (same as default). No exception is expected.
"""
for m in range(0, 121, 15):
dt = pendulum.datetime(2024, 6, 1, 10, 7, tz="UTC").add(minutes=m)
await sequence.insert_by_datetime(self.create_test_record(dt, float(m)))
array = await sequence.key_to_array(
key="data_value",
start_datetime=None,
end_datetime=pendulum.datetime(2024, 6, 1, 12, 7, tz="UTC"),
interval=to_duration("15 minutes"),
fill_method="time",
boundary="strict",
align_to_interval=True,
)
assert isinstance(array, np.ndarray)
assert len(array) > 0
async def test_key_to_array_align_true_output_within_requested_window(self, sequence):
"""align_to_interval=True truncates output to [start_datetime, end_datetime).
The epoch-floor origin may generate a bucket before start_datetime (e.g. 10:00
when start is 10:07), but key_to_array must truncate it away. The surviving
buckets are verified directly by reconstructing the index from the first
surviving timestamp (the first epoch-aligned bucket >= start_datetime).
Also checks that all surviving buckets are on 15-min clock boundaries.
"""
start_dt = pendulum.datetime(2024, 6, 1, 10, 7, tz="UTC")
end_dt = pendulum.datetime(2024, 6, 1, 13, 7, tz="UTC")
for m in range(0, 181):
dt = pendulum.datetime(2024, 6, 1, 10, 7, tz="UTC").add(minutes=m)
await sequence.insert_by_datetime(self.create_test_record(dt, float(m)))
array = await sequence.key_to_array(
key="data_value",
start_datetime=start_dt,
end_datetime=end_dt,
interval=to_duration("15 minutes"),
fill_method="time",
boundary="strict",
align_to_interval=True,
)
assert len(array) > 0
# The first surviving bucket is the first epoch-aligned timestamp >= start_dt.
# Compute it the same way key_to_array does: floor then step forward if needed.
epoch = int(start_dt.timestamp())
floored_epoch = (epoch // 900) * 900
first_bucket = pd.Timestamp(floored_epoch, unit="s", tz="UTC")
if first_bucket < pd.Timestamp(start_dt):
first_bucket += pd.Timedelta(seconds=900)
idx = pd.date_range(start=first_bucket, periods=len(array), freq="900s")
start_pd = pd.Timestamp(start_dt)
end_pd = pd.Timestamp(end_dt)
for ts in idx:
assert ts >= start_pd, f"Bucket {ts} is before start_datetime {start_pd}"
assert ts < end_pd, f"Bucket {ts} is at or after end_datetime {end_pd}"
assert ts.minute % 15 == 0, f"Bucket {ts} is not on a 15-min boundary"
assert ts.second == 0, f"Bucket {ts} has non-zero seconds"
async def test_key_to_array_align_true_preserves_mean_values(self, sequence):
"""align_to_interval=True does not corrupt resampled values.
A constant-valued series must resample to the same constant regardless
of bucket alignment.
"""
# 1-min records with constant value 42.0, starting off-boundary
start_dt = pendulum.datetime(2024, 6, 1, 10, 7, tz="UTC")
end_dt = pendulum.datetime(2024, 6, 1, 12, 7, tz="UTC")
for m in range(0, 121):
dt = pendulum.datetime(2024, 6, 1, 10, 7, tz="UTC").add(minutes=m)
await sequence.insert_by_datetime(self.create_test_record(dt, 42.0))
array = await sequence.key_to_array(
key="data_value",
start_datetime=start_dt,
end_datetime=end_dt,
interval=to_duration("15 minutes"),
fill_method="time",
boundary="strict",
align_to_interval=True,
)
assert len(array) > 0
for v in array:
if v is not None:
assert abs(v - 42.0) < 1e-6, f"Expected 42.0, got {v}"
async def test_key_to_array_align_true_compaction_call_pattern(self, sequence):
"""Verify the call pattern used by _db_compact_tier produces clock-aligned timestamps.
_db_compact_tier calls key_to_array with boundary='strict', fill_method='time',
align_to_interval=True on a window whose start has arbitrary sub-second precision.
All output buckets must land on 15-min boundaries so that compacted records are
stored at predictable, human-readable timestamps.
"""
# Non-round base time: 08:43 — chosen to expose any origin-alignment bug
base_dt = pendulum.datetime(2024, 6, 1, 8, 43, tz="UTC")
window_end = pendulum.datetime(2024, 6, 1, 11, 43, tz="UTC")
for m in range(0, 181):
dt = base_dt.add(minutes=m)
await sequence.insert_by_datetime(self.create_test_record(dt, float(m)))
array = await sequence.key_to_array(
key="data_value",
start_datetime=base_dt,
end_datetime=window_end,
interval=to_duration("15 minutes"),
fill_method="time",
boundary="strict",
align_to_interval=True,
)
assert len(array) > 0
epoch = int(base_dt.timestamp())
floored_epoch = (epoch // 900) * 900
idx = pd.date_range(
start=pd.Timestamp(floored_epoch, unit="s", tz="UTC"),
periods=len(array),
freq="900s",
)
for ts in idx:
assert ts.minute % 15 == 0, (
f"Compacted record at {ts} is not on a 15-min boundary (minute={ts.minute})"
)
assert ts.second == 0, (
f"Compacted record at {ts} has non-zero seconds ({ts.second})"
)
async def test_delete_by_datetime_range(self, sequence):
dt1 = to_datetime("2023-11-05")
dt2 = to_datetime("2023-11-06")
dt3 = to_datetime("2023-11-07")
record1 = self.create_test_record(dt1, 0.8)
record2 = self.create_test_record(dt2, 0.9)
record3 = self.create_test_record(dt3, 1.0)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
await sequence.insert_by_datetime(record3)
assert len(sequence) == 3
await sequence.delete_by_datetime(start_datetime=dt2, end_datetime=dt3)
assert len(sequence) == 2
assert sequence.records[0].date_time == dt1
assert sequence.records[1].date_time == dt3
async def test_delete_by_datetime_start(self, sequence):
dt1 = to_datetime("2023-11-05")
dt2 = to_datetime("2023-11-06")
record1 = self.create_test_record(dt1, 0.8)
record2 = self.create_test_record(dt2, 0.9)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
assert len(sequence) == 2
await sequence.delete_by_datetime(start_datetime=dt2)
assert len(sequence) == 1
assert sequence.records[0].date_time == dt1
async def test_delete_by_datetime_end(self, sequence):
dt1 = to_datetime("2023-11-05")
dt2 = to_datetime("2023-11-06")
record1 = self.create_test_record(dt1, 0.8)
record2 = self.create_test_record(dt2, 0.9)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
assert len(sequence) == 2
await sequence.delete_by_datetime(end_datetime=dt2)
assert len(sequence) == 1
assert sequence.records[0].date_time == dt2
async def test_to_dict_async(self, sequence):
dt = to_datetime("2023-11-06")
record = self.create_test_record(dt, 0.8)
await sequence.insert_by_datetime(record)
data_dict = await sequence.to_dict_async()
assert isinstance(data_dict, dict)
# We need a new class - Sequences are singletons
sequence2 = await DerivedSequence2.from_dict_async(data_dict)
assert sequence2.model_dump() == sequence.model_dump()
async def test_to_json_async(self, sequence):
dt = to_datetime("2023-11-06")
record = self.create_test_record(dt, 0.8)
await sequence.insert_by_datetime(record)
json_str = await sequence.to_json_async()
assert isinstance(json_str, str)
assert "2023-11-06" in json_str
assert ": 0.8" in json_str
async def test_from_json_async(self, sequence, sequence2):
json_str = sequence2.to_json()
sequence = await sequence.from_json_async(json_str)
assert len(sequence) == len(sequence2)
assert sequence.records[0].date_time == sequence2.records[0].date_time
assert sequence.records[0].data_value == sequence2.records[0].data_value
async def test_key_to_value_exact_match(self, sequence):
"""Test key_to_value returns exact match when datetime matches a record."""
dt = to_datetime("2023-11-05")
record = self.create_test_record(dt, 0.75)
await sequence.insert_by_datetime(record)
result = await sequence.key_to_value("data_value", dt)
assert result == 0.75
async def test_key_to_value_nearest(self, sequence):
"""Test key_to_value returns value closest in time to the given datetime."""
record1 = self.create_test_record(datetime(2023, 11, 5, 12), 0.6)
record2 = self.create_test_record(datetime(2023, 11, 6, 12), 0.9)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
dt = datetime(2023, 11, 6, 10) # closer to record2
result = await sequence.key_to_value("data_value", dt, time_window=to_duration("48 hours"))
assert result == 0.9
async def test_key_to_value_nearest_after(self, sequence):
"""Test key_to_value returns value nearest after the given datetime."""
record1 = self.create_test_record(datetime(2023, 11, 5, 10), 0.7)
record2 = self.create_test_record(datetime(2023, 11, 5, 15), 0.8)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
dt = datetime(2023, 11, 5, 14) # closer to record2
result = await sequence.key_to_value("data_value", dt, time_window=to_duration("48 hours"))
assert result == 0.8
async def test_key_to_value_empty_sequence(self, sequence):
"""Test key_to_value returns None when sequence is empty."""
result = await sequence.key_to_value("data_value", datetime(2023, 11, 5))
assert result is None
async def test_key_to_value_missing_key(self, sequence):
"""Test key_to_value returns None when key is missing in records."""
record = self.create_test_record(datetime(2023, 11, 5), None)
await sequence.insert_by_datetime(record)
result = await sequence.key_to_value("data_value", datetime(2023, 11, 5))
assert result is None
async def test_key_to_value_multiple_records_with_none(self, sequence):
"""Test key_to_value skips records with None values."""
r1 = self.create_test_record(datetime(2023, 11, 5), None)
r2 = self.create_test_record(datetime(2023, 11, 6), 1.0)
await sequence.insert_by_datetime(r1)
await sequence.insert_by_datetime(r2)
result = await sequence.key_to_value("data_value", datetime(2023, 11, 5, 12), time_window=to_duration("48 hours"))
assert result == 1.0
async def test_key_to_dict(self, sequence):
record1 = self.create_test_record(datetime(2023, 11, 5), 0.8)
record2 = self.create_test_record(datetime(2023, 11, 6), 0.9)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
data_dict = await sequence.key_to_dict("data_value")
assert isinstance(data_dict, dict)
assert data_dict[to_datetime(datetime(2023, 11, 5), as_string=True)] == 0.8
assert data_dict[to_datetime(datetime(2023, 11, 6), as_string=True)] == 0.9
async def test_key_to_lists(self, sequence):
record1 = self.create_test_record(datetime(2023, 11, 5), 0.8)
record2 = self.create_test_record(datetime(2023, 11, 6), 0.9)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
dates, values = await sequence.key_to_lists("data_value")
assert dates == [to_datetime(datetime(2023, 11, 5)), to_datetime(datetime(2023, 11, 6))]
assert values == [0.8, 0.9]
async def test_to_dataframe_full_data(self, sequence):
"""Test conversion of all records to a DataFrame without filtering."""
record1 = self.create_test_record("2024-01-01T12:00:00Z", 10)
record2 = self.create_test_record("2024-01-01T13:00:00Z", 20)
record3 = self.create_test_record("2024-01-01T14:00:00Z", 30)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
await sequence.insert_by_datetime(record3)
df = await sequence.to_dataframe()
# Validate DataFrame structure
assert isinstance(df, pd.DataFrame)
assert not df.empty
assert len(df) == 3 # All records should be included
assert "data_value" in df.columns
async def test_to_dataframe_with_filter(self, sequence):
"""Test filtering records by datetime range."""
record1 = self.create_test_record("2024-01-01T12:00:00Z", 10)
record2 = self.create_test_record("2024-01-01T13:00:00Z", 20)
record3 = self.create_test_record("2024-01-01T14:00:00Z", 30)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
await sequence.insert_by_datetime(record3)
start = to_datetime("2024-01-01T12:30:00Z")
end = to_datetime("2024-01-01T14:00:00Z")
df = await sequence.to_dataframe(start_datetime=start, end_datetime=end)
assert isinstance(df, pd.DataFrame)
assert not df.empty
assert len(df) == 1 # Only one record should match the range
assert df.index[0] == pd.Timestamp("2024-01-01T13:00:00Z")
async def test_to_dataframe_no_matching_records(self, sequence):
"""Test when no records match the given datetime filter."""
record1 = self.create_test_record("2024-01-01T12:00:00Z", 10)
record2 = self.create_test_record("2024-01-01T13:00:00Z", 20)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
start = to_datetime("2024-01-01T14:00:00Z") # Start time after all records
end = to_datetime("2024-01-01T15:00:00Z")
df = await sequence.to_dataframe(start_datetime=start, end_datetime=end)
assert isinstance(df, pd.DataFrame)
assert df.empty # No records should match
async def test_to_dataframe_empty_sequence(self, sequence):
"""Test when DataSequence has no records."""
sequence = DataSequence(records=[])
df = await sequence.to_dataframe()
assert isinstance(df, pd.DataFrame)
assert df.empty # Should return an empty DataFrame
async def test_to_dataframe_no_start_datetime(self, sequence):
"""Test when only end_datetime is given (all past records should be included)."""
record1 = self.create_test_record("2024-01-01T12:00:00Z", 10)
record2 = self.create_test_record("2024-01-01T13:00:00Z", 20)
record3 = self.create_test_record("2024-01-01T14:00:00Z", 30)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
await sequence.insert_by_datetime(record3)
end = to_datetime("2024-01-01T13:00:00Z") # Include only first record
df = await sequence.to_dataframe(end_datetime=end)
assert isinstance(df, pd.DataFrame)
assert not df.empty
assert len(df) == 1
assert df.index[0] == pd.Timestamp("2024-01-01T12:00:00Z")
async def test_to_dataframe_no_end_datetime(self, sequence):
"""Test when only start_datetime is given (all future records should be included)."""
record1 = self.create_test_record("2024-01-01T12:00:00Z", 10)
record2 = self.create_test_record("2024-01-01T13:00:00Z", 20)
record3 = self.create_test_record("2024-01-01T14:00:00Z", 30)
await sequence.insert_by_datetime(record1)
await sequence.insert_by_datetime(record2)
await sequence.insert_by_datetime(record3)
start = to_datetime("2024-01-01T13:00:00Z") # Include last two records
df = await sequence.to_dataframe(start_datetime=start)
assert isinstance(df, pd.DataFrame)
assert not df.empty
assert len(df) == 2
assert df.index[0] == pd.Timestamp("2024-01-01T13:00:00Z")

701
tests/test_dataabcsequencedb.py Normal file
View File

@@ -0,0 +1,701 @@
"""Pytest tests for async DataSequence with persistence.
Tests the async DataSequence with database persistence.
"""
from __future__ import annotations
import asyncio
import shutil
import tempfile
import time
from pathlib import Path
from typing import AsyncIterator, Optional, Type
import pytest
import pytest_asyncio
from pydantic import Field
from akkudoktoreos.core.coreabc import get_database
from akkudoktoreos.core.dataabc import DataProvider, DataRecord, DataSequence
from akkudoktoreos.core.database import Database, LMDBDatabase, SQLiteDatabase
from akkudoktoreos.core.databaseabc import (
DatabaseRecordProtocolLoadPhase,
DatabaseTimestamp,
)
from akkudoktoreos.utils.datetimeutil import (
DateTime,
Duration,
to_datetime,
to_duration,
)
# ==================== Test Fixtures ====================
@pytest.fixture
def temp_dir():
"""Create a temporary directory for test databases."""
temp_path = Path(tempfile.mkdtemp())
yield temp_path
shutil.rmtree(temp_path, ignore_errors=True)
@pytest.fixture(params=["LMDB", "SQLite"])
def database_provider(request) -> str:
"""Parametrize all database backend tests."""
return request.param
@pytest_asyncio.fixture
async def async_database_instance(
config_eos,
database_provider: str,
) -> AsyncIterator[Database]:
"""Open a database instance for testing and close it afterwards."""
config_eos.database.compression_level = 6
config_eos.database.provider = database_provider
db = get_database()
await db.open()
assert db.provider_id() == database_provider
assert db.is_open is True
yield db
await db.close()
config_eos.database.provider = None
# ==================== Helpers ====================
async def _clear_sequence_state(sequence) -> None:
"""Clear runtime DB state without re-instantiating the singleton.
Does _NOT_ initialize the DB state.
"""
await sequence.db_delete_records()
try:
sequence._db_metadata = None
await sequence.database().set_metadata(None, namespace=sequence.db_namespace())
except Exception:
# Database may not be available, just skip
pass
try:
del sequence._db_initialized
except Exception:
# May not be set
pass
async def _reset_sequence_state(sequence) -> None:
"""Reset runtime DB state without re-instantiating the singleton."""
try:
sequence.records = []
del sequence._db_initialized
except Exception:
# May not be set
pass
await sequence._db_ensure_initialized()
# Sample Data
class SampleDataRecord(DataRecord):
"""Minimal DataRecord for testing."""
temperature: float = Field(default=0.0)
humidity: float = Field(default=0.0)
pressure: float = Field(default=0.0)
class SampleDataSequence(DataSequence):
"""DataSequence subclass with database support."""
records: list[SampleDataRecord] = Field(default_factory=list)
@classmethod
def record_class(cls) -> Type[SampleDataRecord]:
return SampleDataRecord
def db_namespace(self) -> str:
return "SampleDataSequence"
class SampleDataProvider(DataProvider):
"""DataProvider subclass with database support."""
records: list[SampleDataRecord] = Field(default_factory=list)
@classmethod
def record_class(cls) -> Type[SampleDataRecord]:
return SampleDataRecord
def provider_id(self) -> str:
return "SampleDataProvider"
def enabled(self) -> bool:
return True
async def _update_data(self, force_update: Optional[bool] = False) -> None:
pass
def db_namespace(self) -> str:
return "SampleDataProvider"
# ==================== DatabaseRecordProtocolMixin Tests ====================
@pytest.mark.asyncio
class TestDataSequenceDatabaseProtocol:
"""Tests for DatabaseRecordProtocolMixin via SampleDataSequence."""
async def test_db_enabled_when_db_open(self, async_database_instance):
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
assert sequence.db_enabled is True
async def test_db_disabled_when_db_closed(self, config_eos):
config_eos.database.provider = None
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
assert sequence.db_enabled is False
async def test_insert_and_save_records(self, async_database_instance):
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
for i in range(10):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=20.0 + i)
)
# All 10 are dirty/new, none persisted yet
assert len(sequence.records) == 10
assert len(sequence._db_new_timestamps) == 10
saved = await sequence.db_save_records()
assert saved == 10 # 10 inserts + 0 deletes
assert len(sequence._db_dirty_timestamps) == 0
assert len(sequence._db_new_timestamps) == 0
async def test_save_returns_insert_plus_delete_count(self, async_database_instance):
"""db_save_records() return value = saved_inserts + deleted_count."""
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
for i in range(5):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=float(i))
)
# Persist the 5 records
await sequence.db_save_records()
# Delete 2 of them
db_start = DatabaseTimestamp.from_datetime(base_time.add(hours=2))
db_end = DatabaseTimestamp.from_datetime(base_time.add(hours=4))
deleted = await sequence.db_delete_records(start_timestamp=db_start, end_timestamp=db_end)
# Insert 3 new ones
for i in range(10, 13):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=float(i))
)
result = await sequence.db_save_records()
# 3 inserts + 2 deletes = 5
assert result == 5
async def test_load_records_from_db(self, async_database_instance):
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
for i in range(10):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=20.0 + i)
)
await sequence.db_save_records()
# Clear memory, then reload from DB
await _reset_sequence_state(sequence)
loaded = await sequence.db_load_records()
assert loaded == 10
assert len(sequence.records) == 10
for i, record in enumerate(sequence.records):
assert record.temperature == 20.0 + i
async def test_load_records_with_range(self, async_database_instance):
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
for i in range(10):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=20.0 + i)
)
await sequence.db_save_records()
await _reset_sequence_state(sequence)
# Load [hours=3, hours=7) → 4 records (3, 4, 5, 6)
db_start = DatabaseTimestamp.from_datetime(base_time.add(hours=3))
db_end = DatabaseTimestamp.from_datetime(base_time.add(hours=7))
loaded = await sequence.db_load_records(start_timestamp=db_start, end_timestamp=db_end)
assert loaded == 4
assert sequence.records[0].temperature == 23.0
assert sequence.records[-1].temperature == 26.0
async def test_iterate_records_triggers_lazy_load(self, async_database_instance):
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
for i in range(10):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=20.0 + i)
)
await sequence.db_save_records()
await _reset_sequence_state(sequence)
# db_iterate_records calls _db_ensure_loaded internally
db_start = DatabaseTimestamp.from_datetime(base_time.add(hours=2))
db_end = DatabaseTimestamp.from_datetime(base_time.add(hours=5))
records = [record async for record in sequence.db_iterate_records(start_timestamp=db_start, end_timestamp=db_end)]
assert len(records) == 3
assert all(base_time.add(hours=2) <= r.date_time < base_time.add(hours=5) for r in records)
async def test_delete_records(self, async_database_instance):
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
for i in range(6):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=20.0)
)
await sequence.db_save_records()
db_start = DatabaseTimestamp.from_datetime(base_time.add(hours=2))
db_end = DatabaseTimestamp.from_datetime(base_time.add(hours=5))
deleted = await sequence.db_delete_records(start_timestamp=db_start, end_timestamp=db_end)
assert deleted == 3
# Persist the deletions
await sequence.db_save_records()
await _reset_sequence_state(sequence)
await sequence.db_load_records()
assert len(sequence.records) == 3
async def test_delete_tombstone_prevents_resurrection(self, async_database_instance):
"""Deleted records must not re-appear when db_load_records is called."""
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
for i in range(3):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=float(i))
)
await sequence.db_save_records()
# Delete middle record
db_start = DatabaseTimestamp.from_datetime(base_time.add(hours=1))
db_end = DatabaseTimestamp.from_datetime(base_time.add(hours=2))
deleted = await sequence.db_delete_records(start_timestamp=db_start, end_timestamp=db_end)
assert deleted == 1
# Do NOT persist yet — tombstone lives only in memory
# Loading should not resurrect the tombstoned record
loaded = await sequence.db_load_records()
assert all(r.date_time != base_time.add(hours=1) for r in sequence.records)
async def test_insert_after_delete_clears_tombstone(self, async_database_instance):
"""Re-inserting a deleted datetime must clear its tombstone."""
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
dt = base_time.add(hours=5)
await sequence.db_insert_record(SampleDataRecord(date_time=dt, temperature=10.0))
await sequence.db_save_records()
db_start = DatabaseTimestamp.from_datetime(dt)
db_end = sequence._db_timestamp_after(db_start)
deleted = await sequence.db_delete_records(start_timestamp=db_start, end_timestamp=db_end)
assert deleted == 1
await sequence.db_save_records()
# Re-insert the same datetime
await sequence.db_insert_record(SampleDataRecord(date_time=dt, temperature=99.0))
assert dt not in sequence._db_deleted_timestamps
await sequence.db_save_records()
await _reset_sequence_state(sequence)
await sequence.db_load_records()
assert any(r.date_time == dt and r.temperature == 99.0 for r in sequence.records)
async def test_db_count_records_memory_only(self):
"""When db is disabled, count reflects memory only."""
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
# Without a live DB, db_enabled is False
if sequence.db_enabled:
pytest.skip("DB is open; this test requires it to be closed")
base_time = to_datetime("2024-01-01T00:00:00Z")
for i in range(5):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=float(i)),
mark_dirty=False,
)
count = await sequence.db_count_records()
assert count == 5
async def test_db_count_records_combined(self, async_database_instance):
"""db_count_records = storage + new_unpersisted - pending_deletes."""
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
# Persist 10 records
for i in range(10):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=float(i))
)
await sequence.db_save_records()
# Add 3 new unpersisted records
for i in range(10, 13):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=float(i))
)
# Delete 2 persisted records (not yet saved)
db_start = DatabaseTimestamp.from_datetime(base_time.add(hours=0))
db_end = DatabaseTimestamp.from_datetime(base_time.add(hours=2))
deleted = await sequence.db_delete_records(start_timestamp=db_start, end_timestamp=db_end)
assert deleted == 2
# storage=10, new=3, pending_deletes=2 → expected=11
count = await sequence.db_count_records()
assert count == 11
async def test_db_timestamp_range_empty(self, async_database_instance):
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
min_dt, max_dt = await sequence.db_timestamp_range()
assert min_dt is None
assert max_dt is None
async def test_db_timestamp_range_with_records(self, async_database_instance):
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
for hours in [0, 5, 10]:
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=hours), temperature=20.0)
)
await sequence.db_save_records()
await _reset_sequence_state(sequence)
min_dt, max_dt = await sequence.db_timestamp_range()
assert min_dt == DatabaseTimestamp.from_datetime(base_time)
assert max_dt == DatabaseTimestamp.from_datetime(base_time.add(hours=10))
async def test_db_mark_dirty_triggers_save(self, async_database_instance):
"""Marking a record dirty causes it to be re-saved."""
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
record = SampleDataRecord(date_time=base_time, temperature=20.0)
await sequence.db_insert_record(record)
await sequence.db_save_records()
# Mutate and mark dirty
record.temperature = 99.0
await sequence.db_mark_dirty_record(record)
await sequence.db_save_records()
# Reload and verify update was persisted
await _reset_sequence_state(sequence)
await sequence.db_load_records()
assert sequence.records[0].temperature == 99.0
async def test_db_vacuum_keep_hours(self, async_database_instance):
"""db_vacuum(keep_hours=N) retains only the last N hours of records."""
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
# 240 hourly records = 10 days
for i in range(240):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=20.0)
)
await sequence.db_save_records()
await _reset_sequence_state(sequence)
keep_hours = 5 * 24 # keep last 5 days
deleted = await sequence.db_vacuum(keep_hours=keep_hours)
assert deleted == 240 - keep_hours
count = await sequence.db_count_records()
assert count == keep_hours
async def test_db_vacuum_keep_timestamp(self, async_database_instance):
"""db_vacuum(keep_timestamp=T) deletes everything before T (exclusive)."""
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
for i in range(10):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=float(i))
)
await sequence.db_save_records()
await _reset_sequence_state(sequence)
# Keep from hours=5 onward — delete [0, 5), i.e. 5 records
cutoff = base_time.add(hours=5)
db_cutoff = DatabaseTimestamp.from_datetime(cutoff)
deleted = await sequence.db_vacuum(keep_timestamp=db_cutoff)
assert deleted == 5
count = await sequence.db_count_records()
assert count == 5
# Verify the boundary record (hours=5) was NOT deleted
await _reset_sequence_state(sequence)
await sequence.db_load_records()
assert any(r.date_time == cutoff for r in sequence.records)
async def test_db_vacuum_no_argument(self, async_database_instance, config_eos):
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
record = SampleDataRecord(date_time=base_time, temperature=20.0)
await sequence.db_insert_record(record)
await sequence.db_save_records()
config_eos.database.keep_duration_h = None
deleted = await sequence.db_vacuum()
assert deleted == 0
config_eos.database.keep_duration_h = 0
deleted = await sequence.db_vacuum()
assert deleted == 1
async def test_db_vacuum_keep_hours_zero_deletes_all(self, async_database_instance):
"""keep_hours=0 should delete all records."""
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
for i in range(5):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=float(i))
)
await sequence.db_save_records()
await _reset_sequence_state(sequence)
deleted = await sequence.db_vacuum(keep_hours=0)
assert deleted == 5
count = await sequence.db_count_records()
assert count == 0
async def test_db_get_stats(self, async_database_instance):
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
stats = await sequence.db_get_stats()
assert stats["enabled"] is True
assert "backend" in stats
assert "path" in stats
assert "memory_records" in stats
assert "total_records" in stats
assert "compression_enabled" in stats
assert "timestamp_range" in stats
assert stats["timestamp_range"]["min"] == "None"
assert stats["timestamp_range"]["max"] == "None"
async def test_db_get_stats_disabled(self, config_eos):
config_eos.database.provider = None
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
stats = await sequence.db_get_stats()
assert stats == {"enabled": False}
async def test_lazy_load_phase_none_to_initial(self, async_database_instance):
"""Phase transitions from NONE to INITIAL when a range is loaded via ensure_loaded."""
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
assert sequence._db_load_phase is DatabaseRecordProtocolLoadPhase.NONE
base_time = to_datetime("2024-01-01T00:00:00Z")
for i in range(10):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=float(i))
)
await sequence.db_save_records()
await _reset_sequence_state(sequence)
# Use db_iterate_records — it calls _db_ensure_loaded which owns phase transitions
db_start = DatabaseTimestamp.from_datetime(base_time.add(hours=3))
db_end = DatabaseTimestamp.from_datetime(base_time.add(hours=7))
records = [record async for record in sequence.db_iterate_records(start_timestamp=db_start, end_timestamp=db_end)]
assert sequence._db_load_phase is DatabaseRecordProtocolLoadPhase.INITIAL
async def test_lazy_load_phase_initial_to_full(self, async_database_instance):
"""Phase transitions from INITIAL to FULL when iterate is called without range."""
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
for i in range(10):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=float(i))
)
await sequence.db_save_records()
await _reset_sequence_state(sequence)
# Load partial range → INITIAL
# Use db_iterate_records — it calls _db_ensure_loaded which owns phase transitions
db_start = DatabaseTimestamp.from_datetime(base_time.add(hours=3))
db_end = DatabaseTimestamp.from_datetime(base_time.add(hours=7))
records = [record async for record in sequence.db_iterate_records(start_timestamp=db_start, end_timestamp=db_end)]
assert sequence._db_load_phase is DatabaseRecordProtocolLoadPhase.INITIAL
# Iterate without range → escalates to FULL
records = [record async for record in sequence.db_iterate_records()]
assert sequence._db_load_phase is DatabaseRecordProtocolLoadPhase.FULL
async def test_range_covered_skips_redundant_load(self, async_database_instance):
"""_db_range_covered prevents a second DB query for the same range."""
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
for i in range(10):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=float(i))
)
await sequence.db_save_records()
await _reset_sequence_state(sequence)
db_start = DatabaseTimestamp.from_datetime(base_time.add(hours=2))
db_end = DatabaseTimestamp.from_datetime(base_time.add(hours=8))
records = [record async for record in sequence.db_iterate_records(start_timestamp=db_start, end_timestamp=db_end)]
# Loaded range is now set
assert sequence._db_loaded_range is not None
assert sequence._db_range_covered(db_start, db_end) is True
db_start = DatabaseTimestamp.from_datetime(base_time.add(hours=0))
db_end = DatabaseTimestamp.from_datetime(base_time.add(hours=20))
assert sequence._db_range_covered(db_start, db_end) is False
async def test_loaded_range_not_clobbered_by_expansion(self, async_database_instance):
"""Expanding left or right must not narrow the tracked loaded range."""
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T00:00:00Z")
for i in range(24):
await sequence.db_insert_record(
SampleDataRecord(date_time=base_time.add(hours=i), temperature=float(i))
)
await sequence.db_save_records()
await _reset_sequence_state(sequence)
# Initial window: hours 8–16
db_start = DatabaseTimestamp.from_datetime(base_time.add(hours=8))
db_end = DatabaseTimestamp.from_datetime(base_time.add(hours=16))
records = [record async for record in sequence.db_iterate_records(start_timestamp=db_start, end_timestamp=db_end)]
assert sequence._db_loaded_range is not None
initial_start, initial_end = sequence._db_loaded_range
assert initial_start is not None
assert initial_end is not None
# Expand left: load hours 4–8
db_start = DatabaseTimestamp.from_datetime(base_time.add(hours=4))
db_end = DatabaseTimestamp.from_datetime(base_time.add(hours=16))
records = [record async for record in sequence.db_iterate_records(start_timestamp=db_start, end_timestamp=db_end)]
assert sequence._db_loaded_range is not None
expanded_start, expanded_end = sequence._db_loaded_range
assert expanded_start is not None
assert expanded_end is not None
# Left boundary must have moved left; right must not have shrunk
assert expanded_start <= initial_start
assert expanded_end >= initial_end
async def test_duplicate_insert_raises(self, async_database_instance):
sequence = SampleDataSequence()
await _reset_sequence_state(sequence)
dt = to_datetime("2024-01-01T00:00:00Z")
await sequence.db_insert_record(SampleDataRecord(date_time=dt, temperature=1.0))
with pytest.raises(ValueError, match="Duplicate timestamp"):
await sequence.db_insert_record(SampleDataRecord(date_time=dt, temperature=2.0))
async def test_metadata_round_trip(self, async_database_instance):
"""Metadata can be saved and loaded back correctly."""
sequence = SampleDataSequence()
await _clear_sequence_state(sequence)
assert sequence._db_metadata is None
await _reset_sequence_state(sequence)
assert sequence._db_metadata is not None
created = sequence._db_metadata["created"]
assert sequence._db_metadata["version"] == 1
await _reset_sequence_state(sequence)
assert sequence._db_metadata is not None
assert sequence._db_metadata["created"] == created
assert sequence._db_metadata["version"] == 1
async def test_initial_load_window_respected(self, async_database_instance):
"""db_initial_time_window limits the initial load from DB."""
class WindowedSequence(SampleDataSequence):
def db_namespace(self) -> str:
return "WindowedSequence"
def db_initial_time_window(self) -> Optional[Duration]:
return to_duration("2 hours")
sequence = WindowedSequence()
await _reset_sequence_state(sequence)
base_time = to_datetime("2024-01-01T12:00:00Z")
# Store 24 hourly records centred on base_time
for i in range(24):
await sequence.db_insert_record(
SampleDataRecord(
date_time=base_time.subtract(hours=12).add(hours=i),
temperature=float(i),
)
)
await sequence.db_save_records()
await _reset_sequence_state(sequence)
# Trigger initial window load centred on base_time
sequence.config.database.initial_load_window_h = 2
db_center = DatabaseTimestamp.from_datetime(base_time)
await sequence._db_load_initial_window(center_timestamp=db_center)
# Only records within ±2h of base_time should be in memory
assert len(sequence.records) <= 5 # at most 4h window = 4–5 records
assert sequence._db_load_phase is DatabaseRecordProtocolLoadPhase.INITIAL

1604
tests/test_database.py
View File

File diff suppressed because it is too large Load Diff

615
tests/test_databaseabc.py
View File

File diff suppressed because it is too large Load Diff

69
tests/test_docsphinx.py
View File

@@ -20,16 +20,52 @@ DIR_SRC = DIR_PROJECT_ROOT / "src"
HASH_FILE = DIR_BUILD / ".sphinx_hash.json"
def find_sphinx_build() -> str:
venv = os.getenv("VIRTUAL_ENV")
paths = [Path(venv)] if venv else []
paths.append(DIR_PROJECT_ROOT / ".venv")
import os
import subprocess
from pathlib import Path
for base in paths:
cmd = base / ("Scripts" if os.name == "nt" else "bin") / ("sphinx-build.exe" if os.name == "nt" else "sphinx-build")
if cmd.exists():
return str(cmd)
return "sphinx-build"
def find_sphinx_build() -> list[str]:
"""Return command to invoke sphinx-build via virtualenv, uv, or globally."""
candidates = []
# 1️⃣ Currently active virtualenv
venv = os.getenv("VIRTUAL_ENV")
if venv:
candidates.append(Path(venv))
# 2️⃣ uv‑managed virtualenv
uv_venv = Path(".uv") / "venv"
if uv_venv.exists():
candidates.append(uv_venv)
# 3️⃣ traditional .venv
dot_venv = Path(".venv")
if dot_venv.exists():
candidates.append(dot_venv)
# Check each candidate for the sphinx‑build binary
for base in candidates:
sphinx_build_path = base / ("Scripts" if os.name == "nt" else "bin") / (
"sphinx-build.exe" if os.name == "nt" else "sphinx-build"
)
if sphinx_build_path.exists():
return [str(sphinx_build_path)]
# 4️⃣ fallback to uv run sphinx‑build
try:
subprocess.run(
["uv", "run", "sphinx-build", "--version"],
check=True,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
return ["uv", "run", "sphinx-build"]
except (subprocess.CalledProcessError, FileNotFoundError):
pass
# 5️⃣ final fallback to system sphinx‑build
return ["sphinx-build"]
@pytest.fixture(scope="session")
@@ -61,8 +97,7 @@ class TestSphinxDocumentation:
Ensures no major warnings are emitted.
"""
SPHINX_CMD = [
find_sphinx_build(),
SPHINX_CMD = find_sphinx_build() + [
"-M",
"html",
str(DIR_DOCS),
@@ -105,8 +140,6 @@ class TestSphinxDocumentation:
env["EOS_CONFIG_DIR"] = eos_dir
try:
# Run sphinx-build
project_dir = Path(__file__).parent.parent
process = subprocess.run(
self.SPHINX_CMD,
check=True,
@@ -114,13 +147,15 @@ class TestSphinxDocumentation:
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
cwd=project_dir,
cwd=DIR_PROJECT_ROOT, # use the existing constant
)
# Combine output
output = process.stdout + "\n" + process.stderr
returncode = process.returncode
except:
output = f"ERROR: Could not start sphinx-build - {self.SPHINX_CMD}"
except subprocess.CalledProcessError as e:
output = e.stdout + "\n" + e.stderr if e.stdout else ""
returncode = e.returncode
except Exception as e:
output = f"Failed to execute command: {e}"
returncode = -1
# Remove temporary EOS_DIR

293
tests/test_elecpriceakkudoktor.py
View File

@@ -1,3 +1,4 @@
import asyncio
import json
from pathlib import Path
from unittest.mock import Mock, patch
@@ -47,177 +48,181 @@ def cache_store():
return CacheFileStore()
# ------------------------------------------------
# General forecast
# ------------------------------------------------
class TestElecPriceAkkudokor:
# ------------------------------------------------
# General forecast
# ------------------------------------------------
def test_singleton_instance(self, provider):
"""Test that ElecPriceForecast behaves as a singleton."""
another_instance = ElecPriceAkkudoktor()
assert provider is another_instance
def test_singleton_instance(provider):
"""Test that ElecPriceForecast behaves as a singleton."""
another_instance = ElecPriceAkkudoktor()
assert provider is another_instance
def test_invalid_provider(self, provider, monkeypatch):
"""Test requesting an unsupported provider."""
monkeypatch.setenv("EOS_ELECPRICE__ELECPRICE_PROVIDER", "<invalid>")
provider.config.reset_settings()
assert not provider.enabled()
def test_invalid_provider(provider, monkeypatch):
"""Test requesting an unsupported provider."""
monkeypatch.setenv("EOS_ELECPRICE__ELECPRICE_PROVIDER", "<invalid>")
provider.config.reset_settings()
assert not provider.enabled()
# ------------------------------------------------
# Akkudoktor
# ------------------------------------------------
# ------------------------------------------------
# Akkudoktor
# ------------------------------------------------
@patch("akkudoktoreos.prediction.elecpriceakkudoktor.logger.error")
def test_validate_data_invalid_format(self, mock_logger, provider):
"""Test validation for invalid Akkudoktor data."""
invalid_data = '{"invalid": "data"}'
with pytest.raises(ValueError):
provider._validate_data(invalid_data)
mock_logger.assert_called_once_with(mock_logger.call_args[0][0])
@patch("akkudoktoreos.prediction.elecpriceakkudoktor.logger.error")
def test_validate_data_invalid_format(mock_logger, provider):
"""Test validation for invalid Akkudoktor data."""
invalid_data = '{"invalid": "data"}'
with pytest.raises(ValueError):
provider._validate_data(invalid_data)
mock_logger.assert_called_once_with(mock_logger.call_args[0][0])
@patch("requests.get")
def test_request_forecast(self, mock_get, provider, sample_akkudoktor_1_json):
"""Test requesting forecast from Akkudoktor."""
# Mock response object
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(sample_akkudoktor_1_json)
mock_get.return_value = mock_response
# Test function
akkudoktor_data = provider._request_forecast()
assert isinstance(akkudoktor_data, AkkudoktorElecPrice)
assert akkudoktor_data.values[0].model_dump() == AkkudoktorElecPriceValue(
start_timestamp=1733785200000,
end_timestamp=1733788800000,
start="2024-12-09T23:00:00.000Z",
end="2024-12-10T00:00:00.000Z",
marketprice=92.85,
unit="Eur/MWh",
marketpriceEurocentPerKWh=9.29,
).model_dump()
@patch("requests.get")
def test_request_forecast(mock_get, provider, sample_akkudoktor_1_json):
"""Test requesting forecast from Akkudoktor."""
# Mock response object
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(sample_akkudoktor_1_json)
mock_get.return_value = mock_response
@pytest.mark.asyncio
@patch("requests.get")
async def test_update_data(self, mock_get, provider, sample_akkudoktor_1_json, cache_store):
"""Test fetching forecast from Akkudoktor."""
# Mock response object
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(sample_akkudoktor_1_json)
mock_get.return_value = mock_response
# Test function
akkudoktor_data = provider._request_forecast()
cache_store.clear(clear_all=True)
assert isinstance(akkudoktor_data, AkkudoktorElecPrice)
assert akkudoktor_data.values[0].model_dump() == AkkudoktorElecPriceValue(
start_timestamp=1733785200000,
end_timestamp=1733788800000,
start="2024-12-09T23:00:00.000Z",
end="2024-12-10T00:00:00.000Z",
marketprice=92.85,
unit="Eur/MWh",
marketpriceEurocentPerKWh=9.29,
).model_dump()
# Call the method
ems_eos = get_ems()
ems_eos.set_start_datetime(to_datetime("2024-12-11 00:00:00", in_timezone="Europe/Berlin"))
await provider.update_data(force_enable=True, force_update=True)
# Assert: Verify the result is as expected
mock_get.assert_called_once()
assert (
len(provider) == 73
) # we have 48 datasets in the api response, we want to know 48h into the future. The data we get has already 23h into the future so we need only 25h more. 48+25=73
# Assert we get hours prioce values by resampling
np_price_array = await provider.key_to_array(
key="elecprice_marketprice_wh",
start_datetime=provider.ems_start_datetime,
end_datetime=provider.end_datetime,
)
assert len(np_price_array) == provider.total_hours
# with open(FILE_TESTDATA_ELECPRICEAKKUDOKTOR_2_JSON, "w") as f_out:
# f_out.write(provider.to_json())
@patch("requests.get")
def test_update_data(mock_get, provider, sample_akkudoktor_1_json, cache_store):
"""Test fetching forecast from Akkudoktor."""
# Mock response object
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(sample_akkudoktor_1_json)
mock_get.return_value = mock_response
@pytest.mark.asyncio
@patch("requests.get")
async def test_update_data_with_incomplete_forecast(self, mock_get, provider):
"""Test `_update_data` with incomplete or missing forecast data."""
incomplete_data: dict = {"meta": {}, "values": []}
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(incomplete_data)
mock_get.return_value = mock_response
logger.info("The following errors are intentional and part of the test.")
with pytest.raises(ValueError):
await provider._update_data(force_update=True)
cache_store.clear(clear_all=True)
# Call the method
ems_eos = get_ems()
ems_eos.set_start_datetime(to_datetime("2024-12-11 00:00:00", in_timezone="Europe/Berlin"))
provider.update_data(force_enable=True, force_update=True)
# Assert: Verify the result is as expected
mock_get.assert_called_once()
assert (
len(provider) == 73
) # we have 48 datasets in the api response, we want to know 48h into the future. The data we get has already 23h into the future so we need only 25h more. 48+25=73
# Assert we get hours prioce values by resampling
np_price_array = provider.key_to_array(
key="elecprice_marketprice_wh",
start_datetime=provider.ems_start_datetime,
end_datetime=provider.end_datetime,
@pytest.mark.parametrize(
"status_code, exception",
[(400, requests.exceptions.HTTPError), (500, requests.exceptions.HTTPError), (200, None)],
)
assert len(np_price_array) == provider.total_hours
# with open(FILE_TESTDATA_ELECPRICEAKKUDOKTOR_2_JSON, "w") as f_out:
# f_out.write(provider.to_json())
@patch("requests.get")
def test_update_data_with_incomplete_forecast(mock_get, provider):
"""Test `_update_data` with incomplete or missing forecast data."""
incomplete_data: dict = {"meta": {}, "values": []}
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(incomplete_data)
mock_get.return_value = mock_response
logger.info("The following errors are intentional and part of the test.")
with pytest.raises(ValueError):
provider._update_data(force_update=True)
@pytest.mark.parametrize(
"status_code, exception",
[(400, requests.exceptions.HTTPError), (500, requests.exceptions.HTTPError), (200, None)],
)
@patch("requests.get")
def test_request_forecast_status_codes(
mock_get, provider, sample_akkudoktor_1_json, status_code, exception
):
"""Test handling of various API status codes."""
mock_response = Mock()
mock_response.status_code = status_code
mock_response.content = json.dumps(sample_akkudoktor_1_json)
mock_response.raise_for_status.side_effect = (
requests.exceptions.HTTPError if exception else None
)
mock_get.return_value = mock_response
if exception:
with pytest.raises(exception):
@patch("requests.get")
def test_request_forecast_status_codes(
self, mock_get, provider, sample_akkudoktor_1_json, status_code, exception
):
"""Test handling of various API status codes."""
mock_response = Mock()
mock_response.status_code = status_code
mock_response.content = json.dumps(sample_akkudoktor_1_json)
mock_response.raise_for_status.side_effect = (
requests.exceptions.HTTPError if exception else None
)
mock_get.return_value = mock_response
if exception:
with pytest.raises(exception):
provider._request_forecast()
else:
provider._request_forecast()
else:
provider._request_forecast()
@patch("requests.get")
@patch("akkudoktoreos.core.cache.CacheFileStore")
def test_cache_integration(mock_cache, mock_get, provider, sample_akkudoktor_1_json):
"""Test caching of 8-day electricity price data."""
# Mock response object
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(sample_akkudoktor_1_json)
mock_get.return_value = mock_response
@pytest.mark.asyncio
@patch("requests.get")
@patch("akkudoktoreos.core.cache.CacheFileStore")
async def test_cache_integration(self, mock_cache, mock_get, provider, sample_akkudoktor_1_json):
"""Test caching of 8-day electricity price data."""
# Mock response object
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(sample_akkudoktor_1_json)
mock_get.return_value = mock_response
# Mock cache object
mock_cache_instance = mock_cache.return_value
mock_cache_instance.get.return_value = None # Simulate no cache
# Mock cache object
mock_cache_instance = mock_cache.return_value
mock_cache_instance.get.return_value = None # Simulate no cache
provider._update_data(force_update=True)
mock_cache_instance.create.assert_called_once()
mock_cache_instance.get.assert_called_once()
await provider._update_data(force_update=True)
mock_cache_instance.create.assert_called_once()
mock_cache_instance.get.assert_called_once()
def test_key_to_array_resampling(provider):
"""Test resampling of forecast data to NumPy array."""
provider.update_data(force_update=True)
array = provider.key_to_array(
key="elecprice_marketprice_wh",
start_datetime=provider.ems_start_datetime,
end_datetime=provider.end_datetime,
)
assert isinstance(array, np.ndarray)
assert len(array) == provider.total_hours
@pytest.mark.asyncio
async def test_key_to_array_resampling(self, provider):
"""Test resampling of forecast data to NumPy array."""
await provider.update_data(force_update=True)
array = await provider.key_to_array(
key="elecprice_marketprice_wh",
start_datetime=provider.ems_start_datetime,
end_datetime=provider.end_datetime,
)
assert isinstance(array, np.ndarray)
assert len(array) == provider.total_hours
# ------------------------------------------------
# Development Akkudoktor
# ------------------------------------------------
# ------------------------------------------------
# Development Akkudoktor
# ------------------------------------------------
@pytest.mark.skip(reason="For development only")
def test_akkudoktor_development_forecast_data(provider):
"""Fetch data from real Akkudoktor server."""
# Preset, as this is usually done by update_data()
provider.ems_start_datetime = to_datetime("2024-10-26 00:00:00")
@pytest.mark.skip(reason="For development only")
def test_akkudoktor_development_forecast_data(self, provider):
"""Fetch data from real Akkudoktor server."""
# Preset, as this is usually done by update_data()
provider.ems_start_datetime = to_datetime("2024-10-26 00:00:00")
akkudoktor_data = provider._request_forecast()
akkudoktor_data = provider._request_forecast()
with FILE_TESTDATA_ELECPRICEAKKUDOKTOR_1_JSON.open(
"w", encoding="utf-8", newline="\n"
) as f_out:
json.dump(akkudoktor_data, f_out, indent=4)
with FILE_TESTDATA_ELECPRICEAKKUDOKTOR_1_JSON.open(
"w", encoding="utf-8", newline="\n"
) as f_out:
json.dump(akkudoktor_data, f_out, indent=4)

328
tests/test_elecpriceenergycharts.py
View File

@@ -1,3 +1,4 @@
import asyncio
import json
from pathlib import Path
from unittest.mock import Mock, patch
@@ -51,204 +52,205 @@ def cache_store():
return CacheFileStore()
# ------------------------------------------------
# General forecast
# ------------------------------------------------
@pytest.mark.asyncio
class TestElecPriceEnergyCharts:
# ------------------------------------------------
# General forecast
# ------------------------------------------------
def test_singleton_instance(self, provider):
"""Test that ElecPriceForecast behaves as a singleton."""
another_instance = ElecPriceEnergyCharts()
assert provider is another_instance
def test_singleton_instance(provider):
"""Test that ElecPriceForecast behaves as a singleton."""
another_instance = ElecPriceEnergyCharts()
assert provider is another_instance
def test_invalid_provider(self, provider, monkeypatch):
"""Test requesting an unsupported provider."""
monkeypatch.setenv("EOS_ELECPRICE__ELECPRICE_PROVIDER", "<invalid>")
provider.config.reset_settings()
assert not provider.enabled()
def test_invalid_provider(provider, monkeypatch):
"""Test requesting an unsupported provider."""
monkeypatch.setenv("EOS_ELECPRICE__ELECPRICE_PROVIDER", "<invalid>")
provider.config.reset_settings()
assert not provider.enabled()
# ------------------------------------------------
# Akkudoktor
# ------------------------------------------------
# ------------------------------------------------
# Akkudoktor
# ------------------------------------------------
@patch("akkudoktoreos.prediction.elecpriceenergycharts.logger.error")
def test_validate_data_invalid_format(self, mock_logger, provider):
"""Test validation for invalid Energy-Charts data."""
invalid_data = '{"invalid": "data"}'
with pytest.raises(ValueError):
provider._validate_data(invalid_data)
mock_logger.assert_called_once_with(mock_logger.call_args[0][0])
@patch("akkudoktoreos.prediction.elecpriceenergycharts.logger.error")
def test_validate_data_invalid_format(mock_logger, provider):
"""Test validation for invalid Energy-Charts data."""
invalid_data = '{"invalid": "data"}'
with pytest.raises(ValueError):
provider._validate_data(invalid_data)
mock_logger.assert_called_once_with(mock_logger.call_args[0][0])
@patch("requests.get")
def test_request_forecast(self, mock_get, provider, sample_energycharts_json):
"""Test requesting forecast from Energy-Charts."""
# Mock response object
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(sample_energycharts_json)
mock_get.return_value = mock_response
# Test function
energy_charts_data = provider._request_forecast()
assert isinstance(energy_charts_data, EnergyChartsElecPrice)
assert energy_charts_data.unix_seconds[0] == 1733785200
assert energy_charts_data.price[0] == 92.85
@patch("requests.get")
def test_request_forecast(mock_get, provider, sample_energycharts_json):
"""Test requesting forecast from Energy-Charts."""
# Mock response object
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(sample_energycharts_json)
mock_get.return_value = mock_response
@patch("requests.get")
async def test_update_data(self, mock_get, provider, sample_energycharts_json, cache_store):
"""Test fetching forecast from Energy-Charts."""
# Mock response object
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(sample_energycharts_json)
mock_get.return_value = mock_response
# Test function
energy_charts_data = provider._request_forecast()
cache_store.clear(clear_all=True)
assert isinstance(energy_charts_data, EnergyChartsElecPrice)
assert energy_charts_data.unix_seconds[0] == 1733785200
assert energy_charts_data.price[0] == 92.85
# Call the method
ems_eos = get_ems()
ems_eos.set_start_datetime(to_datetime("2024-12-11 00:00:00", in_timezone="Europe/Berlin"))
await provider.update_data(force_enable=True, force_update=True)
# Assert: Verify the result is as expected
mock_get.assert_called_once()
assert (
len(provider) == 73
) # we have 48 datasets in the api response, we want to know 48h into the future. The data we get has already 23h into the future so we need only 25h more. 48+25=73
# Assert we get hours prioce values by resampling
np_price_array = await provider.key_to_array(
key="elecprice_marketprice_wh",
start_datetime=provider.ems_start_datetime,
end_datetime=provider.end_datetime,
)
assert len(np_price_array) == provider.total_hours
@patch("requests.get")
def test_update_data(mock_get, provider, sample_energycharts_json, cache_store):
"""Test fetching forecast from Energy-Charts."""
# Mock response object
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(sample_energycharts_json)
mock_get.return_value = mock_response
@patch("requests.get")
async def test_update_data_with_incomplete_forecast(self, mock_get, provider):
"""Test `_update_data` with incomplete or missing forecast data."""
incomplete_data: dict = {"license_info": "", "unix_seconds": [], "price": [], "unit": "", "deprecated": False}
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(incomplete_data)
mock_get.return_value = mock_response
logger.info("The following errors are intentional and part of the test.")
with pytest.raises(ValueError):
await provider._update_data(force_update=True)
cache_store.clear(clear_all=True)
# Call the method
ems_eos = get_ems()
ems_eos.set_start_datetime(to_datetime("2024-12-11 00:00:00", in_timezone="Europe/Berlin"))
provider.update_data(force_enable=True, force_update=True)
# Assert: Verify the result is as expected
mock_get.assert_called_once()
assert (
len(provider) == 73
) # we have 48 datasets in the api response, we want to know 48h into the future. The data we get has already 23h into the future so we need only 25h more. 48+25=73
# Assert we get hours prioce values by resampling
np_price_array = provider.key_to_array(
key="elecprice_marketprice_wh",
start_datetime=provider.ems_start_datetime,
end_datetime=provider.end_datetime,
@pytest.mark.parametrize(
"status_code, exception",
[(400, requests.exceptions.HTTPError), (500, requests.exceptions.HTTPError), (200, None)],
)
assert len(np_price_array) == provider.total_hours
@patch("requests.get")
def test_update_data_with_incomplete_forecast(mock_get, provider):
"""Test `_update_data` with incomplete or missing forecast data."""
incomplete_data: dict = {"license_info": "", "unix_seconds": [], "price": [], "unit": "", "deprecated": False}
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(incomplete_data)
mock_get.return_value = mock_response
logger.info("The following errors are intentional and part of the test.")
with pytest.raises(ValueError):
provider._update_data(force_update=True)
@pytest.mark.parametrize(
"status_code, exception",
[(400, requests.exceptions.HTTPError), (500, requests.exceptions.HTTPError), (200, None)],
)
@patch("requests.get")
def test_request_forecast_status_codes(
mock_get, provider, sample_energycharts_json, status_code, exception
):
"""Test handling of various API status codes."""
mock_response = Mock()
mock_response.status_code = status_code
mock_response.content = json.dumps(sample_energycharts_json)
mock_response.raise_for_status.side_effect = (
requests.exceptions.HTTPError if exception else None
)
mock_get.return_value = mock_response
if exception:
with pytest.raises(exception):
@patch("requests.get")
def test_request_forecast_status_codes(
self, mock_get, provider, sample_energycharts_json, status_code, exception
):
"""Test handling of various API status codes."""
mock_response = Mock()
mock_response.status_code = status_code
mock_response.content = json.dumps(sample_energycharts_json)
mock_response.raise_for_status.side_effect = (
requests.exceptions.HTTPError if exception else None
)
mock_get.return_value = mock_response
if exception:
with pytest.raises(exception):
provider._request_forecast()
else:
provider._request_forecast()
else:
provider._request_forecast()
@patch("requests.get")
@patch("akkudoktoreos.core.cache.CacheFileStore")
def test_cache_integration(mock_cache, mock_get, provider, sample_energycharts_json):
"""Test caching of 8-day electricity price data."""
# Mock response object
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(sample_energycharts_json)
mock_get.return_value = mock_response
@patch("requests.get")
@patch("akkudoktoreos.core.cache.CacheFileStore")
async def test_cache_integration(self, mock_cache, mock_get, provider, sample_energycharts_json):
"""Test caching of 8-day electricity price data."""
# Mock response object
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(sample_energycharts_json)
mock_get.return_value = mock_response
# Mock cache object
mock_cache_instance = mock_cache.return_value
mock_cache_instance.get.return_value = None # Simulate no cache
# Mock cache object
mock_cache_instance = mock_cache.return_value
mock_cache_instance.get.return_value = None # Simulate no cache
provider._update_data(force_update=True)
mock_cache_instance.create.assert_called_once()
mock_cache_instance.get.assert_called_once()
await provider._update_data(force_update=True)
mock_cache_instance.create.assert_called_once()
mock_cache_instance.get.assert_called_once()
def test_key_to_array_resampling(provider):
"""Test resampling of forecast data to NumPy array."""
provider.update_data(force_update=True)
array = provider.key_to_array(
key="elecprice_marketprice_wh",
start_datetime=provider.ems_start_datetime,
end_datetime=provider.end_datetime,
)
assert isinstance(array, np.ndarray)
assert len(array) == provider.total_hours
async def test_key_to_array_resampling(self, provider):
"""Test resampling of forecast data to NumPy array."""
await provider.update_data(force_update=True)
array = await provider.key_to_array(
key="elecprice_marketprice_wh",
start_datetime=provider.ems_start_datetime,
end_datetime=provider.end_datetime,
)
assert isinstance(array, np.ndarray)
assert len(array) == provider.total_hours
@patch("requests.get")
def test_request_forecast_url_bidding_zone_is_value(mock_get, provider, sample_energycharts_json):
"""Test that the bidding zone in the API URL uses the enum *value* (e.g. 'DE-LU'),
not the enum repr (e.g. 'EnergyChartsBiddingZones.DE_LU').
@patch("requests.get")
def test_request_forecast_url_bidding_zone_is_value(self, mock_get, provider, sample_energycharts_json):
"""Test that the bidding zone in the API URL uses the enum *value* (e.g. 'DE-LU'),
not the enum repr (e.g. 'EnergyChartsBiddingZones.DE_LU').
Regression test for: bzn=EnergyChartsBiddingZones.DE_LU appearing in the URL
instead of bzn=DE-LU, which caused a 400 Bad Request from the Energy-Charts API.
"""
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(sample_energycharts_json)
mock_get.return_value = mock_response
Regression test for: bzn=EnergyChartsBiddingZones.DE_LU appearing in the URL
instead of bzn=DE-LU, which caused a 400 Bad Request from the Energy-Charts API.
"""
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = json.dumps(sample_energycharts_json)
mock_get.return_value = mock_response
provider._request_forecast(force_update=True)
provider._request_forecast(force_update=True)
assert mock_get.called, "requests.get was never called"
actual_url: str = mock_get.call_args[0][0]
assert mock_get.called, "requests.get was never called"
actual_url: str = mock_get.call_args[0][0]
# Extract the bzn= query parameter value from the URL
from urllib.parse import parse_qs, urlparse
parsed = urlparse(actual_url)
query_params = parse_qs(parsed.query)
# Extract the bzn= query parameter value from the URL
from urllib.parse import parse_qs, urlparse
parsed = urlparse(actual_url)
query_params = parse_qs(parsed.query)
assert "bzn" in query_params, f"'bzn' parameter missing from URL: {actual_url}"
bzn_value = query_params["bzn"][0]
assert "bzn" in query_params, f"'bzn' parameter missing from URL: {actual_url}"
bzn_value = query_params["bzn"][0]
# Must be the raw enum value, never contain a class name or dot notation
assert "." not in bzn_value, (
f"Bidding zone in URL looks like an enum repr: '{bzn_value}'. "
f"Use .value when building the URL, not str(enum)."
)
assert bzn_value == provider.config.elecprice.energycharts.bidding_zone.value, (
f"Expected bzn='{provider.config.elecprice.energycharts.bidding_zone.value}' "
f"but got bzn='{bzn_value}' in URL: {actual_url}"
)
# Must be the raw enum value, never contain a class name or dot notation
assert "." not in bzn_value, (
f"Bidding zone in URL looks like an enum repr: '{bzn_value}'. "
f"Use .value when building the URL, not str(enum)."
)
assert bzn_value == provider.config.elecprice.energycharts.bidding_zone.value, (
f"Expected bzn='{provider.config.elecprice.energycharts.bidding_zone.value}' "
f"but got bzn='{bzn_value}' in URL: {actual_url}"
)
# ------------------------------------------------
# Development Energy Charts
# ------------------------------------------------
# ------------------------------------------------
# Development Energy Charts
# ------------------------------------------------
@pytest.mark.skip(reason="For development only")
def test_energycharts_development_forecast_data(provider):
"""Fetch data from real Energy-Charts server."""
# Preset, as this is usually done by update_data()
provider.ems_start_datetime = to_datetime("2024-10-26 00:00:00")
@pytest.mark.skip(reason="For development only")
def test_energycharts_development_forecast_data(self, provider):
"""Fetch data from real Energy-Charts server."""
# Preset, as this is usually done by update_data()
provider.ems_start_datetime = to_datetime("2024-10-26 00:00:00")
energy_charts_data = provider._request_forecast()
energy_charts_data = provider._request_forecast()
with FILE_TESTDATA_ELECPRICE_ENERGYCHARTS_JSON.open(
"w", encoding="utf-8", newline="\n"
) as f_out:
json.dump(energy_charts_data, f_out, indent=4)
with FILE_TESTDATA_ELECPRICE_ENERGYCHARTS_JSON.open(
"w", encoding="utf-8", newline="\n"
) as f_out:
json.dump(energy_charts_data, f_out, indent=4)

36
tests/test_elecpricefixed.py
View File

@@ -1,5 +1,6 @@
"""Tests for fixed electricity price prediction module."""
import asyncio
import json
from pathlib import Path
from unittest.mock import Mock, patch
@@ -91,6 +92,7 @@ def cache_store():
return CacheFileStore()
@pytest.mark.asyncio
class TestElecPriceFixed:
"""Tests for ElecPriceFixed provider."""
@@ -109,7 +111,7 @@ class TestElecPriceFixed:
provider.config.reset_settings()
assert not provider.enabled()
def test_update_data_hourly_intervals(self, provider, config_eos):
async def test_update_data_hourly_intervals(self, provider, config_eos):
"""Test updating data with hourly intervals (3600s)."""
# Set start datetime
ems_eos = get_ems()
@@ -121,7 +123,7 @@ class TestElecPriceFixed:
config_eos.prediction.hours = 24
# Update data
provider.update_data(force_enable=True, force_update=True)
await provider.update_data(force_enable=True, force_update=True)
# Verify data was generated
assert len(provider) == 24 # 24 hours * 1 interval per hour
@@ -140,7 +142,7 @@ class TestElecPriceFixed:
for i in range(8, 24):
assert abs(records[i].elecprice_marketprice_wh - 0.00034) < 1e-6
def test_update_data_15min_intervals(self, provider, config_eos):
async def test_update_data_15min_intervals(self, provider, config_eos):
"""Test updating data with 15-minute intervals (900s)."""
ems_eos = get_ems()
start_dt = to_datetime("2024-01-01 00:00:00", in_timezone="Europe/Berlin")
@@ -149,7 +151,7 @@ class TestElecPriceFixed:
config_eos.optimization.interval = 900
config_eos.prediction.hours = 10 # spans both windows: 00:00–10:00 = 40 intervals
provider.update_data(force_enable=True, force_update=True)
await provider.update_data(force_enable=True, force_update=True)
# 10 hours * 4 intervals per hour = 40 intervals
assert len(provider) == 40
@@ -173,7 +175,7 @@ class TestElecPriceFixed:
f"Expected day rate at interval {i}, got {records[i].elecprice_marketprice_wh}"
)
def test_update_data_30min_intervals(self, provider, config_eos):
async def test_update_data_30min_intervals(self, provider, config_eos):
"""Test updating data with 30-minute intervals (1800s)."""
ems_eos = get_ems()
start_dt = to_datetime("2024-01-01 00:00:00", in_timezone="Europe/Berlin")
@@ -182,7 +184,7 @@ class TestElecPriceFixed:
config_eos.optimization.interval = 1800
config_eos.prediction.hours = 10 # spans both windows: 00:00–10:00 = 20 intervals
provider.update_data(force_enable=True, force_update=True)
await provider.update_data(force_enable=True, force_update=True)
# 10 hours * 2 intervals per hour = 20 intervals
assert len(provider) == 20
@@ -206,24 +208,24 @@ class TestElecPriceFixed:
f"Expected day rate at interval {i}, got {records[i].elecprice_marketprice_wh}"
)
def test_update_data_without_config(self, provider, config_eos):
async def test_update_data_without_config(self, provider, config_eos):
"""Test update_data fails without configuration."""
# Remove elecpricefixed settings
config_eos.elecprice.elecpricefixed = {}
with pytest.raises(ValueError, match="No time windows configured"):
provider.update_data(force_enable=True, force_update=True)
await provider.update_data(force_enable=True, force_update=True)
def test_update_data_without_time_windows(self, provider, config_eos):
async def test_update_data_without_time_windows(self, provider, config_eos):
"""Test update_data fails without time windows."""
# Set empty time windows
empty_settings = ElecPriceFixedCommonSettings(time_windows=ValueTimeWindowSequence(windows=[]))
config_eos.elecprice.elecpricefixed = empty_settings
with pytest.raises(ValueError, match="No time windows configured"):
provider.update_data(force_enable=True, force_update=True)
await provider.update_data(force_enable=True, force_update=True)
def test_key_to_array_resampling(self, provider, config_eos):
async def test_key_to_array_resampling(self, provider, config_eos):
"""Test that key_to_array can resample to different intervals."""
# Setup provider with hourly data
ems_eos = get_ems()
@@ -233,10 +235,10 @@ class TestElecPriceFixed:
config_eos.optimization.interval = 3600
config_eos.prediction.hours = 24
provider.update_data(force_enable=True, force_update=True)
await provider.update_data(force_enable=True, force_update=True)
# Get data as hourly array (original)
hourly_array = provider.key_to_array(
hourly_array = await provider.key_to_array(
key="elecprice_marketprice_wh",
start_datetime=start_dt,
end_datetime=start_dt.add(hours=24)
@@ -247,7 +249,7 @@ class TestElecPriceFixed:
assert abs(hourly_array[8] - 0.00034) < 1e-6 # Day rate
# Resample to 15-minute intervals
quarter_hour_array = provider.key_to_array(
quarter_hour_array = await provider.key_to_array(
key="elecprice_marketprice_wh",
start_datetime=start_dt,
end_datetime=start_dt.add(hours=24),
@@ -260,7 +262,7 @@ class TestElecPriceFixed:
assert abs(quarter_hour_array[i] - 0.000288) < 1e-6
# Resample to 30-minute intervals
half_hour_array = provider.key_to_array(
half_hour_array = await provider.key_to_array(
key="elecprice_marketprice_wh",
start_datetime=start_dt,
end_datetime=start_dt.add(hours=24),
@@ -277,7 +279,7 @@ class TestElecPriceFixedIntegration:
"""Integration tests for ElecPriceFixed."""
@pytest.mark.skip(reason="For development only")
def test_fixed_price_development(self, config_eos):
async def test_fixed_price_development(self, config_eos):
"""Test fixed price provider with real configuration."""
# Create provider with config
provider = ElecPriceFixed()
@@ -308,7 +310,7 @@ class TestElecPriceFixedIntegration:
config_eos.optimization.interval = 900 # 15 minutes
# Update data
provider.update_data(force_enable=True, force_update=True)
await provider.update_data(force_enable=True, force_update=True)
# Verify data
expected_intervals = 168 * 4 # 7 days * 24h * 4 intervals

129
tests/test_elecpriceimport.py
View File

@@ -1,3 +1,4 @@
import asyncio
import json
from pathlib import Path
@@ -39,76 +40,78 @@ def sample_import_1_json():
return input_data
# ------------------------------------------------
# General forecast
# ------------------------------------------------
@pytest.mark.asyncio
class TestElecPriceImport:
# ------------------------------------------------
# General forecast
# ------------------------------------------------
def test_singleton_instance(provider):
"""Test that ElecPriceForecast behaves as a singleton."""
another_instance = ElecPriceImport()
assert provider is another_instance
def test_singleton_instance(self, provider):
"""Test that ElecPriceForecast behaves as a singleton."""
another_instance = ElecPriceImport()
assert provider is another_instance
def test_invalid_provider(provider, config_eos):
"""Test requesting an unsupported provider."""
settings = {
"elecprice": {
"provider": "<invalid>",
"elecpriceimport": {
"import_file_path": str(FILE_TESTDATA_ELECPRICEIMPORT_1_JSON),
},
def test_invalid_provider(self, provider, config_eos):
"""Test requesting an unsupported provider."""
settings = {
"elecprice": {
"provider": "<invalid>",
"elecpriceimport": {
"import_file_path": str(FILE_TESTDATA_ELECPRICEIMPORT_1_JSON),
},
}
}
}
with pytest.raises(ValueError, match="not a valid electricity price provider"):
config_eos.merge_settings_from_dict(settings)
with pytest.raises(ValueError, match="not a valid electricity price provider"):
config_eos.merge_settings_from_dict(settings)
# ------------------------------------------------
# Import
# ------------------------------------------------
# ------------------------------------------------
# Import
# ------------------------------------------------
@pytest.mark.parametrize(
"start_datetime, from_file",
[
("2024-11-10 00:00:00", True), # No DST in Germany
("2024-08-10 00:00:00", True), # DST in Germany
("2024-03-31 00:00:00", True), # DST change in Germany (23 hours/ day)
("2024-10-27 00:00:00", True), # DST change in Germany (25 hours/ day)
("2024-11-10 00:00:00", False), # No DST in Germany
("2024-08-10 00:00:00", False), # DST in Germany
("2024-03-31 00:00:00", False), # DST change in Germany (23 hours/ day)
("2024-10-27 00:00:00", False), # DST change in Germany (25 hours/ day)
],
)
def test_import(provider, sample_import_1_json, start_datetime, from_file, config_eos):
"""Test fetching forecast from Import."""
key = "elecprice_marketprice_wh"
ems_eos = get_ems()
ems_eos.set_start_datetime(to_datetime(start_datetime, in_timezone="Europe/Berlin"))
if from_file:
config_eos.elecprice.elecpriceimport.import_json = None
assert config_eos.elecprice.elecpriceimport.import_json is None
else:
config_eos.elecprice.elecpriceimport.import_file_path = None
assert config_eos.elecprice.elecpriceimport.import_file_path is None
provider.delete_by_datetime(start_datetime=None, end_datetime=None)
# Call the method
provider.update_data()
# Assert: Verify the result is as expected
assert provider.ems_start_datetime is not None
assert provider.total_hours is not None
assert compare_datetimes(provider.ems_start_datetime, ems_eos.start_datetime).equal
expected_values = sample_import_1_json[key]
result_values = provider.key_to_array(
key=key,
start_datetime=provider.ems_start_datetime,
end_datetime=provider.ems_start_datetime + to_duration(f"{len(expected_values)} hours"),
interval=to_duration("1 hour"),
@pytest.mark.parametrize(
"start_datetime, from_file",
[
("2024-11-10 00:00:00", True), # No DST in Germany
("2024-08-10 00:00:00", True), # DST in Germany
("2024-03-31 00:00:00", True), # DST change in Germany (23 hours/ day)
("2024-10-27 00:00:00", True), # DST change in Germany (25 hours/ day)
("2024-11-10 00:00:00", False), # No DST in Germany
("2024-08-10 00:00:00", False), # DST in Germany
("2024-03-31 00:00:00", False), # DST change in Germany (23 hours/ day)
("2024-10-27 00:00:00", False), # DST change in Germany (25 hours/ day)
],
)
# Allow for some difference due to value calculation on DST change
npt.assert_allclose(result_values, expected_values, rtol=0.001)
async def test_import(self, provider, sample_import_1_json, start_datetime, from_file, config_eos):
"""Test fetching forecast from Import."""
key = "elecprice_marketprice_wh"
ems_eos = get_ems()
ems_eos.set_start_datetime(to_datetime(start_datetime, in_timezone="Europe/Berlin"))
if from_file:
config_eos.elecprice.elecpriceimport.import_json = None
assert config_eos.elecprice.elecpriceimport.import_json is None
else:
config_eos.elecprice.elecpriceimport.import_file_path = None
assert config_eos.elecprice.elecpriceimport.import_file_path is None
await provider.delete_by_datetime(start_datetime=None, end_datetime=None)
# Call the method
await provider.update_data()
# Assert: Verify the result is as expected
assert provider.ems_start_datetime is not None
assert provider.total_hours is not None
assert compare_datetimes(provider.ems_start_datetime, ems_eos.start_datetime).equal
expected_values = sample_import_1_json[key]
result_values = await provider.key_to_array(
key=key,
start_datetime=provider.ems_start_datetime,
end_datetime=provider.ems_start_datetime + to_duration(f"{len(expected_values)} hours"),
interval=to_duration("1 hour"),
)
# Allow for some difference due to value calculation on DST change
npt.assert_allclose(result_values, expected_values, rtol=0.001)

1
tests/test_geneticoptimize.py
View File

@@ -37,6 +37,7 @@ def compare_dict(actual: dict[str, Any], expected: dict[str, Any]):
assert actual[key] == pytest.approx(value)
@pytest.mark.asyncio
@pytest.mark.parametrize(
"fn_in, fn_out, ngen, break_even",
[

6
tests/test_inverter_efficiency.py
View File

@@ -257,8 +257,8 @@ class TestAcChargingInSimulation:
battery=akku,
)
sim = GeneticSimulation()
sim.prepare(
simulation = GeneticSimulation()
simulation.prepare(
GeneticEnergyManagementParameters(
pv_prognose_wh=[0.0] * prediction_hours, # No PV
strompreis_euro_pro_wh=[0.0003] * prediction_hours, # ~30ct/kWh
@@ -272,7 +272,7 @@ class TestAcChargingInSimulation:
ev=None,
home_appliance=None,
)
return sim, akku, inverter
return simulation, akku, inverter
return _build

315
tests/test_loadakkudoktor.py
View File

@@ -1,8 +1,10 @@
import asyncio
from unittest.mock import patch
import numpy as np
import pendulum
import pytest
import pytest_asyncio
from akkudoktoreos.core.coreabc import get_ems, get_measurement
from akkudoktoreos.measurement.measurement import MeasurementDataRecord
@@ -49,8 +51,8 @@ def loadakkudoktoradjusted(config_eos):
assert config_eos.load.loadakkudoktor.loadakkudoktor_year_energy_kwh == 1000
return LoadAkkudoktorAdjusted()
@pytest.fixture
def measurement_eos():
@pytest_asyncio.fixture
async def measurement_eos():
"""Fixture to initialise the Measurement instance."""
# Load meter readings are in kWh
measurement = get_measurement()
@@ -59,7 +61,7 @@ def measurement_eos():
dt = to_datetime("2024-01-01T00:00:00")
interval = to_duration("1 hour")
for i in range(25):
measurement.insert_by_datetime(
await measurement.insert_by_datetime(
MeasurementDataRecord(
date_time=dt,
load0_mr=load0_mr,
@@ -70,8 +72,10 @@ def measurement_eos():
# 0.05 kWh = 50 Wh
load0_mr += 0.05
load1_mr += 0.05
assert compare_datetimes(measurement.min_datetime, to_datetime("2024-01-01T00:00:00")).equal
assert compare_datetimes(measurement.max_datetime, to_datetime("2024-01-02T00:00:00")).equal
min_dt = await measurement.min_datetime()
max_dt = await measurement.max_datetime()
assert compare_datetimes(min_dt, to_datetime("2024-01-01T00:00:00")).equal
assert compare_datetimes(max_dt, to_datetime("2024-01-02T00:00:00")).equal
return measurement
@@ -87,163 +91,166 @@ def mock_load_profiles_file(tmp_path):
return load_profiles_path
def test_loadakkudoktor_settings_validator():
"""Test the field validator for `loadakkudoktor_year_energy_kwh`."""
settings = LoadAkkudoktorCommonSettings(loadakkudoktor_year_energy_kwh=1234)
assert isinstance(settings.loadakkudoktor_year_energy_kwh, float)
assert settings.loadakkudoktor_year_energy_kwh == 1234.0
@pytest.mark.asyncio
class TestLoadAkkudoktor:
settings = LoadAkkudoktorCommonSettings(loadakkudoktor_year_energy_kwh=1234.56)
assert isinstance(settings.loadakkudoktor_year_energy_kwh, float)
assert settings.loadakkudoktor_year_energy_kwh == 1234.56
async def test_loadakkudoktor_settings_validator(self):
"""Test the field validator for `loadakkudoktor_year_energy_kwh`."""
settings = LoadAkkudoktorCommonSettings(loadakkudoktor_year_energy_kwh=1234)
assert isinstance(settings.loadakkudoktor_year_energy_kwh, float)
assert settings.loadakkudoktor_year_energy_kwh == 1234.0
settings = LoadAkkudoktorCommonSettings(loadakkudoktor_year_energy_kwh=1234.56)
assert isinstance(settings.loadakkudoktor_year_energy_kwh, float)
assert settings.loadakkudoktor_year_energy_kwh == 1234.56
async def test_loadakkudoktor_provider_id(self, loadakkudoktor):
"""Test the `provider_id` class method."""
assert loadakkudoktor.provider_id() == "LoadAkkudoktor"
@patch("akkudoktoreos.prediction.loadakkudoktor.np.load")
async def test_load_data_from_mock(self, mock_np_load, mock_load_profiles_file, loadakkudoktor):
"""Test the `load_data` method."""
# Mock numpy load to return data similar to what would be in the file
mock_np_load.return_value = {
"yearly_profiles": np.ones((365, 24)),
"yearly_profiles_std": np.zeros((365, 24)),
}
# Test data loading
data_year_energy = loadakkudoktor.load_data()
assert data_year_energy is not None
assert data_year_energy.shape == (365, 2, 24)
async def test_load_data_from_file(self, loadakkudoktor):
"""Test `load_data` loads data from the profiles file."""
data_year_energy = loadakkudoktor.load_data()
assert data_year_energy is not None
@patch("akkudoktoreos.prediction.loadakkudoktor.LoadAkkudoktor.load_data")
async def test_update_data(self, mock_load_data, loadakkudoktor):
"""Test the `_update` method."""
mock_load_data.return_value = np.random.rand(365, 2, 24)
# Mock methods for updating values
ems_eos = get_ems()
ems_eos.set_start_datetime(pendulum.datetime(2024, 1, 1))
# Assure there are no prediction records
await loadakkudoktor.delete_by_datetime(start_datetime=None, end_datetime=None)
assert len(loadakkudoktor) == 0
# Execute the method
await loadakkudoktor._update_data()
# Validate that update_value is called
assert len(loadakkudoktor) > 0
def test_loadakkudoktor_provider_id(loadakkudoktor):
"""Test the `provider_id` class method."""
assert loadakkudoktor.provider_id() == "LoadAkkudoktor"
@pytest.mark.asyncio
class TestLoadAkkudoktorAdjusted:
async def test_calculate_adjustment(self, loadakkudoktoradjusted, measurement_eos):
"""Test `_calculate_adjustment` for various scenarios."""
data_year_energy = np.random.rand(365, 2, 24)
@patch("akkudoktoreos.prediction.loadakkudoktor.np.load")
def test_load_data_from_mock(mock_np_load, mock_load_profiles_file, loadakkudoktor):
"""Test the `load_data` method."""
# Mock numpy load to return data similar to what would be in the file
mock_np_load.return_value = {
"yearly_profiles": np.ones((365, 24)),
"yearly_profiles_std": np.zeros((365, 24)),
}
# Check the test setup
assert loadakkudoktoradjusted.measurement is measurement_eos
min_dt = await measurement_eos.min_datetime()
assert min_dt == to_datetime("2024-01-01T00:00:00")
max_dt = await measurement_eos.max_datetime()
assert max_dt == to_datetime("2024-01-02T00:00:00")
# Use same calculation as in _calculate_adjustment
compare_start = max_dt - to_duration("7 days")
if compare_datetimes(compare_start, min_dt).lt:
# Not enough measurements for 7 days - use what is available
compare_start = min_dt
compare_end = max_dt
compare_interval = to_duration("1 hour")
load_total_kwh_array = await measurement_eos.load_total_kwh(
start_datetime=compare_start,
end_datetime=compare_end,
interval=compare_interval,
)
np.testing.assert_allclose(load_total_kwh_array, [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1])
# Test data loading
data_year_energy = loadakkudoktor.load_data()
assert data_year_energy is not None
assert data_year_energy.shape == (365, 2, 24)
# Call the method and validate results
weekday_adjust, weekend_adjust = await loadakkudoktoradjusted._calculate_adjustment(data_year_energy)
assert weekday_adjust.shape == (24,)
assert weekend_adjust.shape == (24,)
data_year_energy = np.zeros((365, 2, 24))
weekday_adjust, weekend_adjust = await loadakkudoktoradjusted._calculate_adjustment(data_year_energy)
def test_load_data_from_file(loadakkudoktor):
"""Test `load_data` loads data from the profiles file."""
data_year_energy = loadakkudoktor.load_data()
assert data_year_energy is not None
assert weekday_adjust.shape == (24,)
expected = np.array(
[
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
]
)
np.testing.assert_allclose(weekday_adjust, expected)
assert weekend_adjust.shape == (24,)
expected = np.array(
[
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
]
)
np.testing.assert_array_equal(weekend_adjust, expected)
@patch("akkudoktoreos.prediction.loadakkudoktor.LoadAkkudoktor.load_data")
def test_update_data(mock_load_data, loadakkudoktor):
"""Test the `_update` method."""
mock_load_data.return_value = np.random.rand(365, 2, 24)
async def test_provider_adjustments_with_mock_data(self, loadakkudoktoradjusted):
"""Test full integration of adjustments with mock data."""
with patch(
"akkudoktoreos.prediction.loadakkudoktor.LoadAkkudoktorAdjusted._calculate_adjustment"
) as mock_adjust:
mock_adjust.return_value = (np.zeros(24), np.zeros(24))
# Mock methods for updating values
ems_eos = get_ems()
ems_eos.set_start_datetime(pendulum.datetime(2024, 1, 1))
# Assure there are no prediction records
loadakkudoktor.delete_by_datetime(start_datetime=None, end_datetime=None)
assert len(loadakkudoktor) == 0
# Execute the method
loadakkudoktor._update_data()
# Validate that update_value is called
assert len(loadakkudoktor) > 0
def test_calculate_adjustment(loadakkudoktoradjusted, measurement_eos):
"""Test `_calculate_adjustment` for various scenarios."""
data_year_energy = np.random.rand(365, 2, 24)
# Check the test setup
assert loadakkudoktoradjusted.measurement is measurement_eos
assert measurement_eos.min_datetime == to_datetime("2024-01-01T00:00:00")
assert measurement_eos.max_datetime == to_datetime("2024-01-02T00:00:00")
# Use same calculation as in _calculate_adjustment
compare_start = measurement_eos.max_datetime - to_duration("7 days")
if compare_datetimes(compare_start, measurement_eos.min_datetime).lt:
# Not enough measurements for 7 days - use what is available
compare_start = measurement_eos.min_datetime
compare_end = measurement_eos.max_datetime
compare_interval = to_duration("1 hour")
load_total_kwh_array = measurement_eos.load_total_kwh(
start_datetime=compare_start,
end_datetime=compare_end,
interval=compare_interval,
)
np.testing.assert_allclose(load_total_kwh_array, [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1])
# Call the method and validate results
weekday_adjust, weekend_adjust = loadakkudoktoradjusted._calculate_adjustment(data_year_energy)
assert weekday_adjust.shape == (24,)
assert weekend_adjust.shape == (24,)
data_year_energy = np.zeros((365, 2, 24))
weekday_adjust, weekend_adjust = loadakkudoktoradjusted._calculate_adjustment(data_year_energy)
assert weekday_adjust.shape == (24,)
expected = np.array(
[
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
100.0,
]
)
np.testing.assert_allclose(weekday_adjust, expected)
assert weekend_adjust.shape == (24,)
expected = np.array(
[
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
]
)
np.testing.assert_array_equal(weekend_adjust, expected)
def test_provider_adjustments_with_mock_data(loadakkudoktoradjusted):
"""Test full integration of adjustments with mock data."""
with patch(
"akkudoktoreos.prediction.loadakkudoktor.LoadAkkudoktorAdjusted._calculate_adjustment"
) as mock_adjust:
mock_adjust.return_value = (np.zeros(24), np.zeros(24))
# Test execution
loadakkudoktoradjusted._update_data()
assert mock_adjust.called
# Test execution
await loadakkudoktoradjusted._update_data()
assert mock_adjust.called

99
tests/test_loadvrm.py
View File

@@ -1,3 +1,4 @@
import asyncio
import json
from unittest.mock import call, patch
@@ -48,69 +49,67 @@ def mock_forecast_response():
totals={}
)
@pytest.mark.asyncio
class TestLoadVRM:
def test_update_data_calls_update_value(load_vrm_instance):
with patch.object(load_vrm_instance, "_request_forecast", return_value=mock_forecast_response()), \
patch.object(LoadVrm, "update_value") as mock_update:
async def test_update_data_calls_update_value(self, load_vrm_instance):
with patch.object(load_vrm_instance, "_request_forecast", return_value=mock_forecast_response()), \
patch.object(LoadVrm, "update_value") as mock_update:
load_vrm_instance._update_data()
await load_vrm_instance._update_data()
assert mock_update.call_count == 2
assert mock_update.call_count == 2
expected_calls = [
call(
pendulum.datetime(2025, 1, 1, 0, 0, 0, tz='Europe/Berlin'),
{"loadforecast_power_w": 100.5,}
),
call(
pendulum.datetime(2025, 1, 1, 1, 0, 0, tz='Europe/Berlin'),
{"loadforecast_power_w": 101.2,}
),
]
expected_calls = [
call(
pendulum.datetime(2025, 1, 1, 0, 0, 0, tz='Europe/Berlin'),
{"loadforecast_power_w": 100.5,}
),
call(
pendulum.datetime(2025, 1, 1, 1, 0, 0, tz='Europe/Berlin'),
{"loadforecast_power_w": 101.2,}
),
]
mock_update.assert_has_calls(expected_calls, any_order=False)
mock_update.assert_has_calls(expected_calls, any_order=False)
def test_validate_data_accepts_valid_json(self):
"""Test that _validate_data doesn't raise with valid input."""
response = mock_forecast_response()
json_data = response.model_dump_json()
def test_validate_data_accepts_valid_json():
"""Test that _validate_data doesn't raise with valid input."""
response = mock_forecast_response()
json_data = response.model_dump_json()
validated = LoadVrm._validate_data(json_data)
assert validated.success
assert len(validated.records.vrm_consumption_fc) == 2
validated = LoadVrm._validate_data(json_data)
assert validated.success
assert len(validated.records.vrm_consumption_fc) == 2
def test_validate_data_raises_on_invalid_json(self):
"""_validate_data should raise ValueError on schema mismatch."""
invalid_json = json.dumps({"success": True}) # missing 'records'
with pytest.raises(ValueError) as exc_info:
LoadVrm._validate_data(invalid_json)
def test_validate_data_raises_on_invalid_json():
"""_validate_data should raise ValueError on schema mismatch."""
invalid_json = json.dumps({"success": True}) # missing 'records'
assert "Field:" in str(exc_info.value)
assert "records" in str(exc_info.value)
with pytest.raises(ValueError) as exc_info:
LoadVrm._validate_data(invalid_json)
def test_request_forecast_raises_on_http_error(self, load_vrm_instance):
with patch("requests.get", side_effect=requests.Timeout("Request timed out")) as mock_get:
with pytest.raises(RuntimeError) as exc_info:
load_vrm_instance._request_forecast(0, 1)
assert "Field:" in str(exc_info.value)
assert "records" in str(exc_info.value)
assert "Failed to fetch load forecast" in str(exc_info.value)
mock_get.assert_called_once()
async def test_update_data_does_nothing_on_empty_forecast(self, load_vrm_instance):
empty_response = VrmForecastResponse(
success=True,
records=VrmForecastRecords(vrm_consumption_fc=[], solar_yield_forecast=[]),
totals={}
)
def test_request_forecast_raises_on_http_error(load_vrm_instance):
with patch("requests.get", side_effect=requests.Timeout("Request timed out")) as mock_get:
with pytest.raises(RuntimeError) as exc_info:
load_vrm_instance._request_forecast(0, 1)
with patch.object(load_vrm_instance, "_request_forecast", return_value=empty_response), \
patch.object(LoadVrm, "update_value") as mock_update:
assert "Failed to fetch load forecast" in str(exc_info.value)
mock_get.assert_called_once()
await load_vrm_instance._update_data()
def test_update_data_does_nothing_on_empty_forecast(load_vrm_instance):
empty_response = VrmForecastResponse(
success=True,
records=VrmForecastRecords(vrm_consumption_fc=[], solar_yield_forecast=[]),
totals={}
)
with patch.object(load_vrm_instance, "_request_forecast", return_value=empty_response), \
patch.object(LoadVrm, "update_value") as mock_update:
load_vrm_instance._update_data()
mock_update.assert_not_called()
mock_update.assert_not_called()

62
tests/test_measurement.py
View File

@@ -1,5 +1,6 @@
import numpy as np
import pytest
import pytest_asyncio
from pendulum import datetime, duration
from akkudoktoreos.config.config import SettingsEOS
@@ -219,16 +220,17 @@ class TestMeasurementDataRecord:
assert key in keys
@pytest.mark.asyncio
class TestMeasurement:
"""Test suite for the Measuremen class."""
@pytest.fixture
def measurement_eos(self, config_eos):
@pytest_asyncio.fixture
async def measurement_eos(self, config_eos):
"""Fixture to create a Measurement instance."""
# Load meter readings are in kWh
config_eos.measurement.load_emr_keys = ["load0_mr", "load1_mr", "load2_mr", "load3_mr"]
measurement = get_measurement()
measurement.delete_by_datetime(None, None)
await measurement.delete_by_datetime(None, None)
record0 = MeasurementDataRecord(
date_time=datetime(2023, 1, 1, hour=0),
load0_mr=100,
@@ -269,10 +271,10 @@ class TestMeasurement:
),
]
for record in records:
measurement.insert_by_datetime(record)
await measurement.insert_by_datetime(record)
return measurement
def test_interval_count(self, measurement_eos):
async def test_interval_count(self, measurement_eos):
"""Test interval count calculation."""
start = to_datetime("2023-01-01T00:00:00")
end = to_datetime("2023-01-01T03:00:00")
@@ -280,7 +282,7 @@ class TestMeasurement:
assert measurement_eos._interval_count(start, end, interval) == 3
def test_interval_count_invalid_end_before_start(self, measurement_eos):
async def test_interval_count_invalid_end_before_start(self, measurement_eos):
"""Test interval count raises ValueError when end_datetime is before start_datetime."""
start = to_datetime("2023-01-01T03:00:00")
end = to_datetime("2023-01-01T00:00:00")
@@ -289,7 +291,7 @@ class TestMeasurement:
with pytest.raises(ValueError, match="end_datetime must be after start_datetime"):
measurement_eos._interval_count(start, end, interval)
def test_interval_count_invalid_non_positive_interval(self, measurement_eos):
async def test_interval_count_invalid_non_positive_interval(self, measurement_eos):
"""Test interval count raises ValueError when interval is non-positive."""
start = to_datetime("2023-01-01T00:00:00")
end = to_datetime("2023-01-01T03:00:00")
@@ -297,21 +299,21 @@ class TestMeasurement:
with pytest.raises(ValueError, match="interval must be positive"):
measurement_eos._interval_count(start, end, duration(hours=0))
def test_energy_from_meter_readings_valid_input(self, measurement_eos):
async def test_energy_from_meter_readings_valid_input(self, measurement_eos):
"""Test _energy_from_meter_readings with valid inputs and proper alignment of load data."""
key = "load0_mr"
start_datetime = to_datetime("2023-01-01T00:00:00")
end_datetime = to_datetime("2023-01-01T05:00:00")
interval = duration(hours=1)
load_array = measurement_eos._energy_from_meter_readings(
load_array = await measurement_eos._energy_from_meter_readings(
key, start_datetime, end_datetime, interval
)
expected_load_array = np.array([50, 50, 50, 50, 50]) # Differences between consecutive readings
np.testing.assert_array_equal(load_array, expected_load_array)
def test_energy_from_meter_readings_empty_array(self, measurement_eos):
async def test_energy_from_meter_readings_empty_array(self, measurement_eos):
"""Test _energy_from_meter_readings with no data (empty array)."""
key = "load0_mr"
start_datetime = to_datetime("2023-01-01T00:00:00")
@@ -319,9 +321,9 @@ class TestMeasurement:
interval = duration(hours=1)
# Use empyt records array
measurement_eos.delete_by_datetime(start_datetime, end_datetime)
await measurement_eos.delete_by_datetime(start_datetime, end_datetime)
load_array = measurement_eos._energy_from_meter_readings(
load_array = await measurement_eos._energy_from_meter_readings(
key, start_datetime, end_datetime, interval
)
@@ -332,7 +334,7 @@ class TestMeasurement:
expected_load_array = np.zeros(expected_size)
np.testing.assert_array_equal(load_array, expected_load_array)
def test_energy_from_meter_readings_misaligned_array(self, measurement_eos):
async def test_energy_from_meter_readings_misaligned_array(self, measurement_eos):
"""Test _energy_from_meter_readings with misaligned array size."""
key = "load1_mr"
interval = duration(hours=1)
@@ -342,15 +344,15 @@ class TestMeasurement:
# Use misaligned array, latest interval set to 2 hours (instead of 1 hour)
latest_record_datetime = to_datetime("2023-01-01T05:00:00")
new_record_datetime = to_datetime("2023-01-01T06:00:00")
record = measurement_eos.get_by_datetime(latest_record_datetime)
record = await measurement_eos.get_by_datetime(latest_record_datetime)
assert record is not None
measurement_eos.delete_by_datetime(start_datetime = latest_record_datetime,
end_datetime = new_record_datetime)
await measurement_eos.delete_by_datetime(start_datetime = latest_record_datetime,
end_datetime = new_record_datetime)
record.date_time = new_record_datetime
measurement_eos.insert_by_datetime(record)
await measurement_eos.insert_by_datetime(record)
# Check test setup
dates, values = measurement_eos.key_to_lists(key, start_datetime, None)
dates, values = await measurement_eos.key_to_lists(key, start_datetime, None)
assert dates == [
to_datetime("2023-01-01T00:00:00"),
to_datetime("2023-01-01T01:00:00"),
@@ -360,17 +362,17 @@ class TestMeasurement:
to_datetime("2023-01-01T06:00:00"),
]
assert values == [200, 250, 300, 350, 400, 450]
array = measurement_eos.key_to_array(key, start_datetime, end_datetime + interval, interval=interval)
array = await measurement_eos.key_to_array(key, start_datetime, end_datetime + interval, interval=interval)
np.testing.assert_array_equal(array, [200, 250, 300, 350, 400, 425])
load_array = measurement_eos._energy_from_meter_readings(
load_array = await measurement_eos._energy_from_meter_readings(
key, start_datetime, end_datetime, interval
)
expected_load_array = np.array([50., 50., 50., 50., 25.]) # Differences between consecutive readings
np.testing.assert_array_equal(load_array, expected_load_array)
def test_energy_from_meter_readings_partial_data(self, measurement_eos, caplog):
async def test_energy_from_meter_readings_partial_data(self, measurement_eos, caplog):
"""Test _energy_from_meter_readings with partial data (misaligned but empty array)."""
key = "load2_mr"
start_datetime = to_datetime("2023-01-01T00:00:00")
@@ -378,7 +380,7 @@ class TestMeasurement:
interval = duration(hours=1)
with caplog.at_level("DEBUG"):
load_array = measurement_eos._energy_from_meter_readings(
load_array = await measurement_eos._energy_from_meter_readings(
key, start_datetime, end_datetime, interval
)
@@ -388,7 +390,7 @@ class TestMeasurement:
expected_load_array = np.zeros(expected_size)
np.testing.assert_array_equal(load_array, expected_load_array)
def test_energy_from_meter_readings_negative_interval(self, measurement_eos):
async def test_energy_from_meter_readings_negative_interval(self, measurement_eos):
"""Test _energy_from_meter_readings with a negative interval."""
key = "load3_mr"
start_datetime = to_datetime("2023-01-01T00:00:00")
@@ -396,37 +398,37 @@ class TestMeasurement:
interval = duration(hours=-1)
with pytest.raises(ValueError, match="interval must be positive"):
measurement_eos._energy_from_meter_readings(key, start_datetime, end_datetime, interval)
await measurement_eos._energy_from_meter_readings(key, start_datetime, end_datetime, interval)
def test_load_total_kwh(self, measurement_eos):
async def test_load_total_kwh(self, measurement_eos):
"""Test total load calculation."""
start_datetime = to_datetime("2023-01-01T03:00:00")
end_datetime = to_datetime("2023-01-01T05:00:00")
interval = duration(hours=1)
result = measurement_eos.load_total_kwh(start_datetime=start_datetime, end_datetime=end_datetime, interval=interval)
result = await measurement_eos.load_total_kwh(start_datetime=start_datetime, end_datetime=end_datetime, interval=interval)
# Expected total load per interval
expected = np.array([100, 100]) # Differences between consecutive meter readings
np.testing.assert_array_equal(result, expected)
def test_load_total_kwh_no_data(self, measurement_eos):
async def test_load_total_kwh_no_data(self, measurement_eos):
"""Test total load calculation with no data."""
measurement_eos.records = []
start_datetime = to_datetime("2023-01-01T00:00:00")
end_datetime = to_datetime("2023-01-01T03:00:00")
interval = duration(hours=1)
result = measurement_eos.load_total_kwh(start_datetime=start_datetime, end_datetime=end_datetime, interval=interval)
result = await measurement_eos.load_total_kwh(start_datetime=start_datetime, end_datetime=end_datetime, interval=interval)
expected = np.zeros(3) # No data, so all intervals are zero
np.testing.assert_array_equal(result, expected)
def test_load_total_kwh_partial_intervals(self, measurement_eos):
async def test_load_total_kwh_partial_intervals(self, measurement_eos):
"""Test total load calculation with partial intervals."""
start_datetime = to_datetime("2023-01-01T00:30:00") # Start in the middle of an interval
end_datetime = to_datetime("2023-01-01T01:30:00") # End in the middle of another interval
interval = duration(hours=1)
result = measurement_eos.load_total_kwh(start_datetime=start_datetime, end_datetime=end_datetime, interval=interval)
result = await measurement_eos.load_total_kwh(start_datetime=start_datetime, end_datetime=end_datetime, interval=interval)
expected = np.array([100]) # Only one complete interval covered
np.testing.assert_array_equal(result, expected)

5
tests/test_prediction.py
View File

@@ -133,13 +133,14 @@ def test_prediction_repr(prediction):
assert "WeatherImport" in result
def test_empty_providers(prediction, forecast_providers):
@pytest.mark.asyncio
async def test_empty_providers(prediction, forecast_providers):
"""Test behavior when Prediction does not have providers."""
# Clear all prediction providers from prediction
providers_bkup = prediction.providers.copy()
prediction.providers.clear()
assert prediction.providers == []
prediction.update_data() # Should not raise an error even with no providers
await prediction.update_data() # Should not raise an error even with no providers
# Cleanup after Test
prediction.providers = providers_bkup

96
tests/test_predictionabc.py
View File

@@ -5,6 +5,7 @@ from typing import Any, ClassVar, List, Optional, Union
import pandas as pd
import pendulum
import pytest
import pytest_asyncio
from pydantic import Field
from akkudoktoreos.core.coreabc import get_ems
@@ -69,7 +70,7 @@ class DerivedPredictionProvider(PredictionProvider):
def enabled(self) -> bool:
return self.provider_enabled
def _update_data(self, force_update: Optional[bool] = False) -> None:
async def _update_data(self, force_update: Optional[bool] = False) -> None:
# Simulate update logic
DerivedPredictionProvider.provider_updated = True
@@ -127,6 +128,7 @@ class TestPredictionABC:
# --------------------------------------------------------
@pytest.mark.asyncio
class TestPredictionProvider:
# Fixtures and helper functions
@pytest.fixture
@@ -147,7 +149,7 @@ class TestPredictionProvider:
# Tests
def test_singleton_behavior(self, provider):
async def test_singleton_behavior(self, provider):
"""Test that PredictionProvider enforces singleton behavior."""
instance1 = provider
instance2 = DerivedPredictionProvider()
@@ -155,7 +157,7 @@ class TestPredictionProvider:
"Singleton pattern is not enforced; instances are not the same."
)
def test_update_computed_fields(self, provider, sample_start_datetime):
async def test_update_computed_fields(self, provider, sample_start_datetime):
"""Test that computed fields `end_datetime` and `keep_datetime` are correctly calculated."""
ems_eos = get_ems()
ems_eos.set_start_datetime(sample_start_datetime)
@@ -176,7 +178,7 @@ class TestPredictionProvider:
"Keep datetime is not calculated correctly."
)
def test_update_method_with_defaults(
async def test_update_method_with_defaults(
self, provider, sample_start_datetime, config_eos, monkeypatch
):
"""Test the `update` method with default parameters."""
@@ -188,7 +190,7 @@ class TestPredictionProvider:
provider.config.reset_settings()
ems_eos.set_start_datetime(sample_start_datetime)
provider.update_data()
await provider.update_data()
assert provider.config.prediction.hours == config_eos.prediction.hours
assert provider.config.prediction.historic_hours == 2
@@ -198,7 +200,7 @@ class TestPredictionProvider:
)
assert provider.keep_datetime == sample_start_datetime - to_duration("2 hours")
def test_update_method_force_enable(self, provider, monkeypatch):
async def test_update_method_force_enable(self, provider, monkeypatch):
"""Test that `update` executes when `force_enable` is True, even if `enabled` is False."""
# Preset values that are needed by update
monkeypatch.setenv("EOS_GENERAL__LATITUDE", "37.7749")
@@ -207,13 +209,13 @@ class TestPredictionProvider:
# Override enabled to return False for this test
DerivedPredictionProvider.provider_enabled = False
DerivedPredictionProvider.provider_updated = False
provider.update_data(force_enable=True)
await provider.update_data(force_enable=True)
assert provider.enabled() is False, "Provider should be disabled, but enabled() is True."
assert DerivedPredictionProvider.provider_updated is True, (
"Provider should have been executed, but was not."
)
def test_delete_by_datetime(self, provider, sample_start_datetime):
async def test_delete_by_datetime(self, provider, sample_start_datetime):
"""Test `delete_by_datetime` method for removing records by datetime range."""
# Add records to the provider for deletion testing
records = [
@@ -222,9 +224,9 @@ class TestPredictionProvider:
self.create_test_record(sample_start_datetime + to_duration("1 hour"), 3),
]
for record in records:
provider.insert_by_datetime(record)
await provider.insert_by_datetime(record)
provider.delete_by_datetime(
await provider.delete_by_datetime(
start_datetime=sample_start_datetime - to_duration("2 hours"),
end_datetime=sample_start_datetime + to_duration("2 hours"),
)
@@ -236,6 +238,7 @@ class TestPredictionProvider:
)
@pytest.mark.asyncio
class TestPredictionContainer:
# Fixture and helpers
@pytest.fixture
@@ -243,8 +246,8 @@ class TestPredictionContainer:
container = DerivedPredictionContainer()
return container
@pytest.fixture
def container_with_providers(self):
@pytest_asyncio.fixture
async def container_with_providers(self):
records = [
# Test records - include 'prediction_value' key
self.create_test_record(datetime(2023, 11, 5), 1),
@@ -252,10 +255,10 @@ class TestPredictionContainer:
self.create_test_record(datetime(2023, 11, 7), 3),
]
provider = DerivedPredictionProvider()
provider.delete_by_datetime(start_datetime=None, end_datetime=None)
await provider.delete_by_datetime(start_datetime=None, end_datetime=None)
assert len(provider) == 0
for record in records:
provider.insert_by_datetime(record)
await provider.insert_by_datetime(record)
assert len(provider) == 3
container = DerivedPredictionContainer()
container.providers.clear()
@@ -282,7 +285,7 @@ class TestPredictionContainer:
("2024-10-27 00:00:00", 48, "2024-10-29 00:00:00"), # DST change (49 hours/ day)
],
)
def test_end_datetime(self, container, start, hours, end):
async def test_end_datetime(self, container, start, hours, end):
"""Test end datetime calculation from start datetime."""
ems_eos = get_ems()
ems_eos.set_start_datetime(to_datetime(start, in_timezone="Europe/Berlin"))
@@ -312,7 +315,7 @@ class TestPredictionContainer:
),
],
)
def test_keep_datetime(self, container, start, historic_hours, expected_keep):
async def test_keep_datetime(self, container, start, historic_hours, expected_keep):
"""Test the `keep_datetime` property."""
ems_eos = get_ems()
ems_eos.set_start_datetime(to_datetime(start, in_timezone="Europe/Berlin"))
@@ -334,7 +337,7 @@ class TestPredictionContainer:
("2024-10-27 00:00:00", 24, 25), # DST change in Germany (25 hours/ day)
],
)
def test_total_hours(self, container, start, hours, expected_hours):
async def test_total_hours(self, container, start, hours, expected_hours):
"""Test the `total_hours` property."""
ems_eos = get_ems()
ems_eos.set_start_datetime(to_datetime(start, in_timezone="Europe/Berlin"))
@@ -355,7 +358,7 @@ class TestPredictionContainer:
("2024-10-28 00:00:00", 24, 24), # DST change on 2024-10-27 in Germany (25 hours/ day)
],
)
def test_keep_hours(self, container, start, historic_hours, expected_hours):
async def test_keep_hours(self, container, start, historic_hours, expected_hours):
"""Test the `keep_hours` property."""
ems_eos = get_ems()
ems_eos.set_start_datetime(to_datetime(start, in_timezone="Europe/Berlin"))
@@ -367,80 +370,37 @@ class TestPredictionContainer:
container.config.merge_settings_from_dict(settings)
assert container.keep_hours == expected_hours
def test_append_provider(self, container):
async def test_append_provider(self, container):
assert len(container.providers) == 0
container.providers.append(DerivedPredictionProvider())
assert len(container.providers) == 1
assert isinstance(container.providers[0], DerivedPredictionProvider)
@pytest.mark.skip(reason="type check not implemented")
def test_append_provider_invalid_type(self, container):
async def test_append_provider_invalid_type(self, container):
with pytest.raises(ValueError, match="must be an instance of PredictionProvider"):
container.providers.append("not_a_provider")
def test_getitem_existing_key(self, container_with_providers):
assert len(container_with_providers.providers) == 1
# check all keys are available (don't care for position)
for key in ["prediction_value", "date_time"]:
assert key in container_with_providers.record_keys
for key in ["prediction_value", "date_time"]:
assert key in container_with_providers.keys()
series = container_with_providers["prediction_value"]
assert isinstance(series, pd.Series)
assert series.name == "prediction_value"
assert series.tolist() == [1.0, 2.0, 3.0]
def test_getitem_non_existing_key(self, container_with_providers):
with pytest.raises(KeyError, match="No data found for key 'non_existent_key'"):
container_with_providers["non_existent_key"]
def test_setitem_existing_key(self, container_with_providers):
new_series = container_with_providers["prediction_value"]
new_series[:] = [4, 5, 6]
container_with_providers["prediction_value"] = new_series
series = container_with_providers["prediction_value"]
assert series.name == "prediction_value"
assert series.tolist() == [4, 5, 6]
def test_setitem_invalid_value(self, container_with_providers):
with pytest.raises(ValueError, match="Value must be an instance of pd.Series"):
container_with_providers["test_key"] = "not_a_series"
def test_setitem_non_existing_key(self, container_with_providers):
new_series = pd.Series([4, 5, 6], name="non_existent_key")
with pytest.raises(KeyError, match="Key 'non_existent_key' not found"):
container_with_providers["non_existent_key"] = new_series
def test_delitem_existing_key(self, container_with_providers):
del container_with_providers["prediction_value"]
series = container_with_providers["prediction_value"]
assert series.name == "prediction_value"
assert series.tolist() == []
def test_delitem_non_existing_key(self, container_with_providers):
with pytest.raises(KeyError, match="Key 'non_existent_key' not found"):
del container_with_providers["non_existent_key"]
def test_len(self, container_with_providers):
async def test_len(self, container_with_providers):
assert len(container_with_providers) == 2
def test_repr(self, container_with_providers):
async def test_repr(self, container_with_providers):
representation = repr(container_with_providers)
assert representation.startswith("DerivedPredictionContainer(")
assert "DerivedPredictionProvider" in representation
def test_to_json(self, container_with_providers):
async def test_to_json(self, container_with_providers):
json_str = container_with_providers.to_json()
container_other = DerivedPredictionContainer.from_json(json_str)
assert container_other == container_with_providers
def test_from_json(self, container_with_providers):
async def test_from_json(self, container_with_providers):
json_str = container_with_providers.to_json()
container = DerivedPredictionContainer.from_json(json_str)
assert isinstance(container, DerivedPredictionContainer)
assert len(container.providers) == 1
assert container.providers[0] == container_with_providers.providers[0]
def test_provider_by_id(self, container_with_providers):
async def test_provider_by_id(self, container_with_providers):
provider = container_with_providers.provider_by_id("DerivedPredictionProvider")
assert isinstance(provider, DerivedPredictionProvider)

36
tests/test_pvforecastakkudoktor.py
View File

@@ -3,6 +3,7 @@ from pathlib import Path
from unittest.mock import Mock, patch
import pytest
import pytest_asyncio
from loguru import logger
from akkudoktoreos.core.coreabc import get_ems, get_prediction
@@ -132,11 +133,11 @@ def provider():
return provider
@pytest.fixture
def provider_empty_instance():
@pytest_asyncio.fixture
async def provider_empty_instance():
"""Fixture that returns an empty instance of PVForecast."""
empty_instance = PVForecastAkkudoktor()
empty_instance.delete_by_datetime(start_datetime=None, end_datetime=None)
await empty_instance.delete_by_datetime(start_datetime=None, end_datetime=None)
assert len(empty_instance) == 0
return empty_instance
@@ -234,7 +235,8 @@ def test_pvforecast_akkudoktor_data_record():
) # Assuming AC power measured is preferred
def test_pvforecast_akkudoktor_validate_data(provider_empty_instance, sample_forecast_data_raw):
@pytest.mark.asyncio
async def test_pvforecast_akkudoktor_validate_data(provider_empty_instance, sample_forecast_data_raw):
"""Test validation of PV forecast data on sample data."""
logger.info("The following errors are intentional and part of the test.")
with pytest.raises(
@@ -246,7 +248,8 @@ def test_pvforecast_akkudoktor_validate_data(provider_empty_instance, sample_for
# everything worked
def test_pvforecast_akkudoktor_validate_data_single_plane(
@pytest.mark.asyncio
async def test_pvforecast_akkudoktor_validate_data_single_plane(
provider_empty_instance, sample_forecast_data_single_plane_raw
):
"""Test validation of PV forecast data on sample data with a single plane."""
@@ -260,8 +263,9 @@ def test_pvforecast_akkudoktor_validate_data_single_plane(
# everything worked
@pytest.mark.asyncio
@patch("requests.get")
def test_pvforecast_akkudoktor_update_with_sample_forecast(
async def test_pvforecast_akkudoktor_update_with_sample_forecast(
mock_get, sample_settings, sample_forecast_data_raw, sample_forecast_start, provider
):
"""Test data processing using sample forecast data."""
@@ -274,13 +278,14 @@ def test_pvforecast_akkudoktor_update_with_sample_forecast(
# Test that update properly inserts data records
ems_eos = get_ems()
ems_eos.set_start_datetime(sample_forecast_start)
provider.update_data(force_enable=True, force_update=True)
await provider.update_data(force_enable=True, force_update=True)
assert compare_datetimes(provider.ems_start_datetime, sample_forecast_start).equal
assert compare_datetimes(provider.records[0].date_time, to_datetime(sample_forecast_start)).equal
# Report Generation Test
def test_report_ac_power_and_measurement(provider, config_eos):
@pytest.mark.asyncio
async def test_report_ac_power_and_measurement(provider, config_eos):
# Set the configuration
config_eos.merge_settings_from_dict(sample_config_data)
@@ -289,7 +294,7 @@ def test_report_ac_power_and_measurement(provider, config_eos):
pvforecast_dc_power=450.0,
pvforecast_ac_power=400.0,
)
provider.insert_by_datetime(record)
await provider.insert_by_datetime(record)
report = provider.report_ac_power_and_measurement()
assert "DC: 450.0" in report
@@ -300,8 +305,9 @@ def test_report_ac_power_and_measurement(provider, config_eos):
@pytest.mark.skipif(
sys.platform.startswith("win"), reason="'other_timezone' fixture not supported on Windows"
)
@pytest.mark.asyncio
@patch("requests.get")
def test_timezone_behaviour(
async def test_timezone_behaviour(
mock_get,
sample_settings,
sample_forecast_data_raw,
@@ -322,17 +328,17 @@ def test_timezone_behaviour(
expected_datetime = to_datetime("2024-10-06T00:00:00+0200", in_timezone=other_timezone)
assert compare_datetimes(other_start_datetime, expected_datetime).equal
provider.delete_by_datetime(start_datetime=None, end_datetime=None)
await provider.delete_by_datetime(start_datetime=None, end_datetime=None)
assert len(provider) == 0
ems_eos = get_ems()
ems_eos.set_start_datetime(other_start_datetime)
provider.update_data(force_update=True)
await provider.update_data(force_update=True)
assert compare_datetimes(provider.ems_start_datetime, other_start_datetime).equal
# Check wether first record starts at requested sample start time
assert compare_datetimes(provider.records[0].date_time, sample_forecast_start).equal
# Test updating AC power measurement for a specific date.
provider.update_value(sample_forecast_start, "pvforecastakkudoktor_ac_power_measured", 1000)
await provider.update_value(sample_forecast_start, "pvforecastakkudoktor_ac_power_measured", 1000)
# Check wether first record was filled with ac power measurement
assert provider.records[0].pvforecastakkudoktor_ac_power_measured == 1000
@@ -340,7 +346,7 @@ def test_timezone_behaviour(
other_end_datetime = other_start_datetime + to_duration("24 hours")
expected_end_datetime = to_datetime("2024-10-07T00:00:00+0200", in_timezone=other_timezone)
assert compare_datetimes(other_end_datetime, expected_end_datetime).equal
forecast_temps = provider.key_to_series(
forecast_temps = await provider.key_to_series(
"pvforecastakkudoktor_temp_air", other_start_datetime, other_end_datetime
)
assert len(forecast_temps) == 23 # 24-1, first temperature is null
@@ -349,7 +355,7 @@ def test_timezone_behaviour(
# Test fetching AC power forecast
other_end_datetime = other_start_datetime + to_duration("48 hours")
forecast_measured = provider.key_to_series(
forecast_measured = await provider.key_to_series(
"pvforecastakkudoktor_ac_power_measured", other_start_datetime, other_end_datetime
)
assert len(forecast_measured) == 1

10
tests/test_pvforecastimport.py
View File

@@ -72,7 +72,7 @@ def test_invalid_provider(provider, config_eos):
# Import
# ------------------------------------------------
@pytest.mark.asyncio
@pytest.mark.parametrize(
"start_datetime, from_file",
[
@@ -86,7 +86,7 @@ def test_invalid_provider(provider, config_eos):
("2024-10-27 00:00:00", False), # DST change in Germany (25 hours/ day)
],
)
def test_import(provider, sample_import_1_json, start_datetime, from_file, config_eos):
async def test_import(provider, sample_import_1_json, start_datetime, from_file, config_eos):
"""Test fetching forecast from import."""
key = "pvforecast_ac_power"
ems_eos = get_ems()
@@ -97,10 +97,10 @@ def test_import(provider, sample_import_1_json, start_datetime, from_file, confi
else:
config_eos.pvforecast.provider_settings.PVForecastImport.import_file_path = None
assert config_eos.pvforecast.provider_settings.PVForecastImport.import_file_path is None
provider.delete_by_datetime(start_datetime=None, end_datetime=None)
await provider.delete_by_datetime(start_datetime=None, end_datetime=None)
# Call the method
provider.update_data()
await provider.update_data()
# Assert: Verify the result is as expected
assert provider.ems_start_datetime is not None
@@ -108,7 +108,7 @@ def test_import(provider, sample_import_1_json, start_datetime, from_file, confi
assert compare_datetimes(provider.ems_start_datetime, ems_eos.start_datetime).equal
expected_values = sample_import_1_json[key]
result_values = provider.key_to_array(
result_values = await provider.key_to_array(
key=key,
start_datetime=provider.ems_start_datetime,
end_datetime=provider.ems_start_datetime + to_duration(f"{len(expected_values)} hours"),

10
tests/test_pvforecastvrm.py
View File

@@ -51,11 +51,12 @@ def mock_forecast_response():
)
def test_update_data_updates_dc_and_ac_power(pvforecast_instance):
@pytest.mark.asyncio
async def test_update_data_updates_dc_and_ac_power(pvforecast_instance):
with patch.object(pvforecast_instance, "_request_forecast", return_value=mock_forecast_response()), \
patch.object(PVForecastVrm, "update_value") as mock_update:
pvforecast_instance._update_data()
await pvforecast_instance._update_data()
# Check that update_value was called correctly
assert mock_update.call_count == 2
@@ -103,7 +104,8 @@ def test_request_forecast_raises_on_http_error(pvforecast_instance):
mock_get.assert_called_once()
def test_update_data_skips_on_empty_forecast(pvforecast_instance):
@pytest.mark.asyncio
async def test_update_data_skips_on_empty_forecast(pvforecast_instance):
"""Ensure no update_value calls are made if no forecast data is present."""
empty_response = VrmForecastResponse(
success=True,
@@ -114,5 +116,5 @@ def test_update_data_skips_on_empty_forecast(pvforecast_instance):
with patch.object(pvforecast_instance, "_request_forecast", return_value=empty_response), \
patch.object(PVForecastVrm, "update_value") as mock_update:
pvforecast_instance._update_data()
await pvforecast_instance._update_data()
mock_update.assert_not_called()

281
tests/test_system.py
View File

@@ -1,5 +1,6 @@
import json
import os
import random
import signal
import time
from http import HTTPStatus
@@ -8,12 +9,15 @@ from pathlib import Path
import pytest
import requests
from akkudoktoreos.utils.datetimeutil import to_datetime
DIR_TESTDATA = Path(__file__).absolute().parent.joinpath("testdata")
FILE_TESTDATA_EOSSERVER_CONFIG_1 = DIR_TESTDATA.joinpath("eosserver_config_1.json")
class TestSystem:
def test_prediction_brightsky(self, server_setup_for_class, is_system_test):
"""Test weather prediction by BrightSky."""
server = server_setup_for_class["server"]
@@ -412,6 +416,283 @@ class TestSystem:
f"Expected 400 for invalid datetime, got {result.status_code}"
)
def test_measurement_high_frequency_and_duplicates(self, server_setup_for_class):
"""Simulate production-like high-frequency measurement updates."""
server = server_setup_for_class["server"]
# ----------------------------------------------------------------------
# 1. Configure measurement keys
# ----------------------------------------------------------------------
config = {
"database": {
"provider": "LMDB",
},
"measurement": {
"pv_production_emr_keys": [
"pv1_emr_kwh",
"pv2_emr_kwh",
],
"load_emr_keys": [
"load1_emr_kwh",
"load2_emr_kwh",
],
}
}
result = requests.put(f"{server}/v1/config", json=config)
assert result.status_code == HTTPStatus.OK
result = requests.delete(f"{server}/v1/measurement/range", params={"key": "pv1_emr_kwh"})
assert result.status_code in (HTTPStatus.OK, HTTPStatus.NOT_FOUND)
result = requests.delete(f"{server}/v1/measurement/range", params={"key": "pv2_emr_kwh"})
assert result.status_code in (HTTPStatus.OK, HTTPStatus.NOT_FOUND)
result = requests.delete(f"{server}/v1/measurement/range", params={"key": "load1_emr_kwh"})
assert result.status_code in (HTTPStatus.OK, HTTPStatus.NOT_FOUND)
result = requests.delete(f"{server}/v1/measurement/range", params={"key": "load2_emr_kwh"})
assert result.status_code in (HTTPStatus.OK, HTTPStatus.NOT_FOUND)
# ----------------------------------------------------------------------
# 2. Simulate high-frequency writes (1 Hz, mixed keys)
# ----------------------------------------------------------------------
base_time = "2026-04-10T00:00:00"
timestamps = [
"2026-04-10T00:00:16",
"2026-04-10T00:00:46",
"2026-04-10T00:01:46",
"2026-04-10T00:02:32",
"2026-04-10T00:02:32", # duplicate
"2026-04-10T00:03:32",
"2026-04-10T00:03:32", # duplicate
]
test_cases = [
# valid
("pv1_emr_kwh", -687),
("pv2_emr_kwh", 0.76),
("load1_emr_kwh", 500),
("load2_emr_kwh", 0.0),
# invalid
("invalid-key-1", 123),
("invalid-key-2", 456),
]
results = []
for ts in timestamps:
for key, value in test_cases:
response = requests.put(
f"{server}/v1/measurement/value",
params={
"datetime": ts, # NOTE: naive datetime like production
"key": key,
"value": str(value),
},
)
results.append((ts, key, response.status_code))
result = requests.post(f"{server}/v1/admin/database/save")
assert result.status_code == HTTPStatus.OK
# ----------------------------------------------------------------------
# 3. Assertions: system must behave robustly
# ----------------------------------------------------------------------
# A. Invalid keys must consistently return 404
for ts, key, status in results:
if key.startswith("invalid"):
assert status == HTTPStatus.NOT_FOUND
else:
assert status != HTTPStatus.NOT_FOUND
# B. Valid keys must NEVER produce 500
for ts, key, status in results:
if key in ("pv1_emr_kwh", "pv2_emr_kwh"):
assert status != HTTPStatus.INTERNAL_SERVER_ERROR, (
f"500 error for key={key}, ts={ts}"
)
# C. Duplicates must be handled gracefully (200 or 409 or similar, but not 500)
duplicate_failures = [
(ts, key, status)
for ts, key, status in results
if ts in ("2026-04-10T00:02:32", "2026-04-10T00:03:32")
and key == "pv1_emr_kwh"
and status == HTTPStatus.INTERNAL_SERVER_ERROR
]
assert not duplicate_failures, f"Duplicate timestamp caused 500: {duplicate_failures}"
# ----------------------------------------------------------------------
# 4. Verify data integrity (no explosion / corruption)
# ----------------------------------------------------------------------
result = requests.get(
f"{server}/v1/measurement/series",
params={"key": "pv1_emr_kwh"},
)
assert result.status_code == HTTPStatus.OK
data = result.json()["data"]
# Should not contain excessive duplicates
assert len(data) <= len(set(timestamps)), "Duplicate timestamps not handled properly"
@pytest.mark.parametrize("db_provider", ["LMDB", "SQLite", None])
def test_measurement_realtime_stream(self, db_provider, server_setup_for_class):
"""Simulate real production stream: 1 Hz updates with jitter, duplicates, and out-of-order timestamps."""
server = server_setup_for_class["server"]
# ----------------------------------------------------------------------
# 1. Configure measurement keys and measurement database
# ----------------------------------------------------------------------
config = {
"database": {
"provider": db_provider,
},
"measurement": {
"pv_production_emr_keys": ["pv1_emr_kwh"],
"load_emr_keys": ["load1_emr_kwh"],
}
}
result = requests.put(f"{server}/v1/config", json=config)
assert result.status_code == HTTPStatus.OK, f"Failed: {result.status_code} {result}"
result = requests.delete(f"{server}/v1/measurement/range", params={"key": "pv1_emr_kwh"})
assert result.status_code in (HTTPStatus.OK, HTTPStatus.NOT_FOUND)
result = requests.delete(f"{server}/v1/measurement/range", params={"key": "load1_emr_kwh"})
assert result.status_code in (HTTPStatus.OK, HTTPStatus.NOT_FOUND)
# ----------------------------------------------------------------------
# 2. Real-time simulation
# ----------------------------------------------------------------------
start = to_datetime().replace(microsecond=0)
sent_timestamps = []
errors = []
for i in range(20): # run ~20 seconds
now = to_datetime().replace(microsecond=0)
# --- main timestamp ---
ts = str(now)
# --- simulate normal write ---
response = requests.put(
f"{server}/v1/measurement/value",
params={
"datetime": ts,
"key": "pv1_emr_kwh",
"value": str(random.uniform(0, 1000)),
},
)
if response.status_code == HTTPStatus.INTERNAL_SERVER_ERROR:
errors.append(("main", ts, response.text))
sent_timestamps.append(ts)
# ------------------------------------------------------------------
# Inject real-world problems
# ------------------------------------------------------------------
# 0. Other key
if random.random() < 0.5:
now_same_second = now.add(microseconds=430)
ts_same_second = str(now_same_second)
response = requests.put(
f"{server}/v1/measurement/value",
params={
"datetime": ts_same_second,
"key": "load1_emr_kwh",
"value": str(random.uniform(0, 1000)),
},
)
if response.status_code == HTTPStatus.INTERNAL_SERVER_ERROR:
errors.append(("other", ts, response.text))
# 1. Duplicate timestamp (same second, slightly later)
if random.random() < 0.5:
time.sleep(0.05) # slight delay
response = requests.put(
f"{server}/v1/measurement/value",
params={
"datetime": ts,
"key": "pv1_emr_kwh",
"value": str(random.uniform(0, 1000)),
},
)
if response.status_code == HTTPStatus.INTERNAL_SERVER_ERROR:
errors.append(("duplicate", ts, response.text))
# 2. Out-of-order timestamp (older data arrives late)
if len(sent_timestamps) > 2 and random.random() < 0.5:
old_ts = random.choice(sent_timestamps[:-1])
response = requests.put(
f"{server}/v1/measurement/value",
params={
"datetime": old_ts,
"key": "pv1_emr_kwh",
"value": str(random.uniform(0, 1000)),
},
)
if response.status_code == HTTPStatus.INTERNAL_SERVER_ERROR:
errors.append(("out_of_order", old_ts, response.text))
# 3. Same second burst (multiple writes in same second)
if random.random() < 0.5:
for _ in range(random.randint(2, 4)):
response = requests.put(
f"{server}/v1/measurement/value",
params={
"datetime": ts,
"key": "pv1_emr_kwh",
"value": str(random.uniform(0, 1000)),
},
)
if response.status_code == HTTPStatus.INTERNAL_SERVER_ERROR:
errors.append(("burst", ts, response.text))
# 4. Assure database in memory data is saved to database
if i in (0, 4, 9, 14, 19):
response = requests.post(f"{server}/v1/admin/database/save")
assert response.status_code == HTTPStatus.OK
# small delay to simulate real system (~1 Hz)
time.sleep(1)
# ----------------------------------------------------------------------
# 3. Assertions
# ----------------------------------------------------------------------
assert not errors, f"500 errors occurred: {errors}"
# ----------------------------------------------------------------------
# 4. Verify resulting data
# ----------------------------------------------------------------------
result = requests.get(
f"{server}/v1/measurement/series",
params={"key": "pv1_emr_kwh"},
)
assert result.status_code == HTTPStatus.OK
data = result.json()["data"]
# sanity: should not explode in size
assert len(data) == len(set(sent_timestamps))
parsed = [to_datetime(ts) for ts in data.keys()]
assert all(parsed[i] <= parsed[i+1] for i in range(len(parsed)-1)), \
"Timestamps are not sorted"
unique_keys = set(data.keys())
assert len(unique_keys) == len(data), \
"Duplicate timestamps detected in API output"
def test_admin_cache(self, server_setup_for_class, is_system_test):
"""Test whether cache is reconstructed from cached files."""
server = server_setup_for_class["server"]

8
tests/test_visualize.py
View File

@@ -9,17 +9,13 @@ filename = "example_report.pdf"
DIR_TESTDATA = Path(__file__).parent / "testdata"
reference_file = DIR_TESTDATA / "test_example_report.pdf"
output_file = DIR_TESTDATA / "test_example_report_new.pdf"
def test_generate_pdf_example(config_eos):
"""Test generation of example visualization report."""
output_dir = config_eos.general.data_output_path
assert output_dir is not None
output_file = output_dir / filename
assert not output_file.exists()
# Generate PDF
generate_example_report()
generate_example_report(filename=str(output_file))
# Check if the file exists
assert output_file.exists()

10
tests/test_weatherbrightsky.py
View File

@@ -144,8 +144,9 @@ def test_request_forecast(mock_get, provider, sample_brightsky_1_json):
}
@pytest.mark.asyncio
@patch("requests.get")
def test_update_data(mock_get, provider, sample_brightsky_1_json, cache_store):
async def test_update_data(mock_get, provider, sample_brightsky_1_json, cache_store):
"""Test fetching forecast from BrightSky."""
# Mock response object
mock_response = Mock()
@@ -158,7 +159,7 @@ def test_update_data(mock_get, provider, sample_brightsky_1_json, cache_store):
# Call the method
ems_eos = get_ems()
ems_eos.set_start_datetime(to_datetime("2024-10-26 00:00:00", in_timezone="Europe/Berlin"))
provider.update_data(force_enable=True, force_update=True)
await provider.update_data(force_enable=True, force_update=True)
# Assert: Verify the result is as expected
mock_get.assert_called_once()
@@ -170,7 +171,8 @@ def test_update_data(mock_get, provider, sample_brightsky_1_json, cache_store):
# ------------------------------------------------
def test_brightsky_development_forecast_data(provider, config_eos, is_system_test):
@pytest.mark.asyncio
async def test_brightsky_development_forecast_data(provider, config_eos, is_system_test):
"""Fetch data from real BrightSky server."""
if not is_system_test:
return
@@ -186,7 +188,7 @@ def test_brightsky_development_forecast_data(provider, config_eos, is_system_tes
with FILE_TESTDATA_WEATHERBRIGHTSKY_1_JSON.open("w", encoding="utf-8", newline="\n") as f_out:
json.dump(brightsky_data, f_out, indent=4)
provider.update_data(force_enable=True, force_update=True)
await provider.update_data(force_enable=True, force_update=True)
with FILE_TESTDATA_WEATHERBRIGHTSKY_2_JSON.open("w", encoding="utf-8", newline="\n") as f_out:
f_out.write(provider.model_dump_json(indent=4))

17
tests/test_weatherclearoutside.py
View File

@@ -142,8 +142,9 @@ def test_request_forecast(mock_get, provider, sample_clearout_1_html, config_eos
assert response.content == sample_clearout_1_html
@pytest.mark.asyncio
@patch("requests.get")
def test_update_data(mock_get, provider, sample_clearout_1_html, sample_clearout_1_data):
async def test_update_data(mock_get, provider, sample_clearout_1_html, sample_clearout_1_data):
# Mock response object
mock_response = Mock()
mock_response.status_code = 200
@@ -157,7 +158,7 @@ def test_update_data(mock_get, provider, sample_clearout_1_html, sample_clearout
# Call the method
ems_eos = get_ems()
ems_eos.set_start_datetime(expected_start)
provider.update_data()
await provider.update_data()
# Check for correct prediction time window
assert provider.config.prediction.hours == 48
@@ -177,9 +178,10 @@ def test_update_data(mock_get, provider, sample_clearout_1_html, sample_clearout
# # Check additional weather attributes as necessary
@pytest.mark.asyncio
@pytest.mark.skip(reason="Test fixture to be improved")
@patch("requests.get")
def test_cache_forecast(mock_get, provider, sample_clearout_1_html, cache_store):
async def test_cache_forecast(mock_get, provider, sample_clearout_1_html, cache_store):
"""Test that ClearOutside forecast data is cached with TTL.
This can not be tested with mock_get. Mock objects are not pickable and therefor can not be
@@ -193,11 +195,11 @@ def test_cache_forecast(mock_get, provider, sample_clearout_1_html, cache_store)
cache_store.clear(clear_all=True)
provider.update_data()
await provider.update_data()
mock_get.assert_called_once()
forecast_data_first = provider.to_json()
provider.update_data()
await provider.update_data()
forecast_data_second = provider.to_json()
# Verify that cache returns the same object without calling the method again
assert forecast_data_first == forecast_data_second
@@ -210,9 +212,10 @@ def test_cache_forecast(mock_get, provider, sample_clearout_1_html, cache_store)
# ------------------------------------------------
@pytest.mark.asyncio
@pytest.mark.skip(reason="For development only")
@patch("requests.get")
def test_development_forecast_data(mock_get, provider, sample_clearout_1_html):
async def test_development_forecast_data(mock_get, provider, sample_clearout_1_html):
# Mock response object
mock_response = Mock()
mock_response.status_code = 200
@@ -220,7 +223,7 @@ def test_development_forecast_data(mock_get, provider, sample_clearout_1_html):
mock_get.return_value = mock_response
# Fill the instance
provider.update_data(force_enable=True)
await provider.update_data(force_enable=True)
with FILE_TESTDATA_WEATHERCLEAROUTSIDE_1_DATA.open(
"w", encoding="utf-8", newline="\n"

9
tests/test_weatherimport.py
View File

@@ -73,6 +73,7 @@ def test_invalid_provider(provider, config_eos, monkeypatch):
# ------------------------------------------------
@pytest.mark.asyncio
@pytest.mark.parametrize(
"start_datetime, from_file",
[
@@ -86,7 +87,7 @@ def test_invalid_provider(provider, config_eos, monkeypatch):
("2024-10-27 00:00:00", False), # DST change in Germany (25 hours/ day)
],
)
def test_import(provider, sample_import_1_json, start_datetime, from_file, config_eos):
async def test_import(provider, sample_import_1_json, start_datetime, from_file, config_eos):
"""Test fetching forecast from Import."""
key = "weather_temp_air"
ems_eos = get_ems()
@@ -97,10 +98,10 @@ def test_import(provider, sample_import_1_json, start_datetime, from_file, confi
else:
config_eos.weather.provider_settings.WeatherImport.import_file_path = None
assert config_eos.weather.provider_settings.WeatherImport.import_file_path is None
provider.delete_by_datetime(start_datetime=None, end_datetime=None)
await provider.delete_by_datetime(start_datetime=None, end_datetime=None)
# Call the method
provider.update_data()
await provider.update_data()
# Assert: Verify the result is as expected
assert provider.ems_start_datetime is not None
@@ -108,7 +109,7 @@ def test_import(provider, sample_import_1_json, start_datetime, from_file, confi
assert compare_datetimes(provider.ems_start_datetime, ems_eos.start_datetime).equal
expected_values = sample_import_1_json[key]
result_values = provider.key_to_array(
result_values = await provider.key_to_array(
key=key,
start_datetime=provider.ems_start_datetime,
end_datetime=provider.ems_start_datetime + to_duration(f"{len(expected_values)} hours"),

21
tests/test_weatheropenmeteo.py
View File

@@ -135,8 +135,9 @@ def test_request_forecast(mock_get, provider, sample_openmeteo_1_json):
assert "diffuse_radiation" in openmeteo_data["hourly"] # DHI
@pytest.mark.asyncio
@patch("requests.get")
def test_update_data(mock_get, provider, sample_openmeteo_1_json, cache_store):
async def test_update_data(mock_get, provider, sample_openmeteo_1_json, cache_store):
"""Test fetching and processing forecast from Open-Meteo."""
# Mock response object
mock_response = Mock()
@@ -151,7 +152,7 @@ def test_update_data(mock_get, provider, sample_openmeteo_1_json, cache_store):
ems_eos = get_ems()
start_datetime = to_datetime("2026-03-02 09:00:00+01:00", in_timezone="Europe/Berlin")
ems_eos.set_start_datetime(start_datetime)
provider.update_data(force_enable=True, force_update=True)
await provider.update_data(force_enable=True, force_update=True)
# Assert: Verify the result is as expected
mock_get.assert_called_once()
@@ -160,9 +161,12 @@ def test_update_data(mock_get, provider, sample_openmeteo_1_json, cache_store):
# Verify that direct radiation values were properly mapped
# Get the first record and check for irradiance values
value_datetime = to_datetime("2026-03-04 09:00:00+01:00", in_timezone="Europe/Berlin")
assert provider.key_to_value("weather_ghi", target_datetime=start_datetime) == 21.8
assert provider.key_to_value("weather_dni", target_datetime=start_datetime) == 1.2
assert provider.key_to_value("weather_dhi", target_datetime=start_datetime) == 20.5
weather_ghi = await provider.key_to_value("weather_ghi", target_datetime=start_datetime)
weather_dni = await provider.key_to_value("weather_dni", target_datetime=start_datetime)
weather_dhi = await provider.key_to_value("weather_dhi", target_datetime=start_datetime)
assert weather_ghi == 21.8
assert weather_dni == 1.2
assert weather_dhi == 20.5
# ------------------------------------------------
@@ -279,7 +283,8 @@ def test_openmeteo_request_mode_selection(
# ------------------------------------------------
def test_openmeteo_development_forecast_data(provider, config_eos, is_system_test):
@pytest.mark.asyncio
async def test_openmeteo_development_forecast_data(provider, config_eos, is_system_test):
"""Fetch data from real Open-Meteo server for development purposes."""
if not is_system_test:
return
@@ -304,7 +309,7 @@ def test_openmeteo_development_forecast_data(provider, config_eos, is_system_tes
json.dump(openmeteo_data, f_out, indent=4)
# Update and process data
provider.update_data(force_enable=True, force_update=True)
await provider.update_data(force_enable=True, force_update=True)
# Save processed data
with FILE_TESTDATA_WEATHEROPENMETEO_2_JSON.open("w", encoding="utf-8", newline="\n") as f_out:
@@ -312,7 +317,7 @@ def test_openmeteo_development_forecast_data(provider, config_eos, is_system_tes
# Verify radiation values
if len(provider) > 0:
records = list(provider.data_records.values())
records = list(provider.records)
# Check fo radiation values available
has_ghi = any(hasattr(r, 'ghi') and r.ghi is not None for r in records)

25
tests/testdata/docs/_generated/config.md vendored Normal file
View File

@@ -0,0 +1,25 @@
```{toctree}
:maxdepth: 1
:caption: Configuration Table
../_generated/configadapter.md
../_generated/configcache.md
../_generated/configdatabase.md
../_generated/configdevices.md
../_generated/configelecprice.md
../_generated/configems.md
../_generated/configfeedintariff.md
../_generated/configgeneral.md
../_generated/configload.md
../_generated/configlogging.md
../_generated/configmeasurement.md
../_generated/configoptimization.md
../_generated/configprediction.md
../_generated/configpvforecast.md
../_generated/configserver.md
../_generated/configutils.md
../_generated/configweather.md
../_generated/configexample.md
```
Auto generated from source code.

238
tests/testdata/docs/_generated/configadapter.md vendored Normal file
View File

@@ -0,0 +1,238 @@
## Adapter Configuration
<!-- pyml disable line-length -->
:::{table} adapter
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| homeassistant | `EOS_ADAPTER__HOMEASSISTANT` | `HomeAssistantAdapterCommonSettings` | `rw` | `required` | Home Assistant adapter settings. |
| nodered | `EOS_ADAPTER__NODERED` | `NodeREDAdapterCommonSettings` | `rw` | `required` | NodeRED adapter settings. |
| provider | `EOS_ADAPTER__PROVIDER` | `Optional[list[str]]` | `rw` | `None` | List of adapter provider id(s) of provider(s) to be used. |
| providers | | `list[str]` | `ro` | `N/A` | Available adapter provider ids. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"adapter": {
"provider": [
"HomeAssistant"
],
"homeassistant": {
"config_entity_ids": null,
"load_emr_entity_ids": null,
"grid_export_emr_entity_ids": null,
"grid_import_emr_entity_ids": null,
"pv_production_emr_entity_ids": null,
"device_measurement_entity_ids": null,
"device_instruction_entity_ids": null,
"solution_entity_ids": null
},
"nodered": {
"host": "127.0.0.1",
"port": 1880
}
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"adapter": {
"provider": [
"HomeAssistant"
],
"homeassistant": {
"config_entity_ids": null,
"load_emr_entity_ids": null,
"grid_export_emr_entity_ids": null,
"grid_import_emr_entity_ids": null,
"pv_production_emr_entity_ids": null,
"device_measurement_entity_ids": null,
"device_instruction_entity_ids": null,
"solution_entity_ids": null,
"homeassistant_entity_ids": [],
"eos_solution_entity_ids": [],
"eos_device_instruction_entity_ids": []
},
"nodered": {
"host": "127.0.0.1",
"port": 1880
},
"providers": [
"HomeAssistant",
"NodeRED"
]
}
}
```
<!-- pyml enable line-length -->
### Common settings for the NodeRED adapter
The Node-RED adapter sends to HTTP IN nodes.
This is the example flow:
[HTTP In \\<URL\\>] -> [Function (parse payload)] -> [Debug] -> [HTTP Response]
There are two URLs that are used:
- GET /eos/data_aquisition
The GET is issued before the optimization.
- POST /eos/control_dispatch
The POST is issued after the optimization.
<!-- pyml disable line-length -->
:::{table} adapter::nodered
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| host | `Optional[str]` | `rw` | `127.0.0.1` | Node-RED server IP address. Defaults to 127.0.0.1. |
| port | `Optional[int]` | `rw` | `1880` | Node-RED server IP port number. Defaults to 1880. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"adapter": {
"nodered": {
"host": "127.0.0.1",
"port": 1880
}
}
}
```
<!-- pyml enable line-length -->
### Common settings for the home assistant adapter
<!-- pyml disable line-length -->
:::{table} adapter::homeassistant
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| config_entity_ids | `Optional[dict[str, str]]` | `rw` | `None` | Mapping of EOS config keys to Home Assistant entity IDs.
The config key has to be given by a ‘/’-separated path
e.g. devices/batteries/0/capacity_wh |
| device_instruction_entity_ids | `Optional[list[str]]` | `rw` | `None` | Entity IDs for device (resource) instructions to be updated by EOS.
The device ids (resource ids) have to be prepended by 'sensor.eos_' to build the entity_id.
E.g. The instruction for device id 'battery1' becomes the entity_id 'sensor.eos_battery1'. |
| device_measurement_entity_ids | `Optional[dict[str, str]]` | `rw` | `None` | Mapping of EOS measurement keys used by device (resource) simulations to Home Assistant entity IDs. |
| eos_device_instruction_entity_ids | `list[str]` | `ro` | `N/A` | Entity IDs for energy management instructions available at EOS. |
| eos_solution_entity_ids | `list[str]` | `ro` | `N/A` | Entity IDs for optimization solution available at EOS. |
| grid_export_emr_entity_ids | `Optional[list[str]]` | `rw` | `None` | Entity ID(s) of export to grid energy meter readings [kWh] |
| grid_import_emr_entity_ids | `Optional[list[str]]` | `rw` | `None` | Entity ID(s) of import from grid energy meter readings [kWh] |
| homeassistant_entity_ids | `list[str]` | `ro` | `N/A` | Entity IDs available at Home Assistant. |
| load_emr_entity_ids | `Optional[list[str]]` | `rw` | `None` | Entity ID(s) of load energy meter readings [kWh] |
| pv_production_emr_entity_ids | `Optional[list[str]]` | `rw` | `None` | Entity ID(s) of PV production energy meter readings [kWh] |
| solution_entity_ids | `Optional[list[str]]` | `rw` | `None` | Entity IDs for optimization solution keys to be updated by EOS.
The solution keys have to be prepended by 'sensor.eos_' to build the entity_id.
E.g. solution key 'battery1_idle_op_mode' becomes the entity_id 'sensor.eos_battery1_idle_op_mode'. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"adapter": {
"homeassistant": {
"config_entity_ids": {
"devices/batteries/0/capacity_wh": "sensor.battery1_capacity"
},
"load_emr_entity_ids": [
"sensor.load_energy_total_kwh"
],
"grid_export_emr_entity_ids": [
"sensor.grid_export_energy_total_kwh"
],
"grid_import_emr_entity_ids": [
"sensor.grid_import_energy_total_kwh"
],
"pv_production_emr_entity_ids": [
"sensor.pv_energy_total_kwh"
],
"device_measurement_entity_ids": {
"ev11_soc_factor": "sensor.ev11_soc_factor",
"battery1_soc_factor": "sensor.battery1_soc_factor"
},
"device_instruction_entity_ids": [
"sensor.eos_battery1"
],
"solution_entity_ids": [
"sensor.eos_battery1_idle_mode_mode"
]
}
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"adapter": {
"homeassistant": {
"config_entity_ids": {
"devices/batteries/0/capacity_wh": "sensor.battery1_capacity"
},
"load_emr_entity_ids": [
"sensor.load_energy_total_kwh"
],
"grid_export_emr_entity_ids": [
"sensor.grid_export_energy_total_kwh"
],
"grid_import_emr_entity_ids": [
"sensor.grid_import_energy_total_kwh"
],
"pv_production_emr_entity_ids": [
"sensor.pv_energy_total_kwh"
],
"device_measurement_entity_ids": {
"ev11_soc_factor": "sensor.ev11_soc_factor",
"battery1_soc_factor": "sensor.battery1_soc_factor"
},
"device_instruction_entity_ids": [
"sensor.eos_battery1"
],
"solution_entity_ids": [
"sensor.eos_battery1_idle_mode_mode"
],
"homeassistant_entity_ids": [],
"eos_solution_entity_ids": [],
"eos_device_instruction_entity_ids": []
}
}
}
```
<!-- pyml enable line-length -->

28
tests/testdata/docs/_generated/configcache.md vendored Normal file
View File

@@ -0,0 +1,28 @@
## Cache Configuration
<!-- pyml disable line-length -->
:::{table} cache
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| cleanup_interval | `EOS_CACHE__CLEANUP_INTERVAL` | `float` | `rw` | `300.0` | Intervall in seconds for EOS file cache cleanup. |
| subpath | `EOS_CACHE__SUBPATH` | `Optional[pathlib.Path]` | `rw` | `cache` | Sub-path for the EOS cache data directory. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"cache": {
"subpath": "cache",
"cleanup_interval": 300.0
}
}
```
<!-- pyml enable line-length -->

73
tests/testdata/docs/_generated/configdatabase.md vendored Normal file
View File

@@ -0,0 +1,73 @@
## Configuration model for database settings
Attributes:
provider: Optional provider identifier (e.g. "LMDB").
max_records_in_memory: Maximum records kept in memory before auto-save.
auto_save: Whether to auto-save when threshold exceeded.
batch_size: Batch size for batch operations.
<!-- pyml disable line-length -->
:::{table} database
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| autosave_interval_sec | `EOS_DATABASE__AUTOSAVE_INTERVAL_SEC` | `Optional[int]` | `rw` | `10` | Automatic saving interval [seconds].
Set to None to disable automatic saving. |
| batch_size | `EOS_DATABASE__BATCH_SIZE` | `int` | `rw` | `100` | Number of records to process in batch operations. |
| compaction_interval_sec | `EOS_DATABASE__COMPACTION_INTERVAL_SEC` | `Optional[int]` | `rw` | `604800` | Interval in between automatic tiered compaction runs [seconds].
Compaction downsamples old records to reduce storage while retaining coverage. Set to None to disable automatic compaction. |
| compression_level | `EOS_DATABASE__COMPRESSION_LEVEL` | `int` | `rw` | `9` | Compression level for database record data. |
| initial_load_window_h | `EOS_DATABASE__INITIAL_LOAD_WINDOW_H` | `Optional[int]` | `rw` | `None` | Specifies the default duration of the initial load window when loading records from the database, in hours. If set to None, the full available range is loaded. The window is centered around the current time by default, unless a different center time is specified. Different database namespaces may define their own default windows. |
| keep_duration_h | `EOS_DATABASE__KEEP_DURATION_H` | `Optional[int]` | `rw` | `None` | Default maximum duration records shall be kept in database [hours, none].
None indicates forever. Database namespaces may have diverging definitions. |
| provider | `EOS_DATABASE__PROVIDER` | `Optional[str]` | `rw` | `None` | Database provider id of provider to be used. |
| providers | | `List[str]` | `ro` | `N/A` | Return available database provider ids. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"database": {
"provider": "LMDB",
"compression_level": 0,
"initial_load_window_h": 48,
"keep_duration_h": 48,
"autosave_interval_sec": 5,
"compaction_interval_sec": 604800,
"batch_size": 100
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"database": {
"provider": "LMDB",
"compression_level": 0,
"initial_load_window_h": 48,
"keep_duration_h": 48,
"autosave_interval_sec": 5,
"compaction_interval_sec": 604800,
"batch_size": 100,
"providers": [
"LMDB",
"SQLite",
"NoDB"
]
}
}
```
<!-- pyml enable line-length -->

549
tests/testdata/docs/_generated/configdevices.md vendored Normal file
View File

@@ -0,0 +1,549 @@
## Base configuration for devices simulation settings
<!-- pyml disable line-length -->
:::{table} devices
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| batteries | `EOS_DEVICES__BATTERIES` | `Optional[list[akkudoktoreos.devices.devices.BatteriesCommonSettings]]` | `rw` | `None` | List of battery devices |
| electric_vehicles | `EOS_DEVICES__ELECTRIC_VEHICLES` | `Optional[list[akkudoktoreos.devices.devices.BatteriesCommonSettings]]` | `rw` | `None` | List of electric vehicle devices |
| home_appliances | `EOS_DEVICES__HOME_APPLIANCES` | `Optional[list[akkudoktoreos.devices.devices.HomeApplianceCommonSettings]]` | `rw` | `None` | List of home appliances |
| inverters | `EOS_DEVICES__INVERTERS` | `Optional[list[akkudoktoreos.devices.devices.InverterCommonSettings]]` | `rw` | `None` | List of inverters |
| max_batteries | `EOS_DEVICES__MAX_BATTERIES` | `Optional[int]` | `rw` | `None` | Maximum number of batteries that can be set |
| max_electric_vehicles | `EOS_DEVICES__MAX_ELECTRIC_VEHICLES` | `Optional[int]` | `rw` | `None` | Maximum number of electric vehicles that can be set |
| max_home_appliances | `EOS_DEVICES__MAX_HOME_APPLIANCES` | `Optional[int]` | `rw` | `None` | Maximum number of home_appliances that can be set |
| max_inverters | `EOS_DEVICES__MAX_INVERTERS` | `Optional[int]` | `rw` | `None` | Maximum number of inverters that can be set |
| measurement_keys | | `Optional[list[str]]` | `ro` | `N/A` | Return the measurement keys for the resource/ device stati that are measurements. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"devices": {
"batteries": [
{
"device_id": "battery1",
"capacity_wh": 8000,
"charging_efficiency": 0.88,
"discharging_efficiency": 0.88,
"levelized_cost_of_storage_kwh": 0.0,
"max_charge_power_w": 5000,
"min_charge_power_w": 50,
"charge_rates": [
0.0,
0.1,
0.2,
0.3,
0.4,
0.5,
0.6,
0.7,
0.8,
0.9,
1.0
],
"min_soc_percentage": 0,
"max_soc_percentage": 100
}
],
"max_batteries": 1,
"electric_vehicles": [
{
"device_id": "battery1",
"capacity_wh": 8000,
"charging_efficiency": 0.88,
"discharging_efficiency": 0.88,
"levelized_cost_of_storage_kwh": 0.0,
"max_charge_power_w": 5000,
"min_charge_power_w": 50,
"charge_rates": [
0.0,
0.1,
0.2,
0.3,
0.4,
0.5,
0.6,
0.7,
0.8,
0.9,
1.0
],
"min_soc_percentage": 0,
"max_soc_percentage": 100
}
],
"max_electric_vehicles": 1,
"inverters": [],
"max_inverters": 1,
"home_appliances": [],
"max_home_appliances": 1
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"devices": {
"batteries": [
{
"device_id": "battery1",
"capacity_wh": 8000,
"charging_efficiency": 0.88,
"discharging_efficiency": 0.88,
"levelized_cost_of_storage_kwh": 0.0,
"max_charge_power_w": 5000,
"min_charge_power_w": 50,
"charge_rates": [
0.0,
0.1,
0.2,
0.3,
0.4,
0.5,
0.6,
0.7,
0.8,
0.9,
1.0
],
"min_soc_percentage": 0,
"max_soc_percentage": 100,
"measurement_key_soc_factor": "battery1-soc-factor",
"measurement_key_power_l1_w": "battery1-power-l1-w",
"measurement_key_power_l2_w": "battery1-power-l2-w",
"measurement_key_power_l3_w": "battery1-power-l3-w",
"measurement_key_power_3_phase_sym_w": "battery1-power-3-phase-sym-w",
"measurement_keys": [
"battery1-soc-factor",
"battery1-power-l1-w",
"battery1-power-l2-w",
"battery1-power-l3-w",
"battery1-power-3-phase-sym-w"
]
}
],
"max_batteries": 1,
"electric_vehicles": [
{
"device_id": "battery1",
"capacity_wh": 8000,
"charging_efficiency": 0.88,
"discharging_efficiency": 0.88,
"levelized_cost_of_storage_kwh": 0.0,
"max_charge_power_w": 5000,
"min_charge_power_w": 50,
"charge_rates": [
0.0,
0.1,
0.2,
0.3,
0.4,
0.5,
0.6,
0.7,
0.8,
0.9,
1.0
],
"min_soc_percentage": 0,
"max_soc_percentage": 100,
"measurement_key_soc_factor": "battery1-soc-factor",
"measurement_key_power_l1_w": "battery1-power-l1-w",
"measurement_key_power_l2_w": "battery1-power-l2-w",
"measurement_key_power_l3_w": "battery1-power-l3-w",
"measurement_key_power_3_phase_sym_w": "battery1-power-3-phase-sym-w",
"measurement_keys": [
"battery1-soc-factor",
"battery1-power-l1-w",
"battery1-power-l2-w",
"battery1-power-l3-w",
"battery1-power-3-phase-sym-w"
]
}
],
"max_electric_vehicles": 1,
"inverters": [],
"max_inverters": 1,
"home_appliances": [],
"max_home_appliances": 1,
"measurement_keys": [
"battery1-soc-factor",
"battery1-power-l1-w",
"battery1-power-l2-w",
"battery1-power-l3-w",
"battery1-power-3-phase-sym-w",
"battery1-soc-factor",
"battery1-power-l1-w",
"battery1-power-l2-w",
"battery1-power-l3-w",
"battery1-power-3-phase-sym-w"
]
}
}
```
<!-- pyml enable line-length -->
### Inverter devices base settings
<!-- pyml disable line-length -->
:::{table} devices::inverters::list
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| ac_to_dc_efficiency | `float` | `rw` | `1.0` | Efficiency of AC to DC conversion for grid-to-battery AC charging (0-1). Set to 0 to disable AC charging. Default 1.0 (no additional inverter loss). |
| battery_id | `Optional[str]` | `rw` | `None` | ID of battery controlled by this inverter. |
| dc_to_ac_efficiency | `float` | `rw` | `1.0` | Efficiency of DC to AC conversion for battery discharging to AC load/grid (0-1). Default 1.0 (no additional inverter loss). |
| device_id | `str` | `rw` | `<unknown>` | ID of device |
| max_ac_charge_power_w | `Optional[float]` | `rw` | `None` | Maximum AC charging power in watts. null means no additional limit. Set to 0 to disable AC charging. |
| max_power_w | `Optional[float]` | `rw` | `None` | Maximum power [W]. |
| measurement_keys | `Optional[list[str]]` | `ro` | `N/A` | Measurement keys for the inverter stati that are measurements. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"devices": {
"inverters": [
{
"device_id": "battery1",
"max_power_w": 10000.0,
"battery_id": null,
"ac_to_dc_efficiency": 0.95,
"dc_to_ac_efficiency": 0.95,
"max_ac_charge_power_w": null
}
]
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"devices": {
"inverters": [
{
"device_id": "battery1",
"max_power_w": 10000.0,
"battery_id": null,
"ac_to_dc_efficiency": 0.95,
"dc_to_ac_efficiency": 0.95,
"max_ac_charge_power_w": null,
"measurement_keys": []
}
]
}
}
```
<!-- pyml enable line-length -->
### Model defining a daily or date time window with optional localization support
Represents a time interval starting at `start_time` and lasting for `duration`.
Can restrict applicability to a specific day of the week or a specific calendar date.
Supports day names in multiple languages via locale-aware parsing.
Timezone contract:
``start_time`` is always **naive** (no ``tzinfo``). It is interpreted as a
local wall-clock time in whatever timezone the caller's ``date_time`` or
``reference_date`` carries. When those arguments are timezone-aware the
window boundaries are evaluated in that timezone; when they are naive,
arithmetic is performed as-is (no timezone conversion occurs).
``date``, being a calendar ``Date`` object, is inherently timezone-free.
This design avoids the ambiguity that arises when a stored ``start_time``
carries its own timezone that differs from the caller's timezone, and keeps
the model serialisable without timezone state.
<!-- pyml disable line-length -->
:::{table} devices::home_appliances::list::time_windows::windows::list
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| date | `Optional[pydantic_extra_types.pendulum_dt.Date]` | `rw` | `None` | Optional specific calendar date for the time window. Naive — matched against the local date of the datetime passed to contains(). Overrides `day_of_week` if set. |
| day_of_week | `Union[int, str, NoneType]` | `rw` | `None` | Optional day of the week restriction. Can be specified as integer (0=Monday to 6=Sunday) or localized weekday name. If None, applies every day unless `date` is set. |
| duration | `Duration` | `rw` | `required` | Duration of the time window starting from `start_time`. |
| locale | `Optional[str]` | `rw` | `None` | Locale used to parse weekday names in `day_of_week` when given as string. If not set, Pendulum's default locale is used. Examples: 'en', 'de', 'fr', etc. |
| start_time | `Time` | `rw` | `required` | Naive start time of the time window (time of day, no timezone). Interpreted in the timezone of the datetime passed to contains() or earliest_start_time(). |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"devices": {
"home_appliances": [
{
"time_windows": {
"windows": [
{
"start_time": "00:00:00.000000",
"duration": "2 hours",
"day_of_week": null,
"date": null,
"locale": null
}
]
}
}
]
}
}
```
<!-- pyml enable line-length -->
### Model representing a sequence of time windows with collective operations
Manages multiple TimeWindow objects and provides methods to work with them
as a cohesive unit for scheduling and availability checking.
<!-- pyml disable line-length -->
:::{table} devices::home_appliances::list::time_windows
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| windows | `list[akkudoktoreos.config.configabc.TimeWindow]` | `rw` | `required` | List of TimeWindow objects that make up this sequence. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"devices": {
"home_appliances": [
{
"time_windows": {
"windows": []
}
}
]
}
}
```
<!-- pyml enable line-length -->
### Home Appliance devices base settings
<!-- pyml disable line-length -->
:::{table} devices::home_appliances::list
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| consumption_wh | `int` | `rw` | `required` | Energy consumption [Wh]. |
| device_id | `str` | `rw` | `<unknown>` | ID of device |
| duration_h | `int` | `rw` | `required` | Usage duration in hours [0 ... 24]. |
| measurement_keys | `Optional[list[str]]` | `ro` | `N/A` | Measurement keys for the home appliance stati that are measurements. |
| time_windows | `Optional[akkudoktoreos.config.configabc.TimeWindowSequence]` | `rw` | `None` | Sequence of allowed time windows. Defaults to optimization general time window. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"devices": {
"home_appliances": [
{
"device_id": "battery1",
"consumption_wh": 2000,
"duration_h": 1,
"time_windows": {
"windows": [
{
"start_time": "10:00:00.000000",
"duration": "2 hours",
"day_of_week": null,
"date": null,
"locale": null
}
]
}
}
]
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"devices": {
"home_appliances": [
{
"device_id": "battery1",
"consumption_wh": 2000,
"duration_h": 1,
"time_windows": {
"windows": [
{
"start_time": "10:00:00.000000",
"duration": "2 hours",
"day_of_week": null,
"date": null,
"locale": null
}
]
},
"measurement_keys": []
}
]
}
}
```
<!-- pyml enable line-length -->
### Battery devices base settings
<!-- pyml disable line-length -->
:::{table} devices::batteries::list
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| capacity_wh | `int` | `rw` | `8000` | Capacity [Wh]. |
| charge_rates | `Optional[list[float]]` | `rw` | `[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]` | Charge rates as factor of maximum charging power [0.00 ... 1.00]. None triggers fallback to default charge-rates. |
| charging_efficiency | `float` | `rw` | `0.88` | Charging efficiency [0.01 ... 1.00]. |
| device_id | `str` | `rw` | `<unknown>` | ID of device |
| discharging_efficiency | `float` | `rw` | `0.88` | Discharge efficiency [0.01 ... 1.00]. |
| levelized_cost_of_storage_kwh | `float` | `rw` | `0.0` | Levelized cost of storage (LCOS), the average lifetime cost of delivering one kWh [€/kWh]. |
| max_charge_power_w | `Optional[float]` | `rw` | `5000` | Maximum charging power [W]. |
| max_soc_percentage | `int` | `rw` | `100` | Maximum state of charge (SOC) as percentage of capacity [%]. |
| measurement_key_power_3_phase_sym_w | `str` | `ro` | `N/A` | Measurement key for the symmetric 3 phase power the battery is charged or discharged with [W]. |
| measurement_key_power_l1_w | `str` | `ro` | `N/A` | Measurement key for the L1 power the battery is charged or discharged with [W]. |
| measurement_key_power_l2_w | `str` | `ro` | `N/A` | Measurement key for the L2 power the battery is charged or discharged with [W]. |
| measurement_key_power_l3_w | `str` | `ro` | `N/A` | Measurement key for the L3 power the battery is charged or discharged with [W]. |
| measurement_key_soc_factor | `str` | `ro` | `N/A` | Measurement key for the battery state of charge (SoC) as factor of total capacity [0.0 ... 1.0]. |
| measurement_keys | `Optional[list[str]]` | `ro` | `N/A` | Measurement keys for the battery stati that are measurements. |
| min_charge_power_w | `Optional[float]` | `rw` | `50` | Minimum charging power [W]. |
| min_soc_percentage | `int` | `rw` | `0` | Minimum state of charge (SOC) as percentage of capacity [%]. This is the target SoC for charging |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"devices": {
"batteries": [
{
"device_id": "battery1",
"capacity_wh": 8000,
"charging_efficiency": 0.88,
"discharging_efficiency": 0.88,
"levelized_cost_of_storage_kwh": 0.12,
"max_charge_power_w": 5000.0,
"min_charge_power_w": 50.0,
"charge_rates": [
0.0,
0.25,
0.5,
0.75,
1.0
],
"min_soc_percentage": 10,
"max_soc_percentage": 100
}
]
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"devices": {
"batteries": [
{
"device_id": "battery1",
"capacity_wh": 8000,
"charging_efficiency": 0.88,
"discharging_efficiency": 0.88,
"levelized_cost_of_storage_kwh": 0.12,
"max_charge_power_w": 5000.0,
"min_charge_power_w": 50.0,
"charge_rates": [
0.0,
0.25,
0.5,
0.75,
1.0
],
"min_soc_percentage": 10,
"max_soc_percentage": 100,
"measurement_key_soc_factor": "battery1-soc-factor",
"measurement_key_power_l1_w": "battery1-power-l1-w",
"measurement_key_power_l2_w": "battery1-power-l2-w",
"measurement_key_power_l3_w": "battery1-power-l3-w",
"measurement_key_power_3_phase_sym_w": "battery1-power-3-phase-sym-w",
"measurement_keys": [
"battery1-soc-factor",
"battery1-power-l1-w",
"battery1-power-l2-w",
"battery1-power-l3-w",
"battery1-power-3-phase-sym-w"
]
}
]
}
}
```
<!-- pyml enable line-length -->

275
tests/testdata/docs/_generated/configelecprice.md vendored Normal file
View File

@@ -0,0 +1,275 @@
## Electricity Price Prediction Configuration
<!-- pyml disable line-length -->
:::{table} elecprice
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| charges_kwh | `EOS_ELECPRICE__CHARGES_KWH` | `Optional[float]` | `rw` | `None` | Electricity price charges [€/kWh]. Will be added to variable market price. |
| elecpricefixed | `EOS_ELECPRICE__ELECPRICEFIXED` | `ElecPriceFixedCommonSettings` | `rw` | `required` | Fixed electricity price provider settings. |
| elecpriceimport | `EOS_ELECPRICE__ELECPRICEIMPORT` | `ElecPriceImportCommonSettings` | `rw` | `required` | Import provider settings. |
| energycharts | `EOS_ELECPRICE__ENERGYCHARTS` | `ElecPriceEnergyChartsCommonSettings` | `rw` | `required` | Energy Charts provider settings. |
| provider | `EOS_ELECPRICE__PROVIDER` | `Optional[str]` | `rw` | `None` | Electricity price provider id of provider to be used. |
| providers | | `list[str]` | `ro` | `N/A` | Available electricity price provider ids. |
| vat_rate | `EOS_ELECPRICE__VAT_RATE` | `Optional[float]` | `rw` | `1.19` | VAT rate factor applied to electricity price when charges are used. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"elecprice": {
"provider": "ElecPriceAkkudoktor",
"charges_kwh": 0.21,
"vat_rate": 1.19,
"elecpricefixed": {
"time_windows": {
"windows": []
}
},
"elecpriceimport": {
"import_file_path": null,
"import_json": null
},
"energycharts": {
"bidding_zone": "DE-LU"
}
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"elecprice": {
"provider": "ElecPriceAkkudoktor",
"charges_kwh": 0.21,
"vat_rate": 1.19,
"elecpricefixed": {
"time_windows": {
"windows": []
}
},
"elecpriceimport": {
"import_file_path": null,
"import_json": null
},
"energycharts": {
"bidding_zone": "DE-LU"
},
"providers": [
"ElecPriceAkkudoktor",
"ElecPriceEnergyCharts",
"ElecPriceFixed",
"ElecPriceImport"
]
}
}
```
<!-- pyml enable line-length -->
### Common settings for Energy Charts electricity price provider
<!-- pyml disable line-length -->
:::{table} elecprice::energycharts
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| bidding_zone | `<enum 'EnergyChartsBiddingZones'>` | `rw` | `DE-LU` | Bidding Zone: 'AT', 'BE', 'CH', 'CZ', 'DE-LU', 'DE-AT-LU', 'DK1', 'DK2', 'FR', 'HU', 'IT-NORTH', 'NL', 'NO2', 'PL', 'SE4' or 'SI' |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"elecprice": {
"energycharts": {
"bidding_zone": "AT"
}
}
}
```
<!-- pyml enable line-length -->
### Common settings for elecprice data import from file or JSON String
<!-- pyml disable line-length -->
:::{table} elecprice::elecpriceimport
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| import_file_path | `Union[str, pathlib.Path, NoneType]` | `rw` | `None` | Path to the file to import elecprice data from. |
| import_json | `Optional[str]` | `rw` | `None` | JSON string, dictionary of electricity price forecast value lists. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"elecprice": {
"elecpriceimport": {
"import_file_path": null,
"import_json": "{\"elecprice_marketprice_wh\": [0.0003384, 0.0003318, 0.0003284]}"
}
}
}
```
<!-- pyml enable line-length -->
### Value applicable during a specific time window
This model extends `TimeWindow` by associating a value with the defined time interval.
<!-- pyml disable line-length -->
:::{table} elecprice::elecpricefixed::time_windows::windows::list
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| date | `Optional[pydantic_extra_types.pendulum_dt.Date]` | `rw` | `None` | Optional specific calendar date for the time window. Naive — matched against the local date of the datetime passed to contains(). Overrides `day_of_week` if set. |
| day_of_week | `Union[int, str, NoneType]` | `rw` | `None` | Optional day of the week restriction. Can be specified as integer (0=Monday to 6=Sunday) or localized weekday name. If None, applies every day unless `date` is set. |
| duration | `Duration` | `rw` | `required` | Duration of the time window starting from `start_time`. |
| locale | `Optional[str]` | `rw` | `None` | Locale used to parse weekday names in `day_of_week` when given as string. If not set, Pendulum's default locale is used. Examples: 'en', 'de', 'fr', etc. |
| start_time | `Time` | `rw` | `required` | Naive start time of the time window (time of day, no timezone). Interpreted in the timezone of the datetime passed to contains() or earliest_start_time(). |
| value | `Optional[float]` | `rw` | `None` | Value applicable during this time window. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"elecprice": {
"elecpricefixed": {
"time_windows": {
"windows": [
{
"start_time": "00:00:00.000000",
"duration": "2 hours",
"day_of_week": null,
"date": null,
"locale": null,
"value": 0.288
}
]
}
}
}
}
```
<!-- pyml enable line-length -->
### Sequence of value time windows
This model specializes `TimeWindowSequence` to ensure that all
contained windows are instances of `ValueTimeWindow`.
It provides the full set of sequence operations (containment checks,
availability, start time calculations) for value windows.
<!-- pyml disable line-length -->
:::{table} elecprice::elecpricefixed::time_windows
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| windows | `list[akkudoktoreos.config.configabc.ValueTimeWindow]` | `rw` | `required` | Ordered list of value time windows. Each window defines a time interval and an associated value. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"elecprice": {
"elecpricefixed": {
"time_windows": {
"windows": []
}
}
}
}
```
<!-- pyml enable line-length -->
### Common configuration settings for fixed electricity pricing
This model defines a fixed electricity price schedule using a sequence
of time windows. Each window specifies a time interval and the electricity
price applicable during that interval.
<!-- pyml disable line-length -->
:::{table} elecprice::elecpricefixed
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| time_windows | `ValueTimeWindowSequence` | `rw` | `required` | Sequence of time windows defining the fixed price schedule. If not provided, no fixed pricing is applied. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"elecprice": {
"elecpricefixed": {
"time_windows": {
"windows": [
{
"start_time": "00:00:00.000000",
"duration": "8 hours",
"day_of_week": null,
"date": null,
"locale": null,
"value": 0.288
},
{
"start_time": "08:00:00.000000",
"duration": "16 hours",
"day_of_week": null,
"date": null,
"locale": null,
"value": 0.34
}
]
}
}
}
}
```
<!-- pyml enable line-length -->

30
tests/testdata/docs/_generated/configems.md vendored Normal file
View File

@@ -0,0 +1,30 @@
## Energy Management Configuration
<!-- pyml disable line-length -->
:::{table} ems
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| interval | `EOS_EMS__INTERVAL` | `float` | `rw` | `300.0` | Intervall between EOS energy management runs [seconds]. |
| mode | `EOS_EMS__MODE` | `<enum 'EnergyManagementMode'>` | `rw` | `required` | Energy management mode [DISABLED | OPTIMIZATION | PREDICTION]. |
| startup_delay | `EOS_EMS__STARTUP_DELAY` | `float` | `rw` | `5` | Startup delay in seconds for EOS energy management runs. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"ems": {
"startup_delay": 5.0,
"interval": 300.0,
"mode": "OPTIMIZATION"
}
}
```
<!-- pyml enable line-length -->

260
tests/testdata/docs/_generated/configexample.md vendored Normal file
View File

@@ -0,0 +1,260 @@
## Full example Config
<!-- pyml disable line-length -->
```json
{
"adapter": {
"provider": [
"HomeAssistant"
],
"homeassistant": {
"config_entity_ids": null,
"load_emr_entity_ids": null,
"grid_export_emr_entity_ids": null,
"grid_import_emr_entity_ids": null,
"pv_production_emr_entity_ids": null,
"device_measurement_entity_ids": null,
"device_instruction_entity_ids": null,
"solution_entity_ids": null
},
"nodered": {
"host": "127.0.0.1",
"port": 1880
}
},
"cache": {
"subpath": "cache",
"cleanup_interval": 300.0
},
"database": {
"provider": "LMDB",
"compression_level": 0,
"initial_load_window_h": 48,
"keep_duration_h": 48,
"autosave_interval_sec": 5,
"compaction_interval_sec": 604800,
"batch_size": 100
},
"devices": {
"batteries": [
{
"device_id": "battery1",
"capacity_wh": 8000,
"charging_efficiency": 0.88,
"discharging_efficiency": 0.88,
"levelized_cost_of_storage_kwh": 0.0,
"max_charge_power_w": 5000,
"min_charge_power_w": 50,
"charge_rates": [
0.0,
0.1,
0.2,
0.3,
0.4,
0.5,
0.6,
0.7,
0.8,
0.9,
1.0
],
"min_soc_percentage": 0,
"max_soc_percentage": 100
}
],
"max_batteries": 1,
"electric_vehicles": [
{
"device_id": "battery1",
"capacity_wh": 8000,
"charging_efficiency": 0.88,
"discharging_efficiency": 0.88,
"levelized_cost_of_storage_kwh": 0.0,
"max_charge_power_w": 5000,
"min_charge_power_w": 50,
"charge_rates": [
0.0,
0.1,
0.2,
0.3,
0.4,
0.5,
0.6,
0.7,
0.8,
0.9,
1.0
],
"min_soc_percentage": 0,
"max_soc_percentage": 100
}
],
"max_electric_vehicles": 1,
"inverters": [],
"max_inverters": 1,
"home_appliances": [],
"max_home_appliances": 1
},
"elecprice": {
"provider": "ElecPriceAkkudoktor",
"charges_kwh": 0.21,
"vat_rate": 1.19,
"elecpricefixed": {
"time_windows": {
"windows": []
}
},
"elecpriceimport": {
"import_file_path": null,
"import_json": null
},
"energycharts": {
"bidding_zone": "DE-LU"
}
},
"ems": {
"startup_delay": 5.0,
"interval": 300.0,
"mode": "OPTIMIZATION"
},
"feedintariff": {
"provider": "FeedInTariffFixed",
"provider_settings": {
"FeedInTariffFixed": null,
"FeedInTariffImport": null
}
},
"general": {
"config_save_mode": "AUTOMATIC",
"config_save_interval_sec": 60,
"version": "0.0.0",
"data_folder_path": "/home/user/.local/share/net.akkudoktoreos.net",
"data_output_subpath": "output",
"latitude": 52.52,
"longitude": 13.405
},
"load": {
"provider": "LoadAkkudoktor",
"loadakkudoktor": {
"loadakkudoktor_year_energy_kwh": null
},
"loadvrm": {
"load_vrm_token": "your-token",
"load_vrm_idsite": 12345
},
"loadimport": {
"import_file_path": null,
"import_json": null
}
},
"logging": {
"console_level": "TRACE",
"file_level": "TRACE"
},
"measurement": {
"historic_hours": 17520,
"load_emr_keys": [
"load0_emr"
],
"grid_export_emr_keys": [
"grid_export_emr"
],
"grid_import_emr_keys": [
"grid_import_emr"
],
"pv_production_emr_keys": [
"pv1_emr"
]
},
"optimization": {
"horizon_hours": 24,
"interval": 3600,
"algorithm": "GENETIC",
"genetic": {
"individuals": 400,
"generations": 400,
"seed": null,
"penalties": {
"ev_soc_miss": 10
}
}
},
"prediction": {
"hours": 48,
"historic_hours": 48
},
"pvforecast": {
"provider": "PVForecastAkkudoktor",
"provider_settings": {
"PVForecastImport": null,
"PVForecastVrm": null
},
"planes": [
{
"surface_tilt": 10.0,
"surface_azimuth": 180.0,
"userhorizon": [
10.0,
20.0,
30.0
],
"peakpower": 5.0,
"pvtechchoice": "crystSi",
"mountingplace": "free",
"loss": 14.0,
"trackingtype": 0,
"optimal_surface_tilt": false,
"optimalangles": false,
"albedo": null,
"module_model": null,
"inverter_model": null,
"inverter_paco": 6000,
"modules_per_string": 20,
"strings_per_inverter": 2
},
{
"surface_tilt": 20.0,
"surface_azimuth": 90.0,
"userhorizon": [
5.0,
15.0,
25.0
],
"peakpower": 3.5,
"pvtechchoice": "crystSi",
"mountingplace": "free",
"loss": 14.0,
"trackingtype": 1,
"optimal_surface_tilt": false,
"optimalangles": false,
"albedo": null,
"module_model": null,
"inverter_model": null,
"inverter_paco": 4000,
"modules_per_string": 20,
"strings_per_inverter": 2
}
],
"max_planes": 1
},
"server": {
"host": "127.0.0.1",
"port": 8503,
"verbose": false,
"startup_eosdash": true,
"eosdash_host": "127.0.0.1",
"eosdash_port": 8504,
"eosdash_supervise_interval_sec": 10,
"run_as_user": null,
"reload": true
},
"utils": {},
"weather": {
"provider": "WeatherImport",
"provider_settings": {
"WeatherImport": null
}
}
}
```
<!-- pyml enable line-length -->

149
tests/testdata/docs/_generated/configfeedintariff.md vendored Normal file
View File

@@ -0,0 +1,149 @@
## Feed In Tariff Prediction Configuration
<!-- pyml disable line-length -->
:::{table} feedintariff
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| provider | `EOS_FEEDINTARIFF__PROVIDER` | `Optional[str]` | `rw` | `None` | Feed in tariff provider id of provider to be used. |
| provider_settings | `EOS_FEEDINTARIFF__PROVIDER_SETTINGS` | `FeedInTariffCommonProviderSettings` | `rw` | `required` | Provider settings |
| providers | | `list[str]` | `ro` | `N/A` | Available feed in tariff provider ids. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"feedintariff": {
"provider": "FeedInTariffFixed",
"provider_settings": {
"FeedInTariffFixed": null,
"FeedInTariffImport": null
}
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"feedintariff": {
"provider": "FeedInTariffFixed",
"provider_settings": {
"FeedInTariffFixed": null,
"FeedInTariffImport": null
},
"providers": [
"FeedInTariffFixed",
"FeedInTariffImport"
]
}
}
```
<!-- pyml enable line-length -->
### Common settings for feed in tariff data import from file or JSON string
<!-- pyml disable line-length -->
:::{table} feedintariff::provider_settings::FeedInTariffImport
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| import_file_path | `Union[str, pathlib.Path, NoneType]` | `rw` | `None` | Path to the file to import feed in tariff data from. |
| import_json | `Optional[str]` | `rw` | `None` | JSON string, dictionary of feed in tariff forecast value lists. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"feedintariff": {
"provider_settings": {
"FeedInTariffImport": {
"import_file_path": null,
"import_json": "{\"fead_in_tariff_wh\": [0.000078, 0.000078, 0.000023]}"
}
}
}
}
```
<!-- pyml enable line-length -->
### Common settings for elecprice fixed price
<!-- pyml disable line-length -->
:::{table} feedintariff::provider_settings::FeedInTariffFixed
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| feed_in_tariff_kwh | `Optional[float]` | `rw` | `None` | Electricity price feed in tariff [€/kWH]. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"feedintariff": {
"provider_settings": {
"FeedInTariffFixed": {
"feed_in_tariff_kwh": 0.078
}
}
}
}
```
<!-- pyml enable line-length -->
### Feed In Tariff Prediction Provider Configuration
<!-- pyml disable line-length -->
:::{table} feedintariff::provider_settings
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| FeedInTariffFixed | `Optional[akkudoktoreos.prediction.feedintarifffixed.FeedInTariffFixedCommonSettings]` | `rw` | `None` | FeedInTariffFixed settings |
| FeedInTariffImport | `Optional[akkudoktoreos.prediction.feedintariffimport.FeedInTariffImportCommonSettings]` | `rw` | `None` | FeedInTariffImport settings |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"feedintariff": {
"provider_settings": {
"FeedInTariffFixed": null,
"FeedInTariffImport": null
}
}
}
```
<!-- pyml enable line-length -->

67
tests/testdata/docs/_generated/configgeneral.md vendored Normal file
View File

@@ -0,0 +1,67 @@
## General settings
<!-- pyml disable line-length -->
:::{table} general
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| config_file_path | | `Optional[pathlib.Path]` | `ro` | `N/A` | Path to EOS configuration file. |
| config_folder_path | | `Optional[pathlib.Path]` | `ro` | `N/A` | Path to EOS configuration directory. |
| config_save_interval_sec | `EOS_GENERAL__CONFIG_SAVE_INTERVAL_SEC` | `int` | `rw` | `60` | Automatic configuration file saving interval [seconds]. |
| config_save_mode | `EOS_GENERAL__CONFIG_SAVE_MODE` | `<enum 'ConfigSaveMode'>` | `rw` | `AUTOMATIC` | Configuration file save mode for configuration changes ['MANUAL', 'AUTOMATIC']. Defaults to 'AUTOMATIC'. |
| data_folder_path | `EOS_GENERAL__DATA_FOLDER_PATH` | `Path` | `rw` | `required` | Path to EOS data folder. |
| data_output_path | | `Optional[pathlib.Path]` | `ro` | `N/A` | Computed data_output_path based on data_folder_path. |
| data_output_subpath | `EOS_GENERAL__DATA_OUTPUT_SUBPATH` | `Optional[pathlib.Path]` | `rw` | `output` | Sub-path for the EOS output data folder. |
| home_assistant_addon | `EOS_GENERAL__HOME_ASSISTANT_ADDON` | `bool` | `rw` | `required` | EOS is running as home assistant add-on. |
| latitude | `EOS_GENERAL__LATITUDE` | `Optional[float]` | `rw` | `52.52` | Latitude in decimal degrees between -90 and 90. North is positive (ISO 19115) (°) |
| longitude | `EOS_GENERAL__LONGITUDE` | `Optional[float]` | `rw` | `13.405` | Longitude in decimal degrees within -180 to 180 (°) |
| timezone | | `Optional[str]` | `ro` | `N/A` | Computed timezone based on latitude and longitude. |
| version | `EOS_GENERAL__VERSION` | `Optional[str]` | `rw` | `None` | Configuration file version. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"general": {
"config_save_mode": "AUTOMATIC",
"config_save_interval_sec": 60,
"version": "0.0.0",
"data_folder_path": "/home/user/.local/share/net.akkudoktoreos.net",
"data_output_subpath": "output",
"latitude": 52.52,
"longitude": 13.405
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"general": {
"config_save_mode": "AUTOMATIC",
"config_save_interval_sec": 60,
"version": "0.0.0",
"data_folder_path": "/home/user/.local/share/net.akkudoktoreos.net",
"data_output_subpath": "output",
"latitude": 52.52,
"longitude": 13.405,
"timezone": "Europe/Berlin",
"data_output_path": "/home/user/.local/share/net.akkudoktoreos.net/output",
"config_folder_path": "/home/user/.config/net.akkudoktoreos.net",
"config_file_path": "/home/user/.config/net.akkudoktoreos.net/EOS.config.json"
}
}
```
<!-- pyml enable line-length -->

163
tests/testdata/docs/_generated/configload.md vendored Normal file
View File

@@ -0,0 +1,163 @@
## Load Prediction Configuration
<!-- pyml disable line-length -->
:::{table} load
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| loadakkudoktor | `EOS_LOAD__LOADAKKUDOKTOR` | `LoadAkkudoktorCommonSettings` | `rw` | `required` | LoadAkkudoktor provider settings. |
| loadimport | `EOS_LOAD__LOADIMPORT` | `LoadImportCommonSettings` | `rw` | `required` | LoadImport provider settings. |
| loadvrm | `EOS_LOAD__LOADVRM` | `LoadVrmCommonSettings` | `rw` | `required` | LoadVrm provider settings. |
| provider | `EOS_LOAD__PROVIDER` | `Optional[str]` | `rw` | `None` | Load provider id of provider to be used. |
| providers | | `list[str]` | `ro` | `N/A` | Available load provider ids. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"load": {
"provider": "LoadAkkudoktor",
"loadakkudoktor": {
"loadakkudoktor_year_energy_kwh": null
},
"loadvrm": {
"load_vrm_token": "your-token",
"load_vrm_idsite": 12345
},
"loadimport": {
"import_file_path": null,
"import_json": null
}
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"load": {
"provider": "LoadAkkudoktor",
"loadakkudoktor": {
"loadakkudoktor_year_energy_kwh": null
},
"loadvrm": {
"load_vrm_token": "your-token",
"load_vrm_idsite": 12345
},
"loadimport": {
"import_file_path": null,
"import_json": null
},
"providers": [
"LoadAkkudoktor",
"LoadAkkudoktorAdjusted",
"LoadVrm",
"LoadImport"
]
}
}
```
<!-- pyml enable line-length -->
### Common settings for load forecast VRM API
<!-- pyml disable line-length -->
:::{table} load::loadvrm
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| load_vrm_idsite | `int` | `rw` | `12345` | VRM-Installation-ID |
| load_vrm_token | `str` | `rw` | `your-token` | Token for Connecting VRM API |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"load": {
"loadvrm": {
"load_vrm_token": "your-token",
"load_vrm_idsite": 12345
}
}
}
```
<!-- pyml enable line-length -->
### Common settings for load data import from file or JSON string
<!-- pyml disable line-length -->
:::{table} load::loadimport
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| import_file_path | `Union[str, pathlib.Path, NoneType]` | `rw` | `None` | Path to the file to import load data from. |
| import_json | `Optional[str]` | `rw` | `None` | JSON string, dictionary of load forecast value lists. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"load": {
"loadimport": {
"import_file_path": null,
"import_json": "{\"loadforecast_power_w\": [676.71, 876.19, 527.13]}"
}
}
}
```
<!-- pyml enable line-length -->
### Common settings for load data import from file
<!-- pyml disable line-length -->
:::{table} load::loadakkudoktor
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| loadakkudoktor_year_energy_kwh | `Optional[float]` | `rw` | `None` | Yearly energy consumption (kWh). |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"load": {
"loadakkudoktor": {
"loadakkudoktor_year_energy_kwh": 40421.0
}
}
}
```
<!-- pyml enable line-length -->

45
tests/testdata/docs/_generated/configlogging.md vendored Normal file
View File

@@ -0,0 +1,45 @@
## Logging Configuration
<!-- pyml disable line-length -->
:::{table} logging
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| console_level | `EOS_LOGGING__CONSOLE_LEVEL` | `Optional[str]` | `rw` | `None` | Logging level when logging to console. |
| file_level | `EOS_LOGGING__FILE_LEVEL` | `Optional[str]` | `rw` | `None` | Logging level when logging to file. |
| file_path | | `Optional[pathlib.Path]` | `ro` | `N/A` | Computed log file path based on data output path. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"logging": {
"console_level": "TRACE",
"file_level": "TRACE"
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"logging": {
"console_level": "TRACE",
"file_level": "TRACE",
"file_path": "/home/user/.local/share/net.akkudoktor.eos/output/eos.log"
}
}
```
<!-- pyml enable line-length -->

75
tests/testdata/docs/_generated/configmeasurement.md vendored Normal file
View File

@@ -0,0 +1,75 @@
## Measurement Configuration
<!-- pyml disable line-length -->
:::{table} measurement
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| grid_export_emr_keys | `EOS_MEASUREMENT__GRID_EXPORT_EMR_KEYS` | `Optional[list[str]]` | `rw` | `None` | The keys of the measurements that are energy meter readings of energy export to grid [kWh]. |
| grid_import_emr_keys | `EOS_MEASUREMENT__GRID_IMPORT_EMR_KEYS` | `Optional[list[str]]` | `rw` | `None` | The keys of the measurements that are energy meter readings of energy import from grid [kWh]. |
| historic_hours | `EOS_MEASUREMENT__HISTORIC_HOURS` | `Optional[int]` | `rw` | `17520` | Number of hours into the past for measurement data |
| keys | | `list[str]` | `ro` | `N/A` | The keys of the measurements that can be stored. |
| load_emr_keys | `EOS_MEASUREMENT__LOAD_EMR_KEYS` | `Optional[list[str]]` | `rw` | `None` | The keys of the measurements that are energy meter readings of a load [kWh]. |
| pv_production_emr_keys | `EOS_MEASUREMENT__PV_PRODUCTION_EMR_KEYS` | `Optional[list[str]]` | `rw` | `None` | The keys of the measurements that are PV production energy meter readings [kWh]. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"measurement": {
"historic_hours": 17520,
"load_emr_keys": [
"load0_emr"
],
"grid_export_emr_keys": [
"grid_export_emr"
],
"grid_import_emr_keys": [
"grid_import_emr"
],
"pv_production_emr_keys": [
"pv1_emr"
]
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"measurement": {
"historic_hours": 17520,
"load_emr_keys": [
"load0_emr"
],
"grid_export_emr_keys": [
"grid_export_emr"
],
"grid_import_emr_keys": [
"grid_import_emr"
],
"pv_production_emr_keys": [
"pv1_emr"
],
"keys": [
"grid_export_emr",
"grid_import_emr",
"load0_emr",
"pv1_emr"
]
}
}
```
<!-- pyml enable line-length -->

104
tests/testdata/docs/_generated/configoptimization.md vendored Normal file
View File

@@ -0,0 +1,104 @@
## General Optimization Configuration
<!-- pyml disable line-length -->
:::{table} optimization
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| algorithm | `EOS_OPTIMIZATION__ALGORITHM` | `str` | `rw` | `GENETIC` | The optimization algorithm. Defaults to GENETIC |
| genetic | `EOS_OPTIMIZATION__GENETIC` | `GeneticCommonSettings` | `rw` | `required` | Genetic optimization algorithm configuration. |
| horizon | | `int` | `ro` | `N/A` | Number of optimization steps. |
| horizon_hours | `EOS_OPTIMIZATION__HORIZON_HOURS` | `int` | `rw` | `24` | The general time window within which the energy optimization goal shall be achieved [h]. Defaults to 24 hours. |
| interval | `EOS_OPTIMIZATION__INTERVAL` | `int` | `rw` | `3600` | The optimization interval [sec]. Defaults to 3600 seconds (1 hour) |
| keys | | `list[str]` | `ro` | `N/A` | The keys of the solution. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"optimization": {
"horizon_hours": 24,
"interval": 3600,
"algorithm": "GENETIC",
"genetic": {
"individuals": 400,
"generations": 400,
"seed": null,
"penalties": {
"ev_soc_miss": 10
}
}
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"optimization": {
"horizon_hours": 24,
"interval": 3600,
"algorithm": "GENETIC",
"genetic": {
"individuals": 400,
"generations": 400,
"seed": null,
"penalties": {
"ev_soc_miss": 10
}
},
"keys": [],
"horizon": 24
}
}
```
<!-- pyml enable line-length -->
### General Genetic Optimization Algorithm Configuration
<!-- pyml disable line-length -->
:::{table} optimization::genetic
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| generations | `Optional[int]` | `rw` | `400` | Number of generations to evolve [>= 10]. Defaults to 400. |
| individuals | `Optional[int]` | `rw` | `300` | Number of individuals (solutions) in the population [>= 10]. Defaults to 300. |
| penalties | `dict[str, Union[float, int, str]]` | `rw` | `required` | Penalty parameters used in fitness evaluation. |
| seed | `Optional[int]` | `rw` | `None` | Random seed for reproducibility. None = random. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"optimization": {
"genetic": {
"individuals": 300,
"generations": 400,
"seed": null,
"penalties": {
"ev_soc_miss": 10
}
}
}
}
```
<!-- pyml enable line-length -->

28
tests/testdata/docs/_generated/configprediction.md vendored Normal file
View File

@@ -0,0 +1,28 @@
## General Prediction Configuration
<!-- pyml disable line-length -->
:::{table} prediction
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| historic_hours | `EOS_PREDICTION__HISTORIC_HOURS` | `Optional[int]` | `rw` | `48` | Number of hours into the past for historical predictions data |
| hours | `EOS_PREDICTION__HOURS` | `Optional[int]` | `rw` | `48` | Number of hours into the future for predictions |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"prediction": {
"hours": 48,
"historic_hours": 48
}
}
```
<!-- pyml enable line-length -->

346
tests/testdata/docs/_generated/configpvforecast.md vendored Normal file
View File

@@ -0,0 +1,346 @@
## PV Forecast Configuration
<!-- pyml disable line-length -->
:::{table} pvforecast
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| max_planes | `EOS_PVFORECAST__MAX_PLANES` | `Optional[int]` | `rw` | `0` | Maximum number of planes that can be set |
| planes | `EOS_PVFORECAST__PLANES` | `Optional[list[akkudoktoreos.prediction.pvforecast.PVForecastPlaneSetting]]` | `rw` | `None` | Plane configuration. |
| planes_azimuth | | `List[float]` | `ro` | `N/A` | Compute a list of the azimuths per active planes. |
| planes_inverter_paco | | `Any` | `ro` | `N/A` | Compute a list of the maximum power rating of the inverter per active planes. |
| planes_peakpower | | `List[float]` | `ro` | `N/A` | Compute a list of the peak power per active planes. |
| planes_tilt | | `List[float]` | `ro` | `N/A` | Compute a list of the tilts per active planes. |
| planes_userhorizon | | `Any` | `ro` | `N/A` | Compute a list of the user horizon per active planes. |
| provider | `EOS_PVFORECAST__PROVIDER` | `Optional[str]` | `rw` | `None` | PVForecast provider id of provider to be used. |
| provider_settings | `EOS_PVFORECAST__PROVIDER_SETTINGS` | `PVForecastCommonProviderSettings` | `rw` | `required` | Provider settings |
| providers | | `list[str]` | `ro` | `N/A` | Available PVForecast provider ids. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"pvforecast": {
"provider": "PVForecastAkkudoktor",
"provider_settings": {
"PVForecastImport": null,
"PVForecastVrm": null
},
"planes": [
{
"surface_tilt": 10.0,
"surface_azimuth": 180.0,
"userhorizon": [
10.0,
20.0,
30.0
],
"peakpower": 5.0,
"pvtechchoice": "crystSi",
"mountingplace": "free",
"loss": 14.0,
"trackingtype": 0,
"optimal_surface_tilt": false,
"optimalangles": false,
"albedo": null,
"module_model": null,
"inverter_model": null,
"inverter_paco": 6000,
"modules_per_string": 20,
"strings_per_inverter": 2
},
{
"surface_tilt": 20.0,
"surface_azimuth": 90.0,
"userhorizon": [
5.0,
15.0,
25.0
],
"peakpower": 3.5,
"pvtechchoice": "crystSi",
"mountingplace": "free",
"loss": 14.0,
"trackingtype": 1,
"optimal_surface_tilt": false,
"optimalangles": false,
"albedo": null,
"module_model": null,
"inverter_model": null,
"inverter_paco": 4000,
"modules_per_string": 20,
"strings_per_inverter": 2
}
],
"max_planes": 1
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"pvforecast": {
"provider": "PVForecastAkkudoktor",
"provider_settings": {
"PVForecastImport": null,
"PVForecastVrm": null
},
"planes": [
{
"surface_tilt": 10.0,
"surface_azimuth": 180.0,
"userhorizon": [
10.0,
20.0,
30.0
],
"peakpower": 5.0,
"pvtechchoice": "crystSi",
"mountingplace": "free",
"loss": 14.0,
"trackingtype": 0,
"optimal_surface_tilt": false,
"optimalangles": false,
"albedo": null,
"module_model": null,
"inverter_model": null,
"inverter_paco": 6000,
"modules_per_string": 20,
"strings_per_inverter": 2
},
{
"surface_tilt": 20.0,
"surface_azimuth": 90.0,
"userhorizon": [
5.0,
15.0,
25.0
],
"peakpower": 3.5,
"pvtechchoice": "crystSi",
"mountingplace": "free",
"loss": 14.0,
"trackingtype": 1,
"optimal_surface_tilt": false,
"optimalangles": false,
"albedo": null,
"module_model": null,
"inverter_model": null,
"inverter_paco": 4000,
"modules_per_string": 20,
"strings_per_inverter": 2
}
],
"max_planes": 1,
"providers": [
"PVForecastAkkudoktor",
"PVForecastVrm",
"PVForecastImport"
],
"planes_peakpower": [
5.0,
3.5
],
"planes_azimuth": [
180.0,
90.0
],
"planes_tilt": [
10.0,
20.0
],
"planes_userhorizon": [
[
10.0,
20.0,
30.0
],
[
5.0,
15.0,
25.0
]
],
"planes_inverter_paco": [
6000.0,
4000.0
]
}
}
```
<!-- pyml enable line-length -->
### Common settings for PV forecast VRM API
<!-- pyml disable line-length -->
:::{table} pvforecast::provider_settings::PVForecastVrm
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| pvforecast_vrm_idsite | `int` | `rw` | `12345` | VRM-Installation-ID |
| pvforecast_vrm_token | `str` | `rw` | `your-token` | Token for Connecting VRM API |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"pvforecast": {
"provider_settings": {
"PVForecastVrm": {
"pvforecast_vrm_token": "your-token",
"pvforecast_vrm_idsite": 12345
}
}
}
}
```
<!-- pyml enable line-length -->
### Common settings for pvforecast data import from file or JSON string
<!-- pyml disable line-length -->
:::{table} pvforecast::provider_settings::PVForecastImport
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| import_file_path | `Union[str, pathlib.Path, NoneType]` | `rw` | `None` | Path to the file to import PV forecast data from. |
| import_json | `Optional[str]` | `rw` | `None` | JSON string, dictionary of PV forecast value lists. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"pvforecast": {
"provider_settings": {
"PVForecastImport": {
"import_file_path": null,
"import_json": "{\"pvforecast_ac_power\": [0, 8.05, 352.91]}"
}
}
}
}
```
<!-- pyml enable line-length -->
### PV Forecast Provider Configuration
<!-- pyml disable line-length -->
:::{table} pvforecast::provider_settings
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| PVForecastImport | `Optional[akkudoktoreos.prediction.pvforecastimport.PVForecastImportCommonSettings]` | `rw` | `None` | PVForecastImport settings |
| PVForecastVrm | `Optional[akkudoktoreos.prediction.pvforecastvrm.PVForecastVrmCommonSettings]` | `rw` | `None` | PVForecastVrm settings |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"pvforecast": {
"provider_settings": {
"PVForecastImport": null,
"PVForecastVrm": null
}
}
}
```
<!-- pyml enable line-length -->
### PV Forecast Plane Configuration
<!-- pyml disable line-length -->
:::{table} pvforecast::planes::list
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| albedo | `Optional[float]` | `rw` | `None` | Proportion of the light hitting the ground that it reflects back. |
| inverter_model | `Optional[str]` | `rw` | `None` | Model of the inverter of this plane. |
| inverter_paco | `Optional[int]` | `rw` | `None` | AC power rating of the inverter [W]. |
| loss | `Optional[float]` | `rw` | `14.0` | Sum of PV system losses in percent |
| module_model | `Optional[str]` | `rw` | `None` | Model of the PV modules of this plane. |
| modules_per_string | `Optional[int]` | `rw` | `None` | Number of the PV modules of the strings of this plane. |
| mountingplace | `Optional[str]` | `rw` | `free` | Type of mounting for PV system. Options are 'free' for free-standing and 'building' for building-integrated. |
| optimal_surface_tilt | `Optional[bool]` | `rw` | `False` | Calculate the optimum tilt angle. Ignored for two-axis tracking. |
| optimalangles | `Optional[bool]` | `rw` | `False` | Calculate the optimum tilt and azimuth angles. Ignored for two-axis tracking. |
| peakpower | `Optional[float]` | `rw` | `None` | Nominal power of PV system in kW. |
| pvtechchoice | `Optional[str]` | `rw` | `crystSi` | PV technology. One of 'crystSi', 'CIS', 'CdTe', 'Unknown'. |
| strings_per_inverter | `Optional[int]` | `rw` | `None` | Number of the strings of the inverter of this plane. |
| surface_azimuth | `Optional[float]` | `rw` | `180.0` | Orientation (azimuth angle) of the (fixed) plane. Clockwise from north (north=0, east=90, south=180, west=270). |
| surface_tilt | `Optional[float]` | `rw` | `30.0` | Tilt angle from horizontal plane. Ignored for two-axis tracking. |
| trackingtype | `Optional[int]` | `rw` | `None` | Type of suntracking. 0=fixed, 1=single horizontal axis aligned north-south, 2=two-axis tracking, 3=vertical axis tracking, 4=single horizontal axis aligned east-west, 5=single inclined axis aligned north-south. |
| userhorizon | `Optional[List[float]]` | `rw` | `None` | Elevation of horizon in degrees, at equally spaced azimuth clockwise from north. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"pvforecast": {
"planes": [
{
"surface_tilt": 10.0,
"surface_azimuth": 180.0,
"userhorizon": [
10.0,
20.0,
30.0
],
"peakpower": 5.0,
"pvtechchoice": "crystSi",
"mountingplace": "free",
"loss": 14.0,
"trackingtype": 0,
"optimal_surface_tilt": false,
"optimalangles": false,
"albedo": null,
"module_model": null,
"inverter_model": null,
"inverter_paco": 6000,
"modules_per_string": 20,
"strings_per_inverter": 2
}
]
}
}
```
<!-- pyml enable line-length -->

42
tests/testdata/docs/_generated/configserver.md vendored Normal file
View File

@@ -0,0 +1,42 @@
## Server Configuration
<!-- pyml disable line-length -->
:::{table} server
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| eosdash_host | `EOS_SERVER__EOSDASH_HOST` | `Optional[str]` | `rw` | `None` | EOSdash server IP address. Defaults to EOS server IP address. |
| eosdash_port | `EOS_SERVER__EOSDASH_PORT` | `Optional[int]` | `rw` | `None` | EOSdash server IP port number. Defaults to EOS server IP port number + 1. |
| eosdash_supervise_interval_sec | `EOS_SERVER__EOSDASH_SUPERVISE_INTERVAL_SEC` | `int` | `rw` | `10` | Supervision interval for EOS server to supervise EOSdash [seconds]. |
| host | `EOS_SERVER__HOST` | `Optional[str]` | `rw` | `127.0.0.1` | EOS server IP address. Defaults to 127.0.0.1. |
| port | `EOS_SERVER__PORT` | `Optional[int]` | `rw` | `8503` | EOS server IP port number. Defaults to 8503. |
| reload | `EOS_SERVER__RELOAD` | `Optional[bool]` | `rw` | `False` | Enable server auto-reload for debugging or development. Default is False. Monitors the package directory for changes and reloads the server. |
| run_as_user | `EOS_SERVER__RUN_AS_USER` | `Optional[str]` | `rw` | `None` | The name of the target user to switch to. If ``None`` (default), the current effective user is used and no privilege change is attempted. |
| startup_eosdash | `EOS_SERVER__STARTUP_EOSDASH` | `Optional[bool]` | `rw` | `True` | EOS server to start EOSdash server. Defaults to True. |
| verbose | `EOS_SERVER__VERBOSE` | `Optional[bool]` | `rw` | `False` | Enable debug output |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"server": {
"host": "127.0.0.1",
"port": 8503,
"verbose": false,
"startup_eosdash": true,
"eosdash_host": "127.0.0.1",
"eosdash_port": 8504,
"eosdash_supervise_interval_sec": 10,
"run_as_user": null,
"reload": true
}
}
```
<!-- pyml enable line-length -->

23
tests/testdata/docs/_generated/configutils.md vendored Normal file
View File

@@ -0,0 +1,23 @@
## Utils Configuration
<!-- pyml disable line-length -->
:::{table} utils
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"utils": {}
}
```
<!-- pyml enable line-length -->

116
tests/testdata/docs/_generated/configweather.md vendored Normal file
View File

@@ -0,0 +1,116 @@
## Weather Forecast Configuration
<!-- pyml disable line-length -->
:::{table} weather
:widths: 10 20 10 5 5 30
:align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
| ---- | -------------------- | ---- | --------- | ------- | ----------- |
| provider | `EOS_WEATHER__PROVIDER` | `Optional[str]` | `rw` | `None` | Weather provider id of provider to be used. |
| provider_settings | `EOS_WEATHER__PROVIDER_SETTINGS` | `WeatherCommonProviderSettings` | `rw` | `required` | Provider settings |
| providers | | `list[str]` | `ro` | `N/A` | Available weather provider ids. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"weather": {
"provider": "WeatherImport",
"provider_settings": {
"WeatherImport": null
}
}
}
```
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"weather": {
"provider": "WeatherImport",
"provider_settings": {
"WeatherImport": null
},
"providers": [
"BrightSky",
"ClearOutside",
"OpenMeteo",
"WeatherImport"
]
}
}
```
<!-- pyml enable line-length -->
### Common settings for weather data import from file or JSON string
<!-- pyml disable line-length -->
:::{table} weather::provider_settings::WeatherImport
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| import_file_path | `Union[str, pathlib.Path, NoneType]` | `rw` | `None` | Path to the file to import weather data from. |
| import_json | `Optional[str]` | `rw` | `None` | JSON string, dictionary of weather forecast value lists. |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"weather": {
"provider_settings": {
"WeatherImport": {
"import_file_path": null,
"import_json": "{\"weather_temp_air\": [18.3, 17.8, 16.9]}"
}
}
}
}
```
<!-- pyml enable line-length -->
### Weather Forecast Provider Configuration
<!-- pyml disable line-length -->
:::{table} weather::provider_settings
:widths: 10 10 5 5 30
:align: left
| Name | Type | Read-Only | Default | Description |
| ---- | ---- | --------- | ------- | ----------- |
| WeatherImport | `Optional[akkudoktoreos.prediction.weatherimport.WeatherImportCommonSettings]` | `rw` | `None` | WeatherImport settings |
:::
<!-- pyml enable line-length -->
<!-- pyml disable no-emphasis-as-heading -->
**Example Input/Output**
<!-- pyml enable no-emphasis-as-heading -->
<!-- pyml disable line-length -->
```json
{
"weather": {
"provider_settings": {
"WeatherImport": null
}
}
}
```
<!-- pyml enable line-length -->

BIN
tests/testdata/test_example_report.pdf vendored
View File

Binary file not shown.

2463
uv.lock generated
View File

File diff suppressed because it is too large Load Diff