Mirrors/EOS
1
0
Fork 0
mirror of https://github.com/Akkudoktor-EOS/EOS.git synced 2026-07-19 08:18:50 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

fix: move data management to async (#1015)

Browse Source 
FAstAPI is an async framework. Data may be imported and exported, load and save, set and get
asynchronously. Prevent interleaving data operations to corrupt the data. In the previous design
sync and async data access was intermixed leading to data corruption.

The basic data classes DataSequence and DataContainer and the derived classes like Provider and
Measurement now are async. Data access is protected by several async locks.

To support the async design of the data classes the database interface became async.

The energy management is also adapted to the new async design. Optimization is still off-loaded
to another thread, but the prepration for the optimization and the post optimization actions now
follow the async design.

Adapter operations are now also protected by async locks.

Tests were adapted to the async design and new tests were created.

Besides this major fix several other improvements and fixes are included in this PR.

* fix: key_to_dict/list/array only regard data records with key value set.

  Before the exclusion of no value data records was only done if the dropna flag was set.

* fix: test for visual result pdf generation

  Due to updates in the library the generated charts text was a little bit different.
  Adapt the test to create the comaprison pdf in the test data durectory and
  update the reference pdf.

* chore: Remove MutableMapping from DataSequence and DataContainer.

  Mutable Mapping does not fit to the now async design.

* chore: Add NoDB database backend

  This backend implements the full database backend interface but performs
  no actual persistence. It is intended for configurations where database
  persistence is disabled (`provider=None`).

* chore: Improve measurement data import testing with real world scenarios.

  Added two new endpoints to support testing.

* chore: Add mermaid to supported documentation tools

* chore: Add documentation about async design

* chore: Add documentation about generic data handling

  Covers the basics of measurement and prediction time series data handling.

* chore: Add empty lines around markdown lists.

* chore: sync pre-commit config to updated package versions

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
This commit is contained in:
Bobby Noelte
2026-07-15 16:38:53 +02:00
committed by GitHub
parent 38011780c5
commit eb9e966de9
99 changed files with 11971 additions and 5981 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docs/_generated/configdatabase.md
View File

@@ -64,7 +64,8 @@ None indicates forever. Database namespaces may have diverging definitions. |
"batch_size": 100,
"providers": [
"LMDB",
"SQLite"
"SQLite",
"NoDB"
]
}
}

59
docs/_generated/openapi.md
View File

@@ -1,6 +1,6 @@
# Akkudoktor-EOS
**Version**: `v0.3.0.dev2604141105859917`
**Version**: `v0.3.0.dev2607150960221650`
<!-- pyml disable line-length -->
**Description**: This project provides a comprehensive solution for simulating and optimizing an energy system based on renewable energy sources. With a focus on photovoltaic (PV) systems, battery storage (batteries), load management (consumer requirements), heat pumps, electric vehicles, and consideration of electricity price data, this system enables forecasting and optimization of energy flow and costs over a specified period.
@@ -338,6 +338,31 @@ Returns:
---
## POST /v1/admin/database/save
<!-- pyml disable line-length -->
**Links**: [local](http://localhost:8503/docs#/default/fastapi_admin_database_save_post_v1_admin_database_save_post), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_admin_database_save_post_v1_admin_database_save_post)
<!-- pyml enable line-length -->
Fastapi Admin Database Save Post
<!-- pyml disable line-length -->
```python
"""
Save in memory data to database.
Returns:
data (dict): The database stats after saving the records.
"""
```
<!-- pyml enable line-length -->
**Responses**:
- **200**: Successful Response
---
## GET /v1/admin/database/stats
<!-- pyml disable line-length -->
@@ -882,6 +907,38 @@ Fastapi Measurement Keys Get
---
## DELETE /v1/measurement/range
<!-- pyml disable line-length -->
**Links**: [local](http://localhost:8503/docs#/default/fastapi_measurement_range_delete_v1_measurement_range_delete), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_measurement_range_delete_v1_measurement_range_delete)
<!-- pyml enable line-length -->
Fastapi Measurement Range Delete
<!-- pyml disable line-length -->
```python
"""
Delete measurement values for a key within a datetime range.
"""
```
<!-- pyml enable line-length -->
**Parameters**:
- `key` (query, required): Measurement key.
- `start_datetime` (query, optional): Start datetime.
- `end_datetime` (query, optional): End datetime.
**Responses**:
- **200**: Successful Response
- **422**: Validation Error
---
## GET /v1/measurement/series
<!-- pyml disable line-length -->

3
docs/akkudoktoreos/adapter/adapternodered.md
View File

@@ -38,6 +38,7 @@ Typical use cases with Node-RED:
#### 1. Enable and configure the Node-RED adapter
EOS must be configured with access to the Node-RED instance in Config->adapter.
* prerequisite is an already installed and running Node-RED instance
* adapter.nodered.host: 192.168.1.109 (example IP of your Node-RED instance)
* adapter.nodered.port: 1880 (default)
@@ -46,10 +47,12 @@ EOS must be configured with access to the Node-RED instance in Config->adapter.
#### 2. Run energy optimisation
Before the run, EOS receives:
* EOS receives measurement values via HTTP-IN "eos_data_acquisition" before optimisation.
(The HTTP-IN Node "eos_data_acquisition" is NOT yet functional)
After the run, EOS provides:
* The device instruction and solution entities for the current time slot via HTTP-IN "Control Dispatch".
### Configuration steps in NodeRED

4
docs/akkudoktoreos/database.md
View File

@@ -284,6 +284,7 @@ DatabaseTimestamp.from_datetime(dt: DateTime) -> "20241027T123456[Z]"
```
**Properties:**
- Always stored in UTC (timezone-aware required)
- Lexicographically sortable
- Bijective conversion to/from `pendulum.DateTime`
@@ -330,6 +331,7 @@ The system uses a progressive loading model to minimize memory footprint:
### Boundary Extension
When loading a range `[start, end)`, the system automatically extends boundaries to include:
- **First record before** `start` (for interpolation/context)
- **First record at or after** `end` (for closing boundary)
@@ -348,6 +350,7 @@ SELECT * FROM records WHERE namespace='measurement'
```
**Default Namespace:**
- Can be set during `open(namespace="default")`
- Operations with `namespace=None` use the default
- Each record class typically defines its own namespace via `db_namespace()`
@@ -537,6 +540,7 @@ db_vacuum(keep_timestamp=cutoff) # Keep from cutoff onward
```
**Strategy:**
- Computes cutoff relative to `max_timestamp - keep_hours`
- Deletes all records before cutoff
- Immediately persists changes via `db_save_records()`

1
docs/akkudoktoreos/optimpost.md
View File

@@ -223,6 +223,7 @@ penalty = ac_wh_charged × (break_even_price − best_uncovered_price) × factor
```
where:
- `break_even_price = charge_price / η_round_trip`
- `best_uncovered_price` = highest future price not already covered by free PV battery energy
- `factor` = `optimization.genetic.penalties.ac_charge_break_even` (default `1.0`)

4
docs/conf.py
View File

@@ -33,6 +33,7 @@ extensions = [
"sphinx_rtd_theme",
"myst_parser",
"sphinx_tabs.tabs",
"sphinxcontrib.mermaid",
]
templates_path = ["_templates"]
@@ -141,3 +142,6 @@ napoleon_use_rtype = True
napoleon_preprocess_types = False
napoleon_type_aliases = None
napoleon_attr_annotations = True
# -- Options for mermaid -------------------------------------------------
mermaid_output_format = "html" # no mmdc to be installed

184
docs/develop/async.md Normal file
View File

@@ -0,0 +1,184 @@
# Asynchronous Design of Akkudoktor‑EOS
The Akkudoktor‑EOS server is built on **FastAPI** and is transitioning to a fully asynchronous
design to improve scalability, responsiveness, and resource utilisation, especially for
long‑running or I/O‑bound operations.
The server manages a variety of background tasks through a central **Retention Manager**, which
schedules and supervises all periodic and asynchronous work.
## Core Asynchronous Components
- **FastAPI REST Interface**
All HTTP endpoints are defined as `async def` handlers (or synchronous where no I/O waiting is
required), allowing the server to handle many concurrent connections without blocking the event
loop.
- **Retention Manager**
A dedicated component responsible for orchestrating all background tasks. It runs an asynchronous
tick loop that executes registered functions at configurable intervals. The manager also provides
graceful shutdown handling, waiting for in‑flight jobs to finish.
<!-- pyml disable line-length -->
- **Managed Asynchronous Tasks**
The following tasks are registered with the Retention Manager and run periodically:
| Task Name | Function | Interval Configuration | Description |
|-----------|----------|------------------------|-------------|
| `supervise_eosdash` | `supervise_eosdash` | `server/eosdash_supervise_interval_sec` | Monitors and restarts the EOSdash UI process if needed. |
| `autosave_config` | `autosave_config` | `general/config_save_interval_sec` | Saves the current configuration to disk automatically. |
| `cache_clear` | `cache_clear` | `cache/cleanup_interval` | Removes expired entries from the cache. |
| `save_eos_database` | `save_eos_database` | `database/autosave_interval_sec` | Persists in‑memory measurement and prediction data to the database. |
| `compact_eos_database` | `compact_eos_database` | `database/compaction_interval_sec` | Compacts and vacuums the database to reclaim space and improve performance. |
| `manage_energy` | `ems_manage_energy` | `ems/interval` | Core energy management loop: triggers predictions, optimisation, and device control. |
The `manage_energy` task is the central orchestrator that itself calls asynchronous prediction
updates, adapter scheduling and optimisation runs. It uses the `EnergyManagementSystem`
(`get_ems()`) which internally manages concurrency to ensure only one energy management run
happens at a time.
<!-- pyml enable line-length -->
- **On‑Demand Asynchronous Endpoints**
Several REST endpoints are asynchronous and delegate heavy work to the `EnergyManagementSystem`.
Examples include:
- `POST /v1/prediction/update` – updates all prediction providers asynchronously.
- `POST /optimize` – runs a genetic optimisation (deprecated, but still async).
- `POST /v1/admin/server/restart` – spawns a new process and schedules a shutdown task.
## Asynchronous Workflow
The diagram below shows how periodic tasks are registered and executed by the Retention Manager,
and how a client request can trigger an asynchronous update.
```mermaid
sequenceDiagram
participant Client
participant FastAPI as FastAPI (Async)
participant Retention as Retention Manager (Async)
participant EMS as EnergyManagementSystem (Async)
participant Adapter as Adapter (Async)
participant Prediction as Prediction (Async)
participant Measurement as Measurement (Async)
participant DatabaseRecords as DatabaseRecords (Async)
participant Database as Database (Sync)
Note over Retention: On startup (lifespan)
FastAPI->>Retention: register tasks with intervals
Retention-->>FastAPI: tasks registered
Retention->>Retention: start tick loop
loop every EMS interval
Retention->>EMS: run(mode=PREDICTION+OPTIMIZATION)
EMS->>Adapter: update_data, DATA_AQUISITION
Adapter->>Measurement: update_value
Measurement-->>Adapter: done
Adapter-->>EMS: done
EMS->>Prediction: update_data, FORECAST_RETRIEVAL
Prediction-->>EMS: done
EMS->>Adapter: update_data, CONTROL_DISPATCH
Adapter-->>EMS: done
EMS-->>Retention: done
end
loop every DatabaseRecords autosave interval
Retention->>Prediction: save
Prediction->>DatabaseRecords: db_save_records
DatabaseRecords-->>Prediction: done
Prediction-->>Retention: done
Retention->>Measurement: save
Measurement->>DatabaseRecords: db_save_records
DatabaseRecords-->>Measurement: done
Measurement-->>Retention: done
end
loop every DatabaseRecords compaction interval
Retention->>Prediction: db_compact
Prediction->>DatabaseRecords: db_delete_records
DatabaseRecords-->>Prediction: done
Prediction->>DatabaseRecords: db_save_records
DatabaseRecords-->>Prediction: done
Prediction-->>Retention: done
Retention->>Measurement: db_compact
Measurement->>DatabaseRecords: db_delete_records
DatabaseRecords-->>Measurement: done
Measurement->>DatabaseRecords: db_save_records
DatabaseRecords-->>Measurement: done
Measurement-->>Retention: done
end
Client->>FastAPI: POST /v1/prediction/update
FastAPI->>EMS: await run(mode=PREDICTION)
EMS-->>FastAPI: predictions updated
FastAPI-->>Client: 200 OK
```
For server shutdown or restart, the Retention Manager’s task is cancelled, and the server waits
for in‑flight jobs to finish (shutdown timeout = 10 seconds). The state is saved via
`save_eos_state()`.
## Asynchronous vs. Synchronous
- **REST endpoints** that perform I/O or heavy computation are `async def`.
- **Retention Manager** uses `asyncio.create_task()` to run the tick loop and individual task
executions.
- **Database** that synchronizes database access is `async def`, but the database backends
**DataBaseBackendABC** are synchronous.
- **DatabaseRecordProtocolMixin** provides asynchronous access to in memory data records and
database storage.
- **Prediction** and **Measurement** provide asynchronous access to the undelying database using
the **DatabaseRecordProtocolMixin**.
- **Energy management runs** are serialised using an internal lock (via `EMS.run()`) to avoid
overlapping optimisation cycles.
- **Process management** for shutdown/restart uses `asyncio.create_task(server_shutdown_task())`
to gracefully terminate after a delay.
## Benefits of Full Asynchrony
- **Higher throughput** – FastAPI’s event loop can handle thousands of idle keep‑alive connections.
- **Lower latency** – Long‑running tasks (database compaction, prediction updates) do not block HTTP
responses.
- **Easier maintenance** – Uniform async patterns replace mixed sync/async code.
- **Better resource usage** – The Retention Manager can throttle, skip, or prioritise tasks based on
configuration and system load.
- **Graceful shutdown** – All background tasks are cancelled cooperatively, and state is saved
before exit.
## Additional Asynchronous Patterns
Beyond the Retention Manager, the server uses:
- **Asynchronous shutdown/restart** – When a restart is requested, a new process is spawned, and the
current process schedules a delayed termination (`server_shutdown_task`). This ensures zero
downtime if the new process starts before the old one exits.
- **Concurrent request handling** – Multiple clients can call prediction update endpoints
simultaneously; the `EMS.run()` method serialises them internally, preventing race conditions.
- **Non‑blocking logging** – Log entries are written asynchronously (via loguru’s async sinks when
configured).
The combination of FastAPI, the Retention Manager, and asynchronous I/O enables Akkudoktor‑EOS to
run efficiently on resource‑constrained devices (e.g., Raspberry Pi) while maintaining responsive
REST APIs and reliable background data maintenance.
## Asynchronous Detailed Design
### Database
The database backends are synchronous. The selected backend is wrapped in the asynchronous
thread-safe database singleton defined by the Database class.
### DatabaseRecordProtocol, DataRecordProtocol
The DatabaseRecordProtocol completely manages in memory records and database storage. It acesses
the database backend by the database singleton and is therefor asynchronous. The
DatabaseRecordProtocol has a minimum expectation for data records defined by the DataRecordProtocol.
Data records are expected to be synchronous.
### Data Records
Data records are synchronous.
### Data Sequence, Data Provide, Data Container
Data sequences, the derived data provider, and the data provider aggregation data containers are
asynchronous. Data sequences' data can be backed by the database using the
DatabaseRecordProtocol to access the data. That is the reason why all the classes are
asynchronous.

568
docs/develop/dataabc.md Normal file
View File

@@ -0,0 +1,568 @@
# `dataabc` — Generic Data Handling
## Overview
The `dataabc` module provides the foundational abstractions for managing time-series data
in EOS. It defines a layered class hierarchy that covers individual data records, ordered
sequences of records, singleton data providers, and multi-provider containers.
All classes in this module are designed for use in predictive modelling workflows and share
three cross-cutting concerns:
- **Configuration access** via `ConfigMixin`, exposing the global EOS configuration as
`self.config`.
- **Database persistence** via `DatabaseRecordProtocolMixin`, providing optional storage
in a time-series database alongside in-memory records.
- **Async safety** via a three-level locking scheme described in detail in the
[Concurrency and Locking](#concurrency-and-locking) section.
## Class Hierarchy
```text
PydanticBaseModel
└── DataABC (ConfigMixin, StartMixin)
├── DataRecord (MutableMapping)
└── DataSequence (DatabaseRecordProtocolMixin)
└── DataProvider (SingletonMixin)
└── DataImportProvider (DataImportMixin)
DataContainer (SingletonMixin, MutableMapping)
DataImportMixin (StartMixin)
```
## Classes
### `DataABC`
Base class for all data-handling objects. Inherits from `ConfigMixin` and `StartMixin`,
making the global EOS configuration available as `self.config` on every derived instance.
Not intended to be instantiated directly.
### `DataRecord`
A single measurement or forecast point at a specific datetime, implemented as a
`MutableMapping` so that field values can be accessed and mutated both as dictionary
entries (`record["field"]`) and as attributes (`record.field`).
#### Key concepts
**Static fields** are declared as Pydantic model fields in the class body. They are always
present and validated on assignment.
**Configured fields** are dynamic: their names are returned by the classmethod
`configured_data_keys()`, which derived classes override to pull key names from the EOS
configuration. These keys are stored in the internal `configured_data` dict but appear
transparent to callers — they show up in `dir()`, `iter()`, and attribute access just like
static fields.
```python
class MeasurementDataRecord(DataRecord):
@classmethod
def configured_data_keys(cls) -> Optional[list[str]]:
return cls.config.measurement.keys
```
#### Important methods
| Method | Description |
|---|---|
| `record_keys()` | All field names, including configured keys. |
| `record_keys_writable()` | Subset of `record_keys()` that can be written. |
| `key_from_description(desc)` | Fuzzy-matches a description string to a field name. |
| `keys_from_descriptions(descs)` | Batch version of `key_from_description`. |
---
### `DataSequence`
An ordered, mutable collection of `DataRecord` instances with time-series behaviour.
Records are always kept sorted in ascending `date_time` order. `DataSequence` is also the
level at which **async safety is enforced** — see
[Concurrency and Locking](#concurrency-and-locking).
Derived classes must redeclare the `records` field with the concrete record type:
```python
class Measurement(DataSequence):
records: list[MeasurementDataRecord] = Field(default_factory=list)
```
#### Public async interface of `DataSequence`
All methods that mutate sequence state are `async`. Callers in an async context (e.g.
FastAPI endpoint handlers) must `await` them.
| Method | Description |
|---|---|
| `await insert_by_datetime(record)` | Insert or merge a record by its datetime. |
| `await update_value(date, key, value)` | Insert or update a single field value at a datetime. |
| `await key_from_lists(key, dates, values)` | Populate a field from parallel date/value lists. |
| `await key_from_series(key, series)` | Populate a field from a `pd.Series`. |
| `await save()` | Persist all records to the configured storage backend. |
| `await load()` | Load records from the configured storage backend. |
| `await import_from_dict(data)` | Import records from a key-value dictionary. |
| `await import_from_dataframe(df)` | Import records from a `pd.DataFrame`. |
| `await import_from_json(json_str)` | Import records from a JSON string. |
| `await import_from_file(path)` | Import records from a JSON file. |
#### Internal sync interface of `DataSequence`
Each public async method has a private sync counterpart prefixed with `_`. These are
intended **only** for callers that already hold the appropriate locks, or that are running
in a purely sequential context (startup, `_load()` internals, tests). See
[Choosing the right call site](#choosing-the-right-call-site).
| Internal method | Corresponding public method |
|---|---|
| `_insert_by_datetime(record)` | `.insert_by_datetime()` |
| `_update_value(date, ...)` | `.update_value()` |
| `_key_from_lists(key, dates, values)` | `.key_from_lists()` |
| `_key_from_series(key, series)` | `.key_from_series()` |
| `_save()` | `.save()` |
| `_load()` | `.load()` |
| `_import_from_dict(...)` | `.import_from_dict()` |
| `_import_from_dataframe(...)` | `.import_from_dataframe()` |
| `_import_from_json(...)` | `.import_from_json()` |
| `_import_from_file(...)` | `.import_from_file()` |
#### Read-only methods (always sync) of `DataSequence`
These methods do not modify sequence state and require no locking:
| Method | Description |
|---|---|
| `get_by_datetime(dt)` | Exact or nearest record lookup. |
| `get_nearest_by_datetime(dt)` | Nearest record within an optional time window. |
| `key_to_dict(key, ...)` | Extract a `{datetime: value}` dict for a key. |
| `key_to_lists(key, ...)` | Extract parallel date and value lists. |
| `key_to_series(key, ...)` | Extract a `pd.Series` indexed by datetime. |
| `key_to_array(key, ...)` | Extract a resampled `np.ndarray` at a fixed interval. |
| `key_to_value(key, dt)` | Scalar lookup nearest to a datetime. |
| `to_dataframe(...)` | Convert all records to a `pd.DataFrame`. |
| `delete_by_datetime(...)` | Delete records within a datetime range. |
| `key_delete_by_datetime(key, ...)` | Set a field to `None` across a datetime range. |
#### Computed properties of `DataSequence`
| Property | Description |
|---|---|
| `min_datetime` | Earliest datetime in the sequence. |
| `max_datetime` | Latest datetime in the sequence. |
| `record_keys` | All field names for this sequence's record type. |
| `record_keys_writable` | Writable subset of `record_keys`. |
### `DataProvider`
Abstract singleton base class for objects that own and update a `DataSequence`. Each
concrete provider represents one data source (e.g. weather forecast, load measurement,
grid price). `DataProvider` is a specialisation of `DataSequence` and inherits its full
async interface and locking infrastructure — `_record_lock` and `_sequence_lock` — without
adding any new locks of its own.
Derived classes must implement:
| Abstract method | Description |
|---|---|
| `provider_id() -> str` | Unique string identifier for this provider. |
| `enabled() -> bool` | Whether this provider is active per configuration. |
| `_update_data(force_update)` | Custom data fetch/update logic. |
#### `_update_data` contract of `DataProvider`
`_update_data` is always called while the provider's own `_sequence_lock` and
`_record_lock` are both held by `update_data`. Implementations must therefore observe
the following constraints to avoid deadlock:
- Use only the internal sync methods (`_insert_by_datetime`, `_update_value`,
`_key_from_lists`, `_key_from_series`). Never call their public `async` counterparts,
which would attempt to re-acquire `_record_lock`.
- Do not call `save()`, `load()`, `_save()`, or `_load()`. Both locks are already held;
attempting to re-acquire either will deadlock.
- Network or I/O calls are permitted but should be kept short. Offload long-running I/O
to a thread via `asyncio.to_thread` in the caller before entering the lock scope.
```python
# Correct — both _sequence_lock and _record_lock are already held by the caller
def _update_data(self, force_update=False):
for dt, value in self._fetch_from_api():
self._update_value(dt, "temperature_c", value)
# Wrong — would deadlock: _record_lock is not reentrant
def _update_data(self, force_update=False):
for dt, value in self._fetch_from_api():
await self.update_value(dt, "temperature_c", value)
```
#### Public async interface of `DataProvider`
<!-- pyml disable line-length -->
| Method | Description |
|---|---|
| `await update_data(force_enable, force_update)` | Call `_update_data` if enabled or forced, holding both `_sequence_lock` and `_record_lock` for the duration. |
<!-- pyml enable line-length -->
### `DataImportMixin`
Mixin that adds bulk import capability to any class that also provides `update_value` and
`record_keys_writable`. Provides `import_from_dict`, `import_from_dataframe`,
`import_from_json`, and `import_from_file`.
The mixin expects values to be lists aligned to a fixed time interval starting from a
`start_datetime`. Two special dictionary keys are handled automatically:
- `start_datetime` — overrides the start of the import window.
- `interval` — overrides the fixed time step between values.
### `DataImportProvider`
Convenience base class combining `DataImportMixin` and `DataProvider`. Derive from this
when a provider's data arrives via JSON, file, or dict import rather than a live API.
### `DataContainer`
A singleton `MutableMapping` that aggregates multiple `DataProvider` instances and
presents their combined data through a single interface. Providers are tried in order;
the first one that contains the requested key wins.
`DataContainer` carries its own pair of locks (`_record_lock` and `_container_lock`)
that are independent of the locks on each provider. See
[Concurrency and Locking](#concurrency-and-locking) for the full acquisition matrix.
#### Public async interface of `DataContainer`
| Method | Description |
|---|---|
| `await __setitem__(key, series)` | Write a `pd.Series` into the appropriate provider. |
| `await __delitem__(key)` | Clear a field across all providers. |
| `await update_data(force_enable, force_update)` | Update all providers. |
| `await save()` | Save all providers to persistent storage. |
| `await load()` | Load all providers from persistent storage. |
| `await db_vacuum()` | Remove old records from all provider databases. |
| `await db_compact()` | Apply tiered compaction to all provider databases. |
| `await keys_to_dataframe(keys, ...)` | Consistent cross-provider snapshot as a `pd.DataFrame`. |
#### Read-only methods (always sync) of `DataContainer`
| Method | Description |
|---|---|
| `__getitem__(key)` | Return a `pd.Series` for a key from the first matching provider. |
| `__iter__()` | Iterate over all unique keys across enabled providers. |
| `__len__()` | Total number of unique keys. |
| `key_to_series(key, ...)` | Extract a series from the first matching provider. |
| `key_to_array(key, ...)` | Extract a resampled array from the first matching provider. |
| `provider_by_id(provider_id)` | Look up a provider by its string identifier. |
| `enabled_providers` | List of currently active providers. |
| `record_keys` | Union of all record keys across enabled providers. |
| `record_keys_writable` | Union of all writable record keys across enabled providers. |
## Concurrency and Locking
### Problem
EOS runs under FastAPI with an async event loop. Multiple HTTP requests are handled
concurrently as coroutines on a single thread. Because coroutines yield at `await` points,
two coroutines can interleave between a read and a subsequent write, producing a
**check-then-act race condition**:
```text
Coroutine A: db_get_record(ts) → None # record does not exist yet
Coroutine B: db_get_record(ts) → None # same — A has not inserted yet
Coroutine A: db_insert_record(new_rec) # inserts successfully
Coroutine B: db_insert_record(new_rec) # raises: duplicate timestamp
```
This is the root cause of the `ValueError: Duplicate timestamp` errors seen in production.
### Solution: three-level locking
The solution uses three distinct `asyncio.Lock` objects, each protecting a different scope
of state. Each name answers the question "what is being protected":
<!-- pyml disable line-length -->
| Lock | Defined on | Attribute | Protects |
|---|---|---|---|
| Record lock | `DataSequence` | `_record_lock` | A single check-then-act on one record. |
| Sequence lock | `DataSequence` | `_sequence_lock` | The full sequence state during operations that touch many records at once. |
| Container lock | `DataContainer` | `_container_lock` | Cross-provider consistency during container-level bulk operations. |
<!-- pyml enable line-length -->
`DataProvider` inherits `_record_lock` and `_sequence_lock` from `DataSequence` without
adding any new locks of its own. It is a specialisation of `DataSequence`, not a new
locking scope.
The fixed acquisition order across all levels is:
```text
_container_lock → _record_lock → provider _sequence_lock → provider _record_lock
```
No code path may acquire a finer-grained lock and then wait for a coarser one at the
same level. This invariant prevents deadlock.
### Lock creation
All locks are created lazily on first access via `_get_or_create_lock()`, which bypasses
Pydantic's `__setattr__` using `object.__setattr__` directly. This is necessary because
Pydantic v2 rejects attributes that are not declared model fields, and `asyncio.Lock`
cannot be safely created outside a running event loop, making `__init__`-time creation
unsafe. `cached_property` is not used for the same reason — Pydantic v2's tight
`__dict__` control makes it unreliable on model instances.
```python
def _get_or_create_lock(self, attr: str) -> asyncio.Lock:
try:
return object.__getattribute__(self, attr)
except AttributeError:
lock = asyncio.Lock()
object.__setattr__(self, attr, lock)
return lock
@property
def _record_lock(self) -> asyncio.Lock:
return self._get_or_create_lock("_record_lock_instance")
@property
def _sequence_lock(self) -> asyncio.Lock: # DataSequence / DataProvider
return self._get_or_create_lock("_sequence_lock_instance")
@property
def _container_lock(self) -> asyncio.Lock: # DataContainer only
return self._get_or_create_lock("_container_lock_instance")
```
### Lock acquisition rules
**Individual record writes** acquire only `_record_lock`, held for the minimum time
needed to make the check-then-act atomic:
```python
async def update_value(self, date, *args, **kwargs):
async with self._record_lock:
self._update_value(date, *args, **kwargs)
```
**Sequence-level bulk operations** (`save`, `load`, `import_from_*`) acquire
`_sequence_lock` first, then `_record_lock`. This blocks both concurrent bulk operations
and concurrent individual writes for the full duration:
```python
async def load(self):
async with self._sequence_lock:
async with self._record_lock:
self._load()
```
**`update_data`** on a provider acquires both `_sequence_lock` and `_record_lock`,
protecting the provider's full sequence state against concurrent saves, loads, imports,
or individual record writes for the entire duration of `_update_data`:
```python
async def update_data(self, force_enable=False, force_update=False):
if not force_enable and not self.enabled():
return
async with self._sequence_lock:
async with self._record_lock:
self._update_data(force_update=force_update)
```
**Container-level bulk operations** acquire `_container_lock` and `_record_lock` on the
container, then delegate to each provider (which independently acquires its own
`_sequence_lock` and `_record_lock`):
```python
async def save(self): # DataContainer
async with self._container_lock:
async with self._record_lock:
for provider in self.providers:
await provider.save() # provider acquires _sequence_lock + _record_lock
```
**Cross-provider consistent reads** acquire only the container's `_record_lock`, which
prevents concurrent writes from producing an inconsistent snapshot without blocking
other read operations:
```python
async def keys_to_dataframe(self, keys, ...):
async with self._record_lock:
... # read from multiple providers atomically
```
### Lock acquisition matrix
#### `DataSequence` / `DataProvider` lock
| Operation | `_sequence_lock` | `_record_lock` |
|---|---|---|
| `insert_by_datetime` | | ✓ |
| `update_value` | | ✓ |
| `key_from_lists` | | ✓ |
| `key_from_series` | | ✓ |
| `update_data` | ✓ | ✓ |
| `save` | ✓ | ✓ |
| `load` | ✓ | ✓ |
| `import_from_dict` | ✓ | ✓ |
| `import_from_dataframe` | ✓ | ✓ |
| `import_from_json` | ✓ | ✓ |
| `import_from_file` | ✓ | ✓ |
| Read-only methods | | |
#### `DataContainer` lock
<!-- pyml disable line-length -->
| Operation | `_container_lock` | `_record_lock` | Provider `_sequence_lock` | Provider `_record_lock` |
|---|---|---|---|---|
| `update_data` | ✓ | ✓ | ✓ | ✓ |
| `__setitem__` | | ✓ | | |
| `__delitem__` | | ✓ | | |
| `keys_to_dataframe` | | ✓ | | |
| `save` | ✓ | ✓ | ✓ | ✓ |
| `load` | ✓ | ✓ | ✓ | ✓ |
| `db_vacuum` | ✓ | ✓ | | |
| `db_compact` | ✓ | ✓ | | |
| Read-only methods | | | | |
<!-- pyml enable line-length -->
### Why `asyncio.Lock` and not `threading.Lock`
FastAPI's default async workers run all coroutines on a **single OS thread**. Coroutines
interleave at `await` points, not at thread boundaries. `threading.Lock` would not protect
against this interleaving and would risk deadlock if the same coroutine attempts to
re-acquire it. `asyncio.Lock` yields control correctly at `await` points while keeping
other coroutines blocked.
### Why `asyncio.Lock` is not reentrant
Python's `asyncio.Lock` is intentionally non-reentrant. If a coroutine that already holds
`_record_lock` calls a public async method (which also tries to acquire `_record_lock`),
it will deadlock. This is why `_update_data()` implementations and all other internal
callers must use the private `_method()` variants rather than `await self.method()`.
### Choosing the right call site
<!-- pyml disable line-length -->
| Caller context | Correct call |
|---|---|
| FastAPI endpoint or any `async def` | `await sequence.insert_by_datetime(record)` |
| `_update_data()` implementation | `self._insert_by_datetime(record)` |
| Startup, `_load()` internals, tests | `sequence._insert_by_datetime(record)` |
| `DataContainer` delegating to a provider | `await provider.save()` — container holds its own locks; provider independently holds its own |
<!-- pyml enable line-length -->
## Usage Examples
### Reading data (sync, no locking required)
```python
measurement = get_measurement()
# Scalar lookup
value = measurement.key_to_value("grid_import_w", target_datetime=now)
# Resampled array for the next 24 hours
array = measurement.key_to_array(
key="grid_import_w",
start_datetime=now,
end_datetime=now.add(hours=24),
interval=to_duration("1 hour"),
)
# Pandas Series
series = measurement.key_to_series("grid_import_w", start_datetime=now)
```
### Writing a single value from a FastAPI endpoint
```python
@router.put("/measurement/value")
async def put_measurement_value(datetime: str, key: str, value: float):
dt = to_datetime(datetime)
await get_measurement().update_value(dt, key, value)
```
### Bulk import from a FastAPI endpoint
```python
@router.put("/measurement/import")
async def put_measurement_import(data: dict):
await get_measurement().import_from_dict(data)
```
### Writing from a sync context (startup / tests)
```python
def load_initial_data(measurement: Measurement, records: list[MeasurementDataRecord]):
# Sequential — no concurrency, use internal sync methods directly
for record in records:
measurement._insert_by_datetime(record)
```
### Implementing a custom `DataProvider`
```python
class MyProvider(DataProvider):
records: list[MyRecord] = Field(default_factory=list)
def provider_id(self) -> str:
return "MyProvider"
def enabled(self) -> bool:
return self.config.my_provider.enabled
def _update_data(self, force_update: Optional[bool] = False) -> None:
# Both _sequence_lock and _record_lock are already held by the caller.
# Use internal sync methods only — never await public async counterparts.
for dt, reading in fetch_from_hardware():
self._update_value(dt, "sensor_w", reading)
def db_namespace(self) -> str:
return "MyProvider"
def db_keep_datetime(self) -> Optional[DateTime]:
return to_datetime().subtract(hours=48)
```
---
## Design Notes
### Singleton providers and containers
`DataProvider` and `DataContainer` both inherit from `SingletonMixin`. A single instance
is shared across all coroutines handling concurrent requests. This is precisely why
locking is necessary — every concurrent request operates on the same in-memory object.
### Lock naming rationale
The three lock names were chosen to reflect *what is being protected*, not *how coarse
the operation is*:
- `_record_lock` — makes it immediately clear that a single record's check-then-act is
being made atomic. The same name is used at both `DataSequence` and `DataContainer`
level because it plays the same role in both: serialising individual write operations.
- `_sequence_lock` — defined on `DataSequence` and inherited unchanged by `DataProvider`.
The name reflects that the entire sequence state is held exclusively, not just one
record. `DataProvider` is a specialisation of `DataSequence`, so the name remains
accurate at both levels without needing a separate `_provider_lock`.
- `_container_lock` — exclusive to `DataContainer`, reflecting that cross-provider
container-level state is held exclusively during bulk operations.
This naming also makes lock misuse visible in code review: a method that acquires
`_sequence_lock` without also acquiring `_record_lock` (or acquires them in the wrong
order) stands out immediately against the documented fixed acquisition order.
### Database vs in-memory storage
`DataSequence` uses `DatabaseRecordProtocolMixin` to optionally back its records with a
persistent time-series database. When a database is configured, `save()` and `load()`
delegate to it; otherwise they fall back to a JSON file. The in-memory `records` list acts
as a write-through cache. The locking scheme covers both paths — the database operations
are included inside the lock scope so that in-memory state and database state remain
consistent.
### Pydantic v2 compatibility
The `asyncio.Lock` instances are stored on the instance using `object.__setattr__`,
bypassing Pydantic's field validation entirely. This is intentional: `asyncio.Lock` is not
serialisable and must not appear in `model_dump()` or `model_dump_json()` output. The
locks are therefore invisible to Pydantic serialisation and are reconstructed fresh on
each process start, which is correct behaviour for a lock.

1
docs/index.md
View File

@@ -63,6 +63,7 @@ akkudoktoreos/api.rst
develop/develop.md
develop/release.md
develop/async.md
develop/CHANGELOG.md
```