Mirrors/EOS
1
0
Fork 0
mirror of https://github.com/Akkudoktor-EOS/EOS.git synced 2026-10-30 01:31:57 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

fix(database): make delete_by_datetime(None, None) a true full wipe (#1476)

Browse Source 
delete_by_datetime() with no bounds was documented as "delete all
records", but db_delete_records() assumed that
_db_ensure_loaded(start=None, end=None) loads every record into memory
before building the delete list. That is wrong: _db_ensure_loaded()
implements a NONE -> INITIAL -> FULL phase machine, and with both bounds
None in the NONE phase it loads only an *initial window* and leaves the
phase at INITIAL. Records outside that window never enter
_db_sorted_timestamps, so they are never marked for deletion and survive
in the physical store. Symptom: after a "wipe", db_count_records() still
returned N records (observed: 4 leftovers from a previous dense fixture).
A second delete_by_datetime() call happened to escalate INITIAL -> FULL
and deleted the rest, which is why the bug looked intermittent and why
stacked test runs accumulated rows.

Fix: in db_delete_records(), when both bounds are None, call
self._db_load_full() directly; otherwise keep the existing
_db_ensure_loaded(start, end) path. The delete list is then complete and
the "wipe" fixture actually wipes.

delete_by_datetime() remains a soft delete (rows are only added to
_db_deleted_timestamps; the physical DELETE happens on the next flush),
unchanged by this commit.

Besides this an additional fix was added:

* fix(tests): assert UTC-instant ordering in test_timestamps_remain_sorted

The test asserted `dts == sorted(dts)` on seq.records[].date_time, which
are tz-aware DateTime values in Europe/Berlin. Compaction buckets on a
UTC grid but records are materialized in local time, so across a DST
fall-back the local wall-clock sequence is non-monotonic even though the
UTC instants are ordered (e.g. 00:45 UTC -> 02:45 CEST, 01:00 UTC ->
02:00 CET). sorted() compares wall-clock fields, so it flags the pair at
the seam (index 1248 on 2026-10-25 for Europe/Berlin) as out of order.
The assertion is unsatisfiable for any 15-min grid spanning a fall-back
boundary when records are labelled in local time.

The actual invariant is "ascending by UTC instant", which is what
_db_sorted_timestamps guarantees and what the adjacent alignment tests
already assert via int(rec.date_time.timestamp()). Change the assertion
to compare integer UTC epochs:

    epochs = [int(r.date_time.timestamp()) for r in seq.records if r.date_time]
    assert epochs == sorted(epochs)

Note: this test was seasonally flaky. It only fails while a DST
transition lies inside the [now - 4 weeks, now] window used by
dense_energy_seq, which is why it newly started failing without any code
change. Upstream should ideally emit compacted records in UTC (or order
by UTC instant) so that wall-clock ordering is not silently assumed.

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
This commit is contained in:
Bobby Noelte
2026-10-29 15:27:41 +01:00
committed by GitHub
parent c425fd58c4
commit 7554e2dc47
7 changed files with 62 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env
+1 -1
View File
@@ -11,7 +11,7 @@ DOCKER_COMPOSE_DATA_DIR=${HOME}/.local/share/net.akkudoktor.eos
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
# Image / build # Image / build
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
VERSION=0.3.0.dev2610261491542381 VERSION=0.3.0.dev2610291327175670
PYTHON_VERSION=3.13.15 PYTHON_VERSION=3.13.15
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
config.yaml
+1 -1
View File
@@ -6,7 +6,7 @@
# the root directory (no add-on folder as usual). # the root directory (no add-on folder as usual).
name: "Akkudoktor-EOS" name: "Akkudoktor-EOS"
version: "0.3.0.dev2610261491542381" version: "0.3.0.dev2610291327175670"
slug: "eos" slug: "eos"
description: "Akkudoktor-EOS add-on" description: "Akkudoktor-EOS add-on"
url: "https://github.com/Akkudoktor-EOS/EOS" url: "https://github.com/Akkudoktor-EOS/EOS"
docs/_generated/configadapter.md
+5 -5
View File
@@ -140,7 +140,7 @@ e.g. devices/batteries/0/capacity_wh |
| device_instruction_entity_ids | `Optional[list[str]]` | `rw` | `None` | Entity IDs for device (resource) instructions to be updated by EOS. | device_instruction_entity_ids | `Optional[list[str]]` | `rw` | `None` | Entity IDs for device (resource) instructions to be updated by EOS.
The device ids (resource ids) have to be prepended by 'sensor.eos_' to build the entity_id. The device ids (resource ids) have to be prepended by 'sensor.eos_' to build the entity_id.
E.g. The instruction for device id 'battery1' becomes the entity_id 'sensor.eos_battery1'. | E.g. The instruction for device id 'battery1' becomes the entity_id 'sensor.eos_battery1'. |
| device_measurement_entity_ids | `Optional[dict[str, str]]` | `rw` | `None` | Mapping of EOS measurement keys used by device (resource) simulations to Home Assistant entity IDs. | | device_measurement_entity_ids | `Optional[dict[str, str]]` | `rw` | `None` | Mapping of EOS device measurement keys to Home Assistant entity IDs. Registered keys such as 'battery1-soc-factor' are accepted; legacy underscore spellings are resolved when they uniquely match a registered key. |
| eos_device_instruction_entity_ids | `list[str]` | `ro` | `N/A` | Entity IDs for energy management instructions available at EOS. | | eos_device_instruction_entity_ids | `list[str]` | `ro` | `N/A` | Entity IDs for energy management instructions available at EOS. |
| eos_solution_entity_ids | `list[str]` | `ro` | `N/A` | Entity IDs for optimization solution available at EOS. | | eos_solution_entity_ids | `list[str]` | `ro` | `N/A` | Entity IDs for optimization solution available at EOS. |
| grid_export_emr_entity_ids | `Optional[list[str]]` | `rw` | `None` | Entity ID(s) of export to grid energy meter readings [kWh] | | grid_export_emr_entity_ids | `Optional[list[str]]` | `rw` | `None` | Entity ID(s) of export to grid energy meter readings [kWh] |
@@ -179,8 +179,8 @@ E.g. solution key 'battery1_idle_op_mode' becomes the entity_id 'sensor.eos_batt
"sensor.pv_energy_total_kwh" "sensor.pv_energy_total_kwh"
], ],
"device_measurement_entity_ids": { "device_measurement_entity_ids": {
"ev11_soc_factor": "sensor.ev11_soc_factor", "ev11-soc-factor": "sensor.ev11_soc_factor",
"battery1_soc_factor": "sensor.battery1_soc_factor" "battery1-soc-factor": "sensor.battery1_soc_factor"
}, },
"device_instruction_entity_ids": [ "device_instruction_entity_ids": [
"sensor.eos_battery1" "sensor.eos_battery1"
@@ -219,8 +219,8 @@ E.g. solution key 'battery1_idle_op_mode' becomes the entity_id 'sensor.eos_batt
"sensor.pv_energy_total_kwh" "sensor.pv_energy_total_kwh"
], ],
"device_measurement_entity_ids": { "device_measurement_entity_ids": {
"ev11_soc_factor": "sensor.ev11_soc_factor", "ev11-soc-factor": "sensor.ev11_soc_factor",
"battery1_soc_factor": "sensor.battery1_soc_factor" "battery1-soc-factor": "sensor.battery1_soc_factor"
}, },
"device_instruction_entity_ids": [ "device_instruction_entity_ids": [
"sensor.eos_battery1" "sensor.eos_battery1"
docs/_generated/openapi.md
+16 -2
View File
@@ -1,6 +1,6 @@
# Akkudoktor-EOS # Akkudoktor-EOS
**Version**: `v0.3.0.dev2610261491542381` **Version**: `v0.3.0.dev2610291327175670`
<!-- pyml disable line-length --> <!-- pyml disable line-length -->
**Description**: This project provides a comprehensive solution for simulating and optimizing an energy system based on renewable energy sources. With a focus on photovoltaic (PV) systems, battery storage (batteries), load management (consumer requirements), heat pumps, electric vehicles, and consideration of electricity price data, this system enables forecasting and optimization of energy flow and costs over a specified period. **Description**: This project provides a comprehensive solution for simulating and optimizing an energy system based on renewable energy sources. With a focus on photovoltaic (PV) systems, battery storage (batteries), load management (consumer requirements), heat pumps, electric vehicles, and consideration of electricity price data, this system enables forecasting and optimization of energy flow and costs over a specified period.
@@ -2023,7 +2023,7 @@ Return:
**Request Body**: **Request Body**:
- `application/json`: { - `application/json`: {
"anyOf": [ "oneOf": [
{ {
"$ref": "#/components/schemas/PowerMeasurement" "$ref": "#/components/schemas/PowerMeasurement"
}, },
@@ -2052,6 +2052,20 @@ Return:
"$ref": "#/components/schemas/DDBCActuatorStatus" "$ref": "#/components/schemas/DDBCActuatorStatus"
} }
], ],
"discriminator": {
"propertyName": "type",
"mapping": {
"PowerMeasurement": "#/components/schemas/PowerMeasurement",
"EnergyMeasurement": "#/components/schemas/EnergyMeasurement",
"PPBCPowerProfileStatus": "#/components/schemas/PPBCPowerProfileStatus",
"OMBCStatus": "#/components/schemas/OMBCStatus",
"FRBCActuatorStatus": "#/components/schemas/FRBCActuatorStatus",
"FRBCEnergyStatus": "#/components/schemas/FRBCEnergyStatus",
"FRBCStorageStatus": "#/components/schemas/FRBCStorageStatus",
"FRBCTimerStatus": "#/components/schemas/FRBCTimerStatus",
"DDBCActuatorStatus": "#/components/schemas/DDBCActuatorStatus"
}
},
"description": "Resource Status.", "description": "Resource Status.",
"title": "Status" "title": "Status"
} }
openapi.json
+22 -8
View File
@@ -8,7 +8,7 @@
"name": "Apache 2.0", "name": "Apache 2.0",
"url": "https://www.apache.org/licenses/LICENSE-2.0.html" "url": "https://www.apache.org/licenses/LICENSE-2.0.html"
}, },
"version": "v0.3.0.dev2610261491542381" "version": "v0.3.0.dev2610291327175670"
}, },
"paths": { "paths": {
"/v1/measurement/battery-capacity/{battery_id}": { "/v1/measurement/battery-capacity/{battery_id}": {
@@ -1164,7 +1164,7 @@
"content": { "content": {
"application/json": { "application/json": {
"schema": { "schema": {
"anyOf": [ "oneOf": [
{ {
"$ref": "#/components/schemas/PowerMeasurement" "$ref": "#/components/schemas/PowerMeasurement"
}, },
@@ -1193,6 +1193,20 @@
"$ref": "#/components/schemas/DDBCActuatorStatus" "$ref": "#/components/schemas/DDBCActuatorStatus"
} }
], ],
"discriminator": {
"propertyName": "type",
"mapping": {
"PowerMeasurement": "#/components/schemas/PowerMeasurement",
"EnergyMeasurement": "#/components/schemas/EnergyMeasurement",
"PPBCPowerProfileStatus": "#/components/schemas/PPBCPowerProfileStatus",
"OMBCStatus": "#/components/schemas/OMBCStatus",
"FRBCActuatorStatus": "#/components/schemas/FRBCActuatorStatus",
"FRBCEnergyStatus": "#/components/schemas/FRBCEnergyStatus",
"FRBCStorageStatus": "#/components/schemas/FRBCStorageStatus",
"FRBCTimerStatus": "#/components/schemas/FRBCTimerStatus",
"DDBCActuatorStatus": "#/components/schemas/DDBCActuatorStatus"
}
},
"description": "Resource Status.", "description": "Resource Status.",
"title": "Status" "title": "Status"
} }
@@ -10992,11 +11006,11 @@
} }
], ],
"title": "Device Measurement Entity Ids", "title": "Device Measurement Entity Ids",
"description": "Mapping of EOS measurement keys used by device (resource) simulations to Home Assistant entity IDs.", "description": "Mapping of EOS device measurement keys to Home Assistant entity IDs. Registered keys such as 'battery1-soc-factor' are accepted; legacy underscore spellings are resolved when they uniquely match a registered key.",
"examples": [ "examples": [
{ {
"battery1_soc_factor": "sensor.battery1_soc_factor", "battery1-soc-factor": "sensor.battery1_soc_factor",
"ev11_soc_factor": "sensor.ev11_soc_factor" "ev11-soc-factor": "sensor.ev11_soc_factor"
} }
] ]
}, },
@@ -11168,11 +11182,11 @@
} }
], ],
"title": "Device Measurement Entity Ids", "title": "Device Measurement Entity Ids",
"description": "Mapping of EOS measurement keys used by device (resource) simulations to Home Assistant entity IDs.", "description": "Mapping of EOS device measurement keys to Home Assistant entity IDs. Registered keys such as 'battery1-soc-factor' are accepted; legacy underscore spellings are resolved when they uniquely match a registered key.",
"examples": [ "examples": [
{ {
"battery1_soc_factor": "sensor.battery1_soc_factor", "battery1-soc-factor": "sensor.battery1_soc_factor",
"ev11_soc_factor": "sensor.ev11_soc_factor" "ev11-soc-factor": "sensor.ev11_soc_factor"
} }
] ]
}, },
src/akkudoktoreos/core/databaseabc.py
+8 -4
View File
@@ -1601,10 +1601,14 @@ class DatabaseRecordProtocolMixin(
await self._db_ensure_initialized() await self._db_ensure_initialized()
# Deletion is global — ensure we see everything # Deletion is global — ensure we see everything
await self._db_ensure_loaded( if start_timestamp is None and end_timestamp is None:
start_timestamp=start_timestamp, # Full wipe — must escalate to FULL, not stop at INITIAL.
end_timestamp=end_timestamp, await self._db_load_full()
) else:
await self._db_ensure_loaded(
start_timestamp=start_timestamp,
end_timestamp=end_timestamp,
)
to_delete: list[DatabaseTimestamp] = [] to_delete: list[DatabaseTimestamp] = []
tests/test_dataabccompact.py
+9 -2
View File
@@ -672,8 +672,15 @@ class TestDataSequenceCompactIntegrity:
seq, _ = dense_energy_seq seq, _ = dense_energy_seq
await seq.db_compact() await seq.db_compact()
dts = [r.date_time for r in seq.records if r.date_time is not None] # Use UTC timestamp to account for DST change in records sequence
assert dts == sorted(dts) epochs = [
int(r.date_time.timestamp())
for r in seq.records
if r.date_time is not None
]
assert epochs == sorted(epochs), (
"Records must be ascending by UTC instant after compaction"
)
async def test_compacted_old_timestamps_on_1h_boundaries(self, dense_energy_seq): async def test_compacted_old_timestamps_on_1h_boundaries(self, dense_energy_seq):
"""Records older than the floored 2-week cutoff must be on whole-hour UTC boundaries. """Records older than the floored 2-week cutoff must be on whole-hour UTC boundaries.