Commit Graph
1153 Commits
Author SHA1 Message Date
Bobby NoelteandClaude Opus 5.5 e255f987bf feat(devices): add battery profiles
A battery profile describes a battery model: the charge and discharge limit
maps over temperature and SoC, the charge and discharge efficiency curves
over C-rate, the technical SoC bounds, the hard temperature cutoffs and,
for a model, its nominal capacity and power limits. EOS packages the
generic profile generic-lfp; users add their own profiles.

A battery or electric vehicle selects its profile by battery_profile and
describes how the installation differs from it in profile_adjustment
(capacity, power limits of the installation, periodic full charge). The
effective values are resolved from both at the measured operating point
(temperature, SoC, power) and shown as effective_* fields.

The GENETIC simulation uses the limits and efficiencies of the profile over
SoC, with precomputed tables as a fast path; GENETIC0 gets the values at the
operating point. Optimization parameters are always valid: a battery whose
profile is missing, unreadable or invalid uses the packaged generic-lfp
profile, and the former defaults only if that is not usable either; each is
logged as error, and an optimization run is never cancelled because of a
battery.

Additional changes
- Add a generic device profile store: user and packaged profiles, atomic
  saves, compiled profiles cached per file and checked again at most once a
  second, plain file names as profile names ('schema' is reserved). On
  Windows, a save or read that collides with the other is retried briefly.
- Add the REST API /v1/profile to list the profile kinds and to list, read
  (document and JSON schema), save and delete profiles.
- Check the hard temperature cutoffs on the exact temperature.
- Resolve inhibited charging or discharging to a power of 0, e.g. for a cold
  battery; give GENETIC0 its own discharge limit for that.
- Limit generic profiles without own power limits to the former 5000 W.
- Accept only the implemented units; current limits need the nominal
  voltage.
- Migrate legacy battery settings into the profile adjustment, keep the
  former charge power limit as discharge limit (also for a Home Assistant
  entity on it) and report dropped efficiencies as errors.
- Limit GENETIC battery discharge by the discharge power.
- Allow battery profile lookups while the configuration is created.
- Resolve a battery once per configuration dump.
- Package the bundled profiles, document battery profiles and regenerate
  the configuration and OpenAPI documentation.
- Ignore the report PDFs written by the visualization tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: Claude Code (claude-opus-5-5)
2026-11-11 22:08:18 +01:00
Bobby NoelteandClaude Opus 5.5 dd41e5d57c chore(ai): add agent self-review and a CI check for agent-assisted PRs
Additional changes
Agents write a task plan to .scratch/<branch>/plan.md and self-review with code-review
Agents fix hard violations, report judgement calls and rerun the self-review before every push
Agents declare the review in a Self-review section of the pull request body
Add the ai-self-review workflow and scripts/check_pr_self_review.py for pull requests into main
Detect agents by Assisted-by or Anthropic Co-Authored-By trailers, body markers or the AI box
Fail only on a missing Self-review section or a missing, malformed or stale Reviewed SHA
Note an unticked AI-assisted box and malformed optional lines without failing
Comment on the pull request how to fix a failure and update that one comment on later runs
Show the same report in the job summary and accept Windows line endings in the body
Skip dependabot and release pull requests
Run on pull_request_target with the base branch's script, never pull request code
Add the pull request template: the pr skill's sections plus Checklist and Self-review
Require always valid optimization parameters in the energy-domain skill
Require datetimeutil for date and time handling in new and changed code in AGENTS.md
Require the Assisted-by trailer on agent commits in the commit-messages skill
Add tests for the check in tests/test_ai.py
Add GLOSSARY.md and the agent tracker docs in docs/agents/
Add decision record docs/adr/0001 with an ADR index in the docs toctree
Document the workflow in docs/develop/ai.md and CONTRIBUTING.md
Exclude docs/agents from Sphinx and PyMarkdown and ignore .scratch/
Document cz check instead of make gitlint for commit messages in docs/develop/ai.md

Assisted-by: Claude Code (claude-opus-5-5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:37:33 +01:00
62cbf4cf2a fix(homeassistant): apply 0 and off from config entities, never add settings
The configuration sync of the Home Assistant adapter skipped every value that is false in
Python, so an entity state of 0 or off never reached the configuration. It also set any
path it was given: the documented example devices/batteries/0/capacity_wh named a device
"0" that does not exist, so EOS silently added a new battery with a random device id.

Apply every value except an unavailable or unknown entity, and only update settings that
exist; report other paths as an error. The example now addresses the battery by its
device_id.

Signed-off-by: SchaafAlexander <86607129+SchaafAlexander@users.noreply.github.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:38:45 +01:00
elschmanandGitHub 1fda5f9201 fix(doc): compose file in README
Without a docker compose the recommended install part of the documentation will not work at all
2026-11-11 09:36:16 +01:00
7767028a14 fix(config): keep dict entries with "/" in keys during config migration
The automatic field migration recursed into every dict and addressed each
leaf by a "/"-joined path. Keys that themselves contain "/" - e.g. the config
paths used as keys of adapter.homeassistant.config_entity_ids - were split
into path elements. set_nested_value() then created a "devices" entry with
an empty string default and failed further down, so every version migration
replaced the mapping by {"devices": ""}.

Migrate a dict whose keys contain "/" as one value instead of per key.

Signed-off-by: SchaafAlexander <86607129+SchaafAlexander@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 12:50:41 +01:00
Bobby Noelte 93ccb8d592 chore: development version update (incl. home assistant config)
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2026-11-09 17:05:56 +01:00
Bobby NoelteandClaude Opus 5.5 2c58874c52 fix(cache): use a reproducible function hash for cache_in_file keys
Build the cache_in_file key from an MD5 hash of the wrapped function
instead of inspect.getsource(). Reading the source fails for functions
whose source is not on disk, and the key could go stale when the file
changes after import.

The hash covers what decides the result besides the arguments: the
qualified name, the bytecode with the names of called functions,
attributes and globals, the constants including nested functions, the
defaults and the values captured from an enclosing function. Wrappers made
with functools.wraps are unwrapped. Code objects, functions and sets are
hashed without memory addresses or hash-seed dependent order, so the key
stays the same across restarts and the file cache keeps hitting.

Additional fixes
- Build the key from the arguments bound to the signature, so *args,
  **kwargs and the arguments of wrapped functions are part of it.
- Apply control parameters given at call time to that call only; they no
  longer change the decorator settings for later calls.
- Apply all control parameters of a call, so until_date is no longer
  dropped when force_update is also passed; keep the precedence
  until_datetime > with_ttl > until_date.
- Recreate an unreadable cache file in that call only, instead of
  disabling the cache for the rest of the process.
- Pass force_update in two provider tests that relied on a force_update
  left over from an earlier test.
- Fix "Calback" typo in cache_energy_management_store_callback docstring.

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 16:16:26 +01:00
ChenYi-NnergyandGitHub 38b07f5a5b fix: apply inverter_paco cap and system loss in PVForecastPVLib - #1504⁠
Planes sharing one inverter_paco feed a single physical inverter, but their AC output was summed without any cap, so a 15 kW inverter could report 21-25 kW. This PR groups the per-plane AC output by inverter_paco and clips each group's sum to its rating; planes without a rating are left alone. It also applies the configured system loss (PVGIS-style percent, same semantics as PVForecastAkkudoktorLocal) to DC and AC output instead of warning that it is unsupported.
2026-11-09 14:41:16 +01:00
Bobby NoelteandClaude Opus 5.5 b92d115a39 test(eosdash): wait for the htmx swap before checking a tab's content
The browser test read the content area right after the tab's response
arrived. htmx swaps the response in after that, and wait_for_load_state()
did not wait for it, as the page itself was loaded long before. For the
large configuration tab the content area was sometimes still empty, so
the test failed in CI.

Wait until the content area is filled again after a successful response.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 14:21:51 +01:00
c230d9d9de fix: keep tests and doc generators away from the developer's config files
test_docsphinx pointed EOS_DIR/EOS_CONFIG_DIR at an empty temporary
directory. EOS then fell back to the config file in the user config
directory, migrated it and wrote it back with a timestamped backup. The
test now writes an empty config file into the temporary directory, as
server_base does.

Additional fixes
- Decode the Sphinx output in test_sphinx_build as UTF-8; on Windows the
  default code page failed and the test reported a NoneType error.
- Check the user config directory with the app author in
  --check-config-side-effect, as EOS does; on Windows the check looked at
  the wrong directory.
- Let the doc generators create EOS' config file in a temporary directory.
  They left an EOS.config.json in the working directory, and a following
  make test-ci then failed in every test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 13:40:25 +01:00
Bobby NoelteandClaude Opus 5.5 d68f040206 feat(ai): add the mattpocock engineering skills for AI agents
Vendor the engineering skills of https://github.com/mattpocock/skills into
ai/skills/engineering, with their MIT license, and list them in AGENTS.md.
The agent-specific metadata (agents/openai.yaml) is kept for other agents.

Additional changes
- Add scripts/update_mattpocock_skills.py to compare the copy with upstream by
  git blob hashes from a blobless clone and to update it on request.
- Add tests/test_ai.py; it warns when the copy differs from upstream and caches
  the upstream hashes in the pytest cache for a day.
- Document the engineering skills in docs/develop/ai.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 12:45:20 +01:00
Bobby NoelteandClaude Opus 5.5 57132df44e fix(config): enable documentation mode only explicitly
ConfigEOS.documentation_mode() was true as soon as Sphinx was imported
anywhere in the process. A test importing Sphinx therefore switched all
later tests into documentation mode, and test_config_md_current failed
depending on test order. Documentation mode is now enabled only by the
documentation generators (_force_documentation_mode) or by the Sphinx
configuration, which sets sys._called_from_sphinx in docs/conf.py.

Declare _force_documentation_mode as a ClassVar. Without the annotation
pydantic turned it into a private attribute, and the class attribute was
a truthy descriptor object.

Behavior change: because of that descriptor, every running EOS was in
documentation mode, so the get_adapter/config/measurement/prediction/ems/
database/resource_registry() guards never raised "access before init" and
created the singletons lazily on first access. They now raise for every
access before initialization. The worker process of uvicorn's auto-reload
(server.reload, make run-dev) relied on that: the app's lifespan now sets up
the configuration and initializes the singletons that do not exist yet.

Leave runtime state out of the documentation: the Home Assistant entity ID
properties and OptimizationCommonSettings.keys return an empty list in
documentation mode, so entity IDs and solution keys of a running system
no longer reach the generated configuration docs.

Keep the tests isolated: test_doc sets documentation mode on the class
and test_docstringrst restores sys._called_from_sphinx after reading the
Sphinx configuration. Add tests for how documentation mode is enabled and
for the runtime values it leaves out.

Additional changes
- Generate openapi.json in documentation mode, as the config docs.
- Share the documentation mode check as coreabc.documentation_mode().
- Move the logging and server address setup of run_eos() to setup_eos_config().
- Test that EOS starts with server.reload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 12:40:15 +01:00
Bobby NoelteandClaude Opus 5.5 a0e57a18b7 fix(config): validate prediction providers before the prediction exists
The provider settings (weather, PV forecast, load, electricity price, feed-in
tariff, electricity fee) validate their provider id against the providers of
the prediction instance. While the configuration is created, that instance
does not exist yet, and the validation falls back to fixed lists. These were
incomplete: weather knew only 'WeatherImport', load lacked
'LoadAkkudoktorAdjusted'. A configuration file with e.g. weather provider
'OpenMeteo' then fails to load:

  Provider 'OpenMeteo' is not a valid weather provider: ['WeatherImport']

This is hidden today because ConfigEOS.documentation_mode() is always true
(_force_documentation_mode is a pydantic private attribute, whose class
attribute is a truthy descriptor), and get_prediction() then creates the
prediction instance on access. Once documentation mode is enabled only
explicitly, the configuration does not load.

Derive the provider ids from the provider classes instead
(prediction_provider_ids). They no longer depend on whether the prediction
instance exists, so the validation is the same before and after
initialization.

Additional changes
- Add the test fixture uncreated_config, which lets get_config(init=True)
  create a new configuration from the config file and restores the old one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 12:40:15 +01:00
Bobby NoelteandClaude Opus 5.5 4475774c06 test(eosdash): run EOSdash in a headless Chromium browser
Run EOSdash in a headless Chromium browser for testing. Every menu tab must
load with HTTP 200 and render content, in light and dark mode, without
JavaScript errors.

Additional changes
Enable the browser tests by --browser-test (marker browser), skip them otherwise
Add the CI job browser-test that installs Chromium and runs them
Add the make target test-browser, run with --extra dev so playwright is installed
Add playwright to the dev dependencies
Document how to run the browser tests with --extra dev, incl. the browser system
libraries on Linux
Accept a child PID of the started server in the server startup checks, on Windows
only: there the venv python.exe runs the real interpreter as a child process
Share one Chromium per test class, so Playwright stops before other tests run
Request chromium before servers so a missing browser fails before EOS starts
Make the missing-browser hint platform neutral (Linux: add --with-deps)
Set PYTHONIOENCODING=utf-8 for the test server to avoid encoding errors on Windows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 12:39:30 +01:00
2245976de4 docs(ai): move agent rules to ai/skills and add an AI development sandbox (#1498)
The sandbox provides an isolated Docker environment for AI-assisted development
(OpenCode + Claude Code + Qwen Code) against the local AkkudoktorEOS repository.

The sandbox does not protect the repository from harmful changes by the agent.
It limits what a misled agent can reach on the machine, but the agent can still
write any change into the repository.

Primary security defence is to check every change before it is run on the host.
Same holds for opening a pull request or updating one.

Additional changes

- Reduce AGENTS.md to the project summary, baseline rules, completion checklist,
  and an index of the skills.
- Add skills for the Python environment and REST API error responses.
- Add the development guide page docs/develop/ai.md: the AI coding
  workflow, how agents get the project rules from AGENTS.md and ai/skills,
  and how to use the sandbox (start, agent logins, project commands with a
  virtual environment outside the workspace, running EOS and EOSdash,
  GitHub access, persistent data, security model).
- Add a checklist for reviewing agent changes before pushing, starting with
  .git, which stays writable in the sandbox.
- Check out all text files with LF line endings (.gitattributes), also on
  Windows with core.autocrlf, so the sandbox scripts run in the container
  and git in the container agrees with git on the host. Batch files keep CRLF.
- Publish the ports of ai/compose.yaml on request (AI_SANDBOX_PORTS=1).
  docker compose run publishes them only with --service-ports. Off by
  default, so that several sandboxes can run in parallel.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 09:15:12 +01:00
628226d3d9 feat(pvforecast): model snow on the modules in the local backend (#1484)
No PV forecast modelled snow lying on the modules, so after a snowfall the
optimizer planned with PV that never came. The local Open-Meteo/pvlib
backend can now reduce the forecast by the snow coverage (opt-in via
pvforecast.akkudoktor.snow_enabled):

- Request hourly snowfall and snow depth with the weather and run the NREL
  snow coverage model (pvlib.snow) per plane.
- Correct the coverage with measured PV production: production above the
  model clears it at once; low production raises it only while snow is
  plausible, so curtailment and inverter outages are not learned as snow.
  Persisting snow damps the slide rate, fading with
  snow_feedback_memory_hours.
- Rebuild the coverage from snow_lookback_days on every update instead of
  storing state.
- Exclude intervals with possible snow from the calibration fit.
- Warn when PV meter readings are missing or stale; ignore impossible meter
  jumps.

_plane_power keeps its two-value signature (used by the adaptive provider in
#1422); the irradiance comes from the new _plane_power_and_poa.

Tests replay real February 2026 data from one plant (snow on 02-19/20).

Signed-off-by: SchaafAlexander <86607129+SchaafAlexander@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 07:42:20 +01:00
a06473b6d6 feat(battery): plan a periodic full charge of the stationary battery (#1486)
LFP batteries can only recalibrate their state of charge close to full
charge, so many systems expect a full charge every few weeks and some
start one on their own at an arbitrary, possibly expensive time. EOS can
now plan it instead (GENETIC only, off by default).

New battery settings: full_charge_interval_days (unset = off),
full_charge_soc_percentage, full_charge_detect_percentage,
full_charge_allow_grid and the informational cell_chemistry. The last
full charge is read from the measured SoC history; nothing is stored.
When the next one is due within the horizon (or overdue), the maximum
SoC is raised to the target. With full_charge_allow_grid the optimizer
must reach the target once before the deadline, which is at least 24 h
away (penalty battery_full_charge_miss on the best SoC reached), so it
picks the cheapest time itself. Without it, PV surplus may fill the
battery but the target is not enforced.

Signed-off-by: SchaafAlexander <86607129+SchaafAlexander@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 07:40:13 +01:00
NormannandGitHub b33953d445 fix(pvforecast): normalize Forecast.Solar power before resampling (#1487)
Forecast.Solar supplies instantaneous power with irregular sunrise/sunset timestamps. GENETIC interprets stored values as interval averages and infers cadence from the shortest gap, so these samples create artificial coverage gaps. The prediction list API also averages away the sunset zero before interpolating, producing positive nighttime PV.

Integrate each roof plane's piecewise-linear power curve into UTC-contiguous 15-minute interval-average W before summing planes. Both hourly and quarter-hour GENETIC slots then consume the same normalized data as the prediction API, with exactly one downstream W-to-Wh conversion. Explicit sunrise/sunset zero boundaries establish the adjacent night within each forecast day. Daylight gaps longer than an hour, invalid power, absent planes and unavailable forecast days remain missing; the strict GENETIC coverage check stays in place.

Refreshes replace the current forecast and remove stale tails. Because cadence detection also sees historical records, the first refresh clears legacy forecasts containing non-aligned timestamps, including their history. Later refreshes retain normalized history. This upgrade behavior is documented.

Forecast validation errors now identify the first invalid interval and the number of invalid slots.
2026-11-07 19:46:58 +01:00
NormannandGitHub 651301f9e6 build(deps): update Python dependencies (#1497)
Combines Dependabot update into one reviewable change:

- MonsterUI  1.0.47 → 1.0.48
- mypy (dev) 2.4.0 → 2.4.1
- Pydantic  2.14.1 → 2.14.2
- platformdirs 4.13.5 → 4.14.2
- pytest-asyncio (dev)  1.4.0 → 1.4.1
- OAuthLib (transitive) 4.0.0 → 4.0.1
2026-11-06 08:45:12 +01:00
f804d48072 feat(pvforecast): add adaptive per-plane PV learning and diagnostics (#1422)
PV arrays with different orientations and curtailment need separate plant calibration and weather correction. This opt-in provider publishes normal EOS AC/DC forecast keys at 15-minute resolution.

- Preserve daily per-plane energy/shape calibration and validated short-term local/trend selection. Curtailment-sensitive arrays require a fresh release flag or an explicitly configured learning guard.
-  Add switchable empirical solar-angle response relative to separate eligible reference arrays, removing their common weather error. This learns angular residuals, not physical angles or a measured geometric horizon.
-  Archive actually received weather model inputs, first local receipt and frozen per-member PV candidates. Returned past weather is excluded; unknown source issuance remains explicit. Learn model weights and forecast-regime bias separately at 2–6, 6–24 and 24–48 hours.
- New long-horizon candidates remain in shadow until their genuinely issued outputs improve equal-day MAE by the configured margin, win on at least 60% of validation days and pass a matched-support energy-bias guard. Insufficient evidence retains calibration. Daily yields still require complete local days, including DST.
- Expose fitted parameters, independent training/validation day counts, errors, selected methods and persistence in the existing status API, with additional normal prediction keys for Grafana. Retain physical DC through angular/weather corrections an  enforce individual inverter AC limits.

Co-authored-by: Normann <github@koldrack.com>
Co-authored-by: Christin <chrissi@von-perbandt.de>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: YMT <yn276364@dal.ca>
Co-authored-by: sarankumar1325 <120080287+sarankumar1325@users.noreply.github.com>
2026-11-03 15:31:32 +01:00
AndreasandGitHub 298d46a527 fix(battery): preserve energy below minimum state of charge (#1483) 2026-11-01 15:46:45 +01:00
NormannandGitHub 33fc5ed61a build(deps): update pypdf to 6.21.4 (#1482) 2026-10-31 12:12:26 +01:00
Bobby NoelteandGitHub ef852c45a5 chore: development version update (incl. home assistant config) (#1480)
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2026-10-30 17:45:21 +01:00
NormannandGitHub a7e98b5b82 fix(eosdash): render predictions without provider-specific load statistics (#1479)
With load.provider = "LoadImport", EOSdash requests Akkudoktor-only statistics and returns an error instead of rendering any prediction charts when the API correctly responds with 404.

Fetch only keys advertised by /v1/prediction/keys, tolerate individual 404s if availability changes, and render the load mean/stddev lines only when their series exist. Charts without their required series are skipped; empty/all-null responses produce a no-data message instead of invalid axes or concatenation errors. Other HTTP failures remain visible.

Regression coverage uses real Bokeh plotting for LoadImport, LoadAkkudoktor/Adjusted, independently available statistics, missing/empty series, and non-404 errors. The existing timezone/DST regression remains covered.
2026-10-30 16:58:59 +01:00
Bobby NoelteandGitHub e8dcfecf73 feat: add filename linter and AI coding guidelines (#1477)
Add a configurable filename linter to enforce naming conventions and an AGENTS.md to guide
AI coding agents in following project conventions.

The guidelines steer code generation; the linter independently verifies filename compliance.
Together, they improve consistency and reduce unnecessary files and review effort.

Conflicting file names were renamed.

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2026-10-30 15:17:12 +01:00
527c26d424 feat(eosdash): plan page with EMS plan timeline and persisted settings (#1460)
* feat(eosdash): plan page with EMS plan timeline and persisted settings

Fixes:
- SolutionCard no longer appends to the module-level solution_excludes
  list on every render (the list grew without limit).
- All EOSdash pages: keep the scroll position after htmx actions instead
  of jumping to the bottom; Bokeh() reserves the plot height.
- Default visible PV column pvforecast_power_w -> pvforecast_ac_energy_wh
  (the old column no longer exists).
- All EOSdash pages: remove w-screen from components.Page (horizontal
  scrollbar).
- Solution chart: energy is the main axis; the always empty power axis is
  only shown with power columns. Typo "Electricty" fixed.

Features:
- EMS plan timeline per device, plan table as collapsible section.
- Solution chart rework: KPI row, legend, totals view, value axes for the
  displayed columns only, units menu, currency from the timezone, aligned
  time axes with the timeline, themed tooltips, system font.
- Per-column and per-command settings: visibility, alias, color, style;
  device aliases.
- New config section eosdash.plan (server/dash/dashsettings.py) stores the
  plan page settings in the EOS configuration; docs regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: SchaafAlexander <86607129+SchaafAlexander@users.noreply.github.com>
2026-10-30 08:03:19 +01:00
Bobby NoelteandGitHub 7554e2dc47 fix(database): make delete_by_datetime(None, None) a true full wipe (#1476)
delete_by_datetime() with no bounds was documented as "delete all
records", but db_delete_records() assumed that
_db_ensure_loaded(start=None, end=None) loads every record into memory
before building the delete list. That is wrong: _db_ensure_loaded()
implements a NONE -> INITIAL -> FULL phase machine, and with both bounds
None in the NONE phase it loads only an *initial window* and leaves the
phase at INITIAL. Records outside that window never enter
_db_sorted_timestamps, so they are never marked for deletion and survive
in the physical store. Symptom: after a "wipe", db_count_records() still
returned N records (observed: 4 leftovers from a previous dense fixture).
A second delete_by_datetime() call happened to escalate INITIAL -> FULL
and deleted the rest, which is why the bug looked intermittent and why
stacked test runs accumulated rows.

Fix: in db_delete_records(), when both bounds are None, call
self._db_load_full() directly; otherwise keep the existing
_db_ensure_loaded(start, end) path. The delete list is then complete and
the "wipe" fixture actually wipes.

delete_by_datetime() remains a soft delete (rows are only added to
_db_deleted_timestamps; the physical DELETE happens on the next flush),
unchanged by this commit.

Besides this an additional fix was added:

* fix(tests): assert UTC-instant ordering in test_timestamps_remain_sorted

The test asserted `dts == sorted(dts)` on seq.records[].date_time, which
are tz-aware DateTime values in Europe/Berlin. Compaction buckets on a
UTC grid but records are materialized in local time, so across a DST
fall-back the local wall-clock sequence is non-monotonic even though the
UTC instants are ordered (e.g. 00:45 UTC -> 02:45 CEST, 01:00 UTC ->
02:00 CET). sorted() compares wall-clock fields, so it flags the pair at
the seam (index 1248 on 2026-10-25 for Europe/Berlin) as out of order.
The assertion is unsatisfiable for any 15-min grid spanning a fall-back
boundary when records are labelled in local time.

The actual invariant is "ascending by UTC instant", which is what
_db_sorted_timestamps guarantees and what the adjacent alignment tests
already assert via int(rec.date_time.timestamp()). Change the assertion
to compare integer UTC epochs:

    epochs = [int(r.date_time.timestamp()) for r in seq.records if r.date_time]
    assert epochs == sorted(epochs)

Note: this test was seasonally flaky. It only fails while a DST
transition lies inside the [now - 4 weeks, now] window used by
dense_energy_seq, which is why it newly started failing without any code
change. Upstream should ideally emit compacted records in UTC (or order
by UTC instant) so that wall-clock ordering is not silently assumed.

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2026-10-29 15:27:41 +01:00
NormannandGitHub c425fd58c4 build(deps): bump tornado to 6.6 (#1475) 2026-10-29 11:00:39 +01:00
NormannandGitHub 8396bf24ae fix(deps): upgrade pandas to 3.1 and fix EOS deprecations (#1469)
* build(deps): bump pandas from 3.0.6 to 3.1.0

* fix(prediction): avoid pandas 3.1 deprecation warnings
2026-10-29 09:10:21 +01:00
NormannandGitHub dfc62d7e9c fix(deps): update Starlette to avoid deprecated AnyIO alias (#1470) 2026-10-29 08:52:01 +01:00
8e34b1779b fix(genetic): stop parallel evaluation workers reliably at the end of a run (#1473)
Follow-up to #1382. Under the EOS server the end of a GENETIC run with
parallel evaluation (workers >= 2, which is the automatic setting on
machines with 3 or more cores) could hang forever in
Pool.terminate()/join(), so /v1/optimize never answered.

Cause: uvicorn installs a Python SIGTERM/SIGINT handler in the server
process that only asks the server to shut down. The forked evaluation
workers inherit it and survive the SIGTERM of Pool.terminate().
terminate() keeps the task queue's read lock (Pool._help_stuff_finish),
so a surviving worker that has not yet taken its stop sentinel waits for
that lock forever, and join() with it. This always happened after a
batch timeout (busy workers survive) and by chance after a normal run
when a worker was not scheduled in time (several EOS instances or other
load; also seen with a single instance on a 2-vCPU VM). A worker that
got SIGTERM before its initializer ran hit the same path.

- Workers ignore SIGTERM and SIGINT. Sent to the whole process group
  (Ctrl-C, systemd stopping the service) they are meant for the server,
  which finishes or aborts the run and stops the workers itself.
  Restoring the default action instead was measured to be worse: a
  group SIGTERM killed the workers mid-batch, the batch ran into its
  120 s timeout and the server did not exit within 120 s (12 s before).
- _EvaluationPool stops its workers with SIGKILL at the point where
  Pool.terminate() would send SIGTERM (it already holds the read lock).
  This does not depend on handlers, initialization state or whether a
  worker is busy. _stop_evaluation_pool() is unchanged.

Measured (8 EOS servers x 2 workers on 8 vCPU, real configuration,
seed 42): 11 hangs in 160 runs before, 0 in 440 after; pool only under
load: 107/300 before, 0/600 after. Plans and costs are identical, the
stop takes no longer. Ctrl-C, SIGTERM to the server and SIGTERM to the
group behave as before (run completes with HTTP 200, exit after ~12 s,
no orphaned workers).

Tests (tests/test_genetic_parallel_evaluation.py, under uvicorn's real
signal handling): inherited handler, batch timeout, late worker start,
late idle worker (SIGSTOP, Linux) and SIGTERM to the group during a
batch. They fail without the fix.


Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 08:51:36 +01:00
NormannandGitHub aa97e677b6 build(deps): bump dnspython to 2.9.0 (#1472) 2026-10-28 22:12:42 +01:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Normann
bb9c47ea90 build(deps): bump dnspython from 2.8.0 to 2.9.0 (#1471)
Bumps [dnspython](https://github.com/rthalley/dnspython) from 2.8.0 to 2.9.0.
- [Release notes](https://github.com/rthalley/dnspython/releases)
- [Changelog](https://github.com/rthalley/dnspython/blob/main/doc/whatsnew.rst)
- [Commits](https://github.com/rthalley/dnspython/compare/v2.8.0...v2.9.0)

---
updated-dependencies:
- dependency-name: dnspython
  dependency-version: 2.9.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Normann <github@koldrack.com>
2026-10-28 21:35:16 +01:00
NormannandGitHub 5d7198d090 test: consolidate suites and remove redundant cases (#1459)
Regression tests were scattered across feature- and bug-specific files, and some suites repeated the same checks. Group tests by responsibility, isolate feature fixtures in test classes, and remove duplicate or obsolete cases.

The review covered all 127 collected test modules on main. This reduces them and removes or combines definitions:.

 No application code changes.
2026-10-28 20:15:25 +01:00
NormannandGitHub 2863373b17 build(deps): combine FastAPI and three patch updates (#1467)
python-fasthtml 	0.14.13 → 0.14.14
pydantic-settings 	2.16.0 → 2.16.1
FastAPI 	0.143.3 → 0.162.0
Commitizen (dev) 	4.19.2 → 4.19.3
2026-10-28 20:06:41 +01:00
Wu ShuwenandGitHub 727f20fdf0 fix(homeassistant): resolve legacy device measurement keys
Resolve legacy underscore spellings for Home Assistant device measurements only when they uniquely match a registered EOS key; preserve canonical, custom, and ambiguous keys.
Correct the schema examples to show canonical hyphenated keys.
Add a synthetic regression test that exercises data acquisition without contacting Home Assistant.
2026-10-28 19:04:40 +01:00
NormannandBobby Noelte 63e7e7e729 fix: honor DST boundaries in appliance scheduling
Compare elapsed offsets when checking and merging appliance windows so
both occurrences of the autumn hour obey the configured boundaries.
Preserve calendar durations when finding the earliest valid start.

Add optimizer and calendar-day regressions, and consolidate the existing
time-window regression files into test_configabc classes.
2026-10-28 18:59:43 +01:00
NormannandBobby Noelte 7ca66c7dee fix: compare time window boundaries by elapsed time across DST 2026-10-28 18:59:43 +01:00
NormannandBobby Noelte d22628b312 fix: honor effective time window boundaries in cycle scheduling 2026-10-28 18:59:43 +01:00
Bobby Noelte c02d250639 feat: Add month and end_time to time window configuration
Extend TimeWindow with two new optional fields:

- month (1–12): restricts the window to a start date in the given
  calendar month. Useful for seasonal schedules.

- end_time: explicit exclusive end-of-window clock time. The effective
  end is the later of end_time and start_time + duration. Primary
  motivation is DST handling — a wall-clock end_time stays correct
  across DST transitions, whereas a fixed duration does not.

Rework window semantics so date, weekday, and month restrictions apply
to the window's start date, not to every timestamp covered by the
window. This makes overnight windows (e.g. Friday 22:00 → Saturday
06:00) behave intuitively under calendar constraints; contains() now
checks both today's and yesterday's candidate start.

duration becomes optional (default 0). All start/end/duration
computations route through _window_start_end and a new
_matches_start_date helper, so contains, earliest_start_time,
latest_start_time, and available_duration stay consistent.

Dashboard: add end_time and month inputs to the time-window editor,
render the collapsed window summary as icons + pills.

Docs, OpenAPI spec, and generated config docs are regenerated for the
new fields. Tests cover month filtering, end_time boundary cases (incl.
midnight and overnight), and start-date semantics for overnight windows.

Bump version to support automatic updates in Home Assistant.

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2026-10-28 18:59:43 +01:00
Bobby NoelteandGitHub 8a8cd43a19 chore: development version update (incl. home assistant config) (#1456) 2026-10-26 09:09:59 +01:00
sarankumarandGitHub 5ceb21819b fix(genetic0): keep parameter arrays aligned with the start hour on DST days (#1455) 2026-10-26 07:16:21 +01:00
YTM-21andGitHub 8f4fa6fe94 fix(config): report computed fields as read-only on update (#1454)
PUT /v1/config/{path} answered "does not exist in model" when the
target was a computed field such as eos_device_instruction_entity_ids.
Raise an explicit read-only error instead, so API clients can tell
read-only fields from unknown ones.

Closes #913
2026-10-26 07:16:16 +01:00
58b968562e fix(adapter): stamp Home Assistant device states with run time, not slot start (#1452)
* fix(adapter): stamp Home Assistant device states with run time, not slot start

The Home Assistant adapter stored every reading with ems_start_datetime,
which set_start_datetime() rounds down to the optimization slot (e.g.
14:00 for a run at 14:11). The GENETIC configuration request checks SoC
freshness against the unrounded observation_datetime with
measurement_max_age_seconds (default 300 s). Any run starting more than
five minutes into a slot therefore discarded the freshly read SoC and
optimized with 0 % initial SoC.

Stamp device state measurements (device_measurement_entity_ids, e.g.
SoC) with observation_datetime instead. Not to_datetime(): a reading
taken a few seconds after the observation time would be rejected as a
future sample. Meter readings (load, grid import/export, PV) keep the
slot start, because their energy is aggregated per slot.

With device states at their real time, a record holding only some keys
can now be the newest one (e.g. the Home Assistant battery SoC while the
EV SoC or a cycle count comes through the measurement API). Readers
therefore skip records without the value they need:

- GENETIC SoC and completed appliance cycles: such a record no longer
  hides a fresh value (the cycle count case aborted the optimization).
  This also covers a meter reading at the slot start that is newer than
  a SoC supplied through the measurement API. Values stored as pandas NA
  count as missing instead of raising TypeError; an explicit NaN stays
  an invalid sample.
- GENETIC0 SoC: looks up the reading nearest to observation_datetime
  instead of the slot start, skipping records without SoC and converting
  only the selected sample. Before, a run late in the slot (14:40 in the
  14:00 slot) would pick the previous run's reading (13:40). This also
  fixes SoC values PUT via the REST API with their real time.

LoadAkkudoktorAdjusted anchored its hourly comparison window on the
newest record of any key. An off-hour device state would shift every
bucket (e.g. to :11) against hour-aligned meter readings; the window now
starts and ends on full hours, and less than one full hour of data
yields no adjustment instead of an error. (15-minute slot starts and SoC values
PUT with their real time had the same effect before.)

Regenerated openapi.json and docs/_generated/openapi.md (version only).

Fixes #1447

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T

* test(genetic0): use pv_battery instead of deprecated pv_akku

mypy flags attribute access on the pv_akku computed field
(attr-defined), which failed the pre-commit check of the PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 07:14:00 +01:00
NormannandGitHub 5249b389f5 build(deps): combine pypdf and certifi updates (#1451) 2026-10-26 07:12:46 +01:00
NormannandGitHub 5a0763deec build(deps): combine four Dependabot updates and refresh uv.lock (#1444)
Consolidates four currently open Dependabot updates. Only pyproject.toml and uv.lock change.
Included sources
Source 	Dependency 	Change
#1436 	platformdirs 	4.13.3 → 4.13.5
#1437 	Pillow (via matplotlib and bokeh) 	12.3.0 → 13.0.0
#1442 	filelock (development tools) 	3.32.6 → 4.2.1
#1443 	virtualenv (via pre-commit) 	21.7.13 → 21.14.13
2026-10-25 06:13:10 +01:00
sarankumarandGitHub cdbaae45ba fix(elecprice): handle mixed UTC offsets in Tibber prices around DST (#1448)
Once Tibber publishes prices across a DST change, the series holds
timestamps with two UTC offsets and pd.to_datetime() raises "Mixed
timezones detected", so every optimization run is canceled. Parse via
UTC and convert back to the configured timezone.

Fixes #1446
2026-10-25 06:10:48 +01:00
NormannandGitHub baafe570ff build(deps): combine seven Dependabot updates and synchronize uv.lock (#1433)
* build(deps): combine five Dependabot updates

pydantic-settings 2.15.0 → 2.16.0
FastAPI 0.143.0 → 0.143.3
GitPython 3.2.0 → 3.2.1
platformdirs 4.13.0 → 4.13.3
pypdf 6.20.0 → 6.21.1

* build(deps): add Pendulum and pre-commit updates

Pendulum 3.2.0 → 3.3.0
pre-commit 4.6.2 → 4.7.0
2026-10-22 18:13:31 +02:00
Bobby NoelteandGitHub 024706b467 fix: use of inverter config dict in genetic0 (#1432)
Adapt GENETIC0 to inverters configuration in dict - used to be a list.

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
2026-10-20 17:47:00 +02:00
AndreasandGitHub 46a89a6f39 fix(devices): let EVs finish charging within the final slot (#1423) 2026-10-20 12:05:44 +02:00