mirror of
https://github.com/Akkudoktor-EOS/EOS.git
synced 2026-11-13 09:13:30 +01:00
d68f04020647dc5e492d7c3cdaf0936ebdde399b
119
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Bobby NoelteandClaude Opus 5.5
|
a0e57a18b7 |
fix(config): validate prediction providers before the prediction exists
The provider settings (weather, PV forecast, load, electricity price, feed-in tariff, electricity fee) validate their provider id against the providers of the prediction instance. While the configuration is created, that instance does not exist yet, and the validation falls back to fixed lists. These were incomplete: weather knew only 'WeatherImport', load lacked 'LoadAkkudoktorAdjusted'. A configuration file with e.g. weather provider 'OpenMeteo' then fails to load: Provider 'OpenMeteo' is not a valid weather provider: ['WeatherImport'] This is hidden today because ConfigEOS.documentation_mode() is always true (_force_documentation_mode is a pydantic private attribute, whose class attribute is a truthy descriptor), and get_prediction() then creates the prediction instance on access. Once documentation mode is enabled only explicitly, the configuration does not load. Derive the provider ids from the provider classes instead (prediction_provider_ids). They no longer depend on whether the prediction instance exists, so the validation is the same before and after initialization. Additional changes - Add the test fixture uncreated_config, which lets get_config(init=True) create a new configuration from the config file and restores the old one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
 
|
628226d3d9 |
feat(pvforecast): model snow on the modules in the local backend (#1484)
No PV forecast modelled snow lying on the modules, so after a snowfall the optimizer planned with PV that never came. The local Open-Meteo/pvlib backend can now reduce the forecast by the snow coverage (opt-in via pvforecast.akkudoktor.snow_enabled): - Request hourly snowfall and snow depth with the weather and run the NREL snow coverage model (pvlib.snow) per plane. - Correct the coverage with measured PV production: production above the model clears it at once; low production raises it only while snow is plausible, so curtailment and inverter outages are not learned as snow. Persisting snow damps the slide rate, fading with snow_feedback_memory_hours. - Rebuild the coverage from snow_lookback_days on every update instead of storing state. - Exclude intervals with possible snow from the calibration fit. - Warn when PV meter readings are missing or stale; ignore impossible meter jumps. _plane_power keeps its two-value signature (used by the adaptive provider in #1422); the irradiance comes from the new _plane_power_and_poa. Tests replay real February 2026 data from one plant (snow on 02-19/20). Signed-off-by: SchaafAlexander <86607129+SchaafAlexander@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
 NormannandGitHub
|
b33953d445 |
fix(pvforecast): normalize Forecast.Solar power before resampling (#1487)
Forecast.Solar supplies instantaneous power with irregular sunrise/sunset timestamps. GENETIC interprets stored values as interval averages and infers cadence from the shortest gap, so these samples create artificial coverage gaps. The prediction list API also averages away the sunset zero before interpolating, producing positive nighttime PV. Integrate each roof plane's piecewise-linear power curve into UTC-contiguous 15-minute interval-average W before summing planes. Both hourly and quarter-hour GENETIC slots then consume the same normalized data as the prediction API, with exactly one downstream W-to-Wh conversion. Explicit sunrise/sunset zero boundaries establish the adjacent night within each forecast day. Daylight gaps longer than an hour, invalid power, absent planes and unavailable forecast days remain missing; the strict GENETIC coverage check stays in place. Refreshes replace the current forecast and remove stale tails. Because cadence detection also sees historical records, the first refresh clears legacy forecasts containing non-aligned timestamps, including their history. Later refreshes retain normalized history. This upgrade behavior is documented. Forecast validation errors now identify the first invalid interval and the number of invalid slots. |
||
   
|
f804d48072 |
feat(pvforecast): add adaptive per-plane PV learning and diagnostics (#1422)
PV arrays with different orientations and curtailment need separate plant calibration and weather correction. This opt-in provider publishes normal EOS AC/DC forecast keys at 15-minute resolution. - Preserve daily per-plane energy/shape calibration and validated short-term local/trend selection. Curtailment-sensitive arrays require a fresh release flag or an explicitly configured learning guard. - Add switchable empirical solar-angle response relative to separate eligible reference arrays, removing their common weather error. This learns angular residuals, not physical angles or a measured geometric horizon. - Archive actually received weather model inputs, first local receipt and frozen per-member PV candidates. Returned past weather is excluded; unknown source issuance remains explicit. Learn model weights and forecast-regime bias separately at 2–6, 6–24 and 24–48 hours. - New long-horizon candidates remain in shadow until their genuinely issued outputs improve equal-day MAE by the configured margin, win on at least 60% of validation days and pass a matched-support energy-bias guard. Insufficient evidence retains calibration. Daily yields still require complete local days, including DST. - Expose fitted parameters, independent training/validation day counts, errors, selected methods and persistence in the existing status API, with additional normal prediction keys for Grafana. Retain physical DC through angular/weather corrections an enforce individual inverter AC limits. Co-authored-by: Normann <github@koldrack.com> Co-authored-by: Christin <chrissi@von-perbandt.de> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: YMT <yn276364@dal.ca> Co-authored-by: sarankumar1325 <120080287+sarankumar1325@users.noreply.github.com> |
||
|
NormannandGitHub
|
8396bf24ae |
fix(deps): upgrade pandas to 3.1 and fix EOS deprecations (#1469)
* build(deps): bump pandas from 3.0.6 to 3.1.0 * fix(prediction): avoid pandas 3.1 deprecation warnings |
||
|
|
58b968562e |
fix(adapter): stamp Home Assistant device states with run time, not slot start (#1452)
* fix(adapter): stamp Home Assistant device states with run time, not slot start The Home Assistant adapter stored every reading with ems_start_datetime, which set_start_datetime() rounds down to the optimization slot (e.g. 14:00 for a run at 14:11). The GENETIC configuration request checks SoC freshness against the unrounded observation_datetime with measurement_max_age_seconds (default 300 s). Any run starting more than five minutes into a slot therefore discarded the freshly read SoC and optimized with 0 % initial SoC. Stamp device state measurements (device_measurement_entity_ids, e.g. SoC) with observation_datetime instead. Not to_datetime(): a reading taken a few seconds after the observation time would be rejected as a future sample. Meter readings (load, grid import/export, PV) keep the slot start, because their energy is aggregated per slot. With device states at their real time, a record holding only some keys can now be the newest one (e.g. the Home Assistant battery SoC while the EV SoC or a cycle count comes through the measurement API). Readers therefore skip records without the value they need: - GENETIC SoC and completed appliance cycles: such a record no longer hides a fresh value (the cycle count case aborted the optimization). This also covers a meter reading at the slot start that is newer than a SoC supplied through the measurement API. Values stored as pandas NA count as missing instead of raising TypeError; an explicit NaN stays an invalid sample. - GENETIC0 SoC: looks up the reading nearest to observation_datetime instead of the slot start, skipping records without SoC and converting only the selected sample. Before, a run late in the slot (14:40 in the 14:00 slot) would pick the previous run's reading (13:40). This also fixes SoC values PUT via the REST API with their real time. LoadAkkudoktorAdjusted anchored its hourly comparison window on the newest record of any key. An off-hour device state would shift every bucket (e.g. to :11) against hour-aligned meter readings; the window now starts and ends on full hours, and less than one full hour of data yields no adjustment instead of an error. (15-minute slot starts and SoC values PUT with their real time had the same effect before.) Regenerated openapi.json and docs/_generated/openapi.md (version only). Fixes #1447 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * test(genetic0): use pv_battery instead of deprecated pv_akku mypy flags attribute access on the pv_akku computed field (attr-defined), which failed the pre-commit check of the PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
sarankumarandGitHub
|
cdbaae45ba |
fix(elecprice): handle mixed UTC offsets in Tibber prices around DST (#1448)
Once Tibber publishes prices across a DST change, the series holds timestamps with two UTC offsets and pd.to_datetime() raises "Mixed timezones detected", so every optimization run is canceled. Parse via UTC and convert back to the configured timezone. Fixes #1446 |
||
 
|
7dc1ee92fa |
perf(genetic): faster runs with unchanged results (#1385)
* feat(devices): add load-dependent DC-to-AC efficiency curve for GENETIC inverters Real inverters are much less efficient at low load than at rated power (e.g. 0.80 at 2 % vs. 0.93 at 7 % load). A constant dc_to_ac_efficiency overrates night-time battery discharge into a small base load and, with it, arbitrage and grid charging. Add two optional inverter settings for the GENETIC optimizer: - dc_to_ac_efficiency_curve: list of (load_fraction, efficiency) points, load fraction = AC power / max_power_w. Points must be strictly increasing in [0, 1] with efficiencies in (0, 1]; at least two points. Linear interpolation, clamped at both ends. When set, the curve replaces dc_to_ac_efficiency. - dc_to_ac_efficiency_reference_load_fraction (default 0.06): load at which the curve values stored energy without a specific conversion. Battery discharges in Inverter.process_energy() use the efficiency at the AC energy of each conversion. The end-of-horizon value, the terminal and tail value curves and the AC charge break-even penalty use Inverter.reference_dc_to_ac_efficiency. Without a curve, dc_to_ac_efficiency_at() and the reference efficiency return dc_to_ac_efficiency itself, so results are bit-identical to the previous model. GENETIC0 is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): make the fitness cache size configurable The per-run fitness cache of the GENETIC optimizer is unbounded. It grows with individuals x generations (~110k keys at 300 x 400, about 1.1 kB each, 1.6 kB with an EV) and adds 120-170 MB to the peak memory of a run, while only 6-15 % of the lookups hit. On 512 MB boards this is the largest part of the optimizer's memory. Add optimization.genetic.fitness_cache_max_entries: - None (default): unbounded, the previous behaviour. - 0: no cache; every genome is evaluated. - N > 0: at most N keys. The cache is then an OrderedDict and evicts the oldest key first (FIFO, O(1)); LRU would cost bookkeeping on every hit for a cache that rarely hits. A cache hit returns exactly what the deterministic evaluation would return, so the plan for a fixed seed is identical with any setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf: import heavy optional dependencies on first use A GENETIC server with external (import) providers and EOSdash disabled loads several large packages at start that it never uses, which costs a noticeable part of the memory on 512 MB boards: - matplotlib via the GENETIC/GENETIC0 PDF report modules (server/eos.py) - statsmodels via the Holt-Winters fallback in prediction/priceabc.py - pvlib via the weather base class and the pvlib-based providers - bs4/lxml via the ClearOutside provider - scipy via the self-consumption interpolator singletons, which were loaded at import time - the GENETIC0 optimizer, imported by core/ems.py - tzfpy, whose polygon data the computed general.timezone loads Import these where they are used. The interpolator tables are loaded on first use (thread-safe). Nothing changes in what is computed. general.timezone is computed from latitude/longitude and cannot be set. Add general.timezone_override (IANA name, validated with pendulum): when set, general.timezone returns it and the lookup is skipped. When unset (default) the lookup is unchanged. Tests that patched the moved names now patch them at their source. A new test starts the server module in a subprocess and checks that none of these modules is loaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(prediction): evaluate self-consumption tables without SciPy The GENETIC inverter (and the GENETIC0 one) evaluates a pickled scipy.interpolate.RegularGridInterpolator for the self-consumption probabilities. SciPy is used nowhere else in EOS, yet loading it costs about 35 MB RSS (21 MB anonymous) on Linux arm64, which every optimization run pays. Add utils/gridinterpolator.py: - LinearGridInterpolator: 2-D linear interpolation with the same steps as SciPy's linear 2-D path (interval search, normalised distances, corner weights summed in the same order, fill value outside the grid, NaN for NaN input). - load_grid_interpolator(): reads the shipped pickles with a restricted unpickler that maps the SciPy class to a stand-in and accepts only the NumPy globals needed for the arrays. SciPy is not imported, and the data files no longer depend on SciPy's private module paths. Results are bit-identical to SciPy on Linux (aarch64, checked on 3.6M points per table including the query pattern of the inverter). The macOS arm64 SciPy wheel fuses multiply-add, so there SciPy itself differs by <= 1.1e-16; GENETIC plans for fixed seeds are unchanged on both platforms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(prediction): make the SciPy-free table evaluation opt-in Add optimization.self_consumption_interpolator (scipy | numpy, default scipy). The default keeps evaluating the self-consumption tables with SciPy's RegularGridInterpolator, built from the same grid, values and fill value the pickle holds, so results are unchanged. With numpy the tables are evaluated by LinearGridInterpolator and SciPy is never imported. The backend is read on every evaluation, so a config change applies to the next run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(genetic): cache the load distribution, clone genomes directly Two hot spots of a GENETIC run, found by profiling a 15 minute run with EV and two appliances. The solution is unchanged: both repository reference inputs give a byte-identical GeneticSolution at 400 generations. - SelfConsumptionProbabilityInterpolator._load_distribution depends on the mean load only, but ran the table interpolation over all minute-load bins for every new (load, PV) pair - 27 % of the run. It is now kept per (backend, bounded mean load); a run asks for few loads (one per slot, plus appliance combinations) with ever-changing PV values. The table is static, so entries never go stale; the cache is bounded. Scalar bounds use min/max instead of np.clip. - _make_offspring cloned parents with copy.deepcopy (8 % of the run), although the child's fitness, extra_data and immigrant protection are dropped right after. A genome is a flat list of ints, so the child is created from the parent's genes directly. Measured (1 process, Apple M5): eHive run with EV + 2 appliances, 200 generations 33.7 s -> 25.1 s; EV only 34.1 s -> 24.3 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): cheaper slot simulation, lean lookup cache, batch duplicates Four hot spots of a GENETIC run, found by profiling a 15 minute run with EV and two appliances. The solution is unchanged: both repository reference inputs (optimize_input_1/2, seed 42, 400 generations) give a byte-identical solution before and after, as do the fixed-seed runs with 1, 2 and 3 evaluation processes. * Expected direct consumption: the lookup went through the shared cache_energy_management store on every slot of every candidate - a key built from the callable, a callback per call and room for the last 100 results only (18 % misses). It is now a dictionary of the interpolator, dropped together with that store at the start of an energy management run (CacheEnergyManagementStore.generation), bounded to 8192 entries. * Slot simulation: per-slot inputs are read as plain Python numbers and per-slot results are collected in lists, turned into arrays after the loop. Element access on NumPy arrays and NumPy scalar arithmetic were the largest share of the loop. process_energy() skips the charge/export block when a slot has no PV surplus and bounds values without min()/max() calls (exact equivalents, NaN and signed zero included). * Genome keys: bytes() does the 0-255 range check in C, and a genome that is relevant as a whole is packed without copying it first. * Identical genomes within a generation are evaluated once also when the fitness cache is off (fitness_cache_max_entries = 0), by grouping the batch. It needs no memory beyond the batch; the cache costs 85-95 MB. Same run, one process, 60 generations (Apple M-series): 7.8 s -> 5.9 s with appliances, 7.1 s -> 5.0 s without. 200 generations with 3 evaluation processes: 11.4 s -> 8.2 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): keep the battery charge bookkeeping in plain numbers Battery.charge_energy() runs for every slot of every candidate plan. The per-slot charge limit and charged energy were NumPy arrays read and written one element at a time, and the NumPy scalars they return made the state of charge arithmetic slow from then on. They are plain lists now. The lower charge rates tried when a request does not fit are taken from a tuple built once per charge_rates array instead of a searchsorted() plus generator per request. The results are unchanged (same IEEE doubles, same order of operations). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): check the discharge release first, plain discharge bookkeeping Inverter._discharge_battery_to_ac() looked up the conversion efficiency before the battery told it that the plan does not release discharge in this slot. Battery.discharge_released() answers that first; the result of such a slot was (0.0, 0.0) before and is now returned directly. The discharged energy per slot is a plain list instead of a NumPy array read and written one element at a time. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): find the efficiency curve segment by bisection Inverter.dc_to_ac_efficiency_at() runs for every battery discharge of every candidate plan and walked the curve points one by one. The load fractions are split off once per curve and the segment is found by bisection; the segment and the interpolation are those of interpolate_efficiency_curve(). Compared bit for bit against that function for 200000 load values, the curve points themselves, zero, infinities and NaN. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): sum appliance loads once per candidate, read the SoC directly The slot loop asked every appliance for its load in every slot (one method call with a range check each) and converted the state of charge to percent through a method call twice per slot. The aggregate appliance load is now summed once per candidate, in the order the loop added the appliances, and the state of charge is read with the Battery formula in place. Appliances of another class or with a curve that does not cover the simulated slots, and batteries with their own current_soc_percentage(), keep the calls. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): compute the control end slot once per run control_end_slot is a property that reads the interval and the horizon from the configuration on every access - several hundred thousand times per run. The configuration is frozen for the duration of a run, so the value is computed once when the snapshot is taken and dropped with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): work out the energy-shift targets once per source slot _energy_shift_target_slots() runs for every export slot of every energy-shift mutation, in the server process while the evaluation workers wait. Which later slots are worth the shift, and in which order, depends on the forecasts and the source slot only; it is now worked out once per source slot and run, and the individual only filters the ordered slots. Filtering the sorted superset gives the same list as sorting the filtered slots (the sort is stable and its key does not depend on the individual), so mutations and the random number stream are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): compute the battery discharge of a slot in one go Serving load or exporting from the battery went through three nested calls per slot: Inverter._discharge_battery_to_ac() -> dc_to_ac_efficiency_at() -> Battery.discharge_energy(), with the release of the slot checked twice and the efficiency function called also without a curve. For the plain Battery the inverter now computes the release check, the efficiency and the discharge in one go; other battery implementations keep the step-by-step path. The arithmetic is that of Battery.discharge_energy(). The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): decode appliance starts once per gene combination Every candidate decoded its appliance start genes (feasibility repair over cycle windows and minimum gaps) and rebuilt each appliance's load curve over the whole horizon. The decoded starts, the canonical gene values and the curves depend on the appliance genes only, and a run sees few different combinations among tens of thousands of candidates. Each combination is now decoded and built once per layout (bounded to 4096 combinations) and looked up afterwards. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): compute only the terminal credit per candidate Every evaluation called _terminal_value(), which returns the credit for the energy left in the battery together with a TerminalValueResult report (component values, diagnostics, curves). The evaluation used the credit and dropped the report. _terminal_value_credit() computes the same credit without building the report; the report is still built for the final solution. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): decode battery states through a lookup table decode_charge_discharge() decodes every slot on its own from its state, with about twenty array operations per candidate. The four results per state are now worked out once per state layout and looked up for the slots of a candidate. States outside the table keep the explicit decoding. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): check for a full EV before splitting the genome _repair_ev_charge_at_full_soc() split every candidate's genome and walked all slots in Python to find EV charging in slots that begin at full SoC. Nearly no candidate has such a slot. The full slots are now found on the simulated SoC array first; the genome is split and only those slots are looked at when there are any. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * feat(genetic): evaluate candidates in parallel worker processes A GENETIC run evaluated every candidate in the server process, so a run used one CPU core however many the machine had. On small ARM boards (4x Cortex-A55, thermally limited) a 300x200 run took 12-15 minutes. Candidates are now evaluated in up to two worker processes, forked once per run after the run is fully prepared. Each worker owns a copy of the prepared optimizer; only genomes and results cross the process boundary. The fitness cache stays in the server process: hits are answered there, and identical genomes in a batch are evaluated once, as a serial run would hit the cache for the repeats. Results are applied only after the whole batch returned. If a batch fails or times out, the run continues serially. For a fixed seed the result is identical to a serial run. optimization.genetic.workers sets the number of processes. Unset (default) is automatic: one CPU core always stays free and at most two are used (1-2 cores -> 1, 3 or more -> 2), capped by a cgroup v2 CPU limit of the process or any parent (container limit, systemd CPUQuota). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * feat(ems): notify a URL after every completed optimization A client that consumes EOS plans (here DVhub) had to poll /v1/energy-management/optimization/solution to notice a new plan. That delays the plan by up to the poll interval and makes EOS serialize the complete solution on every poll, also while it is computing. ems.notify_url (default None = off) receives an HTTP POST with a small JSON event ({"event": "optimization_completed", "algorithm", "completed_at", "duration_s"}) when a run has published its solution. The POST runs in a daemon thread with a 5 s timeout, so a slow or unreachable receiver never delays energy management; a failure is logged once per failure streak. Only http(s) URLs are accepted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(genetic): fewer calls per slot in the simulation loop Three calls ran for every slot of every candidate although most slots do not need them: the AC charge factor was capped by the inverter also when the slot has no AC charging, the battery was asked for its discharged energy through a method, and process_energy() got its export arguments by keyword. Slots without AC charging skip the cap, the discharged energy is read with the Battery formula in place, and the arguments are positional. reset() of a battery creates its two control arrays with zeros() instead of full(). The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(genetic): freeze the optimization configuration for the duration of a run The genetic optimization reads horizon_hours, tail_horizon_hours, interval_sec and prediction.hours from the live configuration on every access (control_slots, tail_slots, ... are properties). A client that updates the configuration while a run is in progress - DVhub shrinks or grows the control horizon as the known price horizon moves during the night - changed these values mid-run. Genomes and device arrays sized at the start then no longer matched the simulation loop and the final evaluation failed: IndexError: index 84 is out of bounds for axis 0 with size 84 Long runs on slow hardware hit this almost every night. optimize_ems() now takes a deep copy of the genetic configuration and the prediction hours at the start of a run, all reads in genetic.py go through it, and it is released when the run ends (also on errors). New values apply to the next run. Tests: tests/test_genetic_config_frozen_per_run.py reproduces the error without the fix (horizon 20 -> 24 mid-run: index 20 out of bounds) and passes with it; genetic/optimization tests 439 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(genetic): pin evaluation workers, keep the server responsive Workers are pinned to the last `workers` CPU cores (new setting optimization.genetic.pin_workers, default on); the server process stays unpinned. The run never spreads beyond `workers` cores without a CPU quota, and the server and the rest of the system keep the other cores. No pinning without a spare core. With a CPU limit (container cpu.max, systemd CPUQuota) the automatic worker count now keeps a quarter core of it for the server process (limit 2 -> 1 worker, 2.5 -> 2): two workers used a 2-core limit fully. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(ems): tolerate settings without notify_url Test doubles and older configuration objects have no ems.notify_url. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * perf(devices): bound the PV surplus and export values without min()/max() The rest of Inverter.process_energy() - PV surplus, curtailment and battery export - bounded its values with min()/max() calls. They are conditional statements now, as the first half of the method already is: 'if x < 0.0: x = 0.0' is max(x, 0.0) and 'b if b < a else a' is min(a, b), NaN and signed zero included. The results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T * fix(devices): evaluate the efficiency curve at the delivered AC load When the battery limits a discharge (low state of charge or its power limit), the conversion ran with the efficiency of the requested AC energy. With a 1000 Wh battery at 10 % SoC, a 1000 W inverter and the curve [(0, .8), (.5, .95), (1, .95)], a request of 500 Wh AC returned 95 Wh AC from the available 100 Wh DC, although the curve gives 82.5 % at that load. The delivered AC energy now solves a = dc * efficiency(a / rated) exactly on the piecewise-linear curve (solve_ac_for_dc_energy), bounded by the request. Unlimited discharges and inverters without a curve are unchanged. Regression tests use a real Battery: low SoC and a power-limited discharge, each with 60- and 15-minute slots, battery discharge losses, the load path through process_energy, and constant efficiency. * docs: regenerate OpenAPI for the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix * docs: regenerate OpenAPI after merging the efficiency-curve fix --------- Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Andreas <drbacke@gmx.de> |
||
|
|
560370281f |
perf(prediction): optional SciPy-free evaluation of the self-consumption tables (#1378)
* perf(prediction): evaluate self-consumption tables without SciPy The GENETIC inverter (and the GENETIC0 one) evaluates a pickled scipy.interpolate.RegularGridInterpolator for the self-consumption probabilities. SciPy is used nowhere else in EOS, yet loading it costs about 35 MB RSS (21 MB anonymous) on Linux arm64, which every optimization run pays. Add utils/gridinterpolator.py: - LinearGridInterpolator: 2-D linear interpolation with the same steps as SciPy's linear 2-D path (interval search, normalised distances, corner weights summed in the same order, fill value outside the grid, NaN for NaN input). - load_grid_interpolator(): reads the shipped pickles with a restricted unpickler that maps the SciPy class to a stand-in and accepts only the NumPy globals needed for the arrays. SciPy is not imported, and the data files no longer depend on SciPy's private module paths. Results are bit-identical to SciPy on Linux (aarch64, checked on 3.6M points per table including the query pattern of the inverter). The macOS arm64 SciPy wheel fuses multiply-add, so there SciPy itself differs by <= 1.1e-16; GENETIC plans for fixed seeds are unchanged on both platforms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(prediction): make the SciPy-free table evaluation opt-in Add optimization.self_consumption_interpolator (scipy | numpy, default scipy). The default keeps evaluating the self-consumption tables with SciPy's RegularGridInterpolator, built from the same grid, values and fill value the pickle holds, so results are unchanged. With numpy the tables are evaluated by LinearGridInterpolator and SciPy is never imported. The backend is read on every evaluation, so a config change applies to the next run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Andreas <drbacke@gmx.de> |
||
|
|
3651d33915 |
perf: import heavy optional dependencies on first use (#1377)
A GENETIC server with external (import) providers and EOSdash disabled loads several large packages at start that it never uses, which costs a noticeable part of the memory on 512 MB boards: - matplotlib via the GENETIC/GENETIC0 PDF report modules (server/eos.py) - statsmodels via the Holt-Winters fallback in prediction/priceabc.py - pvlib via the weather base class and the pvlib-based providers - bs4/lxml via the ClearOutside provider - scipy via the self-consumption interpolator singletons, which were loaded at import time - the GENETIC0 optimizer, imported by core/ems.py - tzfpy, whose polygon data the computed general.timezone loads Import these where they are used. The interpolator tables are loaded on first use (thread-safe). Nothing changes in what is computed. general.timezone is computed from latitude/longitude and cannot be set. Add general.timezone_override (IANA name, validated with pendulum): when set, general.timezone returns it and the lookup is skipped. When unset (default) the lookup is unchanged. Tests that patched the moved names now patch them at their source. A new test starts the server module in a subprocess and checks that none of these modules is loaded. Co-authored-by: Christin <christin@BigBrother.bikini.bottom.zone> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
NormannandGitHub
|
b7610e8274 |
fix: detect recent Energy-Charts cadence for feed-in coverage (#1362)
* fix: use recent original feed-in cadence for publication coverage * test: cover mixed Energy-Charts feed-in source cadence |
||
|
NormannandGitHub
|
e203a81666 |
fix: surface DVhub feed-in tariff update failures (#1357)
* fix: surface DVhub feed-in tariff update failures * test: annotate DVhub sample records for mypy * fix: require real feed-in tariff coverage in GENETIC0 * test: satisfy GENETIC0 tariff coverage checks * test: patch DVhub storage method on its class |
||
 
|
04f28997ea |
feat: deliver complete GENETIC optimization to main (#1330)
* feat: adapt configuration for multi optimization algorithms Decouple configuration from optimization algorithm parameters. Add to_[algorithm]_param() methods to the configuration that derive optimization algorithm specific parameters from the configuration. Add x-scope tags to the configuration options that describe for which specific algorithms the configuration option is for. The whole device settings are restructured. There are now general settings for the device classes with the afore mentioned to_[algorithm]_param() methods. The general device settings got their own directory `devices/settings`. By this the parameter class also does not have to be a pydantic model which can be used for future optimization/ simulations speed up. Also the parameter class for a device is now part of the device module. This better decouples and also is the natural place for parameters of a device. Besides this feature there are also fixes and improvements: * feat: extend home appliance time window settings and simulation Home appliance can now be configured for multiple runs with per-cycle allowed time windows. The number of remaining cycles to plan is determined at runtime by reading the ``cycles_completed_measurement_key`` from the measurement store. * feat: specialiced CycleTimeWindowSequence for time window sequences Sequence of time windows associated to cycles. This model specializes ``ValueTimeWindowSequence`` so that the ``value`` field of each ``ValueTimeWindow`` encodes the **cycle index** (0-based integer) the window belongs to. Typical use: an appliance that must run ``n`` times per day, each run constrained to a distinct time window. Assign ``value=0`` to windows for the first cycle, ``value=1`` for the second, and so on. Multiple windows may share the same cycle index (their allowed regions are unioned). Windows with ``value=None`` are silently ignored by all cycle-aware methods. * fix: Make test_configmigrate also regard the _ANY_SENTENIEL in key values * chore: Make devices configurations a map instead of a list This makes config paths stable regardless of declaration order and lets each device settings class build its own config path from ``self.device_id`` without needing an external index. Tests are adapted likewise. Devices configurations are automatically migrated from lists to maps. * chore: rename levelized_cost_of_storage_kwh to levelized_cost_of_storage_amt kwh This better fits in the naming scheme and also makes clear the costs are money. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> * fix: runtime config update ignored by config file Runtime settings were handed back to pydantic-settings as init settings, which rank below the config file and the environment. Any key already present in EOS.config.json or in the environment silently discarded the update, so a bulk PUT /v1/config returned 200 without applying anything, while the granular PUT /v1/config/{path} endpoint kept working. Add a dedicated runtime settings source ranked directly below the command line arguments and record granular updates there as well, so both endpoints share one store that survives re-evaluation of the settings sources. Environment variables keep precedence over the config file for all keys that were not set at runtime. Also repairs revert_settings() and update(), which passed their data through the same init settings. Closes #1303 * fix: env vars ignored on first config build ConfigEOS.__init__ passed self as first positional argument to _setup, which forwards it to pydantic_settings.BaseSettings.__init__. Its first positional parameter is _case_sensitive, so the environment source matched the upper case variable names against the lower case field names and returned nothing. Environment settings only took effect after the next configuration setup. * docs: changelog for config priority fixes * fix(config): preserve device identities and storage costs during migration * fix(measurement): restore JSON records into the existing singleton * fix(devices): preserve charge-rate typing and public import compatibility * feat(measurement): integrate typed energy quality and capacity APIs Port the locally backed-up measurement extensions to main async storage and PR #1256 device maps. Preserve runtime capacity estimates across #1305 bulk updates. Confirm JSON singleton restore defect on unchanged main and add regression. No production configuration or measurements included. Co-authored-by: Andreas <drbacke@gmx.de> * docs(measurement): describe household settings and consolidate regression coverage * docs(measurement): regenerate configuration and API contracts * test(measurement): isolate capacity database state between tests * ruff format fix * fix(measurement): restore JSON records into the existing singleton * test(measurement): assert restored timestamps before timezone conversion * test(measurement): assert restored timestamps before timezone conversion * fix: preserve imported feed-in revenue during parameter preparation Cancel GENETIC preparation when imported revenue cannot be read or contains invalid values, preserving the chosen provider instead of replacing it with demo tariffs. Keep valid positive, zero and negative amount/Wh series unchanged. Adapt the revenue-preservation regressions from PRs #1224 and #1304 to the async main API, including real timestamped imports and simulation repricing. The feature-only direct-marketing override remains outside this main fix. Co-authored-by: Christin <info@bikinibottom.capital> Co-authored-by: Normann <github@koldrack.com> * feat(devices): port slot-aware battery export and direct-use physics Port scoped device changes from |
||
|
|
3c862543a1 |
feat(devices): add bounded slot physics for GENETIC (#1327)
* feat: adapt configuration for multi optimization algorithms
Decouple configuration from optimization algorithm parameters. Add to_[algorithm]_param() methods
to the configuration that derive optimization algorithm specific parameters from the configuration.
Add x-scope tags to the configuration options that describe for which specific algorithms the
configuration option is for.
The whole device settings are restructured. There are now general settings for the device classes
with the afore mentioned to_[algorithm]_param() methods. The general device settings got their own
directory `devices/settings`. By this the parameter class also does not have to be a pydantic model
which can be used for future optimization/ simulations speed up.
Also the parameter class for a device is now part of the device module. This better decouples and
also is the natural place for parameters of a device.
Besides this feature there are also fixes and improvements:
* feat: extend home appliance time window settings and simulation
Home appliance can now be configured for multiple runs with per-cycle allowed time windows. The
number of remaining cycles to plan is determined at runtime by reading the
``cycles_completed_measurement_key`` from the measurement store.
* feat: specialiced CycleTimeWindowSequence for time window sequences
Sequence of time windows associated to cycles.
This model specializes ``ValueTimeWindowSequence`` so that the ``value``
field of each ``ValueTimeWindow`` encodes the **cycle index** (0-based
integer) the window belongs to.
Typical use: an appliance that must run ``n`` times per day, each run
constrained to a distinct time window. Assign ``value=0`` to windows
for the first cycle, ``value=1`` for the second, and so on. Multiple
windows may share the same cycle index (their allowed regions are unioned).
Windows with ``value=None`` are silently ignored by all cycle-aware methods.
* fix: Make test_configmigrate also regard the _ANY_SENTENIEL in key values
* chore: Make devices configurations a map instead of a list
This makes config paths stable regardless of declaration order and lets each device settings
class build its own config path from ``self.device_id`` without needing an external index.
Tests are adapted likewise.
Devices configurations are automatically migrated from lists to maps.
* chore: rename levelized_cost_of_storage_kwh to levelized_cost_of_storage_amt kwh
This better fits in the naming scheme and also makes clear the costs are money.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* fix: runtime config update ignored by config file
Runtime settings were handed back to pydantic-settings as init settings,
which rank below the config file and the environment. Any key already
present in EOS.config.json or in the environment silently discarded the
update, so a bulk PUT /v1/config returned 200 without applying anything,
while the granular PUT /v1/config/{path} endpoint kept working.
Add a dedicated runtime settings source ranked directly below the command
line arguments and record granular updates there as well, so both
endpoints share one store that survives re-evaluation of the settings
sources. Environment variables keep precedence over the config file for
all keys that were not set at runtime.
Also repairs revert_settings() and update(), which passed their data
through the same init settings.
Closes #1303
* fix: env vars ignored on first config build
ConfigEOS.__init__ passed self as first positional argument to _setup,
which forwards it to pydantic_settings.BaseSettings.__init__. Its first
positional parameter is _case_sensitive, so the environment source
matched the upper case variable names against the lower case field names
and returned nothing. Environment settings only took effect after the
next configuration setup.
* docs: changelog for config priority fixes
* fix(config): preserve device identities and storage costs during migration
* fix(devices): preserve charge-rate typing and public import compatibility
* ruff format fix
* feat(devices): port slot-aware battery export and direct-use physics
Port scoped device changes from
|
||
|
AndreasandAndreas
|
5a3a3bf07e | style(pvforecast): apply CI import formatting | ||
|
AndreasandAndreas
|
6529c39df4 |
feat(pvforecast): add calibrated local Akkudoktor backend
Port local PV modeling and outage calibration from feature commits |
||
 dr-dimitriandGitHub
|
7ebe6d714b |
fix: detect recent Energy-Charts source cadence (#1315)
Infer interval coverage from consistent recent original price spacings while preserving history and forecasting resolution policies. Reuse source data until a successful refresh and cover cadence transitions, fallback, and history repair. Fixes #1279 |
||
|
|
1abdd345c4 |
fix: unify mypy environments for local checks and CI (#1291)
The isolated pre-commit mypy hook previously omitted runtime type information that make mypy used, hiding errors involving dependencies such as Pydantic and Pendulum. Makefile, pre-commit and CI now run the same full-project typing policy in the development environment defined by uv.lock. - Use uv run --locked --exact --extra dev and the same mypy arguments for Makefile and the local hook. Check all of src and tests, including on configuration-only changes. - Pin Python 3.13 for local development and the pre-commit CI job, and install the locked pre-commit version in CI. - Disable incremental analysis because existing Pendulum cache state changes mypy 2.3.1 diagnostics. Document the policy, the performance tradeoff and the existing typing debt. - Add a regression test that exercises Makefile, the hook and the CI command in a temporary project, accepting valid dependency types and detecting deliberate Pydantic/Pendulum assignment errors. Resolve the newly detected mypy diagnostics. - Enable the numpydantic and Pydantic mypy plugins, retaining strict Pydantic constructor typing with init_typed = true. Validate raw/coercible payloads through model_validate. - Propagate concrete record, provider and time-window types through generic collections, factories and lookup methods. Preserve runtime field inspection and generated time-window documentation. - Align Pendulum annotations with actual factory/arithmetic results while retaining Pydantic validation adapters at runtime. Correct optional values, array boundaries, REST handlers and plotting interfaces. - Add pinned scipy-stubs and types-psutil, update uv.lock, and supply the plugins' dependencies. - Add runtime regression coverage for validated path defaults, normalized time-series metadata, generic field inspection, invalid timestamps and unsupported provider imports. Runtime and compatibility details: - Validate path defaults as Path objects while retaining raw string defaults needed by migration serialization with exclude_defaults. - Normalize feed-in tariff lists and default charge rates to NumPy arrays; reject missing timestamps/uninitialized values explicitly. Importing into a provider without import support returns HTTP 400. - Public JSON schemas and OpenAPI structure match main (excluding the generated version). Signed-off-by: dr-dimitry Signed-off-by: dr-dimitry Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: dr-dimitri <87113560+dr-dimitri@users.noreply.github.com> Co-authored-by: Normann <github@koldrack.com> |
||
|
NormannandGitHub
|
25430bd42d | fix: infer Energy-Charts coverage from recent source intervals (#1280) | ||
|
dr-dimitriandGitHub
|
9d816acbd9 |
fix: honor Energy-Charts electricity price interval coverage (#1278)
* fix: honor Energy-Charts electricity price interval coverage * test: make Energy-Charts coverage checks timezone-independent |
||
|
dr-dimitriandGitHub
|
940aa1021f | fix: honor Energy-Charts feed-in interval coverage (#1275) | ||
  
|
7bafc8d02d |
feat: add PVForecastHomeAssistant provider (#1258)
* feat: add PVForecastHomeAssistant provider
Reads a PV forecast time series directly from a Home Assistant entity
attribute (matching the {"forecast": [{"datetime", "watts"}]} shape
already exposed by common HA PV forecast integrations, e.g. Helios
Forecast and Solcast) and feeds it into pvforecast_ac_power, following
the same self-polling provider pattern as PVForecastVrm.
Closes #1232.
* fix: use MagicMock instead of monkeypatched Response for mypy
requests.Response().json is a typed bound method; reassigning it to a
lambda fails mypy's method-assign check. Use MagicMock(spec=...) instead,
which mocks the response without fighting its static type.
* fix: address PR review on PVForecastHomeAssistant provider
Fixes two issues raised in review on PR #1258:
- _update_data() left stale pvforecast_ac_power values in place when a
refreshed forecast was empty or shorter than a previous one; it now
clears the active forecast window before writing and raises instead
of silently no-op'ing when the response has no usable data.
- pvforecast.homeassistant.entity_id used a "select" widget with no
manual-entry fallback in EOSdash, leaving it unusable in standalone
mode where the entity list can't be resolved without SUPERVISOR_TOKEN;
switched to a plain text field.
Also fills in the config docs and openapi.json for the new
pvforecast.homeassistant.* fields, which were missing from the
original commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix: preserve retained forecast history when clearing stale entries
The previous fix cleared pvforecast_ac_power from start-of-day, but
PredictionProvider deliberately retains historical records back to
keep_datetime (prediction.historic_hours). Since Home Assistant
forecasts are future-only, that clear wiped out retained history
between midnight and the EMS start on every refresh.
Narrow the clear to [ems_start_datetime, end_datetime) - the actual
active forecast window, DST-adjusted - instead of the day boundary.
Addresses review feedback from @NormannK on PR #1258.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Mathias <mathias@Mathiass-MacBook-Air.local>
Co-authored-by: Mathias <mathias@Mathiass-Air.localdomain>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
Bobby NoelteandGitHub
|
ba76087db9 |
feat: electricity fee provider framework and generic providers (#1235)
Add new provider class for electricity fees providers. Add the generic providers: - ElecFeeFixed - ElecFeeImport The providers provide predictions for: - elecfee_consumption_amt_wh: Total fixed fee for consumed energy per Wh [amount/Wh]. This is the accumulation of all fixed per-Wh fees payable on "consumed energy - such as network charge, concession fee, and electricity charge - into a single amount. - elecfee_consumption_percent_amt: Total fixed surcharge on consumed energy, given as a percentage of the monetary amount already charged for that energy [%]. This is the accumulation of all percentage-based surcharges payable on top of the consumed-energy fee - such as VAT - into a single percentage. This is a percentage of the fee amount, not a per-Wh rate. - elecfee_feedin_amt_wh: Total fixed deduction from feed-in energy per Wh [amount/Wh]. This is the accumulation of all fixed per-Wh charges deducted from feed-in energy - such as metering fees or grid-operator handling "charges - into a single amount. Applied after the percentage-based deduction, i.e. it reduces the price by a flat amount per Wh rather than by a share of the raw price. - elecfee_feedin_percent_amt: Total percentage deducted from the raw feed-in price (spot price) [%]. This is the accumulation of all percentage-based deductions payable on the feed-in tariff - such as a marketing or balancing fee retained by the aggregator - into a single percentage. It is applied as `raw_price * (100 - percent) / 100`, i.e. it scales down the raw price rather than adding a surcharge to it. A new _apply_fee() method is added to the base class for ElecPrice and FeedInTariff to be used to add the fees in a consistent way. Fees are taken from the active ElecFee provider and applied to the raw prices given to the _apply_fee() method. The optional application of fees is added to: - ElecPriceAkkudoktor - ElecPriceFixed - ElecPriceEnergyCharts - ElecPriceSMARD - FeedInTariffEnergyCharts - FeedInTariffFixed - FeedInTariffSMARD The import providers ElecPriceImport and FeedInTariffImport do not apply fees by intentention. The following providers currently do not handle fees defined by ElecFee: - ElecPriceTibber - FeedInTariffAkkudoktor - FeedInTariffDvhubOnline - FeedInTariffTibber The tests for this feature are either added or existing tests are extended. The documentation was extended for the electricity fee provider settings. Besides this feature further improvements are added: * feat: add SMARD quarter-hour electricty price and feed-in tariff provider * feat: to_series method for TimeWindows and ValueTimeWindows Additional to to_array the time window sequence can now also produce a pandas series. Test have been extended to cover the series generation. * feat: use time windows in fixed feedin tariff provider Feedin tariff can now be configured by time windows - not a single value. * feat: EOSdash select for PVLib inverters and modules Provide PVLib inverter and module names in config selection. * feat: EOSdash lazy select for big option sets Add a new form for lazy selection of big option sets. Filtering and generation of the option set is done server-side. * fix: use raw data for ETS/ median prediction Use to raw time series data for ETS/ median prediction to avoid interference by e.g. dynamic grid charges. * fix: EOSdash config drops by type only on details resolve Drop configuration by type and path. Prevents dropping of configuration items with same type and level but different path. * fix: EOSdash configuration section closes on update Open section if searching or if last update touched this category — including updates on deeply nested sub-fields. * chore: make elecfeefixed, elecpricefixed and feedintarifffixed warn about no windows and default to 0 Missining configuration creates default 0 value and a warning instead of an exception. * fix: test setup for providers Reset db state on each test run. * chore: improve config option naming for elecpricefixed. * chore: adapt elecpricefixed test to changed time_windows naming * chore: factorized common price provider helpers to priceabc.py Factorized common price provider helpers to priceabc.py. Add tests for these helpers. Reduce/ change testing of elecpriceabc.py and feedintariffabc.py to cover only specifics. Rest of testing is already covered by test_priceabc.py. * chore: update version Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
NormannandGitHub
|
74d31e2fac |
fix PVLib mountingplace None handling (#1220)
Handle an explicit None value for pvforecast.planes[].mountingplace before calling .lower(). |
||
|
NormannandGitHub
|
682e0ccc2b | fix PVLib inverter error message (#1219) | ||
|
Bobby NoelteandGitHub
|
894790f577 |
feat: add pvlib pv forecast provider (#1214)
Add a PV forecast provider that calculates the forecast using a PVLib system model and weather forecast from the EOS weather forecast provider. Additional module and inverter models can be easily added as the database is build from PVLib and SAM databases and a bundled csv file. The module model and inververt model names are provided by new endpoints to be used in configuration. The provider is based on the fantastic work of EMHASS. See https://github.com/davidusb-geek/emhass/blob/master/src/emhass/forecast.py A short description of the provider is added to the documentation. Besides the new features there are the fixes and improvements: * feat: improve EOSdash config page * fix: kex_to_series for start_datetime Make key_to_series always start the series at start_datetime. * fix: default provider for GENETIC and GENETIC0 optimization To make the default less dependent on internet servers (with API changes and availability issues) the default for PVForecast is set to PVForecastPVLib and for ElecPrice to ElecPriceFixed. The default weather provider is changed to OpenMeteo. * fix: EOSdash display resampled prediction values Make EOSdash display resampled prediction values where resampling fits to the prediction value type. Use bar width that fits to 15 minutes value samples. * chore: add a UI hints system to EOSdash The UI hints system eases the definition of forms for configuration items. There are also forms for items in maps and lists. These forms allow to add and delete items to/ from maps and lists. The forms ensure that all required fields of newly added items are filled. * chore: Create an enum for valid optimization algorithms * chore. Make config also provide the available energy management modes. Used for configuration hints. * chore: Randomize default device id in configuration Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
Bobby NoelteandGitHub
|
1905682113 |
chore: adapt pdf visualization (#1205)
Change PDF visualization to be created on demand and per optimization algorithm. The PDF for the GENETIC0 optimization is provided by the /visualization_results.pdf endpoint. There is no change in the interface. By this the optimization algorithm is offloaded from the PDF generation which spares some time. To cope with several users may call the /visualization_results.pdf endpoint at the same time the PDF is generated on the fly without any intermediate file taking the stored GENETIC0 solution as an input. SVG picture generation is removed as this would again create intermediate files. Chart pictures can easily be taken from the PDF. To allow on demand creation of the optimization results visualization the optimisation solution stored is extended by several new attributes. To keep the deprecated /optimize endpoint compatible the optimization solution is stripped to the legacy content before returned. Due to the extension of the solution the optimization tests were adapted to cover the extended content. The optimization tests are adapted to test the generated visualization report by the pypdf reader. Pypdf is added to the development dependencies. Besides the adaptation several fixes and improvements are added: * feat: extend /v1/prediction/series endpoint by resampling and filling Add parameters for resampling and filling. Add the processing parameter to control wether raw data or resampled data shall be returned. * feat: extend /v1/measurement/series endpoint by resampling and filling Add parameters for resampling and filling: Add the processing parameter to control wether raw data or resampled data shall be returned. * feat: standardize and improve API error response Use FASTApi exception handlers to provide a standardized API exception handling. All exceptions are logged. Exception traces are only returned if the new logging configuration parameter logging.api_logging_level is set to "DEBUG" or "TRACE". Avoids unwanted leackage of server internals on exceptions. * fix: align to intervall when resampling Ensure resampling is aligned to interval also when the buckets are shifted due to the align_to_intervall parameter is set. * chore: make dropna mandatory and default to True * chore: refactor key_to_xxx data management methods Make key_to_series the central method for data resampling and fill. Add a new key_to_raw_series to retrieve the data as it is stored (without resampling and filling). Users of key_to_series were mostly moved to key_to_raw_series as this resembles the former interface. Especially in predictions and tests this was done. * chore: create test data sub-directory for each optimization algorithm To prevent cluttering the test data directory and ease test data management for optimization algorithms each algorithm got it's own sub-directory. The current test data was moved to these sub-directories. * chore: update version Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
Bobby NoelteandGitHub
|
8344974c16 |
chore: adapt deprecated endpoints to 15-minutes predictions (#1195)
Bump Version / Bump Version Workflow (push) Canceled after 0s
CodeQL Advanced / Analyze (actions) (push) Canceled after 0s
CodeQL Advanced / Analyze (python) (push) Canceled after 0s
docker-build / platform-excludes (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Run Pytest on Pull Request / test (push) Canceled after 0s
docker-build / build (push) Canceled after 0s
docker-build / merge (push) Canceled after 0s
Ensure the deprecated endpoints get /strompreis, post /gesamtlast, get /gesamtlast_simple,
get /pvforecast to work on 1-hour intervalls even if the prediction provides 15-minutes
intervall data. This keeps the interface compliant to the legacy functionality.
Besides this adaptations several other improvements and fixes are included in this PR.
* feat: extend data management method key_to array by resample_method
One can now define the resample method on how to aggregate the values in an interval
for resampling. Three methods are provided:
- "first": Use the first value in each interval.
- "mean": Compute the arithmetic mean of all samples in each interval.
- "interval_mean": Compute the time-weighted mean assuming each value remains valid
until the next timestamp (piecewise-constant signal).
* feat: extend the get /v1/prediction/dataframe endpoint with resampling parameters
Make all parameters for resampling available at the endpoint.
* feat: extend the get /v1/prediction/list endpoint with resampling parameters
Make all parameters for resampling available at the endpoint.
* feat: add new delete /v1/prediction/range endpoint
The endpoint allows to delete prediction values for a given time span.
* fix: adapt for PVForecastAkkudoktor server side cache handling
/api.akkudoktor/forecast does it's own caching on requests. Call it with slightly
randomized request values to avoid getting cached values in the case we need fresh
data. The requests are anyway rate limited to one request per hour on our side.
* chore: add core.types
This module centralizes reusable type definitions shared across multiple
packages. Defining common types here avoids duplication of complex type
annotations (such as Literal aliases), ensures consistent typing across the
code base, and helps prevent circular import dependencies between modules.
* chore: extend cache testing
* chore: add system test for deprecated /strompreis endpoint
* chore: add unit test module for server endpoints
Add a new test module to do unit tests on server endpoints. First test added
for deprecated get /strompreis endpoint.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
|
||
|
Bobby NoelteandGitHub
|
52fe489d4e |
feat: add dvhubonline feed-in tariff provider (#1193)
Bump Version / Bump Version Workflow (push) Has been cancelled
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
docker-build / platform-excludes (push) Has been cancelled
docker-build / build (push) Has been cancelled
docker-build / merge (push) Has been cancelled
pre-commit / pre-commit (push) Has been cancelled
Run Pytest on Pull Request / test (push) Has been cancelled
Close stale pull requests/issues / Find Stale issues and PRs (push) Has been cancelled
Fetches GET /api/prices?start&end&zone (15-min slots, EUR/MWh) and stores the raw market price as feed_in_tariff_wh (EUR/Wh) — no import charges/VAT. Slots beyond the day-ahead horizon are left to the consumer's forward-fill (FeedInTariffImport behaviour). 5 unit tests + opt-in live smoke (EOS_DVHUB_ONLINE_LIVE=1, passing). Signed-off-by: Christin <info@bikinibottom.capital> Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
Bobby NoelteandGitHub
|
e23bb7b497 |
chore: prepare for update of genetic algorithm (#1190)
Bump Version / Bump Version Workflow (push) Has been cancelled
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
docker-build / platform-excludes (push) Has been cancelled
docker-build / build (push) Has been cancelled
docker-build / merge (push) Has been cancelled
pre-commit / pre-commit (push) Has been cancelled
Run Pytest on Pull Request / test (push) Has been cancelled
Andreas will update the genetic algorithm for 15-minutes optimization intervals. Copy the current GENETIC optimization algorithm to GENETIC0 to enable to keep the algorithm with the current functionality. Also copy resources like the load interpolator to the GENETIC0 algorithm to keep them despite possible later changes to the interpolator. Make the deprecated legacy /optimize endpoint use the GENETIC0 optimization algorithm to in-fact behave the same way even if there will later be changes to the GENETIC algorithm by Andreas. Add a new REST endpoint to provide the unprocessed optimisation results of the GENETIC and GENETIC0 algorithm in case one wants to use them as done with the deprecated /optimize endpoint. Adapt the optimization configuration to have distinct configurations for the GENETIC and the GENETIC0 algorithm. Create a copy of the current tests for the GENETIC algorithm to be used for the GENETIC0 algorithm. This avoids the tests for the GENETIC0 algorithm to be influenced by later changes by Andreas. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
Bobby NoelteandGitHub
|
7e5aa2f218 |
feat: add Tibber feed-in tariff provider (#1189)
Bump Version / Bump Version Workflow (push) Has been cancelled
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
docker-build / platform-excludes (push) Has been cancelled
pre-commit / pre-commit (push) Has been cancelled
Run Pytest on Pull Request / test (push) Has been cancelled
docker-build / build (push) Has been cancelled
docker-build / merge (push) Has been cancelled
Close stale pull requests/issues / Find Stale issues and PRs (push) Has been cancelled
The `FeedInTariffTibber` provider requests `priceInfo` and `priceInfoRange` with `resolution: QUARTER_HOURLY` and preserves the native 15-minute timestamps. It uses Tibber's `energy` spot-price component without the `tax` part or EOS electricity-price charges. The end-customer `total` component is deliberately ignored. The provider deliberately rejects hourly API responses instead of silently repeating them. It reuses `elecprice.tibber.access_token` and `elecprice.tibber.home_id`, so no duplicate credentials are needed. Signed-off-by: Andreas Schmitz <akkudoktor.net> Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
Bobby NoelteandGitHub
|
4c8a8e40f5 |
feat: add Akkudoktor feed-in provider (#1187)
Bump Version / Bump Version Workflow (push) Has been cancelled
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
docker-build / platform-excludes (push) Has been cancelled
docker-build / build (push) Has been cancelled
docker-build / merge (push) Has been cancelled
pre-commit / pre-commit (push) Has been cancelled
Run Pytest on Pull Request / test (push) Has been cancelled
The `FeedInTariffAkkudoktor` provider uses raw day-ahead market prices from `https://api.akkudoktor.net/prices` as `feed_in_tariff_wh`. It does not add electricity import charges or VAT. Published prices are extended to the configured prediction horizon with the same seasonal ETS or median fallback used by the Akkudoktor electricity-price provider. The Akkudoktor endpoint currently forwards hourly market prices from aWATTar. With a 15-minute optimization interval, EOS holds each hourly price constant for its four quarter-hour slots. This keeps the slot grid consistent but does not create genuine quarter-hour market prices. Signed-off-by: Andreas Schmitz <akkudoktor.net> Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
Bobby NoelteandGitHub
|
ed61918fe0 |
feat: add EnergyCharts feed-in tariff provider (#1165)
Bump Version / Bump Version Workflow (push) Has been cancelled
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
docker-build / platform-excludes (push) Has been cancelled
docker-build / build (push) Has been cancelled
docker-build / merge (push) Has been cancelled
pre-commit / pre-commit (push) Has been cancelled
Run Pytest on Pull Request / test (push) Has been cancelled
Close stale pull requests/issues / Find Stale issues and PRs (push) Has been cancelled
The `FeedInTariffEnergyCharts` provider uses the raw Energy-Charts day-ahead market price as the feed-in tariff. It stores prices in `feed_in_tariff_wh` without adding electricity import charges or VAT. The data is loaded from the Energy-Charts `/price` endpoint for the configured bidding zone. The native Energy-Charts resolution, including quarter-hour data, is retained. Energy-Charts usually supplies prices only for the published day-ahead period. If that data does not cover the complete configured prediction horizon, the provider extends it as follows: - With more than 800 hours of history, an ETS (Holt-Winters exponential smoothing) forecast with weekly seasonality is used. - With more than 168 hours of history, an ETS forecast with daily seasonality is used. - With less history, the median of the available values is used as a constant fallback. The seasonal periods are adjusted to the source resolution. For example, quarter-hour data uses four values per hour. Values already supplied by Energy-Charts are kept unchanged; only missing future slots after the last published price are forecast. Consequently, a 15-minute optimization uses four forecast values per hour without converting them to hourly averages. Signed-off-by: Andreas Schmitz <akkudoktor.net> Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
Bobby NoelteandGitHub
|
641873d867 |
fix: akkudoktor api requests (#1181)
Bump Version / Bump Version Workflow (push) Has been cancelled
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
docker-build / platform-excludes (push) Has been cancelled
docker-build / build (push) Has been cancelled
docker-build / merge (push) Has been cancelled
pre-commit / pre-commit (push) Has been cancelled
Run Pytest on Pull Request / test (push) Has been cancelled
The akkudoktor api for PV forecast seems to be changed and does not support to concatenate several planes into one request anymore. Make a request for every plane and add up the power results. Do not use the new 15 minutes slots as the returned data does not include the hourly values for windspeed and temperature as does the hourly slots. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
Bobby NoelteandGitHub
|
3bb0e02aed |
fix: provider settings in top level field of configuration (#1161)
Configuration for some providers was given in the sub-field `provider_settings` combining the settings of several providers. Pydantic does not understand this very well and the configuration became cumbersome, especially in EOSdash. All provider settings from the `provider-settings` sub-field are now lifted to the top level field of the configuration. The changes are automatically migrated in the configuration. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
Bobby NoelteandGitHub
|
cc8193216a |
feat: add tibber electricity price provider (#1160)
Signed-off-by: Andreas Schmitz <akkudoktor.net> Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
 |
8bf798daeb |
fix: bare except (#1159)
Bare `except:` catches SystemExit and KeyboardInterrupt, which can: - Mask Ctrl+C handling during long optimization runs - Hide critical system signals - Make debugging harder Replacement by `except Exception:` preserves the same error handling while respecting system-level exceptions. Co-authored-by: Milo @KeloYuan Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
 
|
6933b33542 |
feat: rename genetic optimization API fields to English (#675)
Bump Version / Bump Version Workflow (push) Has been cancelled
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
docker-build / platform-excludes (push) Has been cancelled
pre-commit / pre-commit (push) Has been cancelled
Run Pytest on Pull Request / test (push) Has been cancelled
docker-build / build (push) Has been cancelled
docker-build / merge (push) Has been cancelled
Close stale pull requests/issues / Find Stale issues and PRs (push) Has been cancelled
Rename the German field names of the genetic optimization API to English with full backward compatibility: - English names are canonical and documented in the OpenAPI schema; the German names are still accepted on input via validation aliases and re-emitted in responses as deprecated computed fields - fix visualization receiving the raw German-keyed simulation dict (KeyError: load_wh_per_hour) - rename internal German identifiers (optimize_ems, total_balance, battery_residual_value, self_consumption, extra_data keys) - use amount instead of currency/euro in chart labels and schema descriptions; document currency handling (whole currency units, never cents) in the API description - regenerate openapi.json and generated docs Co-authored-by: Tobias Welz <tobias.wizneteu@gmail.com> |
||
|
|
75548990e1 |
feat: add cloud PV forecast providers: pvnode.com, forecast-solar and solcast (#1150)
Bump Version / Bump Version Workflow (push) Has been cancelled
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
docker-build / platform-excludes (push) Has been cancelled
docker-build / build (push) Has been cancelled
docker-build / merge (push) Has been cancelled
pre-commit / pre-commit (push) Has been cancelled
Run Pytest on Pull Request / test (push) Has been cancelled
This PR adds three native PV power forecast providers, giving operators more
cloud forecast sources to choose from via pvforecast.provider, alongside the
existing PVForecastAkkudoktor, PVForecastVrm and PVForecastImport:
PVForecastPVNode — native 15-minute forecasts from the pvnode.com
V2 API. Saved-site mode (GET /v2/forecast/{site_id}) where the operator enters their API key +
site id, or inline mode (POST /v2/forecast/inline) using the configured planes.
PVForecastForecastSolar — the free Forecast.Solar API (no key
required for the public endpoint). Multi-plane systems issue one request per plane and the
instantaneous powers are summed per timestamp.
PVForecastSolcast — the Solcast rooftop-site API (API key + resource id).
Implementation notes
All three populate the existing pvforecast_ac_power (and mirror pvforecast_dc_power)
prediction keys, so they slot into the optimizer unchanged.
Timezone handling: pvnode and Forecast.Solar return site-local wall-clock timestamps with an
IANA timezone field, which are resolved to absolute instants before resampling; Solcast returns
UTC period_end and is normalised to the period start.
Each provider follows the existing provider pattern (provider_id(), _request_forecast() with
@cache_in_file, _update_data()), is registered in pvforecast.py and prediction.py, and is
upstream-neutral.
Validation
27 unit tests (timezone resolution, null/zero handling, kW→W and period-start conversion,
azimuth conversion, multi-plane summation, request URL/auth, HTTP-error handling).
ruff check (F/D/S/bandit) and ruff format clean.
Each provider was additionally validated against its live API with a real plant, confirming
the response shapes (pvnode: 288 native 15-min slots; Forecast.Solar: instantaneous watts;
Solcast: kW estimates with period_end/PT30M).
Documentation
Provider descriptions and configuration examples added to docs/akkudoktoreos/prediction.md.
CHANGELOG.md entry under Unreleased.
Regenerated docs/_generated/configpvforecast.md and configexample.md.
Notes for reviewers
Forecast.Solar's free endpoint is rate-limited (12 req/hour) and Solcast's free tier limits daily
calls; both providers rely on the standard 1-hour cache_in_file TTL to stay within budget.
Authors:
The code is created by Christin. Only minor adaptions by Bobby.
Signed-off-by: Christin <info@bikinibottom.capital>
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: Christin <info@bikinibottom.capital>
|
||
|
Bobby NoelteandGitHub
|
0306e7e4ed |
fix: prediction import with pydantic-typed bodies (#1152)
Bump Version / Bump Version Workflow (push) Has been cancelled
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
docker-build / platform-excludes (push) Has been cancelled
docker-build / build (push) Has been cancelled
docker-build / merge (push) Has been cancelled
pre-commit / pre-commit (push) Has been cancelled
Run Pytest on Pull Request / test (push) Has been cancelled
Close stale pull requests/issues / Find Stale issues and PRs (push) Has been cancelled
Respect pydantic model in serializing json data for import. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
Bobby NoelteandGitHub
|
eb9e966de9 |
fix: move data management to async (#1015)
FAstAPI is an async framework. Data may be imported and exported, load and save, set and get asynchronously. Prevent interleaving data operations to corrupt the data. In the previous design sync and async data access was intermixed leading to data corruption. The basic data classes DataSequence and DataContainer and the derived classes like Provider and Measurement now are async. Data access is protected by several async locks. To support the async design of the data classes the database interface became async. The energy management is also adapted to the new async design. Optimization is still off-loaded to another thread, but the prepration for the optimization and the post optimization actions now follow the async design. Adapter operations are now also protected by async locks. Tests were adapted to the async design and new tests were created. Besides this major fix several other improvements and fixes are included in this PR. * fix: key_to_dict/list/array only regard data records with key value set. Before the exclusion of no value data records was only done if the dropna flag was set. * fix: test for visual result pdf generation Due to updates in the library the generated charts text was a little bit different. Adapt the test to create the comaprison pdf in the test data durectory and update the reference pdf. * chore: Remove MutableMapping from DataSequence and DataContainer. Mutable Mapping does not fit to the now async design. * chore: Add NoDB database backend This backend implements the full database backend interface but performs no actual persistence. It is intended for configurations where database persistence is disabled (`provider=None`). * chore: Improve measurement data import testing with real world scenarios. Added two new endpoints to support testing. * chore: Add mermaid to supported documentation tools * chore: Add documentation about async design * chore: Add documentation about generic data handling Covers the basics of measurement and prediction time series data handling. * chore: Add empty lines around markdown lists. * chore: sync pre-commit config to updated package versions Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
|
6f28022ed0 |
fix: optimization fail after restart (#1007)
Bump Version / Bump Version Workflow (push) Has been cancelled
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
docker-build / platform-excludes (push) Has been cancelled
pre-commit / pre-commit (push) Has been cancelled
Run Pytest on Pull Request / test (push) Has been cancelled
docker-build / build (push) Has been cancelled
docker-build / merge (push) Has been cancelled
Close stale pull requests/issues / Find Stale issues and PRs (push) Has been cancelled
Fix documentation for the loadforecast_power_w key. Fix documentation to explain the usage of import file/ JSON string to primarily initialise prediction data. Fix code scanning alert no. 6: URL redirection from remote source Enable to automatically save the configuration to the configuration file by default, which is a widespread user expectation. Make the genetic parameters non optional for better pydantic compliance. Update: - bump pytest to 9.0.3 - bump pillow to 12.2.0 - bump platformdirs to 4.9.6 - bump typespyyaml to 6.0.12.20260408 - bump tzfpy to 1.2.0 - bump pydantic to 2.13.0 - bump types-requests to 2.33.0.20260408 Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: Normann <github@koldrack.com> |
||
|
Bobby NoelteandGitHub
|
71e5abce88 |
fix: energy charts bidding zone in request (#948)
Bump Version / Bump Version Workflow (push) Has been cancelled
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
docker-build / platform-excludes (push) Has been cancelled
pre-commit / pre-commit (push) Has been cancelled
Run Pytest on Pull Request / test (push) Has been cancelled
docker-build / build (push) Has been cancelled
docker-build / merge (push) Has been cancelled
Close stale pull requests/issues / Find Stale issues and PRs (push) Has been cancelled
Ensure that the bidding zone in the request is correctly set to a string value (not an enum). This seems to be also an issue with python version < 3.11. Add safeguards to only use python >= 3.11. Still keep a regression test for the enum conversion to string. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
Bobby NoelteandGitHub
|
8a9aec6d57 |
feat: add openmeteo weather provider (#939)
Add OpenMeteo to the selectable weather prediction providers. Also add tests and documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
Bobby NoelteandGitHub
|
cf477d91a3 |
feat: add fixed electricity prediction with time window support (#930)
Bump Version / Bump Version Workflow (push) Has been cancelled
docker-build / platform-excludes (push) Has been cancelled
docker-build / build (push) Has been cancelled
docker-build / merge (push) Has been cancelled
pre-commit / pre-commit (push) Has been cancelled
Run Pytest on Pull Request / test (push) Has been cancelled
Close stale pull requests/issues / Find Stale issues and PRs (push) Has been cancelled
Add a fixed electricity prediction that supports prices per time window.
The time windows may flexible be defined by day or date.
The prediction documentation is updated to also cover the ElecPriceFixed
provider.
The feature includes several changes that are not directly related to the
electricity price prediction implementation but are necessary to keep
EOS running properly and to test and document the changes.
* feat: add value time windows
Add time windows with an associated float value.
* feat: harden eos measurements endpoints error detection and reporting
Cover more errors that may be raised during endpoint access. Report the
errors including trace information to ease debugging.
* feat: extend server configuration to cover all arguments
Make the argument controlled options also available in server configuration.
* fix: eos config configuration by cli arguments
Move the command line argument handling to config eos so that it is
excuted whenever eos config is rebuild or reset.
* chore: extend measurement endpoint system test
* chore: refactor time windows
Move time windows to configabc as they are only used in configurations.
Also move all tests to test_configabc.
* chore: provide config update errors in eosdash with summarized error text
If there is an update error provide the error text as a summary. On click
provide the full error text.
* chore: force eosdash ip address and port in makefile dev run
Ensure eosdash ip address and port are correctly set for development runs.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
|
||
|
04420e66ab |
fix: Improve provider update error handling and add VRM provider settings validation (#887)
Bump Version / Bump Version Workflow (push) Has been cancelled
docker-build / platform-excludes (push) Has been cancelled
pre-commit / pre-commit (push) Has been cancelled
Run Pytest on Pull Request / test (push) Has been cancelled
docker-build / build (push) Has been cancelled
docker-build / merge (push) Has been cancelled
Close stale pull requests/issues / Find Stale issues and PRs (push) Has been cancelled
* fix: improve error handling for provider updates Distinguishes failures of active providers from inactive ones. Propagates errors only for enabled providers, allowing execution to continue if a non-active provider fails, which avoids unnecessary interruptions and improves robustness. * fix: add provider settings validation for forecast requests Prevents potential runtime errors by checking if provider settings are configured before accessing forecast credentials. Raises a clear error when settings are missing to help with debugging misconfigurations. * refactor(load): move provider settings to top-level fields Transitions load provider settings from a nested "provider_settings" object with provider-specific keys to dedicated top-level fields.\n\nRemoves the legacy "provider_settings" mapping and updates migration logic to ensure backward compatibility with existing configurations. * docs: update version numbers and documantation --------- Co-authored-by: Normann <github@koldrack.com> |
||
|
Bobby NoelteandGitHub
|
6498c7dc32 |
Add database support for measurements and historic prediction data. (#848)
The database supports backend selection, compression, incremental data load, automatic data saving to storage, automatic vaccum and compaction. Make SQLite3 and LMDB database backends available. Update tests for new interface conventions regarding data sequences, data containers, data providers. This includes the measurements provider and the prediction providers. Add database documentation. The fix includes several bug fixes that are not directly related to the database implementation but are necessary to keep EOS running properly and to test and document the changes. * fix: config eos test setup Make the config_eos fixture generate a new instance of the config_eos singleton. Use correct env names to setup data folder path. * fix: startup with no config Make cache and measurements complain about missing data path configuration but do not bail out. * fix: soc data preparation and usage for genetic optimization. Search for soc measurments 48 hours around the optimization start time. Only clamp soc to maximum in battery device simulation. * fix: dashboard bailout on zero value solution display Do not use zero values to calculate the chart values adjustment for display. * fix: openapi generation script Make the script also replace data_folder_path and data_output_path to hide real (test) environment pathes. * feat: add make repeated task function make_repeated_task allows to wrap a function to be repeated cyclically. * chore: removed index based data sequence access Index based data sequence access does not make sense as the sequence can be backed by the database. The sequence is now purely time series data. * chore: refactor eos startup to avoid module import startup Avoid module import initialisation expecially of the EOS configuration. Config mutation, singleton initialization, logging setup, argparse parsing, background task definitions depending on config and environment-dependent behavior is now done at function startup. * chore: introduce retention manager A single long-running background task that owns the scheduling of all periodic server-maintenance jobs (cache cleanup, DB autosave, …) * chore: canonicalize timezone name for UTC Timezone names that are semantically identical to UTC are canonicalized to UTC. * chore: extend config file migration for default value handling Extend the config file migration handling values None or nonexisting values that will invoke a default value generation in the new config file. Also adapt test to handle this situation. * chore: extend datetime util test cases * chore: make version test check for untracked files Check for files that are not tracked by git. Version calculation will be wrong if these files will not be commited. * chore: bump pandas to 3.0.0 Pandas 3.0 now performs inference on the appropriate resolution (a.k.a. unit) for the output dtype which may become datetime64[us] (before it was ns). Also numeric dtype detection is now more strict which needs a different detection for numerics. * chore: bump pydantic-settings to 2.12.0 pydantic-settings 2.12.0 under pytest creates a different behaviour. The tests were adapted and a workaround was introduced. Also ConfigEOS was adapted to allow for fine grain initialization control to be able to switch off certain settings such as file settings during test. * chore: remove sci learn kit from dependencies The sci learn kit is not strictly necessary as long as we have scipy. * chore: add documentation mode guarding for sphinx autosummary Sphinx autosummary excecutes functions. Prevent exceptions in case of pure doc mode. * chore: adapt docker-build CI workflow to stricter GitHub handling Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
Bobby NoelteandGitHub
|
39973bf836 |
fix: load prediction adjustment with measurement in kwh (#826)
Use load energy meter reading in kWh for load prediction adjustment. Before the reading was falsely regarded to be in Wh. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |
||
|
Bobby NoelteandGitHub
|
58d70e417b |
feat: add Home Assistant and NodeRED adapters (#764)
Adapters for Home Assistant and NodeRED integration are added. Akkudoktor-EOS can now be run as Home Assistant add-on and standalone. As Home Assistant add-on EOS uses ingress to fully integrate the EOSdash dashboard in Home Assistant. The fix includes several bug fixes that are not directly related to the adapter implementation but are necessary to keep EOS running properly and to test and document the changes. * fix: development version scheme The development versioning scheme is adaptet to fit to docker and home assistant expectations. The new scheme is x.y.z and x.y.z.dev<hash>. Hash is only digits as expected by home assistant. Development version is appended by .dev as expected by docker. * fix: use mean value in interval on resampling for array When downsampling data use the mean value of all values within the new sampling interval. * fix: default battery ev soc and appliance wh Make the genetic simulation return default values for the battery SoC, electric vehicle SoC and appliance load if these assets are not used. * fix: import json string Strip outer quotes from JSON strings on import to be compliant to json.loads() expectation. * fix: default interval definition for import data Default interval must be defined in lowercase human definition to be accepted by pendulum. * fix: clearoutside schema change * feat: add adapters for integrations Adapters for Home Assistant and NodeRED integration are added. Akkudoktor-EOS can now be run as Home Assistant add-on and standalone. As Home Assistant add-on EOS uses ingress to fully integrate the EOSdash dashboard in Home Assistant. * feat: allow eos to be started with root permissions and drop priviledges Home assistant starts all add-ons with root permissions. Eos now drops root permissions if an applicable user is defined by paramter --run_as_user. The docker image defines the user eos to be used. * feat: make eos supervise and monitor EOSdash Eos now not only starts EOSdash but also monitors EOSdash during runtime and restarts EOSdash on fault. EOSdash logging is captured by EOS and forwarded to the EOS log to provide better visibility. * feat: add duration to string conversion Make to_duration to also return the duration as string on request. * chore: Use info logging to report missing optimization parameters In parameter preparation for automatic optimization an error was logged for missing paramters. Log is now down using the info level. * chore: make EOSdash use the EOS data directory for file import/ export EOSdash use the EOS data directory for file import/ export by default. This allows to use the configuration import/ export function also within docker images. * chore: improve EOSdash config tab display Improve display of JSON code and add more forms for config value update. * chore: make docker image file system layout similar to home assistant Only use /data directory for persistent data. This is handled as a docker volume. The /data volume is mapped to ~/.local/share/net.akkudoktor.eos if using docker compose. * chore: add home assistant add-on development environment Add VSCode devcontainer and task definition for home assistant add-on development. * chore: improve documentation |
||
|
Bobby NoelteandGitHub
|
4c2997dbd6 |
feat: add bidding zone to energy charts price prediction (#765)
Energy charts supports bidding zones. Allow to specifiy the bidding zone in the configuration. Extend and simplify ElecPrice configuration structure and setup config migration to automatically update the configuration file. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> |