5 Commits

Include renames

Author	SHA1	Message	Date
4 people	1abdd345c4	fix: unify mypy environments for local checks and CI (#1291) ... The isolated pre-commit mypy hook previously omitted runtime type information that make mypy used, hiding errors involving dependencies such as Pydantic and Pendulum. Makefile, pre-commit and CI now run the same full-project typing policy in the development environment defined by uv.lock. - Use uv run --locked --exact --extra dev and the same mypy arguments for Makefile and the local hook. Check all of src and tests, including on configuration-only changes. - Pin Python 3.13 for local development and the pre-commit CI job, and install the locked pre-commit version in CI. - Disable incremental analysis because existing Pendulum cache state changes mypy 2.3.1 diagnostics. Document the policy, the performance tradeoff and the existing typing debt. - Add a regression test that exercises Makefile, the hook and the CI command in a temporary project, accepting valid dependency types and detecting deliberate Pydantic/Pendulum assignment errors. Resolve the newly detected mypy diagnostics. - Enable the numpydantic and Pydantic mypy plugins, retaining strict Pydantic constructor typing with init_typed = true. Validate raw/coercible payloads through model_validate. - Propagate concrete record, provider and time-window types through generic collections, factories and lookup methods. Preserve runtime field inspection and generated time-window documentation. - Align Pendulum annotations with actual factory/arithmetic results while retaining Pydantic validation adapters at runtime. Correct optional values, array boundaries, REST handlers and plotting interfaces. - Add pinned scipy-stubs and types-psutil, update uv.lock, and supply the plugins' dependencies. - Add runtime regression coverage for validated path defaults, normalized time-series metadata, generic field inspection, invalid timestamps and unsupported provider imports. Runtime and compatibility details: - Validate path defaults as Path objects while retaining raw string defaults needed by migration serialization with exclude_defaults. - Normalize feed-in tariff lists and default charge rates to NumPy arrays; reject missing timestamps/uninitialized values explicitly. Importing into a provider without import support returns HTTP 400. - Public JSON schemas and OpenAPI structure match main (excluding the generated version). Signed-off-by: dr-dimitry Signed-off-by: dr-dimitry Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com> Co-authored-by: dr-dimitri <87113560+dr-dimitri@users.noreply.github.com> Co-authored-by: Normann <github@koldrack.com>	2026-09-10 23:20:35 +02:00
Bobby NoelteandGitHub	eb9e966de9	fix: move data management to async (#1015) ... FAstAPI is an async framework. Data may be imported and exported, load and save, set and get asynchronously. Prevent interleaving data operations to corrupt the data. In the previous design sync and async data access was intermixed leading to data corruption. The basic data classes DataSequence and DataContainer and the derived classes like Provider and Measurement now are async. Data access is protected by several async locks. To support the async design of the data classes the database interface became async. The energy management is also adapted to the new async design. Optimization is still off-loaded to another thread, but the prepration for the optimization and the post optimization actions now follow the async design. Adapter operations are now also protected by async locks. Tests were adapted to the async design and new tests were created. Besides this major fix several other improvements and fixes are included in this PR. * fix: key_to_dict/list/array only regard data records with key value set. Before the exclusion of no value data records was only done if the dropna flag was set. * fix: test for visual result pdf generation Due to updates in the library the generated charts text was a little bit different. Adapt the test to create the comaprison pdf in the test data durectory and update the reference pdf. * chore: Remove MutableMapping from DataSequence and DataContainer. Mutable Mapping does not fit to the now async design. * chore: Add NoDB database backend This backend implements the full database backend interface but performs no actual persistence. It is intended for configurations where database persistence is disabled (`provider=None`). * chore: Improve measurement data import testing with real world scenarios. Added two new endpoints to support testing. * chore: Add mermaid to supported documentation tools * chore: Add documentation about async design * chore: Add documentation about generic data handling Covers the basics of measurement and prediction time series data handling. * chore: Add empty lines around markdown lists. * chore: sync pre-commit config to updated package versions Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-07-15 16:38:53 +02:00
Bobby NoelteandGitHub	378377b1ce	fix: test break docs and on data compaction (2) (#902) ... Ensure that the snapping sequence generated in the test fixture is within the boundaries expected by the test. Ensure we read the _version_date.py info as UTC datetime and do no localtime conversion. Prevent and guard test_version.py to modify the version date file. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-02-24 18:56:11 +01:00
Bobby NoelteandGitHub	90e2e8af7e	fix: test break on docs version and data compaction (#900) ... Upgrade to: - pandas==3.01 - fastapi[standard-no-fastapi-cloud-cli]==0.132.0 - fastapi_cli==0.0.23 - MonsterUI==1.0.44 - uvicorn==0.41.0 Close database in database fixture on teardown. Fix file exclusion in hash and version date generation. Update version information in documentation. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-02-24 07:37:22 +01:00
Bobby NoelteandGitHub	6498c7dc32	Add database support for measurements and historic prediction data. (#848) ... The database supports backend selection, compression, incremental data load, automatic data saving to storage, automatic vaccum and compaction. Make SQLite3 and LMDB database backends available. Update tests for new interface conventions regarding data sequences, data containers, data providers. This includes the measurements provider and the prediction providers. Add database documentation. The fix includes several bug fixes that are not directly related to the database implementation but are necessary to keep EOS running properly and to test and document the changes. * fix: config eos test setup Make the config_eos fixture generate a new instance of the config_eos singleton. Use correct env names to setup data folder path. * fix: startup with no config Make cache and measurements complain about missing data path configuration but do not bail out. * fix: soc data preparation and usage for genetic optimization. Search for soc measurments 48 hours around the optimization start time. Only clamp soc to maximum in battery device simulation. * fix: dashboard bailout on zero value solution display Do not use zero values to calculate the chart values adjustment for display. * fix: openapi generation script Make the script also replace data_folder_path and data_output_path to hide real (test) environment pathes. * feat: add make repeated task function make_repeated_task allows to wrap a function to be repeated cyclically. * chore: removed index based data sequence access Index based data sequence access does not make sense as the sequence can be backed by the database. The sequence is now purely time series data. * chore: refactor eos startup to avoid module import startup Avoid module import initialisation expecially of the EOS configuration. Config mutation, singleton initialization, logging setup, argparse parsing, background task definitions depending on config and environment-dependent behavior is now done at function startup. * chore: introduce retention manager A single long-running background task that owns the scheduling of all periodic server-maintenance jobs (cache cleanup, DB autosave, …) * chore: canonicalize timezone name for UTC Timezone names that are semantically identical to UTC are canonicalized to UTC. * chore: extend config file migration for default value handling Extend the config file migration handling values None or nonexisting values that will invoke a default value generation in the new config file. Also adapt test to handle this situation. * chore: extend datetime util test cases * chore: make version test check for untracked files Check for files that are not tracked by git. Version calculation will be wrong if these files will not be commited. * chore: bump pandas to 3.0.0 Pandas 3.0 now performs inference on the appropriate resolution (a.k.a. unit) for the output dtype which may become datetime64[us] (before it was ns). Also numeric dtype detection is now more strict which needs a different detection for numerics. * chore: bump pydantic-settings to 2.12.0 pydantic-settings 2.12.0 under pytest creates a different behaviour. The tests were adapted and a workaround was introduced. Also ConfigEOS was adapted to allow for fine grain initialization control to be able to switch off certain settings such as file settings during test. * chore: remove sci learn kit from dependencies The sci learn kit is not strictly necessary as long as we have scipy. * chore: add documentation mode guarding for sphinx autosummary Sphinx autosummary excecutes functions. Prevent exceptions in case of pure doc mode. * chore: adapt docker-build CI workflow to stricter GitHub handling Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>	2026-02-22 14:12:42 +01:00