- Get weather data with fully specified end_date datetime argument to not miss data.
- Make preciptable water records generation robust against missing temperature
or humidity values.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* Move the caching module to core.
Add an in memory cache that for caching function and method results
during an energy management run (optimization run). Two decorators
are provided for methods and functions.
* Improve the file cache store by load and save functions.
Make EOS load the cache file store on startup and save it on shutdown.
Add a cyclic task that cleans the cache file store from outdated cache files.
* Improve startup of EOSdash by EOS
Make EOS starting EOSdash adhere to path configuration given in EOS.
The whole environment from EOS is now passed to EOSdash.
Should also prevent test errors due to unwanted/ wrong config file creation.
Both servers now provide a health endpoint that can be used to detect whether
the server is running. This is also used for testing now.
* Improve startup of EOS
EOS now has got an energy management task that runs shortly after startup.
It tries to execute energy management runs with predictions newly fetched
or initialized from cached data on first run.
* Improve shutdown of EOS
EOS has now a shutdown task that shuts EOS down gracefully with some
time delay to allow REST API requests for shutdwon or restart to be fully serviced.
* Improve EMS
Add energy management task for repeated energy management controlled by
startup delay and interval configuration parameters.
Translate EnergieManagementSystem to english EnergyManagement.
* Add administration endpoints
- endpoints to control caching from REST API.
- endpoints to control server restart (will not work on Windows) and shutdown from REST API
* Improve doc generation
Use "\n" linenend convention also on Windows when generating doc files.
Replace Windows specific 127.0.0.1 address by standard 0.0.0.0.
* Improve test support (to be able to test caching)
- Add system test option to pytest for running tests with "real" resources
- Add new test fixture to start server for test class and test function
- Make kill signal adapt to Windows/ Linux
- Use consistently "\n" for lineends when writing text files in doc test
- Fix test_logging under Windows
- Fix conftest config_default_dirs test fixture under Windows
From @Lasall
* Improve Windows support
- Use 127.0.0.1 as default config host (model defaults) and
addionally redirect 0.0.0.0 to localhost on Windows (because default
config file still has 0.0.0.0).
- Update install/startup instructions as package installation is
required atm.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* All config now nested.
- Use default config from model field default values. If providers
should be enabled by default, non-empty default config file could
be provided again.
- Environment variable support with EOS_ prefix and __ between levels,
e.g. EOS_SERVER__EOS_SERVER_PORT=8503 where all values are case
insensitive.
For more information see:
https://docs.pydantic.dev/latest/concepts/pydantic_settings/#parsing-environment-variable-values
- Use devices as registry for configured devices. DeviceBase as base
class with for now just initializion support (in the future expand
to operations during optimization).
- Strip down ConfigEOS to the only configuration instance. Reload
from file or reset to defaults is possible.
* Fix multi-initialization of derived SingletonMixin classes.
* Cleanup: parameters: extra=forbid, optimize: battery, inverter optional
* Don't allow extra fields for parameters/REST-API (at least for now while
changing API).
* Allow both battery and inverter to be set optionally (atm optional
battery not implemented, no API constraints).
* inverter: Remove default max_power_wh
* single_test_optimization: Add more cli-parameters
* Workflow docker-build: Don't try to authenticate for PRs
* Secrets are not available anyway for forks.
* footer with date and version
* ruff
* replace toml module with build in
* using re to extract the string
* optimize re usage
* use of use pendulum in Akkudoktor-EOS
* create_line_chart_date function added
* replace datetime with pendulum
* align ax2 with ax1 and 0 first point
* dynamic ticks
* all charts with dates
* style changes
* mypy fixes
* fix test
* fixed current time
Rename FastAPI server to `eos` and FastHTML server to `eosdash`.
Make an user easily identify what server is meant. FastAPI and FastHTML are
implementation details that may confuse the non-technical user.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Use documentation generation tools that are available for Windows and Linux.
Use python instead of shell scripts to generate documentation.
For ReadTheDocs make generated documentation content static to avoid
running scripts outside of the docs/ path which is the default path for ReadTheDOcs.
Add tests that check if generated content does go out of sync with latest source.
Use tabs to show commands for Windows and Linux to improve user experience.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Add documentation that covers:
- configuration
- prediction
Add Python scripts that support automatic documentation generation for
configuration data defined with pydantic.
Adapt EOS configuration to provide more methods for REST API and
automatic documentation generation.
Adapt REST API to allow for EOS configuration file load and save.
Sort REST API on generation of openapi markdown for docs.
Move logutil to core/logging to allow configuration of logging by standard config.
Make Akkudoktor predictions always start extraction of prediction data at start of day.
Previously extraction started at actual hour. This is to support the code that assumes
prediction data to start at start of day.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Add documentation that covers:
- Prediction
- Measuremnt
- REST API
Add Python scripts that support automatic documentation generation using the Sphinx
sphinxcontrib.eval extension.
Add automatic update/ test for REST API documentation.
Filter proxy endpoints from REST API documentation.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Detect active PV planes by pvforecast_surface_tilt and pvforecast_surface_azimuth to be non None. Assure by default these configuration values are None.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Normalize electricity price prediction to €/Wh.
Provide electricity price prediction by €/kWh for convenience.
Allow to configure electricity price charges by €/kWh.
Also added error page to fastapi rest server to get rid of annoying
unrelated fault messages during testing.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* Discharge Mask Bug, Tests updated, simple Price Forecast with linear weighting
* Price Forecast with linear weighting, last value = highest weighting Discharge enforce when soc = 0 -> mask bug
* Inverter: Self consumption interpolator for better discharge_hour results
* Small penalty when EV 100% and charge >0
* Price Forceast (use mean of last 7 days instead of repeat)
* Price Prediction as JSON simulation output, config fixed electricty fees configurable + MyPy & Ruff
Extend single_test_optimization.py to be able to use real world data from new prediction classes.
- .venv/bin/python single_test_optimization.py --real_world --verbose
Can also be run with profiling "--profile".
Add single_test_prediction.py to fetch predictions from remote prediction providers
- .venv/bin/python single_test_prediction.py --verbose --provider-id PVForecastAkkudoktor | more
- .venv/bin/python single_test_prediction.py --verbose --provider-id LoadAkkudoktor | more
- .venv/bin/python single_test_prediction.py --verbose --provider-id ElecPriceAkkudoktor | more
- .venv/bin/python single_test_prediction.py --verbose --provider-id BrightSky | more
- .venv/bin/python single_test_prediction.py --verbose --provider-id ClearOutside | more
Can also be run with profiling "--profile".
single_test_optimization.py is an example on how to retrieve prediction data for optimization and
use it to set up the optimization parameters.
Changes:
- load: Only one load provider at a time (vs. 5 before)
Bug fixes:
- prediction: only use providers that are enabled to retrieve or set data.
- prediction: fix pre pendulum format strings
- dataabc: Prevent error when resampling data with no datasets.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* optimization states for AC, DC and IDLE now similar probab. Also AC states taken from config. Maybe a single config option for AC and E-Auto States is sensefull.
* test_class_optimize: Update testdata
* Write pdf and json to test/testdata/new.... so it can be analyzed
manually or just copied as new expected result.
* workflow: Upload pytest optimization result artifacts (pdf, json)
* Update utilities in utils submodule.
* Add base configuration modules.
* Add server base configuration modules.
* Add devices base configuration modules.
* Add optimization base configuration modules.
* Add utils base configuration modules.
* Add prediction abstract and base classes plus tests.
* Add PV forecast to prediction submodule.
The PV forecast modules are adapted from the class_pvforecast module and
replace it.
* Add weather forecast to prediction submodule.
The modules provide classes and methods to retrieve, manage, and process weather forecast data
from various sources. Includes are structured representations of weather data and utilities
for fetching forecasts for specific locations and time ranges.
BrightSky and ClearOutside are currently supported.
* Add electricity price forecast to prediction submodule.
* Adapt fastapi server to base config and add fasthtml server.
* Add ems to core submodule.
* Adapt genetic to config.
* Adapt visualize to config.
* Adapt common test fixtures to config.
* Add load forecast to prediction submodule.
* Add core abstract and base classes.
* Adapt single test optimization to config.
* Adapt devices to config.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* Migrate from Flask to FastAPI
* FastAPI migration:
- Use pydantic model classes as input parameters to the
data/calculation classes.
- Interface field names changed to constructor parameter names (for
simplicity only during transition, should be updated in a followup
PR).
- Add basic interface requirements (e.g. some values > 0, etc.).
* Update tests for new data format.
* Python requirement down to 3.9 (TypeGuard no longer needed)
* Makefile: Add helpful targets (e.g. development server with reload)
* Move API doc from README to pydantic model classes (swagger)
* Link to swagger.io with own openapi.yml.
* Commit openapi.json and check with pytest for changes so the
documentation is always up-to-date.
* Streamline docker
* FastAPI: Run startup action on dev server
* Fix config for /strompreis, endpoint still broken however.
* test_openapi: Compare against docs/.../openapi.json
* Move fastapi to server/ submodule
* See #187 for new repository structure.
Report data are floats.
The report is used for unit testing which may be affected by the float precision of the test environment.
Round to make the test result more robust without loosing the general test case.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* Add documentation to class_pv_forecast.py.
Added documentation. Beware mostly generated by ChatGPT.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* Add CacheFileStore, datetime and logger utilities.
The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing
temporary file objects, allowing the creation, retrieval, and management of cache files.
The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle
different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection
(`to_timezone).
- Cache files are automatically valid for the the current date unless specified otherwise.
This is to mimic the current behaviour used in several classes.
- The logger supports rotating log files to prevent excessive log file size.
- The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats.
They provide the time conversion that is e.g. used in PVForecast.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* Improve testability of PVForecast
Improvements for testing of PVForecast
- Use common utility functions to allow for general testing at one spot.
- to_datetime
- CacheFileStore
- Use logging instead of print to easily capture in testing.
- Add validation of the json schema for Akkudoktor PV forecast data.
- Allow to create an empty PVForecast instance as base instance for testing.
- Make process_data() complete for filling a PVForecast instance for testing.
- Normalize forecast datetime to timezone of system given in loaded data.
- Do not print report but provide report for test checks.
- Get rid of cache file path using the CachFileStore to automate cache file usage.
- Improved module documentation.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* Add test for PVForecast and newly extracted utility modules.
- Add test for PVForecast
- Add test for CacheFileStore in the new cachefilestore module
- Add test for to_datetime, to_timestamp, to_timezone in the new
datetimeutil module
- Add test for get_logger in the new logutil module
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
---------
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: Normann <github@koldrack.com>
* Integrated single_test_optimization into pytest to run a basic optimization test with tolerance set to 1e-6, ensuring quick detection of deviations.
* Added a long-run test (400 generations, like single_test_optimization), which can be triggered using --full-run in pytest.
* Mocked PDF creation in optimization tests and added a new PDF generation test with image comparison validation.
Note: Current tolerance is set to 1e-6; feedback on whether this tolerance is tight enough is welcome.
---------
Co-authored-by: Normann <github@koldrack.com>
Co-authored-by: Michael Osthege <michael.osthege@outlook.com>
