This fix implements the long term goal to have the EOS server run optimization (or
energy management) on regular intervals automatically. Thus clients can request
the current energy management plan at any time and it is updated on regular
intervals without interaction by the client.
This fix started out to "only" make automatic optimization (or energy management)
runs working. It turned out there are several endpoints that in some way
update predictions or run the optimization. To lock against such concurrent attempts
the code had to be refactored to allow control of execution. During refactoring it
became clear that some classes and files are named without a proper reference
to their usage. Thus not only refactoring but also renaming became necessary.
The names are still not the best, but I hope they are more intuitive.
The fix includes several bug fixes that are not directly related to the automatic optimization
but are necessary to keep EOS running properly to do the automatic optimization and
to test and document the changes.
This is a breaking change as the configuration structure changed once again and
the server API was also enhanced and streamlined. The server API that is used by
Andreas and Jörg in their videos has not changed.
* fix: automatic optimization
Allow optimization to automatically run on configured intervals gathering all
optimization parameters from configuration and predictions. The automatic run
can be configured to only run prediction updates skipping the optimization.
Extend documentaion to also cover automatic optimization. Lock automatic runs
against runs initiated by the /optimize or other endpoints. Provide new
endpoints to retrieve the energy management plan and the genetic solution
of the latest automatic optimization run. Offload energy management to thread
pool executor to keep the app more responsive during the CPU heavy optimization
run.
* fix: EOS servers recognize environment variables on startup
Force initialisation of EOS configuration on server startup to assure
all sources of EOS configuration are properly set up and read. Adapt
server tests and configuration tests to also test for environment
variable configuration.
* fix: Remove 0.0.0.0 to localhost translation under Windows
EOS imposed a 0.0.0.0 to localhost translation under Windows for
convenience. This caused some trouble in user configurations. Now, as the
default IP address configuration is 127.0.0.1, the user is responsible
for to set up the correct Windows compliant IP address.
* fix: allow names for hosts additional to IP addresses
* fix: access pydantic model fields by class
Access by instance is deprecated.
* fix: down sampling key_to_array
* fix: make cache clear endpoint clear all cache files
Make /v1/admin/cache/clear clear all cache files. Before it only cleared
expired cache files by default. Add new endpoint /v1/admin/clear-expired
to only clear expired cache files.
* fix: timezonefinder returns Europe/Paris instead of Europe/Berlin
timezonefinder 8.10 got more inaccurate for timezones in europe as there is
a common timezone. Use new package tzfpy instead which is still returning
Europe/Berlin if you are in Germany. tzfpy also claims to be faster than
timezonefinder.
* fix: provider settings configuration
Provider configuration used to be a union holding the settings for several
providers. Pydantic union handling does not always find the correct type
for a provider setting. This led to exceptions in specific configurations.
Now provider settings are explicit comfiguration items for each possible
provider. This is a breaking change as the configuration structure was
changed.
* fix: ClearOutside weather prediction irradiance calculation
Pvlib needs a pandas time index. Convert time index.
* fix: test config file priority
Do not use config_eos fixture as this fixture already creates a config file.
* fix: optimization sample request documentation
Provide all data in documentation of optimization sample request.
* fix: gitlint blocking pip dependency resolution
Replace gitlint by commitizen. Gitlint is not actively maintained anymore.
Gitlint dependencies blocked pip from dependency resolution.
* fix: sync pre-commit config to actual dependency requirements
.pre-commit-config.yaml was out of sync, also requirements-dev.txt.
* fix: missing babel in requirements.txt
Add babel to requirements.txt
* feat: setup default device configuration for automatic optimization
In case the parameters for automatic optimization are not fully defined a
default configuration is setup to allow the automatic energy management
run. The default configuration may help the user to correctly define
the device configuration.
* feat: allow configuration of genetic algorithm parameters
The genetic algorithm parameters for number of individuals, number of
generations, the seed and penalty function parameters are now avaliable
as configuration options.
* feat: allow configuration of home appliance time windows
The time windows a home appliance is allowed to run are now configurable
by the configuration (for /v1 API) and also by the home appliance parameters
(for the classic /optimize API). If there is no such configuration the
time window defaults to optimization hours, which was the standard before
the change. Documentation on how to configure time windows is added.
* feat: standardize mesaurement keys for battery/ ev SoC measurements
The standardized measurement keys to report battery SoC to the device
simulations can now be retrieved from the device configuration as a
read-only config option.
* feat: feed in tariff prediction
Add feed in tarif predictions needed for automatic optimization. The feed in
tariff can be retrieved as fixed feed in tarif or can be imported. Also add
tests for the different feed in tariff providers. Extend documentation to
cover the feed in tariff providers.
* feat: add energy management plan based on S2 standard instructions
EOS can generate an energy management plan as a list of simple instructions.
May be retrieved by the /v1/energy-management/plan endpoint. The instructions
loosely follow the S2 energy management standard.
* feat: make measurement keys configurable by EOS configuration.
The fixed measurement keys are replaced by configurable measurement keys.
* feat: make pendulum DateTime, Date, Duration types usable for pydantic models
Use pydantic_extra_types.pendulum_dt to get pydantic pendulum types. Types are
added to the datetimeutil utility. Remove custom made pendulum adaptations
from EOS pydantic module. Make EOS modules use the pydantic pendulum types
managed by the datetimeutil module instead of the core pendulum types.
* feat: Add Time, TimeWindow, TimeWindowSequence and to_time to datetimeutil.
The time windows are are added to support home appliance time window
configuration. All time classes are also pydantic models. Time is the base
class for time definition derived from pendulum.Time.
* feat: Extend DataRecord by configurable field like data.
Configurable field like data was added to support the configuration of
measurement records.
* feat: Add additional information to health information
Version information is added to the health endpoints of eos and eosDash.
The start time of the last optimization and the latest run time of the energy
management is added to the EOS health information.
* feat: add pydantic merge model tests
* feat: add plan tab to EOSdash
The plan tab displays the current energy management instructions.
* feat: add predictions tab to EOSdash
The predictions tab displays the current predictions.
* feat: add cache management to EOSdash admin tab
The admin tab is extended by a section for cache management. It allows to
clear the cache.
* feat: add about tab to EOSdash
The about tab resembles the former hello tab and provides extra information.
* feat: Adapt changelog and prepare for release management
Release management using commitizen is added. The changelog file is adapted and
teh changelog and a description for release management is added in the
documentation.
* feat(doc): Improve install and devlopment documentation
Provide a more concise installation description in Readme.md and add extra
installation page and development page to documentation.
* chore: Use memory cache for interpolation instead of dict in inverter
Decorate calculate_self_consumption() with @cachemethod_until_update to cache
results in memory during an energy management/ optimization run. Replacement
of dict type caching in inverter is now possible because all optimization
runs are properly locked and the memory cache CacheUntilUpdateStore is properly
cleared at the start of any energy management/ optimization operation.
* chore: refactor genetic
Refactor the genetic algorithm modules for enhanced module structure and better
readability. Removed unnecessary and overcomplex devices singleton. Also
split devices configuration from genetic algorithm parameters to allow further
development independently from genetic algorithm parameter format. Move
charge rates configuration for electric vehicles from optimization to devices
configuration to allow to have different charge rates for different cars in
the future.
* chore: Rename memory cache to CacheEnergyManagementStore
The name better resembles the task of the cache to chache function and method
results for an energy management run. Also the decorator functions are renamed
accordingly: cachemethod_energy_management, cache_energy_management
* chore: use class properties for config/ems/prediction mixin classes
* chore: skip debug logs from mathplotlib
Mathplotlib is very noisy in debug mode.
* chore: automatically sync bokeh js to bokeh python package
bokeh was updated to 3.8.0, make JS CDN automatically follow the package version.
* chore: rename hello.py to about.py
Make hello.py the adapted EOSdash about page.
* chore: remove demo page from EOSdash
As no the plan and prediction pages are working without configuration, the demo
page is no longer necessary
* chore: split test_server.py for system test
Split test_server.py to create explicit test_system.py for system tests.
* chore: move doc utils to generate_config_md.py
The doc utils are only used in scripts/generate_config_md.py. Move it there to
attribute for strong cohesion.
* chore: improve pydantic merge model documentation
* chore: remove pendulum warning from readme
* chore: remove GitHub discussions from contributing documentation
Github discussions is to be replaced by Akkudoktor.net.
* chore(release): bump version to 0.1.0+dev for development
* build(deps): bump fastapi[standard] from 0.115.14 to 0.117.1
bump fastapi and make coverage version (for pytest-cov) explicit to avoid pip break.
* build(deps): bump uvicorn from 0.36.0 to 0.37.0
BREAKING CHANGE: EOS configuration changed. V1 API changed.
- The available_charge_rates_percent configuration is removed from optimization.
Use the new charge_rate configuration for the electric vehicle
- Optimization configuration parameter hours renamed to horizon_hours
- Device configuration now has to provide the number of devices and device
properties per device.
- Specific prediction provider configuration to be provided by explicit
configuration item (no union for all providers).
- Measurement keys to be provided as a list.
- New feed in tariff providers have to be configured.
- /v1/measurement/loadxxx endpoints are removed. Use generic mesaurement endpoints.
- /v1/admin/cache/clear now clears all cache files. Use
/v1/admin/cache/clear-expired to only clear all expired cache files.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* feat(ElecPriceEnergyCharts): Add new electricity price provider: Energy-Charts
* feat(ElecPriceEnergyCharts): update data only if needed
* test(elecpriceforecast): add test for energycharts
* docs(predictions.md): add ElecPriceEnergyCharts Provider
Signed-off-by: redmoon2711 <redmoon2711@gmx.de>
Added bandit checks to continuous integration.
Updated sources to pass bandit checks:
- replaced asserts
- added timeouts to requests
- added checks for process command execution
- changed to 127.0.0.1 as default IP address for EOS and EOSdash for security reasons
Added a rudimentary check for outdated config files.
BREAKING CHANGE: Default IP address for EOS and EOSdash changed to 127.0.0.1
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* fix: pvforecast fails when there is only a single plane
* fix: formatting
* fix: formatting
* fix: add type annotations
* add testdata and validation test for single plane
* fix: formatting
EOS now enforces the general azimuth definition as e.g. defined in ISO 19111:
north=0, east=90, south=180, west=270. This is the convention that is and was
in the EOS documentation.
As the PV forecast of akkudoktor.net follows a different convention
(north=+-180, east=-90, south=0, west=90) the values from EOS are now converted
before the request is sent to akkudoktor.net.
BREAKING CHANGE: Azimuth configurations that followed the PVForecastAkkudoktor convention
(north=+-180, east=-90, south=0, west=90) must be converted to the general azimuth definition:
north=0, east=90, south=180, west=270.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
- Get weather data with fully specified end_date datetime argument to not miss data.
- Make preciptable water records generation robust against missing temperature
or humidity values.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* Move the caching module to core.
Add an in memory cache that for caching function and method results
during an energy management run (optimization run). Two decorators
are provided for methods and functions.
* Improve the file cache store by load and save functions.
Make EOS load the cache file store on startup and save it on shutdown.
Add a cyclic task that cleans the cache file store from outdated cache files.
* Improve startup of EOSdash by EOS
Make EOS starting EOSdash adhere to path configuration given in EOS.
The whole environment from EOS is now passed to EOSdash.
Should also prevent test errors due to unwanted/ wrong config file creation.
Both servers now provide a health endpoint that can be used to detect whether
the server is running. This is also used for testing now.
* Improve startup of EOS
EOS now has got an energy management task that runs shortly after startup.
It tries to execute energy management runs with predictions newly fetched
or initialized from cached data on first run.
* Improve shutdown of EOS
EOS has now a shutdown task that shuts EOS down gracefully with some
time delay to allow REST API requests for shutdwon or restart to be fully serviced.
* Improve EMS
Add energy management task for repeated energy management controlled by
startup delay and interval configuration parameters.
Translate EnergieManagementSystem to english EnergyManagement.
* Add administration endpoints
- endpoints to control caching from REST API.
- endpoints to control server restart (will not work on Windows) and shutdown from REST API
* Improve doc generation
Use "\n" linenend convention also on Windows when generating doc files.
Replace Windows specific 127.0.0.1 address by standard 0.0.0.0.
* Improve test support (to be able to test caching)
- Add system test option to pytest for running tests with "real" resources
- Add new test fixture to start server for test class and test function
- Make kill signal adapt to Windows/ Linux
- Use consistently "\n" for lineends when writing text files in doc test
- Fix test_logging under Windows
- Fix conftest config_default_dirs test fixture under Windows
From @Lasall
* Improve Windows support
- Use 127.0.0.1 as default config host (model defaults) and
addionally redirect 0.0.0.0 to localhost on Windows (because default
config file still has 0.0.0.0).
- Update install/startup instructions as package installation is
required atm.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* All config now nested.
- Use default config from model field default values. If providers
should be enabled by default, non-empty default config file could
be provided again.
- Environment variable support with EOS_ prefix and __ between levels,
e.g. EOS_SERVER__EOS_SERVER_PORT=8503 where all values are case
insensitive.
For more information see:
https://docs.pydantic.dev/latest/concepts/pydantic_settings/#parsing-environment-variable-values
- Use devices as registry for configured devices. DeviceBase as base
class with for now just initializion support (in the future expand
to operations during optimization).
- Strip down ConfigEOS to the only configuration instance. Reload
from file or reset to defaults is possible.
* Fix multi-initialization of derived SingletonMixin classes.
* Cleanup: parameters: extra=forbid, optimize: battery, inverter optional
* Don't allow extra fields for parameters/REST-API (at least for now while
changing API).
* Allow both battery and inverter to be set optionally (atm optional
battery not implemented, no API constraints).
* inverter: Remove default max_power_wh
* single_test_optimization: Add more cli-parameters
* Workflow docker-build: Don't try to authenticate for PRs
* Secrets are not available anyway for forks.
* footer with date and version
* ruff
* replace toml module with build in
* using re to extract the string
* optimize re usage
* use of use pendulum in Akkudoktor-EOS
* create_line_chart_date function added
* replace datetime with pendulum
* align ax2 with ax1 and 0 first point
* dynamic ticks
* all charts with dates
* style changes
* mypy fixes
* fix test
* fixed current time
Rename FastAPI server to `eos` and FastHTML server to `eosdash`.
Make an user easily identify what server is meant. FastAPI and FastHTML are
implementation details that may confuse the non-technical user.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Use documentation generation tools that are available for Windows and Linux.
Use python instead of shell scripts to generate documentation.
For ReadTheDocs make generated documentation content static to avoid
running scripts outside of the docs/ path which is the default path for ReadTheDOcs.
Add tests that check if generated content does go out of sync with latest source.
Use tabs to show commands for Windows and Linux to improve user experience.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Add documentation that covers:
- configuration
- prediction
Add Python scripts that support automatic documentation generation for
configuration data defined with pydantic.
Adapt EOS configuration to provide more methods for REST API and
automatic documentation generation.
Adapt REST API to allow for EOS configuration file load and save.
Sort REST API on generation of openapi markdown for docs.
Move logutil to core/logging to allow configuration of logging by standard config.
Make Akkudoktor predictions always start extraction of prediction data at start of day.
Previously extraction started at actual hour. This is to support the code that assumes
prediction data to start at start of day.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Add documentation that covers:
- Prediction
- Measuremnt
- REST API
Add Python scripts that support automatic documentation generation using the Sphinx
sphinxcontrib.eval extension.
Add automatic update/ test for REST API documentation.
Filter proxy endpoints from REST API documentation.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Detect active PV planes by pvforecast_surface_tilt and pvforecast_surface_azimuth to be non None. Assure by default these configuration values are None.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Normalize electricity price prediction to €/Wh.
Provide electricity price prediction by €/kWh for convenience.
Allow to configure electricity price charges by €/kWh.
Also added error page to fastapi rest server to get rid of annoying
unrelated fault messages during testing.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* Discharge Mask Bug, Tests updated, simple Price Forecast with linear weighting
* Price Forecast with linear weighting, last value = highest weighting Discharge enforce when soc = 0 -> mask bug
* Inverter: Self consumption interpolator for better discharge_hour results
* Small penalty when EV 100% and charge >0
* Price Forceast (use mean of last 7 days instead of repeat)
* Price Prediction as JSON simulation output, config fixed electricty fees configurable + MyPy & Ruff
Extend single_test_optimization.py to be able to use real world data from new prediction classes.
- .venv/bin/python single_test_optimization.py --real_world --verbose
Can also be run with profiling "--profile".
Add single_test_prediction.py to fetch predictions from remote prediction providers
- .venv/bin/python single_test_prediction.py --verbose --provider-id PVForecastAkkudoktor | more
- .venv/bin/python single_test_prediction.py --verbose --provider-id LoadAkkudoktor | more
- .venv/bin/python single_test_prediction.py --verbose --provider-id ElecPriceAkkudoktor | more
- .venv/bin/python single_test_prediction.py --verbose --provider-id BrightSky | more
- .venv/bin/python single_test_prediction.py --verbose --provider-id ClearOutside | more
Can also be run with profiling "--profile".
single_test_optimization.py is an example on how to retrieve prediction data for optimization and
use it to set up the optimization parameters.
Changes:
- load: Only one load provider at a time (vs. 5 before)
Bug fixes:
- prediction: only use providers that are enabled to retrieve or set data.
- prediction: fix pre pendulum format strings
- dataabc: Prevent error when resampling data with no datasets.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* optimization states for AC, DC and IDLE now similar probab. Also AC states taken from config. Maybe a single config option for AC and E-Auto States is sensefull.
* test_class_optimize: Update testdata
* Write pdf and json to test/testdata/new.... so it can be analyzed
manually or just copied as new expected result.
* workflow: Upload pytest optimization result artifacts (pdf, json)
* Update utilities in utils submodule.
* Add base configuration modules.
* Add server base configuration modules.
* Add devices base configuration modules.
* Add optimization base configuration modules.
* Add utils base configuration modules.
* Add prediction abstract and base classes plus tests.
* Add PV forecast to prediction submodule.
The PV forecast modules are adapted from the class_pvforecast module and
replace it.
* Add weather forecast to prediction submodule.
The modules provide classes and methods to retrieve, manage, and process weather forecast data
from various sources. Includes are structured representations of weather data and utilities
for fetching forecasts for specific locations and time ranges.
BrightSky and ClearOutside are currently supported.
* Add electricity price forecast to prediction submodule.
* Adapt fastapi server to base config and add fasthtml server.
* Add ems to core submodule.
* Adapt genetic to config.
* Adapt visualize to config.
* Adapt common test fixtures to config.
* Add load forecast to prediction submodule.
* Add core abstract and base classes.
* Adapt single test optimization to config.
* Adapt devices to config.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* Migrate from Flask to FastAPI
* FastAPI migration:
- Use pydantic model classes as input parameters to the
data/calculation classes.
- Interface field names changed to constructor parameter names (for
simplicity only during transition, should be updated in a followup
PR).
- Add basic interface requirements (e.g. some values > 0, etc.).
* Update tests for new data format.
* Python requirement down to 3.9 (TypeGuard no longer needed)
* Makefile: Add helpful targets (e.g. development server with reload)
* Move API doc from README to pydantic model classes (swagger)
* Link to swagger.io with own openapi.yml.
* Commit openapi.json and check with pytest for changes so the
documentation is always up-to-date.
* Streamline docker
* FastAPI: Run startup action on dev server
* Fix config for /strompreis, endpoint still broken however.
* test_openapi: Compare against docs/.../openapi.json
* Move fastapi to server/ submodule
* See #187 for new repository structure.
Report data are floats.
The report is used for unit testing which may be affected by the float precision of the test environment.
Round to make the test result more robust without loosing the general test case.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* Add documentation to class_pv_forecast.py.
Added documentation. Beware mostly generated by ChatGPT.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* Add CacheFileStore, datetime and logger utilities.
The `CacheFileStore` class is a singleton-based, thread-safe key-value store for managing
temporary file objects, allowing the creation, retrieval, and management of cache files.
The utility modules offer a flexible logging setup (`get_logger`) and utilities to handle
different date-time formats (`to_datetime`, `to_timestamp`) and timezone detection
(`to_timezone).
- Cache files are automatically valid for the the current date unless specified otherwise.
This is to mimic the current behaviour used in several classes.
- The logger supports rotating log files to prevent excessive log file size.
- The `to_datetime` and `to_timestamp`functions support a wide variety of input types and formats.
They provide the time conversion that is e.g. used in PVForecast.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* Improve testability of PVForecast
Improvements for testing of PVForecast
- Use common utility functions to allow for general testing at one spot.
- to_datetime
- CacheFileStore
- Use logging instead of print to easily capture in testing.
- Add validation of the json schema for Akkudoktor PV forecast data.
- Allow to create an empty PVForecast instance as base instance for testing.
- Make process_data() complete for filling a PVForecast instance for testing.
- Normalize forecast datetime to timezone of system given in loaded data.
- Do not print report but provide report for test checks.
- Get rid of cache file path using the CachFileStore to automate cache file usage.
- Improved module documentation.
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
* Add test for PVForecast and newly extracted utility modules.
- Add test for PVForecast
- Add test for CacheFileStore in the new cachefilestore module
- Add test for to_datetime, to_timestamp, to_timezone in the new
datetimeutil module
- Add test for get_logger in the new logutil module
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
---------
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: Normann <github@koldrack.com>
* Integrated single_test_optimization into pytest to run a basic optimization test with tolerance set to 1e-6, ensuring quick detection of deviations.
* Added a long-run test (400 generations, like single_test_optimization), which can be triggered using --full-run in pytest.
* Mocked PDF creation in optimization tests and added a new PDF generation test with image comparison validation.
Note: Current tolerance is set to 1e-6; feedback on whether this tolerance is tight enough is welcome.
---------
Co-authored-by: Normann <github@koldrack.com>
Co-authored-by: Michael Osthege <michael.osthege@outlook.com>
