fix: optimization fail after restart (#1007)

Fix documentation for the loadforecast_power_w key.

Fix documentation to explain the usage of import file/ JSON string to
primarily initialise prediction data.

Fix code scanning alert no. 6: URL redirection from remote source

Enable to automatically save the configuration to the configuration file
by default, which is a widespread user expectation.

Make the genetic parameters non optional for better pydantic compliance.

Update:
- bump pytest to 9.0.3
- bump pillow to 12.2.0
- bump platformdirs to 4.9.6
- bump typespyyaml to 6.0.12.20260408
- bump tzfpy to 1.2.0
- bump pydantic to 2.13.0
- bump types-requests to 2.33.0.20260408

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Co-authored-by: Normann <github@koldrack.com>

docs/_generated/configexample.md

@@ -125,6 +125,8 @@
            }
        },
        "general": {
+           "config_save_mode": "AUTOMATIC",
+           "config_save_interval_sec": 60,
            "version": "0.0.0",
            "data_folder_path": "/home/user/.local/share/net.akkudoktoreos.net",
            "data_output_subpath": "output",

docs/_generated/configgeneral.md

@@ -9,6 +9,8 @@
 | ---- | -------------------- | ---- | --------- | ------- | ----------- |
 | config_file_path | | `Optional[pathlib.Path]` | `ro` | `N/A` | Path to EOS configuration file. |
 | config_folder_path | | `Optional[pathlib.Path]` | `ro` | `N/A` | Path to EOS configuration directory. |
+| config_save_interval_sec | `EOS_GENERAL__CONFIG_SAVE_INTERVAL_SEC` | `int` | `rw` | `60` | Automatic configuration file saving interval [seconds]. |
+| config_save_mode | `EOS_GENERAL__CONFIG_SAVE_MODE` | `<enum 'ConfigSaveMode'>` | `rw` | `AUTOMATIC` | Configuration file save mode for configuration changes ['MANUAL', 'AUTOMATIC']. Defaults to 'AUTOMATIC'. |
 | data_folder_path | `EOS_GENERAL__DATA_FOLDER_PATH` | `Path` | `rw` | `required` | Path to EOS data folder. |
 | data_output_path | | `Optional[pathlib.Path]` | `ro` | `N/A` | Computed data_output_path based on data_folder_path. |
 | data_output_subpath | `EOS_GENERAL__DATA_OUTPUT_SUBPATH` | `Optional[pathlib.Path]` | `rw` | `output` | Sub-path for the EOS output data folder. |
@@ -28,6 +30,8 @@
 ```json
    {
        "general": {
+           "config_save_mode": "AUTOMATIC",
+           "config_save_interval_sec": 60,
            "version": "0.0.0",
            "data_folder_path": "/home/user/.local/share/net.akkudoktoreos.net",
            "data_output_subpath": "output",
@@ -46,6 +50,8 @@
 ```json
    {
        "general": {
+           "config_save_mode": "AUTOMATIC",
+           "config_save_interval_sec": 60,
            "version": "0.0.0",
            "data_folder_path": "/home/user/.local/share/net.akkudoktoreos.net",
            "data_output_subpath": "output",

docs/_generated/configload.md

@@ -126,7 +126,7 @@
        "load": {
            "loadimport": {
                "import_file_path": null,
-               "import_json": "{\"load0_mean\": [676.71, 876.19, 527.13]}"
+               "import_json": "{\"loadforecast_power_w\": [676.71, 876.19, 527.13]}"
            }
        }
    }

docs/_generated/configoptimization.md

@@ -7,10 +7,11 @@
 
 | Name | Environment Variable | Type | Read-Only | Default | Description |
 | ---- | -------------------- | ---- | --------- | ------- | ----------- |
-| algorithm | `EOS_OPTIMIZATION__ALGORITHM` | `Optional[str]` | `rw` | `GENETIC` | The optimization algorithm. |
-| genetic | `EOS_OPTIMIZATION__GENETIC` | `Optional[akkudoktoreos.optimization.optimization.GeneticCommonSettings]` | `rw` | `None` | Genetic optimization algorithm configuration. |
-| horizon_hours | `EOS_OPTIMIZATION__HORIZON_HOURS` | `Optional[int]` | `rw` | `24` | The general time window within which the energy optimization goal shall be achieved [h]. Defaults to 24 hours. |
-| interval | `EOS_OPTIMIZATION__INTERVAL` | `Optional[int]` | `rw` | `3600` | The optimization interval [sec]. |
+| algorithm | `EOS_OPTIMIZATION__ALGORITHM` | `str` | `rw` | `GENETIC` | The optimization algorithm. Defaults to GENETIC |
+| genetic | `EOS_OPTIMIZATION__GENETIC` | `GeneticCommonSettings` | `rw` | `required` | Genetic optimization algorithm configuration. |
+| horizon | | `int` | `ro` | `N/A` | Number of optimization steps. |
+| horizon_hours | `EOS_OPTIMIZATION__HORIZON_HOURS` | `int` | `rw` | `24` | The general time window within which the energy optimization goal shall be achieved [h]. Defaults to 24 hours. |
+| interval | `EOS_OPTIMIZATION__INTERVAL` | `int` | `rw` | `3600` | The optimization interval [sec]. Defaults to 3600 seconds (1 hour) |
 | keys | | `list[str]` | `ro` | `N/A` | The keys of the solution. |
 :::
 <!-- pyml enable line-length -->
@@ -58,7 +59,8 @@
                    "ev_soc_miss": 10
                }
            },
-           "keys": []
+           "keys": [],
+           "horizon": 24
        }
    }
 ```
@@ -73,10 +75,10 @@
 
 | Name | Type | Read-Only | Default | Description |
 | ---- | ---- | --------- | ------- | ----------- |
-| generations | `Optional[int]` | `rw` | `400` | Number of generations to evaluate the optimal solution [>= 10]. Defaults to 400. |
-| individuals | `Optional[int]` | `rw` | `300` | Number of individuals (solutions) to generate for the (initial) generation [>= 10]. Defaults to 300. |
-| penalties | `dict[str, Union[float, int, str]]` | `rw` | `required` | A dictionary of penalty function parameters consisting of a penalty function parameter name and the associated value. |
-| seed | `Optional[int]` | `rw` | `None` | Fixed seed for genetic algorithm. Defaults to 'None' which means random seed. |
+| generations | `Optional[int]` | `rw` | `400` | Number of generations to evolve [>= 10]. Defaults to 400. |
+| individuals | `Optional[int]` | `rw` | `300` | Number of individuals (solutions) in the population [>= 10]. Defaults to 300. |
+| penalties | `dict[str, Union[float, int, str]]` | `rw` | `required` | Penalty parameters used in fitness evaluation. |
+| seed | `Optional[int]` | `rw` | `None` | Random seed for reproducibility. None = random. |
 :::
 <!-- pyml enable line-length -->
 

docs/_generated/openapi.md

@@ -1,6 +1,6 @@
 # Akkudoktor-EOS
 
-**Version**: `v0.3.0.dev2603180781250771`
+**Version**: `v0.3.0.dev2604141105859917`
 
 <!-- pyml disable line-length -->
 **Description**: This project provides a comprehensive solution for simulating and optimizing an energy system based on renewable energy sources. With a focus on photovoltaic (PV) systems, battery storage (batteries), load management (consumer requirements), heat pumps, electric vehicles, and consideration of electricity price data, this system enables forecasting and optimization of energy flow and costs over a specified period.
@@ -1348,13 +1348,13 @@ Return:
 - `application/json`: {
   "anyOf": [
     {
-      "$ref": "#/components/schemas/PowerMeasurement-Input"
+      "$ref": "#/components/schemas/PowerMeasurement"
     },
     {
-      "$ref": "#/components/schemas/EnergyMeasurement-Input"
+      "$ref": "#/components/schemas/EnergyMeasurement"
     },
     {
-      "$ref": "#/components/schemas/PPBCPowerProfileStatus-Input"
+      "$ref": "#/components/schemas/PPBCPowerProfileStatus"
     },
     {
       "$ref": "#/components/schemas/OMBCStatus"
@@ -1363,7 +1363,7 @@ Return:
       "$ref": "#/components/schemas/FRBCActuatorStatus"
     },
     {
-      "$ref": "#/components/schemas/FRBCEnergyStatus-Input"
+      "$ref": "#/components/schemas/FRBCEnergyStatus"
     },
     {
       "$ref": "#/components/schemas/FRBCStorageStatus"

docs/akkudoktoreos/configuration.md

@@ -8,9 +8,7 @@ management.
 
 ## Storing Configuration
 
-EOS stores configuration data in a `nested structure`. Note that configuration changes inside EOS
-are updated in memory, meaning all changes will be lost upon restarting the EOS REST server if not
-saved to the `EOS configuration file`.
+EOS stores configuration data in a `nested structure`.
 
 Some `configuration keys` are read-only and cannot be altered. These keys are either set up by other
 means, such as environment variables, or determined from other information.
@@ -18,8 +16,18 @@ means, such as environment variables, or determined from other information.
 Several endpoints of the EOS REST server allow for the management and retrieval of configuration
 data.
 
+:::{admonition} Note
+:class: note
+Configuration changes inside EOS are updated in memory, meaning all changes will be lost upon
+restarting the EOS REST server unless the configuration is saved to the `EOS configuration file`.
+This can be done manually or is done automatically by default.
+:::
+
 ### Save Configuration File
 
+Configure EOS for `AUTOMATIC` (vs. `MANUAL`) configuration file update in case of configuration
+changes.
+
 Use endpoint `PUT /v1/config/file` to save the current configuration to the
 `EOS configuration file`.
 
@@ -39,13 +47,18 @@ The configuration sources and their priorities are as follows:
 
 ### Runtime Config Updates
 
-The EOS configuration can be updated at runtime. Note that those updates are not persistent
-automatically. However it is possible to save the configuration to the `EOS configuration file`.
+The EOS configuration can be updated at runtime.
 
 Use the following endpoints to change the current runtime configuration:
 
 - `PUT /v1/config`: Update the entire or parts of the configuration.
 
+:::{admonition} Note
+:class: note
+Those updates are not persistent automatically. However it is possible to save the configuration to
+the `EOS configuration file`. See [Save Configuration File](#save-configuration-file) above.
+:::
+
 ### Environment Variables
 
 All `configuration keys` can be set by environment variables prefixed with `EOS_` and separated by

docs/akkudoktoreos/prediction.md

@@ -38,9 +38,22 @@ to prediction keys like:
 
 ### Prediction Import Providers
 
-The prediction import providers are designed to import prediction data from a file or a JSON
-string. An external entity should update the file or JSON string whenever new prediction data
-becomes available.
+Prediction import providers allow you to import prediction data from:
+
+- A file or a JSON string (primarily for initialization), or
+- The **PUT** `/v1/prediction/import/{provider_id}` endpoint (recommended for dynamic updates).
+
+An external entity may update the file or JSON string whenever new prediction data becomes
+available. However, for production use or regular updates, you should prefer the **PUT** endpoint
+over file or JSON string based imports.
+
+:::{admonition} Warning
+:class: warning
+Be aware that providing dynamic values via a file and/or JSON string **alongside other value
+sources** can lead to unintended data overwrites. Moreover, after a restart, even outdated values
+from the configuration may be reloaded. To avoid these issues, use the **PUT** endpoint for live
+data and rely on file/JSON imports only for initial setup.
+:::
 
 The prediction data must be provided in one of the following formats:
 
@@ -124,7 +137,7 @@ Configuration options:
     - `ElecPriceAkkudoktor`: Retrieves from Akkudoktor.net.
     - `ElecPriceEnergyCharts`: Retrieves from Energy-Charts.info.
     - `ElecPriceFixed`: Caluclates from configured time window prices.
-    - `ElecPriceImport`: Imports from a file or JSON string.
+    - `ElecPriceImport`: Imports from a file or JSON string or by endpoint data provision.
 
   - `charges_kwh`: Electricity price charges (€/kWh).
   - `vat_rate`: VAT rate factor applied to electricity price when charges are used (default: 1.19).
@@ -166,20 +179,30 @@ of electricity price time windows set up by the user.
 
 ### ElecPriceImport Provider
 
-The `ElecPriceImport` provider is designed to import electricity prices from a file or a JSON
-string. An external entity should update the file or JSON string whenever new prediction data
-becomes available.
+The `ElecPriceImport` provider is designed to import electricity prices from:
+
+- A file or a JSON string (primarily for initialization). The data source can be given in the
+  `import_file_path` or `import_json` configuration option.
+- The **PUT** `/v1/prediction/import/ElecPriceImport` endpoint (recommended for dynamic updates).
 
 The prediction key for the electricity price forecast data is:
 
 - `elecprice_marketprice_wh`: Electricity market price per Wh (€/Wh).
 
-The electricity proce forecast data must be provided in one of the formats described in
-<project:#prediction-import-providers>. The data source can be given in the
-`import_file_path` or `import_json` configuration option.
+The electricity price forecast data must be provided in one of the formats described in
+<project:#prediction-import-providers>.
 
-The data may additionally or solely be provided by the
-**PUT** `/v1/prediction/import/ElecPriceImport` endpoint.
+An external entity may update the file or JSON string whenever new prediction data becomes
+available. However, for production use or regular updates, you should prefer the **PUT** endpoint
+over file or JSON string based imports.
+
+:::{admonition} Warning
+:class: warning
+Be aware that providing dynamic values via a file and/or JSON string **alongside other value
+sources** can lead to unintended data overwrites. Moreover, after a restart, even outdated values
+from the configuration may be reloaded. To avoid these issues, use the **PUT** endpoint for live
+data and rely on file/JSON imports only for initial setup.
+:::
 
 ## Feed In Tariff Prediction
 
@@ -195,19 +218,44 @@ Configuration options:
   - `provider`: Feed in tariff provider id of provider to be used.
 
     - `FeedInTariffFixed`: Provides fixed feed in tariff values.
-    - `FeedInTariffImport`: Imports from a file or JSON string.
+    - `FeedInTariffImport`: Imports from a file or JSON string or by endpoint data provision.
 
   - `provider_settings.feed_in_tariff_kwh`: Fixed feed in tariff (€/kWh).
   - `provider_settings.import_file_path`: Path to the file to import feed in tariff forecast data from.
   - `provider_settings.import_json`: JSON string, dictionary of feed in tariff value lists.
 
+### FeedInTariffImport Provider
+
+The `FeedInTariffImport` provider is designed to import feed in tariff prices from:
+
+- A file or a JSON string (primarily for initialization). The data source can be given in the
+  `import_file_path` or `import_json` configuration option.
+- The **PUT** `/v1/prediction/import/FeedInTariffImport` endpoint (recommended for dynamic updates).
+
+The prediction key for the feed in tariff price forecast data is:
+
+- `feed_in_tariff_wh`: Feed in tariff price per Wh (€/Wh).
+
+The feed in tariff price forecast data must be provided in one of the formats described in
+<project:#prediction-import-providers>.
+
+An external entity may update the file or JSON string whenever new prediction data becomes
+available. However, for production use or regular updates, you should prefer the **PUT** endpoint
+over file or JSON string based imports.
+
+:::{admonition} Warning
+:class: warning
+Be aware that providing dynamic values via a file and/or JSON string **alongside other value
+sources** can lead to unintended data overwrites. Moreover, after a restart, even outdated values
+from the configuration may be reloaded. To avoid these issues, use the **PUT** endpoint for live
+data and rely on file/JSON imports only for initial setup.
+:::
+
 ## Load Prediction
 
 Prediction keys:
 
 - `loadforecast_power_w`: Predicted load mean value (W).
-- `load_std`: Predicted load standard deviation (W).
-- `load_mean_adjusted`: Predicted load mean value adjusted by load measurement (W).
 
 Configuration options:
 
@@ -217,7 +265,7 @@ Configuration options:
 
     - `LoadAkkudoktor`: Retrieves from local database.
     - `LoadVrm`: Retrieves data from the VRM API by Victron Energy.
-    - `LoadImport`: Imports from a file or JSON string.
+    - `LoadImport`: Imports from a file or JSON string or by endpoint data provision.
 
   - `provider_settings.LoadAkkudoktor.loadakkudoktor_year_energy_kwh`: Yearly energy consumption (kWh).
   - `provider_settings.LoadVRM.load_vrm_token`: API token.
@@ -231,6 +279,12 @@ The `LoadAkkudoktor` provider retrieves generic load data from the local databas
 it to match the annual energy consumption specified in the
 `LoadAkkudoktor.loadakkudoktor_year_energy` configuration option.
 
+Prediction keys:
+
+- `loadforecast_power_w`: Predicted load mean value (W).
+- `loadakkudoktor_mean_power_w`: Predicted load mean value (W). Same as `loadforecast_power_w`.
+- `loadakkudoktor_std_power_w`: Predicted load standard deviation (W).
+
 ### LoadAkkudoktorAdjusted Provider
 
 The `LoadAkkudoktorAdjusted` provider retrieves generic load data from the local database and scales
@@ -239,14 +293,20 @@ it to match the annual energy consumption specified in the
 the forecast by incorporating available measured load data, ensuring a more realistic and
 site-specific consumption profile.
 
+Prediction keys:
+
+- `loadforecast_power_w`: Adjusted load mean value (W).
+- `loadakkudoktor_mean_power_w`: Predicted load mean value (W).
+- `loadakkudoktor_std_power_w`: Predicted load standard deviation (W).
+
 For details on how to supply load measurements, see the [Measurements](measurement-page) section.
 
 ### LoadVrm Provider
 
 The `LoadVrm` provider retrieves load forecast data from the VRM API by Victron Energy.
 To receive forecasts, the system data must be configured under Dynamic ESS in the VRM portal.
-To query the forecasts, an API token is required, which can also be created in the VRM portal under Preferences.
-This token must be stored in the EOS configuration along with the VRM-Installations-ID.
+To query the forecasts, an API token is required, which can also be created in the VRM portal under
+Preferences. This token must be stored in the EOS configuration along with the VRM-Installations-ID.
 
 ```json
     {
@@ -262,28 +322,36 @@ This token must be stored in the EOS configuration along with the VRM-Installati
     }
 ```
 
-The prediction keys for the load forecast data are:
+The prediction key for the load forecast data is:
 
-- `load_mean`: Predicted load mean value (W).
+- `loadforecast_power_w`: Predicted load mean value (W).
 
 ### LoadImport Provider
 
-The `LoadImport` provider is designed to import load forecast data from a file or a JSON
-string. An external entity should update the file or JSON string whenever new prediction data
-becomes available.
+The `LoadImport` provider is designed to import load forecast data from:
 
-The prediction keys for the load forecast data are:
+- A file or a JSON string (primarily for initialization). The data source can be given in the
+  `loadimport_file_path` or `loadimport_json` configuration option.
+- The **PUT** `/v1/prediction/import/LoadImport` endpoint (recommended for dynamic updates).
 
-- `load_mean`: Predicted load mean value (W).
-- `load_std`: Predicted load standard deviation (W).
-- `load_mean_adjusted`: Predicted load mean value adjusted by load measurement (W).
+The prediction key for the load forecast data is:
+
+- `loadforecast_power_w`: Predicted load mean value (W).
 
 The load forecast data must be provided in one of the formats described in
-<project:#prediction-import-providers>. The data source can be given in the `loadimport_file_path`
-or `loadimport_json` configuration option.
+<project:#prediction-import-providers>.
 
-The data may additionally or solely be provided by the
-**PUT** `/v1/prediction/import/LoadImport` endpoint.
+An external entity may update the file or JSON string whenever new prediction data becomes
+available. However, for production use or regular updates, you should prefer the **PUT** endpoint
+over file or JSON string based imports.
+
+:::{admonition} Warning
+:class: warning
+Be aware that providing dynamic values via a file and/or JSON string **alongside other value
+sources** can lead to unintended data overwrites. Moreover, after a restart, even outdated values
+from the configuration may be reloaded. To avoid these issues, use the **PUT** endpoint for live
+data and rely on file/JSON imports only for initial setup.
+:::
 
 ## PV Power Prediction
 
@@ -305,7 +373,7 @@ Configuration options:
 
     - `PVForecastAkkudoktor`: Retrieves from Akkudoktor.net.
     - `PVForecastVrm`: Retrieves data from the VRM API by Victron Energy.
-    - `PVForecastImport`: Imports from a file or JSON string.
+    - `PVForecastImport`: Imports from a file or JSON string or by endpoint data provision.
 
   - `planes[].surface_tilt`: Tilt angle from horizontal plane. Ignored for two-axis tracking.
   - `planes[].surface_azimuth`: Orientation (azimuth angle) of the (fixed) plane.
@@ -572,7 +640,7 @@ Configuration options:
     - `BrightSky`: Retrieves from [BrightSky](https://api.brightsky.dev).
     - `ClearOutside`: Retrieves from [ClearOutside](https://clearoutside.com/forecast).
     - `OpenMeteo`: Retrieves from [OpenMeteo](https://api.open-meteo.com/v1/forecast).
-    - `LoadImport`: Imports from a file or JSON string.
+    - `LoadImport`: Imports from a file or JSON string or by endpoint data provision.
 
   - `provider_settings.import_file_path`: Path to the file to import weatherforecast data from.
   - `provider_settings.import_json`: JSON string, dictionary of weather forecast value lists.