fix: logging, prediction update, multiple bugs (#584)

* Fix logging configuration issues that made logging stop operation. Switch to Loguru
  logging (from Python logging). Enable console and file logging with different log levels.
  Add logging documentation.

* Fix logging configuration and EOS configuration out of sync. Added tracking support
  for nested value updates of Pydantic models. This used to update the logging configuration
  when the EOS configurationm for logging is changed. Should keep logging config and EOS
  config in sync as long as all changes to the EOS logging configuration are done by
  set_nested_value(), which is the case for the REST API.

* Fix energy management task looping endlessly after the second update when trying to update
  the last_update datetime.

* Fix get_nested_value() to correctly take values from the dicts in a Pydantic model instance.

* Fix usage of model classes instead of model instances in nested value access when evaluation
  the value type that is associated to each key.

* Fix illegal json format in prediction documentation for PVForecastAkkudoktor provider.

* Fix documentation qirks and add EOS Connect to integrations.

* Support deprecated fields in configuration in documentation generation and EOSdash.

* Enhance EOSdash demo to show BrightSky humidity data (that is often missing)

* Update documentation reference to German EOS installation videos.

Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>

docs/_generated/config.md

@@ -15,10 +15,6 @@ Properties:
     timezone (Optional[str]): Computed time zone string based on the specified latitude
         and longitude.
 
-Validators:
-    validate_latitude (float): Ensures `latitude` is within the range -90 to 90.
-    validate_longitude (float): Ensures `longitude` is within the range -180 to 180.
-
 :::{table} general
 :widths: 10 20 10 5 5 30
 :align: left
@@ -127,8 +123,10 @@ Validators:
 
 | Name | Environment Variable | Type | Read-Only | Default | Description |
 | ---- | -------------------- | ---- | --------- | ------- | ----------- |
-| level | `EOS_LOGGING__LEVEL` | `Optional[str]` | `rw` | `None` | EOS default logging level. |
-| root_level | | `str` | `ro` | `N/A` | Root logger logging level. |
+| level | `EOS_LOGGING__LEVEL` | `Optional[str]` | `rw` | `None` | This is deprecated. Use console_level and file_level instead. |
+| console_level | `EOS_LOGGING__CONSOLE_LEVEL` | `Optional[str]` | `rw` | `None` | Logging level when logging to console. |
+| file_level | `EOS_LOGGING__FILE_LEVEL` | `Optional[str]` | `rw` | `None` | Logging level when logging to file. |
+| file_path | | `Optional[pathlib.Path]` | `ro` | `N/A` | Computed log file path based on data output path. |
 :::
 
 ### Example Input
@@ -138,7 +136,9 @@ Validators:
 
    {
        "logging": {
-           "level": "INFO"
+           "level": null,
+           "console_level": "TRACE",
+           "file_level": "TRACE"
        }
    }
 ```
@@ -150,8 +150,10 @@ Validators:
 
    {
        "logging": {
-           "level": "INFO",
-           "root_level": "INFO"
+           "level": null,
+           "console_level": "TRACE",
+           "file_level": "TRACE",
+           "file_path": "/home/user/.local/share/net.akkudoktoreos.net/output/eos.log"
        }
    }
 ```
@@ -946,7 +948,9 @@ Validators:
            "interval": 300.0
        },
        "logging": {
-           "level": "INFO"
+           "level": null,
+           "console_level": "TRACE",
+           "file_level": "TRACE"
        },
        "devices": {
            "batteries": [

docs/_generated/openapi.md

@@ -464,6 +464,55 @@ Health check endpoint to verify that the EOS server is alive.
 
 ---
 
+## GET /v1/logging/log
+
+**Links**: [local](http://localhost:8503/docs#/default/fastapi_logging_get_log_v1_logging_log_get), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_logging_get_log_v1_logging_log_get)
+
+Fastapi Logging Get Log
+
+```
+Get structured log entries from the EOS log file.
+
+Filters and returns log entries based on the specified query parameters. The log
+file is expected to contain newline-delimited JSON entries.
+
+Args:
+    limit (int): Maximum number of entries to return.
+    level (Optional[str]): Filter logs by severity level (e.g., DEBUG, INFO).
+    contains (Optional[str]): Return only logs that include this string in the message.
+    regex (Optional[str]): Return logs that match this regular expression in the message.
+    from_time (Optional[str]): ISO 8601 timestamp to filter logs not older than this.
+    to_time (Optional[str]): ISO 8601 timestamp to filter logs not newer than this.
+    tail (bool): If True, fetch the most recent log entries (like `tail`).
+
+Returns:
+    JSONResponse: A JSON list of log entries.
+```
+
+**Parameters**:
+
+- `limit` (query, optional): Maximum number of log entries to return.
+
+- `level` (query, optional): Filter by log level (e.g., INFO, ERROR).
+
+- `contains` (query, optional): Filter logs containing this substring.
+
+- `regex` (query, optional): Filter logs by matching regex in message.
+
+- `from_time` (query, optional): Start time (ISO format) for filtering logs.
+
+- `to_time` (query, optional): End time (ISO format) for filtering logs.
+
+- `tail` (query, optional): If True, returns the most recent lines (tail mode).
+
+**Responses**:
+
+- **200**: Successful Response
+
+- **422**: Validation Error
+
+---
+
 ## PUT /v1/measurement/data
 
 **Links**: [local](http://localhost:8503/docs#/default/fastapi_measurement_data_put_v1_measurement_data_put), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_measurement_data_put_v1_measurement_data_put)

docs/akkudoktoreos/architecture.md

@@ -28,7 +28,7 @@ management.
 Energy management is the overall process to provide planning data for scheduling the different
 devices in your system in an optimal way. Energy management cares for the update of predictions and
 the optimization of the planning based on the simulated behavior of the devices. The planning is on
-the hour. Sub-hour energy management is left
+the hour.
 
 ### Optimization
 

docs/akkudoktoreos/integration.md

@@ -20,8 +20,8 @@ Andreas Schmitz uses [Node-RED](https://nodered.org/) as part of his home automa
 
 ### Node-Red Resources
 
-- [Installation Guide (German)](https://meintechblog.de/2024/09/05/andreas-schmitz-joerg-installiert-mein-energieoptimierungssystem/)
-  \— A detailed guide on integrating an early version of EOS with `Node-RED`.
+- [Installation Guide (German)](https://www.youtube.com/playlist?list=PL8_vk9A-s7zLD865Oou6y3EeQLlNtu-Hn)
+  \— A detailed guide on integrating EOS with `Node-RED`.
 
 ## Home Assistant
 
@@ -34,3 +34,8 @@ emphasizes local control and user privacy.
 
 - Duetting's [EOS Home Assistant Addon](https://github.com/Duetting/ha_eos_addon) — Additional
   details can be found in this [discussion thread](https://github.com/Akkudoktor-EOS/EOS/discussions/294).
+
+## EOS Connect
+
+[EOS connect](https://github.com/ohAnd/EOS_connect) uses `EOS` for energy management and optimization,
+and connects to smart home platforms to monitor, forecast, and control energy flows.

docs/akkudoktoreos/introduction.md

@@ -174,7 +174,7 @@ but usually find good local optima very quickly in a large solution space.
 
 ## Links
 
-- [German Video explaining the basic concept and installation process for the early version of EOS (YouTube)](https://www.youtube.com/live/ftQULW4-1ts?si=oDdBBifCpUmiCXaY)
+- [German Videos explaining the basic concept and installation process of EOS (YouTube)](https://www.youtube.com/playlist?list=PL8_vk9A-s7zLD865Oou6y3EeQLlNtu-Hn)
 - [German Forum of Akkudoktor EOS](https://akkudoktor.net/c/der-akkudoktor/eos)
 - [Akkudoktor-EOS GitHub Repository](https://github.com/Akkudoktor-EOS/EOS)
 - [Latest EOS Documentation](https://akkudoktor-eos.readthedocs.io/en/latest/)

docs/akkudoktoreos/logging.md

@@ -0,0 +1,75 @@
+% SPDX-License-Identifier: Apache-2.0
+(logging-page)=
+
+# Logging
+
+EOS automatically records important events and messages to help you understand what’s happening and
+to troubleshoot problems.
+
+## How Logging Works
+
+- By default, logs are shown in your terminal (console).
+- You can also save logs to a file for later review.
+- Log files are rotated automatically to avoid becoming too large.
+
+## Controlling Log Details
+
+### 1. Command-Line Option
+
+Set the amount of log detail shown on the console by using `--log-level` when starting EOS.
+
+Example:
+
+```{eval-rst}
+.. tabs::
+
+  .. tab:: Windows
+
+     .. code-block:: powershell
+
+        .venv\Scripts\python src/akkudoktoreos/server/eos.py --log-level DEBUG
+
+  .. tab:: Linux
+
+     .. code-block:: bash
+
+        .venv/bin/python src/akkudoktoreos/server/eos.py --log-level DEBUG
+
+```
+
+Common levels:
+
+- DEBUG (most detail)
+- INFO (default)
+- WARNING
+- ERROR
+- CRITICAL (least detail)
+
+### 2. Configuration File
+
+You can also set logging options in your EOS configuration file (EOS.config.json).
+
+```Json
+{
+  "logging": {
+    "console_level": "INFO",
+    "file_level": "DEBUG"
+  }
+}
+```
+
+### 3. Environment Variable
+
+You can also control the log level by setting the `EOS_LOGGING__CONSOLE_LEVEL` and the
+`EOS_LOGGING__FILE_LEVEL` environment variables.
+
+```bash
+  EOS_LOGGING__CONSOLE_LEVEL="INFO"
+  EOS_LOGGING__FILE_LEVEL="DEBUG"
+```
+
+## File Logging
+
+If the `file_level` configuration is set, log records are written to a rotating log file. The log
+file is in the data output directory and named `eos.log`. You may directly read the file or use
+the `/v1/logging/log` endpoint to access the file log.

docs/akkudoktoreos/prediction.md

@@ -375,28 +375,28 @@ Example:
         "surface_azimuth": -10,
         "surface_tilt": 7,
         "userhorizon": [20, 27, 22, 20],
-        "inverter_paco": 10000,
+        "inverter_paco": 10000
       },
       {
         "peakpower": 4.8,
         "surface_azimuth": -90,
         "surface_tilt": 7,
         "userhorizon": [30, 30, 30, 50],
-        "inverter_paco": 10000,
+        "inverter_paco": 10000
       },
       {
         "peakpower": 1.4,
         "surface_azimuth": -40,
         "surface_tilt": 60,
         "userhorizon": [60, 30, 0, 30],
-        "inverter_paco": 2000,
+        "inverter_paco": 2000
       },
       {
         "peakpower": 1.6,
         "surface_azimuth": 5,
         "surface_tilt": 45,
         "userhorizon": [45, 25, 30, 60],
-        "inverter_paco": 1400,
+        "inverter_paco": 1400
       }
     ]
   }

docs/index.md

@@ -40,6 +40,7 @@ akkudoktoreos/optimization.md
 akkudoktoreos/prediction.md
 akkudoktoreos/measurement.md
 akkudoktoreos/integration.md
+akkudoktoreos/logging.md
 akkudoktoreos/serverapi.md
 akkudoktoreos/api.rst