mirror of
https://github.com/Akkudoktor-EOS/EOS.git
synced 2026-07-24 18:58:12 +00:00
2ed04d5573
These fixes were taken from https://github.com/arneman/EOS/tree/refactor/economic-objective. 1. Fix compaction job registration bug in eos.py: - compact_eos_database was registered with save_eos_database function - Now correctly calls compact_eos_database() - Root cause: compaction/vacuum never ran, allowing records to grow unbounded 2. Optimize db_iterate_records() from O(n) to O(log n): - Previous: linear scan from index 0 for every key_to_array() call - Now: uses bisect_left on _db_sorted_timestamps to skip to start position - Critical for large datasets: 9400 measurement records → 94s/call overhead → 5 calls/optimization 3. Reduce compaction_interval_sec default from 604800s (7 days) to 3600s (1 hour): - With 7-day interval: 65000 records accumulate before first cleanup - Bridge pushes ~1 record/9s (grid_export_emr) - Hourly compaction + 2h data window → stable ~950 records Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
3.2 KiB
3.2 KiB
Configuration model for database settings
Attributes: provider: Optional provider identifier (e.g. "LMDB"). max_records_in_memory: Maximum records kept in memory before auto-save. auto_save: Whether to auto-save when threshold exceeded. batch_size: Batch size for batch operations.
:::{table} database :widths: 10 20 10 5 5 30 :align: left
| Name | Environment Variable | Type | Read-Only | Default | Description |
|---|---|---|---|---|---|
| autosave_interval_sec | EOS_DATABASE__AUTOSAVE_INTERVAL_SEC |
Optional[int] |
rw |
10 |
Automatic saving interval [seconds]. |
| Set to None to disable automatic saving. | |||||
| batch_size | EOS_DATABASE__BATCH_SIZE |
int |
rw |
100 |
Number of records to process in batch operations. |
| compaction_interval_sec | EOS_DATABASE__COMPACTION_INTERVAL_SEC |
Optional[int] |
rw |
3600 |
Interval in between automatic tiered compaction runs [seconds]. |
| Compaction downsamples old records to reduce storage while retaining coverage. Set to None to disable automatic compaction. | |||||
| compression_level | EOS_DATABASE__COMPRESSION_LEVEL |
int |
rw |
9 |
Compression level for database record data. |
| initial_load_window_h | EOS_DATABASE__INITIAL_LOAD_WINDOW_H |
Optional[int] |
rw |
None |
Specifies the default duration of the initial load window when loading records from the database, in hours. If set to None, the full available range is loaded. The window is centered around the current time by default, unless a different center time is specified. Different database namespaces may define their own default windows. |
| keep_duration_h | EOS_DATABASE__KEEP_DURATION_H |
Optional[int] |
rw |
None |
Default maximum duration records shall be kept in database [hours, none]. |
| None indicates forever. Database namespaces may have diverging definitions. | |||||
| provider | EOS_DATABASE__PROVIDER |
Optional[str] |
rw |
None |
Database provider id of provider to be used. |
| providers | List[str] |
ro |
N/A |
Return available database provider ids. | |
| ::: |
Example Input
{
"database": {
"provider": "LMDB",
"compression_level": 0,
"initial_load_window_h": 48,
"keep_duration_h": 48,
"autosave_interval_sec": 5,
"compaction_interval_sec": 3600,
"batch_size": 100
}
}
Example Output
{
"database": {
"provider": "LMDB",
"compression_level": 0,
"initial_load_window_h": 48,
"keep_duration_h": 48,
"autosave_interval_sec": 5,
"compaction_interval_sec": 3600,
"batch_size": 100,
"providers": [
"LMDB",
"SQLite",
"NoDB"
]
}
}