fix(optimization): make the genetic diversity boost an actual intervention

On a converged population the boost was permanently on, so there was nothing
left for it to intervene in. Its trigger, DIVERSITY_BOOST_THRESHOLD at 0.35, sat
above SELECTION_DIVERSITY_FLOOR at 0.30 - the floor the selection itself
guarantees - so `diversity < threshold` was true in every generation after
convergence. The threshold now sits below the floor.

The immigrants the boost injects are by construction the worst individuals in
the pool, and `_select_diverse` ran a plain tournament over parents and
offspring together, so they were removed in the very generation that created
them and their genes never recombined. A bounded share of seats
(IMMIGRANT_PROTECTION_FRACTION) is now reserved for them for
IMMIGRANT_PROTECTION_GENERATIONS selections. The incumbent is protected by
genome key, so no immigrant can evict the best solution or an equal-genome twin,
and offspring do not inherit the protection.

The log line was edge-triggered on `diversity_boost_active`, which was also
cleared on every fitness improvement, so a running boost re-announced itself
with "stagnation 0" while a boost that never stopped looked like several short
ones. It now tracks the boost alone, and the end of a boost is logged too.

Measured on a converged population: without protection 0 of 12 immigrants
survive the selection, with it all 12 do, and the incumbent is kept either way.

CHANGELOG.md

@@ -184,6 +184,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
   failed evaluations and results from previous runs are never reused.
 - `max_home_appliances` is now purely an upper bound. No demo appliance is created when
   no `home_appliances` are configured, and the number is no longer used as an on/off switch.
+- The genetic diversity boost is an intervention again instead of the steady state. Its trigger
+  (`DIVERSITY_BOOST_THRESHOLD`) sat above the floor the selection guarantees
+  (`SELECTION_DIVERSITY_FLOOR`), so on a converged population it was permanently true; it now sits
+  below the floor. Freshly injected immigrants are also the worst individuals in the pool and were
+  removed by the very tournament of the generation that created them, so their genes never
+  recombined - a bounded share of seats is now reserved for them for two selections. The log line
+  is edge-triggered on the boost itself rather than on the last fitness improvement, and the end
+  of a boost is logged too.
 - Required forecasts are no longer silently replaced by demo providers. Previously a missing PV,
   price, load, feed-in or weather forecast rewrote the configured provider to a demo one and
   retried, so a run could quietly optimize against invented data. Missing values now stay missing:

src/akkudoktoreos/optimization/genetic/genetic.py

@@ -602,10 +602,20 @@ class GeneticOptimization(OptimizationBase):
     STAGNATION_MUTATION_PROBABILITY = 0.80
     STAGNATION_GENERATIONS = 8
     SOFT_RESTART_GENERATIONS = 20
-    DIVERSITY_BOOST_THRESHOLD = 0.35
+    # The selection keeps SELECTION_DIVERSITY_FLOOR of the population unique, so a
+    # boost threshold at or above that floor would fire in every converged
+    # generation and make the boost the normal operating state instead of an
+    # intervention. Keep it strictly below the floor.
     SELECTION_DIVERSITY_FLOOR = 0.30
+    DIVERSITY_BOOST_THRESHOLD = 0.25
     SOFT_RESTART_DIVERSITY_THRESHOLD = 0.10
     IMMIGRANT_FRACTION = 0.12
+    # Fresh immigrants are the worst individuals in the pool, so a plain
+    # tournament removes them in the generation they are born and their genes
+    # never get a chance to recombine. Keep a bounded number of them for a few
+    # selections so a boost can actually explore.
+    IMMIGRANT_PROTECTION_GENERATIONS = 2
+    IMMIGRANT_PROTECTION_FRACTION = 0.25
     SOFT_RESTART_SURVIVOR_FRACTION = 0.20
     POINT_MUTATION_EXPECTED_GENES = 3.0
 
@@ -2153,6 +2163,9 @@ class GeneticOptimization(OptimizationBase):
             del individual.fitness.values
         if hasattr(individual, "extra_data"):
             del individual.extra_data
+        # A child of a protected immigrant is an ordinary offspring.
+        if hasattr(individual, "immigrant_protection"):
+            del individual.immigrant_protection
 
     def _evaluate_invalid(self, population: list[Any]) -> int:
         """Evaluate invalid individuals and return the number of cache lookups."""
@@ -2186,6 +2199,62 @@ class GeneticOptimization(OptimizationBase):
                 break
         return selected
 
+    def _reserve_immigrant_slots(
+        self,
+        candidates: list[Any],
+        selected: list[Any],
+        selected_keys: list[tuple[int, ...]],
+        best_key: tuple[int, ...],
+    ) -> bool:
+        """Carry still-protected immigrants into ``selected`` in place.
+
+        The tournament judges immigrants on the fitness they have before any
+        recombination, which they lose. Reserving a bounded share of the seats
+        gives their genes the generations they need to be crossed into the
+        incumbents.
+
+        Returns whether any seat was reassigned.
+        """
+        protected = [
+            candidate
+            for candidate in candidates
+            if getattr(candidate, "immigrant_protection", 0) > 0
+        ]
+        if not protected:
+            return False
+
+        limit = max(1, int(len(selected) * self.IMMIGRANT_PROTECTION_FRACTION))
+        chosen = {id(candidate) for candidate in selected}
+        seated = sum(1 for candidate in protected if id(candidate) in chosen)
+        missing = [candidate for candidate in protected if id(candidate) not in chosen]
+        if seated >= limit or not missing:
+            return False
+
+        # Evict the weakest seats that carry neither the incumbent genome nor a
+        # protection of their own, worst first.
+        evictable = sorted(
+            (
+                index
+                for index, candidate in enumerate(selected)
+                if selected_keys[index] != best_key
+                and getattr(candidate, "immigrant_protection", 0) <= 0
+            ),
+            key=lambda index: selected[index].fitness.values[0],
+            reverse=True,
+        )
+        reassigned = False
+        for immigrant, index in zip(missing[: limit - seated], evictable):
+            selected[index] = immigrant
+            reassigned = True
+        return reassigned
+
+    def _age_immigrant_protection(self, population: list[Any]) -> None:
+        """Spend one generation of the surviving immigrants' protection."""
+        for individual in population:
+            remaining = getattr(individual, "immigrant_protection", 0)
+            if remaining > 0:
+                individual.immigrant_protection = remaining - 1
+
     def _select_diverse(self, candidates: list[Any], count: int) -> list[Any]:
         """Tournament-select while repairing only severe duplicate takeover."""
         if not candidates or count <= 0:
@@ -2203,6 +2272,9 @@ class GeneticOptimization(OptimizationBase):
             selected[worst_index] = best
             selected_keys[worst_index] = best_key
 
+        if self._reserve_immigrant_slots(candidates, selected, selected_keys, best_key):
+            selected_keys = [self._fitness_key(candidate) for candidate in selected]
+
         # Duplicates are useful for exploitation and cache hits. Replace only
         # enough duplicate selections to keep a minimum search breadth.
         target_unique = min(
@@ -2331,6 +2403,7 @@ class GeneticOptimization(OptimizationBase):
                 total_immigrants += immigrants
                 stagnation = 0
                 diversity_boost_active = False
+                self._age_immigrant_protection(population)
                 logger.info(
                     "Genetic soft restart at generation {}: kept {} unique survivors, "
                     "injected {} immigrants (diversity {:.1%}).",
@@ -2352,6 +2425,14 @@ class GeneticOptimization(OptimizationBase):
                         stagnation,
                         diversity,
                     )
+                elif diversity_boost_active and not diversity_boost:
+                    logger.info(
+                        "Genetic diversity boost ended at generation {}: stagnation {}, "
+                        "diversity {:.1%}.",
+                        generation,
+                        stagnation,
+                        diversity,
+                    )
                 diversity_boost_active = diversity_boost
                 mutation_probability = (
                     self.STAGNATION_MUTATION_PROBABILITY
@@ -2365,22 +2446,20 @@ class GeneticOptimization(OptimizationBase):
                     lambda_ - immigrants,
                     mutation_probability=mutation_probability,
                 )
-                offspring.extend(
-                    self._fresh_population(
-                        immigrants,
-                        educated_fraction=0.50,
-                    )
-                )
+                fresh = self._fresh_population(immigrants, educated_fraction=0.50)
+                for immigrant in fresh:
+                    immigrant.immigrant_protection = self.IMMIGRANT_PROTECTION_GENERATIONS
+                offspring.extend(fresh)
                 nevals = self._evaluate_invalid(offspring)
                 halloffame.update(offspring)
                 population = self._select_diverse(population + offspring, mu)
+                self._age_immigrant_protection(population)
                 total_immigrants += immigrants
 
             current_best = float(halloffame[0].fitness.values[0])
             if current_best < best_fitness - 1e-9:
                 best_fitness = current_best
                 stagnation = 0
-                diversity_boost_active = False
             elif not soft_restart:
                 stagnation += 1
 

tests/test_genetic_seeding.py

@@ -410,3 +410,94 @@ def test_flat_feed_in_tariff_does_not_seed_direct_marketing(config_eos: ConfigEO
     export_state = 5
     assert all(export_state not in guess for guess in guesses)
 
+
+def _rated(genome: list[int], fitness: float, *, protection: int = 0):
+    """Build an evaluated individual, optionally a protected immigrant."""
+    individual = creator.Individual(genome)
+    individual.fitness.values = (fitness,)
+    if protection:
+        individual.immigrant_protection = protection
+    return individual
+
+
+def test_diversity_boost_threshold_stays_below_selection_floor():
+    # The selection guarantees SELECTION_DIVERSITY_FLOOR unique genomes, so a
+    # boost threshold at or above the floor would fire in every converged
+    # generation and turn the boost into the normal operating state.
+    assert (
+        GeneticOptimization.DIVERSITY_BOOST_THRESHOLD
+        < GeneticOptimization.SELECTION_DIVERSITY_FLOOR
+    )
+
+
+def _immigrant_selection_pool(opt: GeneticOptimization, protection: int):
+    """Converged incumbents plus fresh immigrants that the tournament dislikes.
+
+    The incumbents already carry more unique genomes than
+    ``SELECTION_DIVERSITY_FLOOR`` demands, so the duplicate repair has no reason
+    to reach for an immigrant and only the protection can seat one.
+    """
+    slots = opt.control_slots
+    incumbents = [
+        _rated([1, index] + [0] * (slots - 2), -5.73 + index * 1e-4) for index in range(100)
+    ]
+    offspring = [
+        _rated([2, index] + [0] * (slots - 2), -5.72 + index * 1e-4) for index in range(88)
+    ]
+    offspring.extend(
+        _rated([3, index, index] + [0] * (slots - 3), 3.0 + index, protection=protection)
+        for index in range(12)
+    )
+    return incumbents, offspring
+
+
+def test_protected_immigrants_survive_the_selection(config_eos: ConfigEOS):
+    _configure_hourly_grid(config_eos)
+    opt = GeneticOptimization(fixed_seed=42)
+    opt.optimize_ev = False
+    opt.setup_deap_environment({"home_appliance": 0}, start_hour=0)
+
+    seated = {}
+    for protection in (0, opt.IMMIGRANT_PROTECTION_GENERATIONS):
+        incumbents, offspring = _immigrant_selection_pool(opt, protection)
+        selected = opt._select_diverse(incumbents + offspring, 100)
+        seated[protection] = sum(1 for candidate in selected if candidate[0] == 3)
+        # The incumbent is never evicted to make room for an immigrant.
+        assert min(candidate.fitness.values[0] for candidate in selected) == pytest.approx(-5.73)
+
+    # Without protection the tournament removes every immigrant in the
+    # generation it is born, so its genes never get to recombine.
+    assert seated[0] == 0
+    assert seated[opt.IMMIGRANT_PROTECTION_GENERATIONS] == 12
+
+
+def test_immigrant_protection_expires_after_its_generations(config_eos: ConfigEOS):
+    _configure_hourly_grid(config_eos)
+    opt = GeneticOptimization(fixed_seed=42)
+    opt.optimize_ev = False
+    opt.setup_deap_environment({"home_appliance": 0}, start_hour=0)
+
+    immigrants = [_rated([3, 0, 0], 3.0, protection=opt.IMMIGRANT_PROTECTION_GENERATIONS)]
+    for _ in range(opt.IMMIGRANT_PROTECTION_GENERATIONS):
+        assert immigrants[0].immigrant_protection > 0
+        opt._age_immigrant_protection(immigrants)
+    assert immigrants[0].immigrant_protection == 0
+
+    # Aging is idempotent once the protection is spent.
+    opt._age_immigrant_protection(immigrants)
+    assert immigrants[0].immigrant_protection == 0
+
+
+def test_offspring_do_not_inherit_immigrant_protection(config_eos: ConfigEOS):
+    _configure_hourly_grid(config_eos)
+    opt = GeneticOptimization(fixed_seed=42)
+    opt.optimize_ev = False
+    opt.setup_deap_environment({"home_appliance": 0}, start_hour=0)
+    opt.toolbox.register("evaluate", lambda individual: (float(sum(individual)),))
+
+    parents = [
+        _rated([0] * opt.control_slots, 0.0, protection=opt.IMMIGRANT_PROTECTION_GENERATIONS)
+        for _ in range(4)
+    ]
+    offspring = opt._make_offspring(parents, 8, mutation_probability=1.0)
+    assert all(getattr(child, "immigrant_protection", 0) == 0 for child in offspring)