fix(genetic): stop parallel evaluation workers reliably at the end of a run (#1473)

Follow-up to #1382. Under the EOS server the end of a GENETIC run with
parallel evaluation (workers >= 2, which is the automatic setting on
machines with 3 or more cores) could hang forever in
Pool.terminate()/join(), so /v1/optimize never answered.

Cause: uvicorn installs a Python SIGTERM/SIGINT handler in the server
process that only asks the server to shut down. The forked evaluation
workers inherit it and survive the SIGTERM of Pool.terminate().
terminate() keeps the task queue's read lock (Pool._help_stuff_finish),
so a surviving worker that has not yet taken its stop sentinel waits for
that lock forever, and join() with it. This always happened after a
batch timeout (busy workers survive) and by chance after a normal run
when a worker was not scheduled in time (several EOS instances or other
load; also seen with a single instance on a 2-vCPU VM). A worker that
got SIGTERM before its initializer ran hit the same path.

- Workers ignore SIGTERM and SIGINT. Sent to the whole process group
  (Ctrl-C, systemd stopping the service) they are meant for the server,
  which finishes or aborts the run and stops the workers itself.
  Restoring the default action instead was measured to be worse: a
  group SIGTERM killed the workers mid-batch, the batch ran into its
  120 s timeout and the server did not exit within 120 s (12 s before).
- _EvaluationPool stops its workers with SIGKILL at the point where
  Pool.terminate() would send SIGTERM (it already holds the read lock).
  This does not depend on handlers, initialization state or whether a
  worker is busy. _stop_evaluation_pool() is unchanged.

Measured (8 EOS servers x 2 workers on 8 vCPU, real configuration,
seed 42): 11 hangs in 160 runs before, 0 in 440 after; pool only under
load: 107/300 before, 0/600 after. Plans and costs are identical, the
stop takes no longer. Ctrl-C, SIGTERM to the server and SIGTERM to the
group behave as before (run completes with HTTP 200, exit after ~12 s,
no orphaned workers).

Tests (tests/test_genetic_parallel_evaluation.py, under uvicorn's real
signal handling): inherited handler, batch timeout, late worker start,
late idle worker (SIGSTOP, Linux) and SIGTERM to the group during a
batch. They fail without the fix.


Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

src/akkudoktoreos/optimization/genetic/genetic.py

@@ -5,8 +5,10 @@ import ctypes.util
 import gc
 import math
 import multiprocessing
+import multiprocessing.pool
 import os
 import random
+import signal
 import sys
 import time
 from array import array
@@ -277,7 +279,17 @@ def worker_cpu_set(workers: int, allowed: Optional[set[int]] = None) -> Optional
 
 
 def _evaluation_worker_init(cpus: Optional[set[int]] = None) -> None:
-    """Evaluate without the run cache in a worker; the parent owns the cache."""
+    """Prepare a forked evaluation worker; the parent owns the run cache.
+
+    The worker ignores SIGTERM and SIGINT. Sent to the whole process group
+    (Ctrl-C, a service manager stopping the server) they are meant for the
+    server process, which finishes or aborts the run and stops the workers
+    itself (see ``_EvaluationPool``). Without this, a worker would run the
+    signal handler it inherited from the server - uvicorn's only asks the
+    server to shut down - or die in the middle of a batch.
+    """
+    signal.signal(signal.SIGTERM, signal.SIG_IGN)
+    signal.signal(signal.SIGINT, signal.SIG_IGN)
     if _WORKER_OPTIMIZER is not None:
         _WORKER_OPTIMIZER._fitness_cache_enabled = False
     if cpus:
@@ -287,6 +299,24 @@ def _evaluation_worker_init(cpus: Optional[set[int]] = None) -> None:
             pass
 
 
+class _EvaluationPool(multiprocessing.pool.Pool):
+    """Process pool whose ``terminate()`` stops the workers with SIGKILL.
+
+    ``Pool.terminate()`` keeps the task queue's read lock and then stops the
+    workers with SIGTERM. A worker that survives it - because it ignores the
+    signal, still runs the server's inherited handler (uvicorn's only asks the
+    server to shut down) or got it before its initializer ran - waits for that
+    lock forever, and ``Pool.join()`` with it. SIGKILL cannot be caught; the
+    pool sends it at the same point it would send SIGTERM.
+    """
+
+    @staticmethod
+    def Process(ctx: Any, *args: Any, **kwds: Any) -> Any:  # noqa: N802 - Pool API
+        process = ctx.Process(*args, **kwds)
+        setattr(process, "terminate", process.kill)
+        return process
+
+
 def _evaluation_worker(genome: list[int]) -> tuple[list[int], tuple[float], Any]:
     """Evaluate one genome in a worker and return the (repaired) genome and results."""
     individual = creator.Individual(genome)
@@ -2954,10 +2984,11 @@ class GeneticOptimization(OptimizationBase):
         cpus = worker_cpu_set(self._evaluation_workers) if self._pin_workers() else None
         _WORKER_OPTIMIZER = self
         try:
-            self._evaluation_pool = context.Pool(
+            self._evaluation_pool = _EvaluationPool(
                 self._evaluation_workers,
                 initializer=_evaluation_worker_init,
                 initargs=(cpus,),
+                context=context,
             )
         except Exception as exc:
             _WORKER_OPTIMIZER = None