Follow-up to #1382. Under the EOS server the end of a GENETIC run with
parallel evaluation (workers >= 2, which is the automatic setting on
machines with 3 or more cores) could hang forever in
Pool.terminate()/join(), so /v1/optimize never answered.
Cause: uvicorn installs a Python SIGTERM/SIGINT handler in the server
process that only asks the server to shut down. The forked evaluation
workers inherit it and survive the SIGTERM of Pool.terminate().
terminate() keeps the task queue's read lock (Pool._help_stuff_finish),
so a surviving worker that has not yet taken its stop sentinel waits for
that lock forever, and join() with it. This always happened after a
batch timeout (busy workers survive) and by chance after a normal run
when a worker was not scheduled in time (several EOS instances or other
load; also seen with a single instance on a 2-vCPU VM). A worker that
got SIGTERM before its initializer ran hit the same path.
- Workers ignore SIGTERM and SIGINT. Sent to the whole process group
(Ctrl-C, systemd stopping the service) they are meant for the server,
which finishes or aborts the run and stops the workers itself.
Restoring the default action instead was measured to be worse: a
group SIGTERM killed the workers mid-batch, the batch ran into its
120 s timeout and the server did not exit within 120 s (12 s before).
- _EvaluationPool stops its workers with SIGKILL at the point where
Pool.terminate() would send SIGTERM (it already holds the read lock).
This does not depend on handlers, initialization state or whether a
worker is busy. _stop_evaluation_pool() is unchanged.
Measured (8 EOS servers x 2 workers on 8 vCPU, real configuration,
seed 42): 11 hangs in 160 runs before, 0 in 440 after; pool only under
load: 107/300 before, 0/600 after. Plans and costs are identical, the
stop takes no longer. Ctrl-C, SIGTERM to the server and SIGTERM to the
group behave as before (run completes with HTTP 200, exit after ~12 s,
no orphaned workers).
Tests (tests/test_genetic_parallel_evaluation.py, under uvicorn's real
signal handling): inherited handler, batch timeout, late worker start,
late idle worker (SIGSTOP, Linux) and SIGTERM to the group during a
batch. They fail without the fix.
Claude-Session: https://claude.ai/code/session_01EYWR1HVM8a8src23xuyL3T
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
8e34b1779b