tao.chenandClaude Fable 5
4acfbb162f
S2: schedule worker add janitor task that force-terminals node_runs whose deadline (timeout_seconds + retry_count*retry_interval + 120s slack from started_at) has passed. Closes the gap where outbox retry exhaustion (5 tries, capped 30s backoff) marked the *event* failed but left the *node_run* stuck in queued/running forever. Re-reads the row under FOR UPDATE before writing so a worker that races us to a real terminal state is not overwritten; idempotency key uses :timed_out variant so the :finished path cannot collide. R1: extract _reap_once() from _reap_loop for testability; in the dead-process branch, re-verify (process.pid, started_at) against the live JUPYTER_PROCESSES entry before del. A start_workspace that replaced the dead record mid-cycle used to have its new entry silently erased by the reaper's stale snapshot — leaked the port. R2: delete _drop_workspace_lock and its two call sites (stop_workspace tail, get_workspace 404 path). Popping the lock object after release breaks mutual exclusion for any coroutine still holding the old reference while a fresh caller gets a new lock object — same ws_id can race two starts. The dict is bounded by the number of workspaces so the leak is negligible; invariant lives on WORKSPACE_LOCKS in a comment. Tests: - schedule/tests/test_janitor.py — 8 tests covering normal kill / healthy-skip / worker-race / never-started / multi-row batch / cancellation propagation / per-iteration self-heal - runtime/tests/test_process.py — 7 tests covering reaper identity match / replacement-skip / alive-preserved + lock same-object / concurrent-serialize / survives-stop / helper-removed guard uv run --package schedule pytest schedule/tests → 14 passed uv run --package runtime pytest runtime/tests → 7 passed Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Runtime
独立 Runtime/Jupyter 管理服务：
- 使用共享 Jupyter Server；
- 在 MySQL 中维护 Runtime 实例和编辑会话租约；
- 创建、心跳和释放文件编辑锁；
- 创建短期 Jupyter 访问票据；
- 为 Nginx
auth_request校验票据并注入内部 Jupyter Token。
当前简化部署要求 Runtime 单副本运行。