Stage 3 of the layered refactor. Relocate the two scheduling components
into their own package so that domain / application / scheduling /
execution / infrastructure boundaries actually exist on disk.
- Add schedule/src/schedule/scheduling/__init__.py
- Move scheduler.py (232 lines) -> scheduling/scheduler.py
(byte-identical via diff; CronScheduler class name unchanged)
- Move orchestrator.py (946 lines) -> scheduling/orchestrator.py
(byte-identical via diff; DispatchOrchestrator + event constants
unchanged; NOT further split this round, per plan)
- service.py lines 39-40: import paths rewritten to the new module
- tests/test_janitor.py: rewrite the import + 5 patch() string targets
The 5 patch() targets ("schedule.orchestrator.session_scope" x3,
"schedule.orchestrator.asyncio.sleep" x2) were NOT caught by the
import-line grep — they patch module attributes at runtime and would
have become dead no-ops after the move (and would hard-raise once
the old module is deleted in stage 6). Rewriting them to
"schedule.scheduling.orchestrator.*" keeps the janitor tests meaningfully
exercising the new module.
- old flat scheduler.py / orchestrator.py left on disk; stage 6 deletes
them once all layers are extracted.
Validation:
- uv run --package schedule pytest schedule/tests -q: 18 passed
- uv run python -m compileall schedule/src: zero errors
- grep 'from schedule.(scheduler|orchestrator)\\b' (old paths): 0 matches
- grep '"schedule.orchestrator.' (old patch targets): 0 matches
- main.py / worker.py / domain/ / infrastructure/ / pyproject.toml
byte-identical to HEAD
Co-Authored-By: Claude <noreply@anthropic.com>
S2: schedule worker add janitor task that force-terminals node_runs
whose deadline (timeout_seconds + retry_count*retry_interval + 120s
slack from started_at) has passed. Closes the gap where outbox retry
exhaustion (5 tries, capped 30s backoff) marked the *event* failed
but left the *node_run* stuck in queued/running forever. Re-reads the
row under FOR UPDATE before writing so a worker that races us to a
real terminal state is not overwritten; idempotency key uses
:timed_out variant so the :finished path cannot collide.
R1: extract _reap_once() from _reap_loop for testability; in the
dead-process branch, re-verify (process.pid, started_at) against the
live JUPYTER_PROCESSES entry before del. A start_workspace that
replaced the dead record mid-cycle used to have its new entry
silently erased by the reaper's stale snapshot — leaked the port.
R2: delete _drop_workspace_lock and its two call sites
(stop_workspace tail, get_workspace 404 path). Popping the lock
object after release breaks mutual exclusion for any coroutine still
holding the old reference while a fresh caller gets a new lock
object — same ws_id can race two starts. The dict is bounded by the
number of workspaces so the leak is negligible; invariant lives on
WORKSPACE_LOCKS in a comment.
Tests:
- schedule/tests/test_janitor.py — 8 tests covering normal kill /
healthy-skip / worker-race / never-started / multi-row batch /
cancellation propagation / per-iteration self-heal
- runtime/tests/test_process.py — 7 tests covering reaper identity
match / replacement-skip / alive-preserved + lock
same-object / concurrent-serialize / survives-stop /
helper-removed guard
uv run --package schedule pytest schedule/tests → 14 passed
uv run --package runtime pytest runtime/tests → 7 passed
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
