perf: decouple notebook execution and tune pools

schedule:
- New _execution_loop runs alongside _database_event_loop. It claims
  job.node.execute rows, sets a 30-min lease on available_at, then
  dispatches each as asyncio.create_task under a Semaphore(N).
  Polling loop is back to sub-millisecond turnaround for
  schedule.run.requested and job.node.finished. Long notebook
  execution no longer blocks DAG advance events.
- _process_pending_events filters by event_type IN
  ('schedule.run.requested', 'job.node.finished'); the executor
  loop owns job.node.execute exclusively.
- _process_outbox_event builds a plain dict envelope before
  handler dispatch; the previous ORM-row handoff risked
  DetachedInstanceError once the outer session closed.
- _sync_once uses get_job + reschedule_job for existing job ids
  instead of add_job(replace_existing=True). Each cron schedule
  no longer removed-and-readded every 5s.
- service.py threads settings.schedule_execution_concurrency into
  the orchestrator (default 4).

common:
- create_async_engine gets explicit pool_size=10, max_overflow=20,
  pool_recycle=1800. No more relying on SQLAlchemy defaults.
- New schedule_execution_concurrency setting.

runtime:
- scan_workspaces: add missing 'import os' (NameError on startup)
  and switch to asyncio.gather bounded by Semaphore(4) so N
  workspaces start in parallel instead of sequentially.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
tao.chen
2026-07-31 15:07:57 +08:00
co-authored by Claude
parent b0f93d976f
commit 9fc886a55e
6 changed files with 185 additions and 18 deletions
+12 -6
View File
@@ -9,6 +9,7 @@ a thin wrapper over ``start_workspace``.
from __future__ import annotations
import asyncio
import os
import secrets
import subprocess
import time
@@ -249,13 +250,18 @@ async def scan_workspaces() -> None:
logger.error(f"scan workspace failed: {e}")
return
for entry in entries:
sem = asyncio.Semaphore(4)
async def _start(entry: str) -> None:
path = WORKSPACES_ROOT / entry
if not path.is_dir():
continue
return
logger.info(f"Found workspace: {entry}")
logger.info(f"Auto-starting Jupyter for workspace ID: '{entry}'")
try:
await start_workspace(entry)
except Exception as err:
logger.error(f"Startup failed for workspace '{entry}': {err}")
async with sem:
try:
await start_workspace(entry)
except Exception as err:
logger.error(f"Startup failed for workspace '{entry}': {err}")
await asyncio.gather(*[_start(entry) for entry in entries])