perf: decouple notebook execution and tune pools
schedule:
- New _execution_loop runs alongside _database_event_loop. It claims
job.node.execute rows, sets a 30-min lease on available_at, then
dispatches each as asyncio.create_task under a Semaphore(N).
Polling loop is back to sub-millisecond turnaround for
schedule.run.requested and job.node.finished. Long notebook
execution no longer blocks DAG advance events.
- _process_pending_events filters by event_type IN
('schedule.run.requested', 'job.node.finished'); the executor
loop owns job.node.execute exclusively.
- _process_outbox_event builds a plain dict envelope before
handler dispatch; the previous ORM-row handoff risked
DetachedInstanceError once the outer session closed.
- _sync_once uses get_job + reschedule_job for existing job ids
instead of add_job(replace_existing=True). Each cron schedule
no longer removed-and-readded every 5s.
- service.py threads settings.schedule_execution_concurrency into
the orchestrator (default 4).
common:
- create_async_engine gets explicit pool_size=10, max_overflow=20,
pool_recycle=1800. No more relying on SQLAlchemy defaults.
- New schedule_execution_concurrency setting.
runtime:
- scan_workspaces: add missing 'import os' (NameError on startup)
and switch to asyncio.gather bounded by Semaphore(4) so N
workspaces start in parallel instead of sequentially.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -9,6 +9,7 @@ a thin wrapper over ``start_workspace``.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import secrets
|
||||
import subprocess
|
||||
import time
|
||||
@@ -249,13 +250,18 @@ async def scan_workspaces() -> None:
|
||||
logger.error(f"scan workspace failed: {e}")
|
||||
return
|
||||
|
||||
for entry in entries:
|
||||
sem = asyncio.Semaphore(4)
|
||||
|
||||
async def _start(entry: str) -> None:
|
||||
path = WORKSPACES_ROOT / entry
|
||||
if not path.is_dir():
|
||||
continue
|
||||
return
|
||||
logger.info(f"Found workspace: {entry}")
|
||||
logger.info(f"Auto-starting Jupyter for workspace ID: '{entry}'")
|
||||
try:
|
||||
await start_workspace(entry)
|
||||
except Exception as err:
|
||||
logger.error(f"Startup failed for workspace '{entry}': {err}")
|
||||
async with sem:
|
||||
try:
|
||||
await start_workspace(entry)
|
||||
except Exception as err:
|
||||
logger.error(f"Startup failed for workspace '{entry}': {err}")
|
||||
|
||||
await asyncio.gather(*[_start(entry) for entry in entries])
|
||||
Reference in New Issue
Block a user