# Model Platform A self-hosted **Jupyter-based model development platform** that combines an interactive workspace, a DAG scheduler, an object-storage-backed artifact store, and per-workspace runtime isolation — all behind a single Nginx gateway. > Stack: React Router SPA · FastAPI · APScheduler · MySQL · S3-compatible > storage (or local filesystem via `STORAGE_BACKEND=local`) · shared > Jupyter · FUSE mount via rclone (s3 mode only) > Single ingress (Nginx :80); all other services are Docker-internal. ## What it does | Capability | Where | |---|---| | Workspace-scoped notebook editing with row-level lock | `backend/jupyter.py` + `scripts.is_locked` | | Authenticated Jupyter routing (browser never sees the runtime token) | `nginx/default.conf` + `auth_request` + `backend/jupyter.py` | | Object storage for notebooks / scripts / versions / run logs (s3 / local toggle) | `common/storage/` + `backend/scripts.py` | | DAG-style scheduling: nodes, edges, cron, manual trigger, retries, snapshots | `backend/schedules.py` + `backend/schedule_runs.py` + `schedule/` (5 modules) | | DAG execution via MySQL Outbox (no Redis, no in-process queues) | `schedule/orchestrator.py` + `schedule/worker.py` | | Per-workspace Jupyter sub-process pool with asyncio locks | `runtime/process.py` | | rclone FUSE mount of the workspace bucket into the runtime (s3 mode) | `runtime/mount.py` | | MySQL-only persistence (26 tables, soft-delete, no foreign keys) | `common/db/models/` | ## Architecture at a glance ``` ┌────────────────────┐ │ Browser (SPA) │ └─────────┬──────────┘ │ HTTPS / WS ┌─────────▼──────────┐ │ Nginx (only :80) │ ← templates/default.conf │ /api/ /jupyter/ /storage/ └────┬───────┬──────┘ │ │ ┌──────────────┘ └─────────────┐ ▼ ▼ ┌──────────────────┐ ┌──────────────────────┐ │ FastAPI Backend │ │ Runtime (Jupyter) │ │ + /internal/v1 │ control │ - rclone FUSE mount │ │ (storage) ├──────────────►│ - subprocess pool │ │ - DAG CRUD │ │ (per workspace) │ │ - script CRUD │ └──────────┬───────────┘ │ - auth_request │ │ FUSE / shared vol │ - /api/v1/... │ ▼ └────┬──────┬──────┘ ┌──────────────────────┐ │ │ │ Object storage │ │ └──────── HTTP ───────►│ (s3: S3 service / │ ▼ │ local: shared vol) │ ┌────────────┐ │ 4 buckets per usage │ │ MySQL │◄───────── poll ─────│ │ │ - 26 tbls │ └──────────────────────┘ │ - outbox │ │ - jobstore │ └────┬───────┘ ▲ │ outbox poll ┌────┴──────────────────────────┐ │ Schedule Executor │ │ - CronScheduler (APScheduler) │ │ - DispatchOrchestrator │ │ - NodeExecutor (worker) │ │ - SchedulerService (facade) │ └───────────────────────────────┘ ``` Object storage is selectable via `STORAGE_BACKEND` (s3 | local). In s3 mode the 4 purpose-named buckets (`workspaces` / `versions` / `run-logs` / `trash`) are S3 buckets; in local mode they're subdirectories of `LOCAL_STORAGE_BASE_DIR`, shared via the `local-storage` Docker volume. See `DEVELOP.md` §Storage. Detailed design lives in `ARCHITECTURE.md`. Implementation deviations and recent refactors are recorded in `HANDOVER.md`. ## Repository layout ```text frontend/ React Router SPA backend/ FastAPI: public API + internal storage API runtime/ Jupyter subprocess manager + rclone FUSE schedule/ DAG scheduler (5 modules: context/scheduler/ orchestrator/worker/service) common/ Settings, SQLAlchemy models, storage SDK, outbox events, jobstore migrations/ Alembic baseline + per-feature revisions nginx/ (concept only — see "Container" below) scripts/ nginx-entrypoint.sh (template renderer) docker-compose.yml 4 services — web / backend / runtime / schedule default.conf Nginx template (mounted, rendered at start) .env.example All env vars consumed by common.config.Settings ``` ## Containers | Service | Image | Exposed | Purpose | |---|---|---|---| | `web` | `nginx:alpine` | host `:8888` → `:80` | SPA, `/api/` reverse-proxy, `/jupyter/{ws}/` auth_request proxy, `/storage/` S3 passthrough (s3 mode only) | | `backend` | `Dockerfile` | internal only | DAG CRUD, script CRUD, schedule triggers, `/api/v1/auth/jupyter`, `/internal/v1/*` storage control plane | | `runtime` | `Dockerfile` | internal only | Per-workspace Jupyter sub-process pool, rclone FUSE mount of `workspaces` bucket (s3 mode) | | `schedule` | `Dockerfile` | internal only | Cron tick + DAG execution via MySQL Outbox polling | The architecture **deliberately has only one host port** (the gateway); all other services are on the Docker internal network. This is enforced in `docker-compose.yml` — no `ports:` on backend / runtime / schedule. ## Quick start ```bash cp .env.example .env # Edit .env — at minimum change MYSQL password and (in s3 mode) S3 credentials. # Static check uv sync --all-packages uv run --frozen --package backend python -m compileall -q backend/src common/src uv run --frozen --package schedule python -m compileall -q schedule/src uv run --frozen --package runtime python -m compileall -q runtime/src # Apply schema uv run --frozen --package backend alembic upgrade head # Bring up the stack docker compose config # validate docker compose up -d --build docker compose ps ``` Visit `http://localhost:8888`. ### Logs ```bash docker compose logs -f backend docker compose logs -f schedule docker compose logs -f runtime ``` ### Tear down (keeps MySQL + S3 / local-storage volumes) ```bash docker compose down ``` ### Wipe data ```bash docker compose down -v ``` ## Configuration All environment variables are declared once in `common/src/common/config.py` as a pydantic-settings `Settings` class, with a `@lru_cache` singleton. Adding a new env var: 1. Add the field to `Settings` in `common/src/common/config.py` (with a sensible default so dev-env "just works"). 2. Add the line to `.env.example` with a comment. 3. Use `settings.` at the call site. Never `os.environ["..."]`. See `DEVELOP.md` for the full list of variables and their meanings. ## Storage layout Four purpose-named buckets. The mapping from `StorageObjects.usage_type` to bucket is decided in **one place** (`backend/storage_api.py:resolve_bucket`): | `usage_type` | Bucket (env var) | Default name | |---|---|---| | `working_copy`, `public_script`, `data_resource`, `snapshot` | `S3_WORKSPACE_BUCKET` | `workspaces` | | `version_artifact` | `S3_VERSION_BUCKET` | `versions` | | `run_log`, `run_result` | `S3_RUN_LOG_BUCKET` | `run-logs` | | (soft-delete target) | `S3_TRASH_BUCKET` | `trash` | In `STORAGE_BACKEND=s3` mode these are 4 separate S3 buckets. In `STORAGE_BACKEND=local` mode they are 4 subdirectories under `LOCAL_STORAGE_BASE_DIR` (default `/data`), so the layout above becomes: ``` /data/ ├── workspace/ # S3_WORKSPACE_BUCKET ├── version/ # S3_VERSION_BUCKET ├── run_log/ # S3_RUN_LOG_BUCKET └── trash/ # S3_TRASH_BUCKET ``` A workspace's `artifact_bucket` column (when non-null) overrides the default for that workspace, regardless of `usage_type` — useful for isolating a paying customer to their own bucket. The object key is a flat two-level path — `workspace_id` and a server- issued `ulid` for the object: ``` //{.} ``` The file name, extension, content type, and logical path live in the `StorageObjects` and `Scripts` rows, not in the object key, so the storage can be re-organised without a database rewrite. Backend code never writes to the container's local filesystem (except in `STORAGE_BACKEND=local` mode, where the shared `local-storage` volume is the canonical store). Schedule Executor stages node artifacts in `tempfile.TemporaryDirectory()` (auto-cleaned). Only the `runtime` container keeps a host volume — required by the rclone FUSE mount in s3 mode, and a no-op pass-through in local mode. ## Documentation - `README.md` (this file) — quick orientation - `ARCHITECTURE.md` — design diagrams + simplification history - `HANDOVER.md` — implementation deviations, recent refactors, pending work - `DEVELOP.md` — developer guide (env vars, code conventions, common tasks) - `CLAUDE.md` — agent-facing conventions for the repo ## License Internal.