Model Platform

A self-hosted Jupyter-based model development platform that combines an interactive workspace, a DAG scheduler, an object-storage-backed artifact store, and per-workspace runtime isolation — all behind a single Nginx gateway.

Stack: React Router SPA · FastAPI · APScheduler · MySQL · RustFS (S3) · shared Jupyter · FUSE mount via rclone Single ingress (Nginx :80); all other services are Docker-internal.

What it does

Capability	Where
Workspace-scoped notebook editing with row-level lock	backend/jupyter.py + scripts.is_locked
Authenticated Jupyter routing (browser never sees the runtime token)	nginx/default.conf + auth_request + backend/jupyter.py
Object storage for notebooks / scripts / versions / run logs (RustFS, S3 API)	common/storage/ + backend/scripts.py
DAG-style scheduling: nodes, edges, cron, manual trigger, retries, snapshots	backend/schedules.py + backend/schedule_runs.py + schedule/ (5 modules)
DAG execution via MySQL Outbox (no Redis, no in-process queues)	schedule/orchestrator.py + schedule/worker.py
Per-workspace Jupyter sub-process pool with asyncio locks	runtime/process.py
rclone FUSE mount of the workspace bucket into the runtime	runtime/mount.py
MySQL-only persistence (26 tables, soft-delete, no foreign keys)	common/db/models/

Architecture at a glance

                        ┌────────────────────┐
                        │  Browser (SPA)     │
                        └─────────┬──────────┘
                                  │ HTTPS / WS
                        ┌─────────▼──────────┐
                        │  Nginx (only :80) │  ← templates/default.conf
                        │   /api/  /jupyter/ /storage/
                        └────┬───────┬──────┘
                             │       │
              ┌──────────────┘       └─────────────┐
              ▼                                     ▼
   ┌──────────────────┐               ┌──────────────────────┐
   │ FastAPI Backend  │               │ Runtime (Jupyter)     │
   │ + /internal/v1   │   control     │ - rclone FUSE mount  │
   │ (storage)        ├──────────────►│ - subprocess pool    │
   │ - DAG CRUD       │               │   (per workspace)    │
   │ - script CRUD    │               └──────────┬───────────┘
   │ - auth_request   │                          │ FUSE
   │ - /api/v1/...    │                          ▼
   └────┬──────┬──────┘               ┌──────────────────────┐
        │      │                       │ RustFS (S3)         │
        │      └──────── HTTP ───────►│ bucket: workspaces   │
        ▼                              │ bucket: versions     │
   ┌────────────┐                      │ bucket: run-logs     │
   │  MySQL     │◄───────── poll ─────│                      │
   │ - 26 tbls  │                      └──────────────────────┘
   │ - outbox   │
   │ - jobstore │
   └────┬───────┘
        ▲
        │ outbox poll
   ┌────┴──────────────────────────┐
   │ Schedule Executor             │
   │ - CronScheduler (APScheduler) │
   │ - DispatchOrchestrator        │
   │ - NodeExecutor (worker)       │
   │ - SchedulerService (facade)   │
   └───────────────────────────────┘

Detailed design lives in ARCHITECTURE.md. Implementation deviations and recent refactors are recorded in HANDOVER.md.

Repository layout

frontend/                    React Router SPA
backend/                     FastAPI: public API + internal storage API
runtime/                     Jupyter subprocess manager + rclone FUSE
schedule/                    DAG scheduler (5 modules: context/scheduler/
                             orchestrator/worker/service)
common/                      Settings, SQLAlchemy models, storage SDK,
                             outbox events, jobstore
migrations/                  Alembic baseline + per-feature revisions
nginx/                       (concept only — see "Container" below)
scripts/                     nginx-entrypoint.sh (template renderer)
docker-compose.yml           4 services — web / backend / runtime / schedule
default.conf                 Nginx template (mounted, rendered at start)
.env.example                 All env vars consumed by common.config.Settings

Containers

Service	Image	Exposed	Purpose
web	nginx:alpine	host :8888 → :80	SPA, /api/ reverse-proxy, /jupyter/{ws}/ auth_request proxy, /storage/ RustFS passthrough
backend	Dockerfile	internal only	DAG CRUD, script CRUD, schedule triggers, /api/v1/auth/jupyter, /internal/v1/* storage control plane
runtime	Dockerfile	internal only	Per-workspace Jupyter sub-process pool, rclone FUSE mount of workspaces bucket
schedule	Dockerfile	internal only	Cron tick + DAG execution via MySQL Outbox polling

The architecture deliberately has only one host port (the gateway); all other services are on the Docker internal network. This is enforced in docker-compose.yml — no ports: on backend / runtime / schedule.

Quick start

cp .env.example .env
# Edit .env — at minimum change MYSQL password and RUSTFS credentials.

# Static check
uv sync --all-packages
uv run --frozen --package backend python -m compileall -q backend/src common/src
uv run --frozen --package schedule python -m compileall -q schedule/src
uv run --frozen --package runtime python -m compileall -q runtime/src

# Apply schema
uv run --frozen --package backend alembic upgrade head

# Bring up the stack
docker compose config          # validate
docker compose up -d --build
docker compose ps

Visit http://localhost:8888.

Logs

docker compose logs -f backend
docker compose logs -f schedule
docker compose logs -f runtime

Tear down (keeps MySQL + RustFS volumes)

docker compose down

Wipe data

docker compose down -v

Configuration

All environment variables are declared once in common/src/common/config.py as a pydantic-settings Settings class, with a @lru_cache singleton. Adding a new env var:

1. Add the field to Settings in common/src/common/config.py (with a sensible default so dev-env "just works").
2. Add the line to .env.example with a comment.
3. Use settings.<name> at the call site. Never os.environ["..."].

See DEVELOP.md for the full list of variables and their meanings.

Storage layout

Three purpose-named RustFS buckets. The mapping from StorageObjects.usage_type to bucket is decided in one place (storage_api.py:resolve_bucket):

usage_type	Bucket (env var)	Default name
working_copy, public_script, data_resource, snapshot	RUSTFS_WORKSPACE_BUCKET	workspaces
version_artifact	RUSTFS_VERSION_BUCKET	versions
run_log, run_result	RUSTFS_RUN_LOG_BUCKET	run-logs

A workspace's artifact_bucket column (when non-null) overrides the default for that workspace, regardless of usage_type — useful for isolating a paying customer to their own bucket.

The object key is a flat two-level path — workspace_id and a server- issued ulid for the object:

s3://workspaces/
└── <workspace_id>/
    ├── <ulid-1>     # working_copy / data_resource / snapshot / ...
    ├── <ulid-2>
    └── ...

s3://versions/<workspace_id>/<ulid>   # immutable script versions
s3://run-logs/<workspace_id>/<ulid>   # node run logs and results

The file name, extension, content type, and logical path live in the StorageObjects and Scripts rows, not in the S3 key, so the bucket can be re-organised without a database rewrite.

Backend code never writes to the container's local filesystem. Schedule Executor stages node artifacts in tempfile.TemporaryDirectory() (auto- cleaned). Only the runtime container keeps a host volume — it is required by the rclone FUSE mount.

Documentation

• README.md (this file) — quick orientation
• ARCHITECTURE.md — design diagrams + simplification history
• HANDOVER.md — implementation deviations, recent refactors, pending work
• DEVELOP.md — developer guide (env vars, code conventions, common tasks)
• CLAUDE.md — agent-facing conventions for the repo

License

Internal.