A self-hosted Jupyter-based model development platform that combines an interactive workspace, a DAG scheduler, an object-storage-backed artifact store, and per-workspace runtime isolation — all behind a single Nginx gateway.

Stack: React Router SPA · FastAPI · APScheduler · MySQL · RustFS (S3) · shared Jupyter · FUSE mount via rclone Single ingress (Nginx :80); all other services are Docker-internal.

What it does

Capability	Where
Workspace-scoped notebook editing with row-level lock	`backend/jupyter.py` + `scripts.is_locked`
Authenticated Jupyter routing (browser never sees the runtime token)	`nginx/default.conf` + `auth_request` + `backend/jupyter.py`
Object storage for notebooks / scripts / versions / run logs (RustFS, S3 API)	`common/storage/` + `backend/scripts.py`
DAG-style scheduling: nodes, edges, cron, manual trigger, retries, snapshots	`backend/schedules.py` + `backend/schedule_runs.py` + `schedule/` (5 modules)
DAG execution via MySQL Outbox (no Redis, no in-process queues)	`schedule/orchestrator.py` + `schedule/worker.py`
Per-workspace Jupyter sub-process pool with asyncio locks	`runtime/process.py`
rclone FUSE mount of the workspace bucket into the runtime	`runtime/mount.py`
MySQL-only persistence (26 tables, soft-delete, no foreign keys)	`common/db/models/`

Architecture at a glance

                        ┌────────────────────┐
                        │  Browser (SPA)     │
                        └─────────┬──────────┘
                                  │ HTTPS / WS
                        ┌─────────▼──────────┐
                        │  Nginx (only :80) │  ← templates/default.conf
                        │   /api/  /jupyter/ /storage/
                        └────┬───────┬──────┘
                             │       │
              ┌──────────────┘       └─────────────┐
              ▼                                     ▼
   ┌──────────────────┐               ┌──────────────────────┐
   │ FastAPI Backend  │               │ Runtime (Jupyter)     │
   │ + /internal/v1   │   control     │ - rclone FUSE mount  │
   │ (storage)        ├──────────────►│ - subprocess pool    │
   │ - DAG CRUD       │               │   (per workspace)    │
   │ - script CRUD    │               └──────────┬───────────┘
   │ - auth_request   │                          │ FUSE
   │ - /api/v1/...    │                          ▼
   └────┬──────┬──────┘               ┌──────────────────────┐
        │      │                       │ RustFS (S3)         │
        │      └──────── HTTP ───────►│ bucket: workspaces   │
        ▼                              │ bucket: versions     │
   ┌────────────┐                      │ bucket: run-logs     │
   │  MySQL     │◄───────── poll ─────│                      │
   │ - 26 tbls  │                      └──────────────────────┘
   │ - outbox   │
   │ - jobstore │
   └────┬───────┘
        ▲
        │ outbox poll
   ┌────┴──────────────────────────┐
   │ Schedule Executor             │
   │ - CronScheduler (APScheduler) │
   │ - DispatchOrchestrator        │
   │ - NodeExecutor (worker)       │
   │ - SchedulerService (facade)   │
   └───────────────────────────────┘

Detailed design lives in ARCHITECTURE.md. Implementation deviations and recent refactors are recorded in HANDOVER.md.

Repository layout

frontend/                    React Router SPA
backend/                     FastAPI: public API + internal storage API
runtime/                     Jupyter subprocess manager + rclone FUSE
schedule/                    DAG scheduler (5 modules: context/scheduler/
                             orchestrator/worker/service)
common/                      Settings, SQLAlchemy models, storage SDK,
                             outbox events, jobstore
migrations/                  Alembic baseline + per-feature revisions
nginx/                       (concept only — see "Container" below)
scripts/                     nginx-entrypoint.sh (template renderer)
docker-compose.yml           4 services — web / backend / runtime / schedule
default.conf                 Nginx template (mounted, rendered at start)
.env.example                 All env vars consumed by common.config.Settings

Containers

Service	Image	Exposed	Purpose
`web`	`nginx:alpine`	host `:8888` → `:80`	SPA, `/api/` reverse-proxy, `/jupyter/{ws}/` auth_request proxy, `/storage/` RustFS passthrough
`backend`	`Dockerfile`	internal only	DAG CRUD, script CRUD, schedule triggers, `/api/v1/auth/jupyter`, `/internal/v1/*` storage control plane
`runtime`	`Dockerfile`	internal only	Per-workspace Jupyter sub-process pool, rclone FUSE mount of `workspaces` bucket
`schedule`	`Dockerfile`	internal only	Cron tick + DAG execution via MySQL Outbox polling

The architecture deliberately has only one host port (the gateway); all other services are on the Docker internal network. This is enforced in docker-compose.yml — no ports: on backend / runtime / schedule.

Quick start

cp .env.example .env
# Edit .env — at minimum change MYSQL password and RUSTFS credentials.

# Static check
uv sync --all-packages
uv run --frozen --package backend python -m compileall -q backend/src common/src
uv run --frozen --package schedule python -m compileall -q schedule/src
uv run --frozen --package runtime python -m compileall -q runtime/src

# Apply schema
uv run --frozen --package backend alembic upgrade head

# Bring up the stack
docker compose config          # validate
docker compose up -d --build
docker compose ps

Visit http://localhost:8888.

Logs

docker compose logs -f backend
docker compose logs -f schedule
docker compose logs -f runtime

Tear down (keeps MySQL + RustFS volumes)

docker compose down

Wipe data

docker compose down -v

Configuration

All environment variables are declared once in common/src/common/config.py as a pydantic-settings Settings class, with a @lru_cache singleton. Adding a new env var:

Add the field to Settings in common/src/common/config.py (with a sensible default so dev-env "just works").
Add the line to .env.example with a comment.
Use settings.<name> at the call site. Never os.environ["..."].

See DEVELOP.md for the full list of variables and their meanings.

Storage layout

Single bucket workspaces (configurable via RUSTFS_WORKSPACE_BUCKET). The object key is a flat two-level path — workspace_id and a server- issued ulid for the object. The file name, extension, content type, and logical path live in the StorageObjects and Scripts rows, not in the S3 key, so the bucket can be re-organised without a database rewrite.

s3://workspaces/
└── <workspace_id>/
    ├── <ulid-1>     # script / notebook / data resource
    ├── <ulid-2>
    └── ...

s3://versions/  (RUSTFS_VERSION_BUCKET — reserved, used by publish_version)
s3://run-logs/  (RUSTFS_RUN_LOG_BUCKET — reserved, used by node executor)

To find the original file name and its logical path for a given bucket object, join StorageObjects.bucket_name + object_key to the row.

Backend code never writes to the container's local filesystem. Schedule Executor stages node artifacts in tempfile.TemporaryDirectory() (auto- cleaned). Only the runtime container keeps a host volume — it is required by the rclone FUSE mount.

Documentation

README.md (this file) — quick orientation
ARCHITECTURE.md — design diagrams + simplification history
HANDOVER.md — implementation deviations, recent refactors, pending work
DEVELOP.md — developer guide (env vars, code conventions, common tasks)
CLAUDE.md — agent-facing conventions for the repo

License

Internal.