# Model Platform

A self-hosted **Jupyter-based model development platform** that combines
an interactive workspace, a DAG scheduler, an object-storage-backed artifact
store, and per-workspace runtime isolation — all behind a single Nginx
gateway.

> Stack: React Router SPA · FastAPI · APScheduler · MySQL · RustFS (S3)
> · shared Jupyter · FUSE mount via rclone
> Single ingress (Nginx :80); all other services are Docker-internal.

## What it does

| Capability | Where |
|---|---|
| Workspace-scoped notebook editing with row-level lock | `backend/jupyter.py` + `scripts.is_locked` |
| Authenticated Jupyter routing (browser never sees the runtime token) | `nginx/default.conf` + `auth_request` + `backend/jupyter.py` |
| Object storage for notebooks / scripts / versions / run logs (RustFS, S3 API) | `common/storage/` + `backend/scripts.py` |
| DAG-style scheduling: nodes, edges, cron, manual trigger, retries, snapshots | `backend/schedules.py` + `backend/schedule_runs.py` + `schedule/` (5 modules) |
| DAG execution via MySQL Outbox (no Redis, no in-process queues) | `schedule/orchestrator.py` + `schedule/worker.py` |
| Per-workspace Jupyter sub-process pool with asyncio locks | `runtime/process.py` |
| rclone FUSE mount of the workspace bucket into the runtime | `runtime/mount.py` |
| MySQL-only persistence (26 tables, soft-delete, no foreign keys) | `common/db/models/` |

## Architecture at a glance

```
                        ┌────────────────────┐
                        │  Browser (SPA)     │
                        └─────────┬──────────┘
                                  │ HTTPS / WS
                        ┌─────────▼──────────┐
                        │  Nginx (only :80) │  ← templates/default.conf
                        │   /api/  /jupyter/ /storage/
                        └────┬───────┬──────┘
                             │       │
              ┌──────────────┘       └─────────────┐
              ▼                                     ▼
   ┌──────────────────┐               ┌──────────────────────┐
   │ FastAPI Backend  │               │ Runtime (Jupyter)     │
   │ + /internal/v1   │   control     │ - rclone FUSE mount  │
   │ (storage)        ├──────────────►│ - subprocess pool    │
   │ - DAG CRUD       │               │   (per workspace)    │
   │ - script CRUD    │               └──────────┬───────────┘
   │ - auth_request   │                          │ FUSE
   │ - /api/v1/...    │                          ▼
   └────┬──────┬──────┘               ┌──────────────────────┐
        │      │                       │ RustFS (S3)         │
        │      └──────── HTTP ───────►│ bucket: workspaces   │
        ▼                              │ bucket: versions     │
   ┌────────────┐                      │ bucket: run-logs     │
   │  MySQL     │◄───────── poll ─────│                      │
   │ - 26 tbls  │                      └──────────────────────┘
   │ - outbox   │
   │ - jobstore │
   └────┬───────┘
        ▲
        │ outbox poll
   ┌────┴──────────────────────────┐
   │ Schedule Executor             │
   │ - CronScheduler (APScheduler) │
   │ - DispatchOrchestrator        │
   │ - NodeExecutor (worker)       │
   │ - SchedulerService (facade)   │
   └───────────────────────────────┘
```

Detailed design lives in `ARCHITECTURE.md`. Implementation deviations and
recent refactors are recorded in `HANDOVER.md`.

## Repository layout

```text
frontend/                    React Router SPA
backend/                     FastAPI: public API + internal storage API
runtime/                     Jupyter subprocess manager + rclone FUSE
schedule/                    DAG scheduler (5 modules: context/scheduler/
                             orchestrator/worker/service)
common/                      Settings, SQLAlchemy models, storage SDK,
                             outbox events, jobstore
migrations/                  Alembic baseline + per-feature revisions
nginx/                       (concept only — see "Container" below)
scripts/                     nginx-entrypoint.sh (template renderer)
docker-compose.yml           4 services — web / backend / runtime / schedule
default.conf                 Nginx template (mounted, rendered at start)
.env.example                 All env vars consumed by common.config.Settings
```

## Containers

| Service | Image | Exposed | Purpose |
|---|---|---|---|
| `web` | `nginx:alpine` | host `:8888` → `:80` | SPA, `/api/` reverse-proxy, `/jupyter/{ws}/` auth_request proxy, `/storage/` RustFS passthrough |
| `backend` | `Dockerfile` | internal only | DAG CRUD, script CRUD, schedule triggers, `/api/v1/auth/jupyter`, `/internal/v1/*` storage control plane |
| `runtime` | `Dockerfile` | internal only | Per-workspace Jupyter sub-process pool, rclone FUSE mount of `workspaces` bucket |
| `schedule` | `Dockerfile` | internal only | Cron tick + DAG execution via MySQL Outbox polling |

The architecture **deliberately has only one host port** (the gateway);
all other services are on the Docker internal network. This is enforced in
`docker-compose.yml` — no `ports:` on backend / runtime / schedule.

## Quick start

```bash
cp .env.example .env
# Edit .env — at minimum change MYSQL password and RUSTFS credentials.

# Static check
uv sync --all-packages
uv run --frozen --package backend python -m compileall -q backend/src common/src
uv run --frozen --package schedule python -m compileall -q schedule/src
uv run --frozen --package runtime python -m compileall -q runtime/src

# Apply schema
uv run --frozen --package backend alembic upgrade head

# Bring up the stack
docker compose config          # validate
docker compose up -d --build
docker compose ps
```

Visit `http://localhost:8888`.

### Logs

```bash
docker compose logs -f backend
docker compose logs -f schedule
docker compose logs -f runtime
```

### Tear down (keeps MySQL + RustFS volumes)

```bash
docker compose down
```

### Wipe data

```bash
docker compose down -v
```

## Configuration

All environment variables are declared once in `common/src/common/config.py`
as a pydantic-settings `Settings` class, with a `@lru_cache` singleton.
Adding a new env var:

1. Add the field to `Settings` in `common/src/common/config.py` (with a
   sensible default so dev-env "just works").
2. Add the line to `.env.example` with a comment.
3. Use `settings.<name>` at the call site. Never `os.environ["..."]`.

See `DEVELOP.md` for the full list of variables and their meanings.

## Storage layout

Single bucket `workspaces` (configurable via `RUSTFS_WORKSPACE_BUCKET`).
The object key is a flat two-level path — `workspace_id` and a server-
issued `ulid` for the object. The file name, extension, content type,
and logical path live in the `StorageObjects` and `Scripts` rows, not in
the S3 key, so the bucket can be re-organised without a database
rewrite.

```
s3://workspaces/
└── <workspace_id>/
    ├── <ulid-1>     # script / notebook / data resource
    ├── <ulid-2>
    └── ...

s3://versions/  (RUSTFS_VERSION_BUCKET — reserved, used by publish_version)
s3://run-logs/  (RUSTFS_RUN_LOG_BUCKET — reserved, used by node executor)
```

To find the original file name and its logical path for a given bucket
object, join `StorageObjects.bucket_name + object_key` to the row.

Backend code never writes to the container's local filesystem. Schedule
Executor stages node artifacts in `tempfile.TemporaryDirectory()` (auto-
cleaned). Only the `runtime` container keeps a host volume — it is required
by the rclone FUSE mount.

## Documentation

- `README.md` (this file) — quick orientation
- `ARCHITECTURE.md` — design diagrams + simplification history
- `HANDOVER.md` — implementation deviations, recent refactors, pending work
- `DEVELOP.md` — developer guide (env vars, code conventions, common tasks)
- `CLAUDE.md` — agent-facing conventions for the repo

## License

Internal.