tao.chen
309b657d35
All operator- and developer-facing docs updated to reflect:
- The unified AsyncStorageBackend abstraction (s3 + local backends).
- The STORAGE_BACKEND toggle ("s3" default, "local" for dev /
single-node / air-gapped deployments).
- The 4-purpose-bucket layout (workspace / version / run_log / trash)
in both modes — 4 separate S3 buckets in s3 mode, 4 subdirectories
of LOCAL_STORAGE_BASE_DIR in local mode.
- The S3_* env var naming (was RUSTFS_*).
- The server-proxied upload flow (was browser-direct presign-PUT):
POST /internal/v1/uploads → PUT /internal/v1/uploads/{id} with
raw bytes → server calls backend.put().
- The factory helpers workspaces_root() (runtime's view of the
workspace bucket on disk) and rclone_remote_spec() (s3-mode mount
source).
- The "two settings describing the same thing" cleanup: the deleted
settings.workspace_root, settings.workspaces_root, and
settings.remote_bucket fields.
Files touched:
- API.md (§5 data-resource upload flow, §9 storage control plane,
§10 readiness example)
- ARCHITECTURE.md (storage layer diagram)
- CLAUDE.md (architecture description + volume-preservation note)
- DEVELOP.md (settings list, Storage section, "Wire a new bucket"
how-to, dev-export example, troubleshooting network hint)
- README.md (architecture diagram, container table, quick-start
credentials note, tear-down note, Storage layout section)
- REFACTOR_NOTES.md (final container list with s3 explanation)
- backend/README.md (storage backend description)
- migrations/data/README.md (step 11/12 record mentioning object
storage)
A handful of historical "RustFS" mentions are intentionally retained
where they name a specific S3-compatible product (e.g. as an example
in REFACTOR_NOTES.md's container list) or document the pre-2026
abstraction name (DEVELOP.md Storage section).
222 lines
9.8 KiB
Markdown
222 lines
9.8 KiB
Markdown
# Model Platform
|
|
|
|
A self-hosted **Jupyter-based model development platform** that combines
|
|
an interactive workspace, a DAG scheduler, an object-storage-backed artifact
|
|
store, and per-workspace runtime isolation — all behind a single Nginx
|
|
gateway.
|
|
|
|
> Stack: React Router SPA · FastAPI · APScheduler · MySQL · S3-compatible
|
|
> storage (or local filesystem via `STORAGE_BACKEND=local`) · shared
|
|
> Jupyter · FUSE mount via rclone (s3 mode only)
|
|
> Single ingress (Nginx :80); all other services are Docker-internal.
|
|
|
|
## What it does
|
|
|
|
| Capability | Where |
|
|
|---|---|
|
|
| Workspace-scoped notebook editing with row-level lock | `backend/jupyter.py` + `scripts.is_locked` |
|
|
| Authenticated Jupyter routing (browser never sees the runtime token) | `nginx/default.conf` + `auth_request` + `backend/jupyter.py` |
|
|
| Object storage for notebooks / scripts / versions / run logs (s3 / local toggle) | `common/storage/` + `backend/scripts.py` |
|
|
| DAG-style scheduling: nodes, edges, cron, manual trigger, retries, snapshots | `backend/schedules.py` + `backend/schedule_runs.py` + `schedule/` (5 modules) |
|
|
| DAG execution via MySQL Outbox (no Redis, no in-process queues) | `schedule/orchestrator.py` + `schedule/worker.py` |
|
|
| Per-workspace Jupyter sub-process pool with asyncio locks | `runtime/process.py` |
|
|
| rclone FUSE mount of the workspace bucket into the runtime (s3 mode) | `runtime/mount.py` |
|
|
| MySQL-only persistence (26 tables, soft-delete, no foreign keys) | `common/db/models/` |
|
|
|
|
## Architecture at a glance
|
|
|
|
```
|
|
┌────────────────────┐
|
|
│ Browser (SPA) │
|
|
└─────────┬──────────┘
|
|
│ HTTPS / WS
|
|
┌─────────▼──────────┐
|
|
│ Nginx (only :80) │ ← templates/default.conf
|
|
│ /api/ /jupyter/ /storage/
|
|
└────┬───────┬──────┘
|
|
│ │
|
|
┌──────────────┘ └─────────────┐
|
|
▼ ▼
|
|
┌──────────────────┐ ┌──────────────────────┐
|
|
│ FastAPI Backend │ │ Runtime (Jupyter) │
|
|
│ + /internal/v1 │ control │ - rclone FUSE mount │
|
|
│ (storage) ├──────────────►│ - subprocess pool │
|
|
│ - DAG CRUD │ │ (per workspace) │
|
|
│ - script CRUD │ └──────────┬───────────┘
|
|
│ - auth_request │ │ FUSE / shared vol
|
|
│ - /api/v1/... │ ▼
|
|
└────┬──────┬──────┘ ┌──────────────────────┐
|
|
│ │ │ Object storage │
|
|
│ └──────── HTTP ───────►│ (s3: S3 service / │
|
|
▼ │ local: shared vol) │
|
|
┌────────────┐ │ 4 buckets per usage │
|
|
│ MySQL │◄───────── poll ─────│ │
|
|
│ - 26 tbls │ └──────────────────────┘
|
|
│ - outbox │
|
|
│ - jobstore │
|
|
└────┬───────┘
|
|
▲
|
|
│ outbox poll
|
|
┌────┴──────────────────────────┐
|
|
│ Schedule Executor │
|
|
│ - CronScheduler (APScheduler) │
|
|
│ - DispatchOrchestrator │
|
|
│ - NodeExecutor (worker) │
|
|
│ - SchedulerService (facade) │
|
|
└───────────────────────────────┘
|
|
```
|
|
|
|
Object storage is selectable via `STORAGE_BACKEND` (s3 | local). In s3 mode
|
|
the 4 purpose-named buckets (`workspaces` / `versions` / `run-logs` / `trash`)
|
|
are S3 buckets; in local mode they're subdirectories of `LOCAL_STORAGE_BASE_DIR`,
|
|
shared via the `local-storage` Docker volume. See `DEVELOP.md` §Storage.
|
|
|
|
Detailed design lives in `ARCHITECTURE.md`. Implementation deviations and
|
|
recent refactors are recorded in `HANDOVER.md`.
|
|
|
|
## Repository layout
|
|
|
|
```text
|
|
frontend/ React Router SPA
|
|
backend/ FastAPI: public API + internal storage API
|
|
runtime/ Jupyter subprocess manager + rclone FUSE
|
|
schedule/ DAG scheduler (5 modules: context/scheduler/
|
|
orchestrator/worker/service)
|
|
common/ Settings, SQLAlchemy models, storage SDK,
|
|
outbox events, jobstore
|
|
migrations/ Alembic baseline + per-feature revisions
|
|
nginx/ (concept only — see "Container" below)
|
|
scripts/ nginx-entrypoint.sh (template renderer)
|
|
docker-compose.yml 4 services — web / backend / runtime / schedule
|
|
default.conf Nginx template (mounted, rendered at start)
|
|
.env.example All env vars consumed by common.config.Settings
|
|
```
|
|
|
|
## Containers
|
|
|
|
| Service | Image | Exposed | Purpose |
|
|
|---|---|---|---|
|
|
| `web` | `nginx:alpine` | host `:8888` → `:80` | SPA, `/api/` reverse-proxy, `/jupyter/{ws}/` auth_request proxy, `/storage/` S3 passthrough (s3 mode only) |
|
|
| `backend` | `Dockerfile` | internal only | DAG CRUD, script CRUD, schedule triggers, `/api/v1/auth/jupyter`, `/internal/v1/*` storage control plane |
|
|
| `runtime` | `Dockerfile` | internal only | Per-workspace Jupyter sub-process pool, rclone FUSE mount of `workspaces` bucket (s3 mode) |
|
|
| `schedule` | `Dockerfile` | internal only | Cron tick + DAG execution via MySQL Outbox polling |
|
|
|
|
The architecture **deliberately has only one host port** (the gateway);
|
|
all other services are on the Docker internal network. This is enforced in
|
|
`docker-compose.yml` — no `ports:` on backend / runtime / schedule.
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
# Edit .env — at minimum change MYSQL password and (in s3 mode) S3 credentials.
|
|
|
|
# Static check
|
|
uv sync --all-packages
|
|
uv run --frozen --package backend python -m compileall -q backend/src common/src
|
|
uv run --frozen --package schedule python -m compileall -q schedule/src
|
|
uv run --frozen --package runtime python -m compileall -q runtime/src
|
|
|
|
# Apply schema
|
|
uv run --frozen --package backend alembic upgrade head
|
|
|
|
# Bring up the stack
|
|
docker compose config # validate
|
|
docker compose up -d --build
|
|
docker compose ps
|
|
```
|
|
|
|
Visit `http://localhost:8888`.
|
|
|
|
### Logs
|
|
|
|
```bash
|
|
docker compose logs -f backend
|
|
docker compose logs -f schedule
|
|
docker compose logs -f runtime
|
|
```
|
|
|
|
### Tear down (keeps MySQL + S3 / local-storage volumes)
|
|
|
|
```bash
|
|
docker compose down
|
|
```
|
|
|
|
### Wipe data
|
|
|
|
```bash
|
|
docker compose down -v
|
|
```
|
|
|
|
## Configuration
|
|
|
|
All environment variables are declared once in `common/src/common/config.py`
|
|
as a pydantic-settings `Settings` class, with a `@lru_cache` singleton.
|
|
Adding a new env var:
|
|
|
|
1. Add the field to `Settings` in `common/src/common/config.py` (with a
|
|
sensible default so dev-env "just works").
|
|
2. Add the line to `.env.example` with a comment.
|
|
3. Use `settings.<name>` at the call site. Never `os.environ["..."]`.
|
|
|
|
See `DEVELOP.md` for the full list of variables and their meanings.
|
|
|
|
## Storage layout
|
|
|
|
Four purpose-named buckets. The mapping from `StorageObjects.usage_type`
|
|
to bucket is decided in **one place** (`backend/storage_api.py:resolve_bucket`):
|
|
|
|
| `usage_type` | Bucket (env var) | Default name |
|
|
|---|---|---|
|
|
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `S3_WORKSPACE_BUCKET` | `workspaces` |
|
|
| `version_artifact` | `S3_VERSION_BUCKET` | `versions` |
|
|
| `run_log`, `run_result` | `S3_RUN_LOG_BUCKET` | `run-logs` |
|
|
| (soft-delete target) | `S3_TRASH_BUCKET` | `trash` |
|
|
|
|
In `STORAGE_BACKEND=s3` mode these are 4 separate S3 buckets. In
|
|
`STORAGE_BACKEND=local` mode they are 4 subdirectories under
|
|
`LOCAL_STORAGE_BASE_DIR` (default `/data`), so the layout above
|
|
becomes:
|
|
|
|
```
|
|
/data/
|
|
├── workspace/ # S3_WORKSPACE_BUCKET
|
|
├── version/ # S3_VERSION_BUCKET
|
|
├── run_log/ # S3_RUN_LOG_BUCKET
|
|
└── trash/ # S3_TRASH_BUCKET
|
|
```
|
|
|
|
A workspace's `artifact_bucket` column (when non-null) overrides the
|
|
default for that workspace, regardless of `usage_type` — useful for
|
|
isolating a paying customer to their own bucket.
|
|
|
|
The object key is a flat two-level path — `workspace_id` and a server-
|
|
issued `ulid` for the object:
|
|
|
|
```
|
|
<workspace_bucket>/<workspace_id>/<ulid>{.<ext>}
|
|
```
|
|
|
|
The file name, extension, content type, and logical path live in the
|
|
`StorageObjects` and `Scripts` rows, not in the object key, so the
|
|
storage can be re-organised without a database rewrite.
|
|
|
|
Backend code never writes to the container's local filesystem (except
|
|
in `STORAGE_BACKEND=local` mode, where the shared `local-storage` volume
|
|
is the canonical store). Schedule Executor stages node artifacts in
|
|
`tempfile.TemporaryDirectory()` (auto-cleaned). Only the `runtime`
|
|
container keeps a host volume — required by the rclone FUSE mount in
|
|
s3 mode, and a no-op pass-through in local mode.
|
|
|
|
## Documentation
|
|
|
|
- `README.md` (this file) — quick orientation
|
|
- `ARCHITECTURE.md` — design diagrams + simplification history
|
|
- `HANDOVER.md` — implementation deviations, recent refactors, pending work
|
|
- `DEVELOP.md` — developer guide (env vars, code conventions, common tasks)
|
|
- `CLAUDE.md` — agent-facing conventions for the repo
|
|
|
|
## License
|
|
|
|
Internal.
|