tao.chen
2874ccdce3
- README: refresh "what it does", "architecture at a glance", "containers" table, "configuration", "storage layout" (with usage_type→bucket routing table), and add "documentation" section pointing to ARCHITECTURE / HANDOVER / DEVELOP / CLAUDE. - DEVELOP: add the developer-facing guide that was previously only in CLAUDE.md. Covers code layout, the Settings singleton, conventions (DB / Storage / Auth / Permission gates / Outbox / async-sync signatures), local dev workflow, common tasks (adding a DAG endpoint, env var, MySQL table, RustFS bucket, schedule node type), tests status, and a troubleshooting section with the four real bugs hit this session (greenlet, MySQL, Jupyter 401, Schedule not advancing).
207 lines
9.1 KiB
Markdown
207 lines
9.1 KiB
Markdown
# Model Platform
|
|
|
|
A self-hosted **Jupyter-based model development platform** that combines
|
|
an interactive workspace, a DAG scheduler, an object-storage-backed artifact
|
|
store, and per-workspace runtime isolation — all behind a single Nginx
|
|
gateway.
|
|
|
|
> Stack: React Router SPA · FastAPI · APScheduler · MySQL · RustFS (S3)
|
|
> · shared Jupyter · FUSE mount via rclone
|
|
> Single ingress (Nginx :80); all other services are Docker-internal.
|
|
|
|
## What it does
|
|
|
|
| Capability | Where |
|
|
|---|---|
|
|
| Workspace-scoped notebook editing with row-level lock | `backend/jupyter.py` + `scripts.is_locked` |
|
|
| Authenticated Jupyter routing (browser never sees the runtime token) | `nginx/default.conf` + `auth_request` + `backend/jupyter.py` |
|
|
| Object storage for notebooks / scripts / versions / run logs (RustFS, S3 API) | `common/storage/` + `backend/scripts.py` |
|
|
| DAG-style scheduling: nodes, edges, cron, manual trigger, retries, snapshots | `backend/schedules.py` + `backend/schedule_runs.py` + `schedule/` (5 modules) |
|
|
| DAG execution via MySQL Outbox (no Redis, no in-process queues) | `schedule/orchestrator.py` + `schedule/worker.py` |
|
|
| Per-workspace Jupyter sub-process pool with asyncio locks | `runtime/process.py` |
|
|
| rclone FUSE mount of the workspace bucket into the runtime | `runtime/mount.py` |
|
|
| MySQL-only persistence (26 tables, soft-delete, no foreign keys) | `common/db/models/` |
|
|
|
|
## Architecture at a glance
|
|
|
|
```
|
|
┌────────────────────┐
|
|
│ Browser (SPA) │
|
|
└─────────┬──────────┘
|
|
│ HTTPS / WS
|
|
┌─────────▼──────────┐
|
|
│ Nginx (only :80) │ ← templates/default.conf
|
|
│ /api/ /jupyter/ /storage/
|
|
└────┬───────┬──────┘
|
|
│ │
|
|
┌──────────────┘ └─────────────┐
|
|
▼ ▼
|
|
┌──────────────────┐ ┌──────────────────────┐
|
|
│ FastAPI Backend │ │ Runtime (Jupyter) │
|
|
│ + /internal/v1 │ control │ - rclone FUSE mount │
|
|
│ (storage) ├──────────────►│ - subprocess pool │
|
|
│ - DAG CRUD │ │ (per workspace) │
|
|
│ - script CRUD │ └──────────┬───────────┘
|
|
│ - auth_request │ │ FUSE
|
|
│ - /api/v1/... │ ▼
|
|
└────┬──────┬──────┘ ┌──────────────────────┐
|
|
│ │ │ RustFS (S3) │
|
|
│ └──────── HTTP ───────►│ bucket: workspaces │
|
|
▼ │ bucket: versions │
|
|
┌────────────┐ │ bucket: run-logs │
|
|
│ MySQL │◄───────── poll ─────│ │
|
|
│ - 26 tbls │ └──────────────────────┘
|
|
│ - outbox │
|
|
│ - jobstore │
|
|
└────┬───────┘
|
|
▲
|
|
│ outbox poll
|
|
┌────┴──────────────────────────┐
|
|
│ Schedule Executor │
|
|
│ - CronScheduler (APScheduler) │
|
|
│ - DispatchOrchestrator │
|
|
│ - NodeExecutor (worker) │
|
|
│ - SchedulerService (facade) │
|
|
└───────────────────────────────┘
|
|
```
|
|
|
|
Detailed design lives in `ARCHITECTURE.md`. Implementation deviations and
|
|
recent refactors are recorded in `HANDOVER.md`.
|
|
|
|
## Repository layout
|
|
|
|
```text
|
|
frontend/ React Router SPA
|
|
backend/ FastAPI: public API + internal storage API
|
|
runtime/ Jupyter subprocess manager + rclone FUSE
|
|
schedule/ DAG scheduler (5 modules: context/scheduler/
|
|
orchestrator/worker/service)
|
|
common/ Settings, SQLAlchemy models, storage SDK,
|
|
outbox events, jobstore
|
|
migrations/ Alembic baseline + per-feature revisions
|
|
nginx/ (concept only — see "Container" below)
|
|
scripts/ nginx-entrypoint.sh (template renderer)
|
|
docker-compose.yml 4 services — web / backend / runtime / schedule
|
|
default.conf Nginx template (mounted, rendered at start)
|
|
.env.example All env vars consumed by common.config.Settings
|
|
```
|
|
|
|
## Containers
|
|
|
|
| Service | Image | Exposed | Purpose |
|
|
|---|---|---|---|
|
|
| `web` | `nginx:alpine` | host `:8888` → `:80` | SPA, `/api/` reverse-proxy, `/jupyter/{ws}/` auth_request proxy, `/storage/` RustFS passthrough |
|
|
| `backend` | `Dockerfile` | internal only | DAG CRUD, script CRUD, schedule triggers, `/api/v1/auth/jupyter`, `/internal/v1/*` storage control plane |
|
|
| `runtime` | `Dockerfile` | internal only | Per-workspace Jupyter sub-process pool, rclone FUSE mount of `workspaces` bucket |
|
|
| `schedule` | `Dockerfile` | internal only | Cron tick + DAG execution via MySQL Outbox polling |
|
|
|
|
The architecture **deliberately has only one host port** (the gateway);
|
|
all other services are on the Docker internal network. This is enforced in
|
|
`docker-compose.yml` — no `ports:` on backend / runtime / schedule.
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
# Edit .env — at minimum change MYSQL password and RUSTFS credentials.
|
|
|
|
# Static check
|
|
uv sync --all-packages
|
|
uv run --frozen --package backend python -m compileall -q backend/src common/src
|
|
uv run --frozen --package schedule python -m compileall -q schedule/src
|
|
uv run --frozen --package runtime python -m compileall -q runtime/src
|
|
|
|
# Apply schema
|
|
uv run --frozen --package backend alembic upgrade head
|
|
|
|
# Bring up the stack
|
|
docker compose config # validate
|
|
docker compose up -d --build
|
|
docker compose ps
|
|
```
|
|
|
|
Visit `http://localhost:8888`.
|
|
|
|
### Logs
|
|
|
|
```bash
|
|
docker compose logs -f backend
|
|
docker compose logs -f schedule
|
|
docker compose logs -f runtime
|
|
```
|
|
|
|
### Tear down (keeps MySQL + RustFS volumes)
|
|
|
|
```bash
|
|
docker compose down
|
|
```
|
|
|
|
### Wipe data
|
|
|
|
```bash
|
|
docker compose down -v
|
|
```
|
|
|
|
## Configuration
|
|
|
|
All environment variables are declared once in `common/src/common/config.py`
|
|
as a pydantic-settings `Settings` class, with a `@lru_cache` singleton.
|
|
Adding a new env var:
|
|
|
|
1. Add the field to `Settings` in `common/src/common/config.py` (with a
|
|
sensible default so dev-env "just works").
|
|
2. Add the line to `.env.example` with a comment.
|
|
3. Use `settings.<name>` at the call site. Never `os.environ["..."]`.
|
|
|
|
See `DEVELOP.md` for the full list of variables and their meanings.
|
|
|
|
## Storage layout
|
|
|
|
Three purpose-named RustFS buckets. The mapping from `StorageObjects.usage_type`
|
|
to bucket is decided in **one place** (`storage_api.py:resolve_bucket`):
|
|
|
|
| `usage_type` | Bucket (env var) | Default name |
|
|
|---|---|---|
|
|
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `RUSTFS_WORKSPACE_BUCKET` | `workspaces` |
|
|
| `version_artifact` | `RUSTFS_VERSION_BUCKET` | `versions` |
|
|
| `run_log`, `run_result` | `RUSTFS_RUN_LOG_BUCKET` | `run-logs` |
|
|
|
|
A workspace's `artifact_bucket` column (when non-null) overrides the
|
|
default for that workspace, regardless of `usage_type` — useful for
|
|
isolating a paying customer to their own bucket.
|
|
|
|
The object key is a flat two-level path — `workspace_id` and a server-
|
|
issued `ulid` for the object:
|
|
|
|
```
|
|
s3://workspaces/
|
|
└── <workspace_id>/
|
|
├── <ulid-1> # working_copy / data_resource / snapshot / ...
|
|
├── <ulid-2>
|
|
└── ...
|
|
|
|
s3://versions/<workspace_id>/<ulid> # immutable script versions
|
|
s3://run-logs/<workspace_id>/<ulid> # node run logs and results
|
|
```
|
|
|
|
The file name, extension, content type, and logical path live in the
|
|
`StorageObjects` and `Scripts` rows, not in the S3 key, so the bucket
|
|
can be re-organised without a database rewrite.
|
|
|
|
Backend code never writes to the container's local filesystem. Schedule
|
|
Executor stages node artifacts in `tempfile.TemporaryDirectory()` (auto-
|
|
cleaned). Only the `runtime` container keeps a host volume — it is required
|
|
by the rclone FUSE mount.
|
|
|
|
## Documentation
|
|
|
|
- `README.md` (this file) — quick orientation
|
|
- `ARCHITECTURE.md` — design diagrams + simplification history
|
|
- `HANDOVER.md` — implementation deviations, recent refactors, pending work
|
|
- `DEVELOP.md` — developer guide (env vars, code conventions, common tasks)
|
|
- `CLAUDE.md` — agent-facing conventions for the repo
|
|
|
|
## License
|
|
|
|
Internal.
|