tao.chen/model-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: align with new storage architecture (s3 + local + server-proxied PUT)

Browse Source 
All operator- and developer-facing docs updated to reflect:

  - The unified AsyncStorageBackend abstraction (s3 + local backends).
  - The STORAGE_BACKEND toggle ("s3" default, "local" for dev /
    single-node / air-gapped deployments).
  - The 4-purpose-bucket layout (workspace / version / run_log / trash)
    in both modes — 4 separate S3 buckets in s3 mode, 4 subdirectories
    of LOCAL_STORAGE_BASE_DIR in local mode.
  - The S3_* env var naming (was RUSTFS_*).
  - The server-proxied upload flow (was browser-direct presign-PUT):
    POST /internal/v1/uploads → PUT /internal/v1/uploads/{id} with
    raw bytes → server calls backend.put().
  - The factory helpers workspaces_root() (runtime's view of the
    workspace bucket on disk) and rclone_remote_spec() (s3-mode mount
    source).
  - The "two settings describing the same thing" cleanup: the deleted
    settings.workspace_root, settings.workspaces_root, and
    settings.remote_bucket fields.

Files touched:
  - API.md (§5 data-resource upload flow, §9 storage control plane,
    §10 readiness example)
  - ARCHITECTURE.md (storage layer diagram)
  - CLAUDE.md (architecture description + volume-preservation note)
  - DEVELOP.md (settings list, Storage section, "Wire a new bucket"
    how-to, dev-export example, troubleshooting network hint)
  - README.md (architecture diagram, container table, quick-start
    credentials note, tear-down note, Storage layout section)
  - REFACTOR_NOTES.md (final container list with s3 explanation)
  - backend/README.md (storage backend description)
  - migrations/data/README.md (step 11/12 record mentioning object
    storage)

A handful of historical "RustFS" mentions are intentionally retained
where they name a specific S3-compatible product (e.g. as an example
in REFACTOR_NOTES.md's container list) or document the pre-2026
abstraction name (DEVELOP.md Storage section).
This commit is contained in:
tao.chen
2026-08-05 13:13:20 +08:00
parent 4e290bd80a
commit 309b657d35
8 changed files with 195 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+47 -32
View File
@@ -5,8 +5,9 @@ an interactive workspace, a DAG scheduler, an object-storage-backed artifact
store, and per-workspace runtime isolation — all behind a single Nginx
gateway.
> Stack: React Router SPA · FastAPI · APScheduler · MySQL · RustFS (S3)
> · shared Jupyter · FUSE mount via rclone
> Stack: React Router SPA · FastAPI · APScheduler · MySQL · S3-compatible
> storage (or local filesystem via `STORAGE_BACKEND=local`) · shared
> Jupyter · FUSE mount via rclone (s3 mode only)
> Single ingress (Nginx :80); all other services are Docker-internal.
## What it does
@@ -15,11 +16,11 @@ gateway.
|---|---|
| Workspace-scoped notebook editing with row-level lock | `backend/jupyter.py` + `scripts.is_locked` |
| Authenticated Jupyter routing (browser never sees the runtime token) | `nginx/default.conf` + `auth_request` + `backend/jupyter.py` |
| Object storage for notebooks / scripts / versions / run logs (RustFS, S3 API) | `common/storage/` + `backend/scripts.py` |
| Object storage for notebooks / scripts / versions / run logs (s3 / local toggle) | `common/storage/` + `backend/scripts.py` |
| DAG-style scheduling: nodes, edges, cron, manual trigger, retries, snapshots | `backend/schedules.py` + `backend/schedule_runs.py` + `schedule/` (5 modules) |
| DAG execution via MySQL Outbox (no Redis, no in-process queues) | `schedule/orchestrator.py` + `schedule/worker.py` |
| Per-workspace Jupyter sub-process pool with asyncio locks | `runtime/process.py` |
| rclone FUSE mount of the workspace bucket into the runtime | `runtime/mount.py` |
| rclone FUSE mount of the workspace bucket into the runtime (s3 mode) | `runtime/mount.py` |
| MySQL-only persistence (26 tables, soft-delete, no foreign keys) | `common/db/models/` |
## Architecture at a glance
@@ -42,13 +43,13 @@ gateway.
│ (storage) ├──────────────►│ - subprocess pool │
│ - DAG CRUD │ │ (per workspace) │
│ - script CRUD │ └──────────┬───────────┘
│ - auth_request │ │ FUSE
│ - auth_request │ │ FUSE / shared vol
│ - /api/v1/... │ ▼
└────┬──────┬──────┘ ┌──────────────────────┐
│ │ │ RustFS (S3) │
│ └──────── HTTP ───────►│ bucket: workspaces │
▼ │ bucket: versions │
┌────────────┐ │ bucket: run-logs │
│ │ │ Object storage │
│ └──────── HTTP ───────►│ (s3: S3 service / │
▼ │ local: shared vol) │
┌────────────┐ │ 4 buckets per usage │
│ MySQL │◄───────── poll ─────│ │
│ - 26 tbls │ └──────────────────────┘
│ - outbox │
@@ -65,6 +66,11 @@ gateway.
└───────────────────────────────┘
```
Object storage is selectable via `STORAGE_BACKEND` (s3 | local). In s3 mode
the 4 purpose-named buckets (`workspaces` / `versions` / `run-logs` / `trash`)
are S3 buckets; in local mode they're subdirectories of `LOCAL_STORAGE_BASE_DIR`,
shared via the `local-storage` Docker volume. See `DEVELOP.md` §Storage.
Detailed design lives in `ARCHITECTURE.md`. Implementation deviations and
recent refactors are recorded in `HANDOVER.md`.
@@ -90,9 +96,9 @@ default.conf Nginx template (mounted, rendered at start)
| Service | Image | Exposed | Purpose |
|---|---|---|---|
| `web` | `nginx:alpine` | host `:8888` → `:80` | SPA, `/api/` reverse-proxy, `/jupyter/{ws}/` auth_request proxy, `/storage/` RustFS passthrough |
| `web` | `nginx:alpine` | host `:8888` → `:80` | SPA, `/api/` reverse-proxy, `/jupyter/{ws}/` auth_request proxy, `/storage/` S3 passthrough (s3 mode only) |
| `backend` | `Dockerfile` | internal only | DAG CRUD, script CRUD, schedule triggers, `/api/v1/auth/jupyter`, `/internal/v1/*` storage control plane |
| `runtime` | `Dockerfile` | internal only | Per-workspace Jupyter sub-process pool, rclone FUSE mount of `workspaces` bucket |
| `runtime` | `Dockerfile` | internal only | Per-workspace Jupyter sub-process pool, rclone FUSE mount of `workspaces` bucket (s3 mode) |
| `schedule` | `Dockerfile` | internal only | Cron tick + DAG execution via MySQL Outbox polling |
The architecture **deliberately has only one host port** (the gateway);
@@ -103,7 +109,7 @@ all other services are on the Docker internal network. This is enforced in
```bash
cp .env.example .env
# Edit .env — at minimum change MYSQL password and RUSTFS credentials.
# Edit .env — at minimum change MYSQL password and (in s3 mode) S3 credentials.
# Static check
uv sync --all-packages
@@ -130,7 +136,7 @@ docker compose logs -f schedule
docker compose logs -f runtime
```
### Tear down (keeps MySQL + RustFS volumes)
### Tear down (keeps MySQL + S3 / local-storage volumes)
```bash
docker compose down
@@ -157,14 +163,28 @@ See `DEVELOP.md` for the full list of variables and their meanings.
## Storage layout
Three purpose-named RustFS buckets. The mapping from `StorageObjects.usage_type`
to bucket is decided in **one place** (`storage_api.py:resolve_bucket`):
Four purpose-named buckets. The mapping from `StorageObjects.usage_type`
to bucket is decided in **one place** (`backend/storage_api.py:resolve_bucket`):
| `usage_type` | Bucket (env var) | Default name |
|---|---|---|
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `RUSTFS_WORKSPACE_BUCKET` | `workspaces` |
| `version_artifact` | `RUSTFS_VERSION_BUCKET` | `versions` |
| `run_log`, `run_result` | `RUSTFS_RUN_LOG_BUCKET` | `run-logs` |
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `S3_WORKSPACE_BUCKET` | `workspaces` |
| `version_artifact` | `S3_VERSION_BUCKET` | `versions` |
| `run_log`, `run_result` | `S3_RUN_LOG_BUCKET` | `run-logs` |
| (soft-delete target) | `S3_TRASH_BUCKET` | `trash` |
In `STORAGE_BACKEND=s3` mode these are 4 separate S3 buckets. In
`STORAGE_BACKEND=local` mode they are 4 subdirectories under
`LOCAL_STORAGE_BASE_DIR` (default `/data`), so the layout above
becomes:
```
/data/
├── workspace/ # S3_WORKSPACE_BUCKET
├── version/ # S3_VERSION_BUCKET
├── run_log/ # S3_RUN_LOG_BUCKET
└── trash/ # S3_TRASH_BUCKET
```
A workspace's `artifact_bucket` column (when non-null) overrides the
default for that workspace, regardless of `usage_type` — useful for
@@ -174,24 +194,19 @@ The object key is a flat two-level path — `workspace_id` and a server-
issued `ulid` for the object:
```
s3://workspaces/
└── <workspace_id>/
├── <ulid-1> # working_copy / data_resource / snapshot / ...
├── <ulid-2>
└── ...
s3://versions/<workspace_id>/<ulid> # immutable script versions
s3://run-logs/<workspace_id>/<ulid> # node run logs and results
<workspace_bucket>/<workspace_id>/<ulid>{.<ext>}
```
The file name, extension, content type, and logical path live in the
`StorageObjects` and `Scripts` rows, not in the S3 key, so the bucket
can be re-organised without a database rewrite.
`StorageObjects` and `Scripts` rows, not in the object key, so the
storage can be re-organised without a database rewrite.
Backend code never writes to the container's local filesystem. Schedule
Executor stages node artifacts in `tempfile.TemporaryDirectory()` (auto-
cleaned). Only the `runtime` container keeps a host volume — it is required
by the rclone FUSE mount.
Backend code never writes to the container's local filesystem (except
in `STORAGE_BACKEND=local` mode, where the shared `local-storage` volume
is the canonical store). Schedule Executor stages node artifacts in
`tempfile.TemporaryDirectory()` (auto-cleaned). Only the `runtime`
container keeps a host volume — required by the rclone FUSE mount in
s3 mode, and a no-op pass-through in local mode.
## Documentation