tao.chen/model-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: align with new storage architecture (s3 + local + server-proxied PUT)

Browse Source 
All operator- and developer-facing docs updated to reflect:

  - The unified AsyncStorageBackend abstraction (s3 + local backends).
  - The STORAGE_BACKEND toggle ("s3" default, "local" for dev /
    single-node / air-gapped deployments).
  - The 4-purpose-bucket layout (workspace / version / run_log / trash)
    in both modes — 4 separate S3 buckets in s3 mode, 4 subdirectories
    of LOCAL_STORAGE_BASE_DIR in local mode.
  - The S3_* env var naming (was RUSTFS_*).
  - The server-proxied upload flow (was browser-direct presign-PUT):
    POST /internal/v1/uploads → PUT /internal/v1/uploads/{id} with
    raw bytes → server calls backend.put().
  - The factory helpers workspaces_root() (runtime's view of the
    workspace bucket on disk) and rclone_remote_spec() (s3-mode mount
    source).
  - The "two settings describing the same thing" cleanup: the deleted
    settings.workspace_root, settings.workspaces_root, and
    settings.remote_bucket fields.

Files touched:
  - API.md (§5 data-resource upload flow, §9 storage control plane,
    §10 readiness example)
  - ARCHITECTURE.md (storage layer diagram)
  - CLAUDE.md (architecture description + volume-preservation note)
  - DEVELOP.md (settings list, Storage section, "Wire a new bucket"
    how-to, dev-export example, troubleshooting network hint)
  - README.md (architecture diagram, container table, quick-start
    credentials note, tear-down note, Storage layout section)
  - REFACTOR_NOTES.md (final container list with s3 explanation)
  - backend/README.md (storage backend description)
  - migrations/data/README.md (step 11/12 record mentioning object
    storage)

A handful of historical "RustFS" mentions are intentionally retained
where they name a specific S3-compatible product (e.g. as an example
in REFACTOR_NOTES.md's container list) or document the pre-2026
abstraction name (DEVELOP.md Storage section).
This commit is contained in:
tao.chen
2026-08-05 13:13:20 +08:00
parent 4e290bd80a
commit 309b657d35
8 changed files with 195 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files
API.md
+58 -29
View File
@@ -92,7 +92,7 @@
### 3.2 `POST /api/v1/workspace-directories` ### 3.2 `POST /api/v1/workspace-directories`
创建一个**逻辑目录**(RustFS 上是隐式前缀,无需落对象)。 创建一个**逻辑目录**(对象存储上是隐式前缀,无需落对象)。
- **请求体**: - **请求体**:
```json ```json
@@ -163,7 +163,7 @@
### 3.7 `POST /api/v1/scripts/upload?file_name=...&parent_path=...&visibility=...` ### 3.7 `POST /api/v1/scripts/upload?file_name=...&parent_path=...&visibility=...`
multipart/binary 形式上传大文件(走 presigned PUT)。 multipart/binary 形式上传大文件(走 server-proxied PUT,详见 §九)。
- **查询参数**: `file_name`(必填)、`parent_path`、`visibility` - **查询参数**: `file_name`(必填)、`parent_path`、`visibility`
- **请求体**: 原始文件字节(`Content-Type` 必须与脚本类型匹配) - **请求体**: 原始文件字节(`Content-Type` 必须与脚本类型匹配)
@@ -203,7 +203,7 @@ multipart/binary 形式上传大文件(走 presigned PUT)。
### 3.11 `POST /api/v1/scripts/{script_id}/versions` ### 3.11 `POST /api/v1/scripts/{script_id}/versions`
发布一个**稳定版本**(immutable,绑定到 `RUSTFS_VERSION_BUCKET`)。门禁同 §3.8。 发布一个**稳定版本**(immutable,绑定到 `S3_VERSION_BUCKET`)。门禁同 §3.8。
- **请求体**: - **请求体**:
```json ```json
@@ -243,11 +243,11 @@ multipart/binary 形式上传大文件(走 presigned PUT)。
### 3.14 `DELETE /api/v1/versions/{versions_id}` ### 3.14 `DELETE /api/v1/versions/{versions_id}`
从调度候选中**隐藏**此版本(不删除 RustFS 对象)。门禁:**owner 校验基于所属 `Scripts` 的 owner**——即"按整本 script 判定",而非"按版本发布者判定"。 从调度候选中**隐藏**此版本(不删除对象存储里的对象)。门禁:**owner 校验基于所属 `Scripts` 的 owner**——即"按整本 script 判定",而非"按版本发布者判定"。
### 3.15 `POST /api/v1/versions/{versions_id}/download-url` ### 3.15 `POST /api/v1/versions/{versions_id}/download-url`
生成 RustFS 的 presigned download URL。 生成对象存储的 presigned download URL(走 S3 兼容协议,local 模式下该 endpoint 在 s3 模式才生效)。
- **请求体**: - **请求体**:
```json ```json
@@ -399,8 +399,8 @@ queued ──→ running ──┬─→ succeeded
| 方法 | 路径 | 说明 | | 方法 | 路径 | 说明 |
|---|---|---| |---|---|---|
| `POST` | `/api/v1/data-resources/uploads` | 创建上传会话,返回 presigned PUT URL + `upload_id` | | `POST` | `/api/v1/data-resources/uploads` | 创建上传会话,返回 `upload_id` + `upload_path` |
| `POST` | `/api/v1/data-resources/uploads/{upload_id}/complete` | 完成上传(写 `StorageObjects` 行) | | `PUT` | `/api/v1/data-resources/uploads/{upload_id}` | 上传字节(请求体即文件内容) |
| `GET` | `/api/v1/data-resources` | 列表(workspace 范围) | | `GET` | `/api/v1/data-resources` | 列表(workspace 范围) |
| `GET` | `/api/v1/data-resources/{id}` | 详情 | | `GET` | `/api/v1/data-resources/{id}` | 详情 |
| `POST` | `/api/v1/data-resources/{id}/download-url` | 生成 presigned GET URL | | `POST` | `/api/v1/data-resources/{id}/download-url` | 生成 presigned GET URL |
@@ -420,15 +420,18 @@ queued ──→ running ──┬─→ succeeded
**完整上传流程(前端应实现的模式)**: **完整上传流程(前端应实现的模式)**:
``` ```
1. POST /uploads → {upload_id, presigned_url} 1. POST /uploads → {upload_id, upload_path, expires_at}
2. PUT presigned_url with file bytes 2. PUT upload_path with raw file bytes (Content-Type: application/octet-stream)
3. POST /uploads/{upload_id}/complete 3. 服务器端走 backend.put() → 200 {data: StorageObjectPayload}
4. 服务器完成 → 200 {data: StorageObjectPayload}
``` ```
字节经过 backend 进程(server-proxied upload),最大 100 MiB,由 backend
直接调 `AsyncStorageBackend.put()` 写入存储(不再走 presigned PUT 直传)。
前端无需关心 S3 协议或签名。
**小对象(<100 KiB)捷径**:直接调 `create_server_object` 把字节 base64 放进 **小对象(<100 KiB)捷径**:直接调 `create_server_object` 把字节 base64 放进
`content_base64` 字段(走 `StorageClient.create_server_object`),前端无需 `content_base64` 字段(JSON 体里走),内部走同一条 `AsyncStorageBackend.put`
走 presigned PUT 三步。 路径,前端无需分两步。
--- ---
@@ -672,9 +675,13 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断
## 九、对象存储控制面 ## 九、对象存储控制面
> 路径前缀 `/internal/v1/...`,**前端不要直接调用**。这是 backend 内部 > 路径前缀 `/internal/v1/...`,**前端不要直接调用**。这是 backend 内部
> 异步消息处理(Schedule worker)用的 RPC 端点,经 `StorageClient` HTTP > 异步消息处理(Schedule worker)用的 RPC 端点,经 in-process ASGI 直接
> 客户端访问。Backend 通过 ASGI `auth_request_set` 路由转发,外部无法 > 转发(`storage_app` 路由被 `app.include_router` 进同一个 backend 进程),
> 访问。 > 外部无法访问。
底层抽象:`common.storage.AsyncStorageBackend`(`put/get/delete/exists/stat/
list/get_url/copy`)。按 `settings.storage_backend` 选实现:`"s3"` 走
S3-兼容服务,`"local"` 走 `LOCAL_STORAGE_BASE_DIR` 子目录。
### 9.1 `POST /internal/v1/uploads` ### 9.1 `POST /internal/v1/uploads`
@@ -689,23 +696,36 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断
"content_type": "text/x-python", "content_type": "text/x-python",
"expected_size_bytes": 1024, "expected_size_bytes": 1024,
"expected_hash": "<optional sha256 hex>", "expected_hash": "<optional sha256 hex>",
"idempotency_key": "..." "idempotency_key": "...",
"visibility": "private",
"is_immutable": false
} }
``` ```
返回 `{upload_id, bucket_name, object_key, presigned_url, expires_in_seconds}`。 返回 `{upload_id, status, upload_path, expires_at}`。`upload_path` 是
第 9.2 步要 PUT 的端点(本进程内 `/internal/v1/uploads/{upload_id}`)。
### 9.2 `POST /internal/v1/uploads/{upload_id}/complete` ### 9.2 `PUT /internal/v1/uploads/{upload_id}`
完成上传。从 RustFS 读 HEAD → 校验 hash → 写 `StorageObjects` 行。 完成上传(server-proxied PUT)。**请求体即原始字节**,`Content-Type:
application/octet-stream`。后端 `await request.body()` 读字节 → 校验
size + sha256 → 调 `await backend.put(key, bytes, content_type=...,
metadata={"sha256": ...})` → 写 `StorageObjects` 行 → 标 session 为
completed。最大 100 MiB。
> 历史:旧版本这一步是 `POST /uploads/{id}/complete`,靠
> presigned-PUT + head() 验证。已被 server-proxied PUT 取代。
### 9.3 `POST /internal/v1/uploads/{upload_id}/abort` ### 9.3 `POST /internal/v1/uploads/{upload_id}/abort`
主动放弃。释放 `UploadSessions` 行,对象不入库。 主动放弃。删除可能已经写了一半的对象字节,释放 `UploadSessions` 行。
### 9.4 `POST /internal/v1/objects` ### 9.4 `POST /internal/v1/objects`
**单步创建**(不走 presigned PUT,字节随请求体直传)。适用 < 100 KiB 对象。 **单步创建**(不走两步上传,字节 base64 进 JSON 体)。适用 < 100 KiB
对象(避免 multipart/大请求体的前端复杂度)。内部直接调
`AsyncStorageBackend.put(key, content, content_type=...,
metadata={"sha256": ...})`。
```json ```json
{ {
@@ -717,28 +737,36 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断
"content_base64": "PHN0ZXAtY29udGVudD4=", "content_base64": "PHN0ZXAtY29udGVudD4=",
"visibility": "private", "visibility": "private",
"is_immutable": false, "is_immutable": false,
"idempotency_key": "..." "idempotency_key": "...",
"relative_path": null
} }
``` ```
### 9.5 `POST /internal/v1/objects/{storage_object_id}/download-url` ### 9.5 `POST /internal/v1/objects/{storage_object_id}/download-url`
生成 presigned GET URL。 生成 presigned GET URL(s3 模式:`AsyncStorageBackend.get_url()`;
local 模式:目前抛 `NotImplementedError`,需要 native FS serving 配合
nginx 静态 location)。
### 9.6 `DELETE /internal/v1/objects/{storage_object_id}` ### 9.6 `DELETE /internal/v1/objects/{storage_object_id}`
软删。`is_immutable == 1` 的对象拒绝删除。 软删。`is_immutable == 1` 的对象拒绝删除。`move_to_trash` 走跨后端
`copy + delete`(同一进程内的两个 backend 实例)。
### 9.7 usage_type → 桶路由(自动) ### 9.7 usage_type → 桶路由(自动)
| usage_type | 实际桶(env var) | 默认桶名 | | usage_type | 实际桶(env var) | 默认桶名 |
|---|---|---| |---|---|---|
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `RUSTFS_WORKSPACE_BUCKET` | `workspaces` | | `working_copy`, `public_script`, `data_resource`, `snapshot` | `S3_WORKSPACE_BUCKET` | `workspaces` |
| `version_artifact` | `RUSTFS_VERSION_BUCKET` | `versions` | | `version_artifact` | `S3_VERSION_BUCKET` | `versions` |
| `run_log`, `run_result` | `RUSTFS_RUN_LOG_BUCKET` | `run-logs` | | `run_log`, `run_result` | `S3_RUN_LOG_BUCKET` | `run-logs` |
| (soft-delete target) | `S3_TRASH_BUCKET` | `trash` |
若 `Workspaces.artifact_bucket` 非空,优先用 per-workspace 桶(覆盖 usage_type 路由)。 若 `Workspaces.artifact_bucket` 非空,优先用 per-workspace 桶(覆盖 usage_type 路由)。
桶在 `STORAGE_BACKEND=s3` 时是 4 个独立 S3 bucket,在
`STORAGE_BACKEND=local` 时是 `LOCAL_STORAGE_BASE_DIR` 下的 4 个子目录。
--- ---
## 十、健康检查 ## 十、健康检查
@@ -751,7 +779,8 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断
| `GET` | `/api/v1/health` | 公开健康检查(前端可访问) | | `GET` | `/api/v1/health` | 公开健康检查(前端可访问) |
`/health/ready` 支持 `READINESS_TARGETS` 环境变量,逗号分隔的 `host:port` `/health/ready` 支持 `READINESS_TARGETS` 环境变量,逗号分隔的 `host:port`
列表,例如 `mysql:3306,rustfs:9000`,全部 TCP 通则返回 200,否则 503。 列表,例如 `mysql:3306,s3:9000`,全部 TCP 通则返回 200,否则 503。
`STORAGE_BACKEND=local` 模式下不需要 S3 host,列表里删掉即可。
--- ---
ARCHITECTURE.md
+1 -1
View File
@@ -24,7 +24,7 @@ Schedule Executor（APScheduler）
|-- MySQL APSchedulerJobStore |-- MySQL APSchedulerJobStore
|-- MySQL Outbox 轮询兜底 |-- MySQL Outbox 轮询兜底
|-- DAG 节点执行与重试 |-- DAG 节点执行与重试
|-- RustFS 日志/结果 |-- S3 日志/结果
+-- Backend 内部 Storage API +-- Backend 内部 Storage API
``` ```
CLAUDE.md
+2 -2
View File
@@ -3,7 +3,7 @@
## Current architecture ## Current architecture
- `frontend`: React Router SPA. Production files are built in `nginx/Dockerfile`. - `frontend`: React Router SPA. Production files are built in `nginx/Dockerfile`.
- `backend`: public FastAPI API and internal RustFS storage API in one process. - `backend`: public FastAPI API and internal S3 storage API in one process.
- `runtime`: shared Jupyter lifecycle, MySQL edit leases and short-lived in-memory access tickets. - `runtime`: shared Jupyter lifecycle, MySQL edit leases and short-lived in-memory access tickets.
- `schedule`: APScheduler, MySQL JobStore, MySQL Outbox polling and DAG execution. - `schedule`: APScheduler, MySQL JobStore, MySQL Outbox polling and DAG execution.
- `common`: SQLAlchemy models, database/session helpers, IDs and object-store helpers. - `common`: SQLAlchemy models, database/session helpers, IDs and object-store helpers.
@@ -51,7 +51,7 @@ pnpm build
- Cron jobs are persisted by APScheduler in MySQL table `apscheduler_jobs`. - Cron jobs are persisted by APScheduler in MySQL table `apscheduler_jobs`.
- Runtime must stay single-replica while file leases and Jupyter tickets use the simplified implementation. - Runtime must stay single-replica while file leases and Jupyter tickets use the simplified implementation.
- Never expose the internal Jupyter token to the browser. - Never expose the internal Jupyter token to the browser.
- Never delete Docker volumes when preserving MySQL or RustFS data is required. - Never delete Docker volumes when preserving MySQL or storage data is required.
## Main entrypoints ## Main entrypoints
DEVELOP.md
+73 -43
View File
@@ -12,7 +12,7 @@ common/ Pure-Python shared library
db/ SQLAlchemy 2.0 async engine, session_scope, Base db/ SQLAlchemy 2.0 async engine, session_scope, Base
db/models/ 26 tables in 9 domain files (zero FK, zero relationship) db/models/ 26 tables in 9 domain files (zero FK, zero relationship)
scheduler/ build_sqlalchemy_jobstore (delayed import) scheduler/ build_sqlalchemy_jobstore (delayed import)
storage/ RustFSObjectStore + StorageClient + Pydantic schemas storage/ AsyncStorageBackend abstraction (s3 + local impls) + Pydantic schemas
eventing.py add_outbox_event / utcnow / event_time eventing.py add_outbox_event / utcnow / event_time
service_app.py /health/ready TCP probe, /api/v1/health service_app.py /health/ready TCP probe, /api/v1/health
schemas.py StrictModel base schemas.py StrictModel base
@@ -21,14 +21,14 @@ common/ Pure-Python shared library
backend/ Public FastAPI service + internal /internal/v1/* sub-app backend/ Public FastAPI service + internal /internal/v1/* sub-app
main.py lifespan + route registration main.py lifespan + route registration
jupyter.py /api/v1/auth/jupyter — the ONLY auth entry jupyter.py /api/v1/auth/jupyter — the ONLY auth entry
scripts.py CRUD for scripts/notebooks (workspace_fs=rustfs) scripts.py CRUD for scripts/notebooks (object storage via AsyncStorageBackend)
schedules.py DAG CRUD: schedules, nodes, edges schedules.py DAG CRUD: schedules, nodes, edges
schedule_runs.py Trigger / list / get runs schedule_runs.py Trigger / list / get runs
schedule_schemas.py Pydantic request/response models schedule_schemas.py Pydantic request/response models
admin.py Admin endpoints admin.py Admin endpoints
resources.py Misc data resources resources.py Misc data resources
storage_api.py /internal/v1/* (sub-app merged into main) storage_api.py /internal/v1/* (sub-app merged into main)
storage_client.py HTTP client for the storage sub-app storage_client.py Stub (HTTP client removed post-migration; rewrite pending)
schedule_client.py Placeholder module (was the HTTP-push executor client) schedule_client.py Placeholder module (was the HTTP-push executor client)
runtime_client.py Self-contained httpx wrapper for the runtime runtime_client.py Self-contained httpx wrapper for the runtime
jupyter.py auth_request handler jupyter.py auth_request handler
@@ -41,7 +41,7 @@ schedule/ Schedule Executor (DAG worker)
worker.py NodeExecutor (notebook / python execution) worker.py NodeExecutor (notebook / python execution)
service.py SchedulerService facade (composes the three) service.py SchedulerService facade (composes the three)
main.py Lifespan + FastAPI app main.py Lifespan + FastAPI app
storage_client.py SchedulerStorageClient (subclass of common StorageClient) storage_client.py Stub (SchedulerStorageClient rewrite pending — use AsyncStorageBackend directly)
execution.py execute_artifact (notebook + python paths) execution.py execute_artifact (notebook + python paths)
notebook_runner.py Subprocess entry point (nbclient) notebook_runner.py Subprocess entry point (nbclient)
@@ -68,16 +68,17 @@ All env vars go through one place: `common/src/common/config.py`.
from common.config import settings from common.config import settings
settings.database_url # str settings.database_url # str
settings.rustfs_endpoint # str (full URL, e.g. "http://rustfs:9000") settings.storage_backend # str: "s3" (default) or "local"
settings.rustfs_access_key # str settings.local_storage_base_dir # str: root dir for storage data (default "/data"); see "Storage" below for per-mode derivation
settings.rustfs_secret_key # str settings.s3_endpoint # str (full URL, e.g. "http://s3:9000"; s3 mode only)
settings.rustfs_workspace_bucket settings.s3_access_key # str (s3 mode only)
settings.rustfs_version_bucket settings.s3_secret_key # str (s3 mode only)
settings.rustfs_run_log_bucket settings.s3_workspace_bucket # str (s3 mode only)
settings.jwt_secret # HS256 secret for the auth_request handler settings.s3_version_bucket # str (s3 mode only)
settings.workspace_root # schedule subprocess cwd; backend ignores settings.s3_run_log_bucket # str (s3 mode only)
settings.workspaces_root # runtime rclone FUSE mount point settings.s3_trash_bucket # str (s3 mode only)
settings.remote_bucket # rclone remote spec (e.g. "rustfs:workspaces") settings.s3_trash_retention_days # int (s3 mode only)
settings.jwt_secret # HS256 secret for the auth_request handler
settings.backend_api_url # schedule → backend HTTP base settings.backend_api_url # schedule → backend HTTP base
settings.runtime_api_url # backend → runtime HTTP base settings.runtime_api_url # backend → runtime HTTP base
settings.public_base_url # runtime public base URL settings.public_base_url # runtime public base URL
@@ -125,12 +126,31 @@ grep -rnE 'os\.(environ\[?["\x27][A-Z_]+|getenv\(["\x27][A-Z_]+)' --include="*.p
### Storage ### Storage
- All object bytes go to **RustFS** via boto3. - All object bytes go through `common.storage.AsyncStorageBackend`,
- Use `StorageClient` (HTTP) or `RustFSObjectStore` (direct) — never created by `create_storage(config)` from `common.storage.factory`.
the local filesystem. - Two backends are registered: `local` (filesystem, local mode) and
- `bucket_name` is one of: `RUSTFS_WORKSPACE_BUCKET` (default `s3` (S3-compatible service, s3 mode). Selection is per-deployment
`workspaces`), `RUSTFS_VERSION_BUCKET` (default `versions`, via `settings.storage_backend` (`"s3"` default, `"local"` for
reserved), `RUSTFS_RUN_LOG_BUCKET` (default `run-logs`, reserved). dev / single-node / air-gapped).
- The factory helper `build_storage_config(bucket_name)` returns the
right `create_storage` kwargs for each of the 4 purpose buckets
(`workspace`, `version`, `run_log`, `trash`). Use it in lifespan code;
route handlers don't see the difference.
- Bucket resolution from `usage_type` is in **one place**
(`backend/storage_api.py:resolve_bucket`); route handlers only know
about `app.state.object_stores[bucket_name]`.
- The runtime's view of the workspace bucket on disk is exposed by
`common.storage.workspaces_root()`:
- `s3` mode: `${settings.local_storage_base_dir}/workspaces`
(default `/data/workspaces`, the rclone FUSE mount target).
- `local` mode: `${settings.local_storage_base_dir}/workspace`
(default `/data/workspace`, a subdir of the shared local-storage
volume).
`settings.local_storage_base_dir` is the **only** path setting; the
helper handles the per-mode suffix. Don't read `settings.workspaces_root`
or any other path setting directly in runtime code — use this helper.
- The pre-2026 abstraction (`RustFSObjectStore` / `common.storage.client`
/ `StorageClient` HTTP wrapper) is gone. Don't reintroduce it.
### Auth ### Auth
@@ -202,9 +222,13 @@ cd frontend && pnpm install && cd ..
```bash ```bash
# Backend (terminal 1) # Backend (terminal 1)
export DATABASE_URL="mysql+asyncmy://model_platform:model_platform@127.0.0.1:3306/model_platform?charset=utf8mb4" export DATABASE_URL="mysql+asyncmy://model_platform:model_platform@127.0.0.1:3306/model_platform?charset=utf8mb4"
export RUSTFS_ACCESS_KEY=modelplatform export STORAGE_BACKEND=s3
export RUSTFS_SECRET_KEY=modelplatformsecret export S3_ACCESS_KEY=modelplatform
export RUSTFS_ENDPOINT=http://127.0.0.1:9000 export S3_SECRET_KEY=modelplatformsecret
export S3_ENDPOINT=http://127.0.0.1:9000
# Or for local mode:
# export STORAGE_BACKEND=local
# export LOCAL_STORAGE_BASE_DIR=/data
uv run --frozen --package backend uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload uv run --frozen --package backend uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload
# Schedule Executor (terminal 2) # Schedule Executor (terminal 2)
@@ -246,7 +270,7 @@ PYTHONPATH="backend/src:common/src" uv run --frozen --package backend python -c
from backend.main import app from backend.main import app
from common.config import settings from common.config import settings
print('backend:', len(app.routes), 'routes') print('backend:', len(app.routes), 'routes')
print('settings ok:', settings.rustfs_endpoint) print('settings ok:', settings.s3_endpoint)
" "
``` ```
@@ -284,36 +308,42 @@ See "Adding a new env var" above.
uv run --frozen --package backend alembic upgrade head uv run --frozen --package backend alembic upgrade head
``` ```
### Wire a new RustFS bucket ### Wire a new storage bucket
The current three buckets are wired in `backend/storage_api.py:resolve_bucket`: The current 4 buckets are wired in `backend/storage_api.py:resolve_bucket`:
```python ```python
BUCKET_FOR_USAGE: dict[str, str] = { BUCKET_FOR_USAGE: dict[str, str] = {
"working_copy": settings.rustfs_workspace_bucket, "working_copy": settings.s3_workspace_bucket,
"public_script": settings.rustfs_workspace_bucket, "public_script": settings.s3_workspace_bucket,
"data_resource": settings.rustfs_workspace_bucket, "data_resource": settings.s3_workspace_bucket,
"snapshot": settings.rustfs_workspace_bucket, "snapshot": settings.s3_workspace_bucket,
"version_artifact": settings.rustfs_version_bucket, "version_artifact": settings.s3_version_bucket,
"run_log": settings.rustfs_run_log_bucket, "run_log": settings.s3_run_log_bucket,
"run_result": settings.rustfs_run_log_bucket, "run_result": settings.s3_run_log_bucket,
} }
``` ```
To add a fourth bucket: The constant `PURPOSE_BUCKETS = ("workspace", "version", "run_log", "trash")`
in `common.storage.factory` enumerates the four backends built in the
backend lifespan. To add a fifth bucket:
1. Add the env var to `Settings`: 1. Add the env var to `Settings` (s3 mode only):
```python ```python
rustfs_<feature>_bucket: str = Field(default="<feature>", description="...") s3_<feature>_bucket: str = Field(default="<feature>", description="...")
``` ```
2. Add to `.env.example` with a one-line comment. 2. Add to `.env.example` with a one-line comment.
3. Extend the `Literal` in `common/storage/schemas.py` (in 3. Append `"<feature>"` to the `PURPOSE_BUCKETS` tuple in
`ServerObjectRequest.usage_type`, `CreateUploadRequest.usage_type`, `common/storage/factory.py`. `build_storage_config("<feature>")`
`CompleteUploadRequest.usage_type`) to include the new value. will then automatically read `settings.s3_<feature>_bucket` (s3
4. Add an entry in `BUCKET_FOR_USAGE` mapping the new `usage_type` to mode) or use `<local_storage_base_dir>/<feature>` (local mode).
4. Extend the `Literal` in `common/storage/schemas.py` (in
`CreateUploadRequest.usage_type`, `ServerObjectRequest.usage_type`)
to include the new value.
5. Add an entry in `BUCKET_FOR_USAGE` mapping the new `usage_type` to
the new bucket env var. the new bucket env var.
5. Add the bucket to the `ensure_bucket` loop in 6. Pre-create the bucket (s3 mode) or subdirectory (local mode) in the
`backend/main.py` lifespan. deployment. The backend no longer auto-creates buckets.
A workspace's `artifact_bucket` column (when non-null) overrides the A workspace's `artifact_bucket` column (when non-null) overrides the
default for that workspace, regardless of `usage_type`. default for that workspace, regardless of `usage_type`.
@@ -354,7 +384,7 @@ contexts. Add `greenlet>=3.0.0` to `common/pyproject.toml` and
Either MySQL isn't running, or the network namespace doesn't allow Either MySQL isn't running, or the network namespace doesn't allow
`mysql:3306` resolution. Inside the Docker network, services reach `mysql:3306` resolution. Inside the Docker network, services reach
each other by service name (`mysql`, `backend`, `runtime`, each other by service name (`mysql`, `backend`, `runtime`,
`schedule`, `rustfs`). `schedule`, `s3`).
### Jupyter routing 401s ### Jupyter routing 401s
README.md
+47 -32
View File
@@ -5,8 +5,9 @@ an interactive workspace, a DAG scheduler, an object-storage-backed artifact
store, and per-workspace runtime isolation — all behind a single Nginx store, and per-workspace runtime isolation — all behind a single Nginx
gateway. gateway.
> Stack: React Router SPA · FastAPI · APScheduler · MySQL · RustFS (S3) > Stack: React Router SPA · FastAPI · APScheduler · MySQL · S3-compatible
> · shared Jupyter · FUSE mount via rclone > storage (or local filesystem via `STORAGE_BACKEND=local`) · shared
> Jupyter · FUSE mount via rclone (s3 mode only)
> Single ingress (Nginx :80); all other services are Docker-internal. > Single ingress (Nginx :80); all other services are Docker-internal.
## What it does ## What it does
@@ -15,11 +16,11 @@ gateway.
|---|---| |---|---|
| Workspace-scoped notebook editing with row-level lock | `backend/jupyter.py` + `scripts.is_locked` | | Workspace-scoped notebook editing with row-level lock | `backend/jupyter.py` + `scripts.is_locked` |
| Authenticated Jupyter routing (browser never sees the runtime token) | `nginx/default.conf` + `auth_request` + `backend/jupyter.py` | | Authenticated Jupyter routing (browser never sees the runtime token) | `nginx/default.conf` + `auth_request` + `backend/jupyter.py` |
| Object storage for notebooks / scripts / versions / run logs (RustFS, S3 API) | `common/storage/` + `backend/scripts.py` | | Object storage for notebooks / scripts / versions / run logs (s3 / local toggle) | `common/storage/` + `backend/scripts.py` |
| DAG-style scheduling: nodes, edges, cron, manual trigger, retries, snapshots | `backend/schedules.py` + `backend/schedule_runs.py` + `schedule/` (5 modules) | | DAG-style scheduling: nodes, edges, cron, manual trigger, retries, snapshots | `backend/schedules.py` + `backend/schedule_runs.py` + `schedule/` (5 modules) |
| DAG execution via MySQL Outbox (no Redis, no in-process queues) | `schedule/orchestrator.py` + `schedule/worker.py` | | DAG execution via MySQL Outbox (no Redis, no in-process queues) | `schedule/orchestrator.py` + `schedule/worker.py` |
| Per-workspace Jupyter sub-process pool with asyncio locks | `runtime/process.py` | | Per-workspace Jupyter sub-process pool with asyncio locks | `runtime/process.py` |
| rclone FUSE mount of the workspace bucket into the runtime | `runtime/mount.py` | | rclone FUSE mount of the workspace bucket into the runtime (s3 mode) | `runtime/mount.py` |
| MySQL-only persistence (26 tables, soft-delete, no foreign keys) | `common/db/models/` | | MySQL-only persistence (26 tables, soft-delete, no foreign keys) | `common/db/models/` |
## Architecture at a glance ## Architecture at a glance
@@ -42,13 +43,13 @@ gateway.
│ (storage) ├──────────────►│ - subprocess pool │ │ (storage) ├──────────────►│ - subprocess pool │
│ - DAG CRUD │ │ (per workspace) │ │ - DAG CRUD │ │ (per workspace) │
│ - script CRUD │ └──────────┬───────────┘ │ - script CRUD │ └──────────┬───────────┘
│ - auth_request │ │ FUSE │ - auth_request │ │ FUSE / shared vol
│ - /api/v1/... │ ▼ │ - /api/v1/... │ ▼
└────┬──────┬──────┘ ┌──────────────────────┐ └────┬──────┬──────┘ ┌──────────────────────┐
│ │ │ RustFS (S3) │ │ │ │ Object storage │
│ └──────── HTTP ───────►│ bucket: workspaces │ │ └──────── HTTP ───────►│ (s3: S3 service / │
▼ │ bucket: versions │ ▼ │ local: shared vol) │
┌────────────┐ │ bucket: run-logs │ ┌────────────┐ │ 4 buckets per usage │
│ MySQL │◄───────── poll ─────│ │ │ MySQL │◄───────── poll ─────│ │
│ - 26 tbls │ └──────────────────────┘ │ - 26 tbls │ └──────────────────────┘
│ - outbox │ │ - outbox │
@@ -65,6 +66,11 @@ gateway.
└───────────────────────────────┘ └───────────────────────────────┘
``` ```
Object storage is selectable via `STORAGE_BACKEND` (s3 | local). In s3 mode
the 4 purpose-named buckets (`workspaces` / `versions` / `run-logs` / `trash`)
are S3 buckets; in local mode they're subdirectories of `LOCAL_STORAGE_BASE_DIR`,
shared via the `local-storage` Docker volume. See `DEVELOP.md` §Storage.
Detailed design lives in `ARCHITECTURE.md`. Implementation deviations and Detailed design lives in `ARCHITECTURE.md`. Implementation deviations and
recent refactors are recorded in `HANDOVER.md`. recent refactors are recorded in `HANDOVER.md`.
@@ -90,9 +96,9 @@ default.conf Nginx template (mounted, rendered at start)
| Service | Image | Exposed | Purpose | | Service | Image | Exposed | Purpose |
|---|---|---|---| |---|---|---|---|
| `web` | `nginx:alpine` | host `:8888` → `:80` | SPA, `/api/` reverse-proxy, `/jupyter/{ws}/` auth_request proxy, `/storage/` RustFS passthrough | | `web` | `nginx:alpine` | host `:8888` → `:80` | SPA, `/api/` reverse-proxy, `/jupyter/{ws}/` auth_request proxy, `/storage/` S3 passthrough (s3 mode only) |
| `backend` | `Dockerfile` | internal only | DAG CRUD, script CRUD, schedule triggers, `/api/v1/auth/jupyter`, `/internal/v1/*` storage control plane | | `backend` | `Dockerfile` | internal only | DAG CRUD, script CRUD, schedule triggers, `/api/v1/auth/jupyter`, `/internal/v1/*` storage control plane |
| `runtime` | `Dockerfile` | internal only | Per-workspace Jupyter sub-process pool, rclone FUSE mount of `workspaces` bucket | | `runtime` | `Dockerfile` | internal only | Per-workspace Jupyter sub-process pool, rclone FUSE mount of `workspaces` bucket (s3 mode) |
| `schedule` | `Dockerfile` | internal only | Cron tick + DAG execution via MySQL Outbox polling | | `schedule` | `Dockerfile` | internal only | Cron tick + DAG execution via MySQL Outbox polling |
The architecture **deliberately has only one host port** (the gateway); The architecture **deliberately has only one host port** (the gateway);
@@ -103,7 +109,7 @@ all other services are on the Docker internal network. This is enforced in
```bash ```bash
cp .env.example .env cp .env.example .env
# Edit .env — at minimum change MYSQL password and RUSTFS credentials. # Edit .env — at minimum change MYSQL password and (in s3 mode) S3 credentials.
# Static check # Static check
uv sync --all-packages uv sync --all-packages
@@ -130,7 +136,7 @@ docker compose logs -f schedule
docker compose logs -f runtime docker compose logs -f runtime
``` ```
### Tear down (keeps MySQL + RustFS volumes) ### Tear down (keeps MySQL + S3 / local-storage volumes)
```bash ```bash
docker compose down docker compose down
@@ -157,14 +163,28 @@ See `DEVELOP.md` for the full list of variables and their meanings.
## Storage layout ## Storage layout
Three purpose-named RustFS buckets. The mapping from `StorageObjects.usage_type` Four purpose-named buckets. The mapping from `StorageObjects.usage_type`
to bucket is decided in **one place** (`storage_api.py:resolve_bucket`): to bucket is decided in **one place** (`backend/storage_api.py:resolve_bucket`):
| `usage_type` | Bucket (env var) | Default name | | `usage_type` | Bucket (env var) | Default name |
|---|---|---| |---|---|---|
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `RUSTFS_WORKSPACE_BUCKET` | `workspaces` | | `working_copy`, `public_script`, `data_resource`, `snapshot` | `S3_WORKSPACE_BUCKET` | `workspaces` |
| `version_artifact` | `RUSTFS_VERSION_BUCKET` | `versions` | | `version_artifact` | `S3_VERSION_BUCKET` | `versions` |
| `run_log`, `run_result` | `RUSTFS_RUN_LOG_BUCKET` | `run-logs` | | `run_log`, `run_result` | `S3_RUN_LOG_BUCKET` | `run-logs` |
| (soft-delete target) | `S3_TRASH_BUCKET` | `trash` |
In `STORAGE_BACKEND=s3` mode these are 4 separate S3 buckets. In
`STORAGE_BACKEND=local` mode they are 4 subdirectories under
`LOCAL_STORAGE_BASE_DIR` (default `/data`), so the layout above
becomes:
```
/data/
├── workspace/ # S3_WORKSPACE_BUCKET
├── version/ # S3_VERSION_BUCKET
├── run_log/ # S3_RUN_LOG_BUCKET
└── trash/ # S3_TRASH_BUCKET
```
A workspace's `artifact_bucket` column (when non-null) overrides the A workspace's `artifact_bucket` column (when non-null) overrides the
default for that workspace, regardless of `usage_type` — useful for default for that workspace, regardless of `usage_type` — useful for
@@ -174,24 +194,19 @@ The object key is a flat two-level path — `workspace_id` and a server-
issued `ulid` for the object: issued `ulid` for the object:
``` ```
s3://workspaces/ <workspace_bucket>/<workspace_id>/<ulid>{.<ext>}
└── <workspace_id>/
├── <ulid-1> # working_copy / data_resource / snapshot / ...
├── <ulid-2>
└── ...
s3://versions/<workspace_id>/<ulid> # immutable script versions
s3://run-logs/<workspace_id>/<ulid> # node run logs and results
``` ```
The file name, extension, content type, and logical path live in the The file name, extension, content type, and logical path live in the
`StorageObjects` and `Scripts` rows, not in the S3 key, so the bucket `StorageObjects` and `Scripts` rows, not in the object key, so the
can be re-organised without a database rewrite. storage can be re-organised without a database rewrite.
Backend code never writes to the container's local filesystem. Schedule Backend code never writes to the container's local filesystem (except
Executor stages node artifacts in `tempfile.TemporaryDirectory()` (auto- in `STORAGE_BACKEND=local` mode, where the shared `local-storage` volume
cleaned). Only the `runtime` container keeps a host volume — it is required is the canonical store). Schedule Executor stages node artifacts in
by the rclone FUSE mount. `tempfile.TemporaryDirectory()` (auto-cleaned). Only the `runtime`
container keeps a host volume — required by the rclone FUSE mount in
s3 mode, and a no-op pass-through in local mode.
## Documentation ## Documentation
REFACTOR_NOTES.md
+7 -2
View File
@@ -28,8 +28,8 @@
```text ```text
mysql mysql
rustfs s3 # 外部 S3-兼容服务（MinIO/RustFS/SeaweedFS/...），由运维在 compose 外启动
jupyter jupyter # 注释保留；当前实现未在 compose 启此独立容器
migrate（一次性） migrate（一次性）
backend backend
runtime runtime
@@ -37,6 +37,11 @@ schedule
gateway gateway
``` ```
> 补充说明：`STORAGE_BACKEND=local` 模式下不需要外部 S3 服务，backend
> 与 runtime 共享 docker 卷 `local-storage`，挂载到
> `LOCAL_STORAGE_BASE_DIR`（默认 `/data/storage`）；runtime 也跳过
> rclone FUSE 挂载（见 `runtime/src/runtime/mount.py`）。
## 已执行检查 ## 已执行检查
- Python 全项目 `compileall` 通过； - Python 全项目 `compileall` 通过；
backend/README.md
+4 -1
View File
@@ -1,5 +1,8 @@
# Backend # Backend
统一 FastAPI 管理服务。包含用户、Workspace、脚本、稳定版本、调度定义、 统一 FastAPI 管理服务。包含用户、Workspace、脚本、稳定版本、调度定义、
立即运行以及 RustFS 对象接口。原 `platform_api` 与 `storage_api` 已在此 立即运行以及 S3 对象接口。原 `platform_api` 与 `storage_api` 已在此
模块合并，外部 REST 契约保持不变。 模块合并，外部 REST 契约保持不变。
底层走的是 `common.storage.AsyncStorageBackend` 抽象，按
`settings.storage_backend` 切换 s3 / local 两种实现。
migrations/data/README.md
+3 -2
View File
@@ -65,5 +65,6 @@ python -m migrations.data.migrate_legacy_workspaces --source "<path>/server.py"
## 第 11、12 小步数据处理决定 ## 第 11、12 小步数据处理决定
根据实施确认，第 11、12 小步不迁移旧版资源、脚本或稳定版本数据。新实现直接 根据实施确认，第 11、12 小步不迁移旧版资源、脚本或稳定版本数据。新实现直接
使用 MySQL、Workspace 文件目录和 RustFS，从空的 `data_resources`、`scripts` 使用 MySQL、Workspace 文件目录和对象存储（s3 模式连 S3-兼容服务，local 模式
及 `versions` 表开始运行。功能验收产生的临时对象和数据库记录均已清理。 走 `/data/storage` 共享卷），从空的 `data_resources`、`scripts` 及
`versions` 表开始运行。功能验收产生的临时对象和数据库记录均已清理。