docs: align with new storage architecture (s3 + local + server-proxied PUT)
All operator- and developer-facing docs updated to reflect:
- The unified AsyncStorageBackend abstraction (s3 + local backends).
- The STORAGE_BACKEND toggle ("s3" default, "local" for dev /
single-node / air-gapped deployments).
- The 4-purpose-bucket layout (workspace / version / run_log / trash)
in both modes — 4 separate S3 buckets in s3 mode, 4 subdirectories
of LOCAL_STORAGE_BASE_DIR in local mode.
- The S3_* env var naming (was RUSTFS_*).
- The server-proxied upload flow (was browser-direct presign-PUT):
POST /internal/v1/uploads → PUT /internal/v1/uploads/{id} with
raw bytes → server calls backend.put().
- The factory helpers workspaces_root() (runtime's view of the
workspace bucket on disk) and rclone_remote_spec() (s3-mode mount
source).
- The "two settings describing the same thing" cleanup: the deleted
settings.workspace_root, settings.workspaces_root, and
settings.remote_bucket fields.
Files touched:
- API.md (§5 data-resource upload flow, §9 storage control plane,
§10 readiness example)
- ARCHITECTURE.md (storage layer diagram)
- CLAUDE.md (architecture description + volume-preservation note)
- DEVELOP.md (settings list, Storage section, "Wire a new bucket"
how-to, dev-export example, troubleshooting network hint)
- README.md (architecture diagram, container table, quick-start
credentials note, tear-down note, Storage layout section)
- REFACTOR_NOTES.md (final container list with s3 explanation)
- backend/README.md (storage backend description)
- migrations/data/README.md (step 11/12 record mentioning object
storage)
A handful of historical "RustFS" mentions are intentionally retained
where they name a specific S3-compatible product (e.g. as an example
in REFACTOR_NOTES.md's container list) or document the pre-2026
abstraction name (DEVELOP.md Storage section).
This commit is contained in:
@@ -92,7 +92,7 @@
|
||||
|
||||
### 3.2 `POST /api/v1/workspace-directories`
|
||||
|
||||
创建一个**逻辑目录**(RustFS 上是隐式前缀,无需落对象)。
|
||||
创建一个**逻辑目录**(对象存储上是隐式前缀,无需落对象)。
|
||||
|
||||
- **请求体**:
|
||||
```json
|
||||
@@ -163,7 +163,7 @@
|
||||
|
||||
### 3.7 `POST /api/v1/scripts/upload?file_name=...&parent_path=...&visibility=...`
|
||||
|
||||
multipart/binary 形式上传大文件(走 presigned PUT)。
|
||||
multipart/binary 形式上传大文件(走 server-proxied PUT,详见 §九)。
|
||||
|
||||
- **查询参数**: `file_name`(必填)、`parent_path`、`visibility`
|
||||
- **请求体**: 原始文件字节(`Content-Type` 必须与脚本类型匹配)
|
||||
@@ -203,7 +203,7 @@ multipart/binary 形式上传大文件(走 presigned PUT)。
|
||||
|
||||
### 3.11 `POST /api/v1/scripts/{script_id}/versions`
|
||||
|
||||
发布一个**稳定版本**(immutable,绑定到 `RUSTFS_VERSION_BUCKET`)。门禁同 §3.8。
|
||||
发布一个**稳定版本**(immutable,绑定到 `S3_VERSION_BUCKET`)。门禁同 §3.8。
|
||||
|
||||
- **请求体**:
|
||||
```json
|
||||
@@ -243,11 +243,11 @@ multipart/binary 形式上传大文件(走 presigned PUT)。
|
||||
|
||||
### 3.14 `DELETE /api/v1/versions/{versions_id}`
|
||||
|
||||
从调度候选中**隐藏**此版本(不删除 RustFS 对象)。门禁:**owner 校验基于所属 `Scripts` 的 owner**——即"按整本 script 判定",而非"按版本发布者判定"。
|
||||
从调度候选中**隐藏**此版本(不删除对象存储里的对象)。门禁:**owner 校验基于所属 `Scripts` 的 owner**——即"按整本 script 判定",而非"按版本发布者判定"。
|
||||
|
||||
### 3.15 `POST /api/v1/versions/{versions_id}/download-url`
|
||||
|
||||
生成 RustFS 的 presigned download URL。
|
||||
生成对象存储的 presigned download URL(走 S3 兼容协议,local 模式下该 endpoint 在 s3 模式才生效)。
|
||||
|
||||
- **请求体**:
|
||||
```json
|
||||
@@ -399,8 +399,8 @@ queued ──→ running ──┬─→ succeeded
|
||||
|
||||
| 方法 | 路径 | 说明 |
|
||||
|---|---|---|
|
||||
| `POST` | `/api/v1/data-resources/uploads` | 创建上传会话,返回 presigned PUT URL + `upload_id` |
|
||||
| `POST` | `/api/v1/data-resources/uploads/{upload_id}/complete` | 完成上传(写 `StorageObjects` 行) |
|
||||
| `POST` | `/api/v1/data-resources/uploads` | 创建上传会话,返回 `upload_id` + `upload_path` |
|
||||
| `PUT` | `/api/v1/data-resources/uploads/{upload_id}` | 上传字节(请求体即文件内容) |
|
||||
| `GET` | `/api/v1/data-resources` | 列表(workspace 范围) |
|
||||
| `GET` | `/api/v1/data-resources/{id}` | 详情 |
|
||||
| `POST` | `/api/v1/data-resources/{id}/download-url` | 生成 presigned GET URL |
|
||||
@@ -420,15 +420,18 @@ queued ──→ running ──┬─→ succeeded
|
||||
|
||||
**完整上传流程(前端应实现的模式)**:
|
||||
```
|
||||
1. POST /uploads → {upload_id, presigned_url}
|
||||
2. PUT presigned_url with file bytes
|
||||
3. POST /uploads/{upload_id}/complete
|
||||
4. 服务器完成 → 200 {data: StorageObjectPayload}
|
||||
1. POST /uploads → {upload_id, upload_path, expires_at}
|
||||
2. PUT upload_path with raw file bytes (Content-Type: application/octet-stream)
|
||||
3. 服务器端走 backend.put() → 200 {data: StorageObjectPayload}
|
||||
```
|
||||
|
||||
字节经过 backend 进程(server-proxied upload),最大 100 MiB,由 backend
|
||||
直接调 `AsyncStorageBackend.put()` 写入存储(不再走 presigned PUT 直传)。
|
||||
前端无需关心 S3 协议或签名。
|
||||
|
||||
**小对象(<100 KiB)捷径**:直接调 `create_server_object` 把字节 base64 放进
|
||||
`content_base64` 字段(走 `StorageClient.create_server_object`),前端无需
|
||||
走 presigned PUT 三步。
|
||||
`content_base64` 字段(JSON 体里走),内部走同一条 `AsyncStorageBackend.put`
|
||||
路径,前端无需分两步。
|
||||
|
||||
---
|
||||
|
||||
@@ -672,9 +675,13 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断
|
||||
## 九、对象存储控制面
|
||||
|
||||
> 路径前缀 `/internal/v1/...`,**前端不要直接调用**。这是 backend 内部
|
||||
> 异步消息处理(Schedule worker)用的 RPC 端点,经 `StorageClient` HTTP
|
||||
> 客户端访问。Backend 通过 ASGI `auth_request_set` 路由转发,外部无法
|
||||
> 访问。
|
||||
> 异步消息处理(Schedule worker)用的 RPC 端点,经 in-process ASGI 直接
|
||||
> 转发(`storage_app` 路由被 `app.include_router` 进同一个 backend 进程),
|
||||
> 外部无法访问。
|
||||
|
||||
底层抽象:`common.storage.AsyncStorageBackend`(`put/get/delete/exists/stat/
|
||||
list/get_url/copy`)。按 `settings.storage_backend` 选实现:`"s3"` 走
|
||||
S3-兼容服务,`"local"` 走 `LOCAL_STORAGE_BASE_DIR` 子目录。
|
||||
|
||||
### 9.1 `POST /internal/v1/uploads`
|
||||
|
||||
@@ -689,23 +696,36 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断
|
||||
"content_type": "text/x-python",
|
||||
"expected_size_bytes": 1024,
|
||||
"expected_hash": "<optional sha256 hex>",
|
||||
"idempotency_key": "..."
|
||||
"idempotency_key": "...",
|
||||
"visibility": "private",
|
||||
"is_immutable": false
|
||||
}
|
||||
```
|
||||
|
||||
返回 `{upload_id, bucket_name, object_key, presigned_url, expires_in_seconds}`。
|
||||
返回 `{upload_id, status, upload_path, expires_at}`。`upload_path` 是
|
||||
第 9.2 步要 PUT 的端点(本进程内 `/internal/v1/uploads/{upload_id}`)。
|
||||
|
||||
### 9.2 `POST /internal/v1/uploads/{upload_id}/complete`
|
||||
### 9.2 `PUT /internal/v1/uploads/{upload_id}`
|
||||
|
||||
完成上传。从 RustFS 读 HEAD → 校验 hash → 写 `StorageObjects` 行。
|
||||
完成上传(server-proxied PUT)。**请求体即原始字节**,`Content-Type:
|
||||
application/octet-stream`。后端 `await request.body()` 读字节 → 校验
|
||||
size + sha256 → 调 `await backend.put(key, bytes, content_type=...,
|
||||
metadata={"sha256": ...})` → 写 `StorageObjects` 行 → 标 session 为
|
||||
completed。最大 100 MiB。
|
||||
|
||||
> 历史:旧版本这一步是 `POST /uploads/{id}/complete`,靠
|
||||
> presigned-PUT + head() 验证。已被 server-proxied PUT 取代。
|
||||
|
||||
### 9.3 `POST /internal/v1/uploads/{upload_id}/abort`
|
||||
|
||||
主动放弃。释放 `UploadSessions` 行,对象不入库。
|
||||
主动放弃。删除可能已经写了一半的对象字节,释放 `UploadSessions` 行。
|
||||
|
||||
### 9.4 `POST /internal/v1/objects`
|
||||
|
||||
**单步创建**(不走 presigned PUT,字节随请求体直传)。适用 < 100 KiB 对象。
|
||||
**单步创建**(不走两步上传,字节 base64 进 JSON 体)。适用 < 100 KiB
|
||||
对象(避免 multipart/大请求体的前端复杂度)。内部直接调
|
||||
`AsyncStorageBackend.put(key, content, content_type=...,
|
||||
metadata={"sha256": ...})`。
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -717,28 +737,36 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断
|
||||
"content_base64": "PHN0ZXAtY29udGVudD4=",
|
||||
"visibility": "private",
|
||||
"is_immutable": false,
|
||||
"idempotency_key": "..."
|
||||
"idempotency_key": "...",
|
||||
"relative_path": null
|
||||
}
|
||||
```
|
||||
|
||||
### 9.5 `POST /internal/v1/objects/{storage_object_id}/download-url`
|
||||
|
||||
生成 presigned GET URL。
|
||||
生成 presigned GET URL(s3 模式:`AsyncStorageBackend.get_url()`;
|
||||
local 模式:目前抛 `NotImplementedError`,需要 native FS serving 配合
|
||||
nginx 静态 location)。
|
||||
|
||||
### 9.6 `DELETE /internal/v1/objects/{storage_object_id}`
|
||||
|
||||
软删。`is_immutable == 1` 的对象拒绝删除。
|
||||
软删。`is_immutable == 1` 的对象拒绝删除。`move_to_trash` 走跨后端
|
||||
`copy + delete`(同一进程内的两个 backend 实例)。
|
||||
|
||||
### 9.7 usage_type → 桶路由(自动)
|
||||
|
||||
| usage_type | 实际桶(env var) | 默认桶名 |
|
||||
|---|---|---|
|
||||
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `RUSTFS_WORKSPACE_BUCKET` | `workspaces` |
|
||||
| `version_artifact` | `RUSTFS_VERSION_BUCKET` | `versions` |
|
||||
| `run_log`, `run_result` | `RUSTFS_RUN_LOG_BUCKET` | `run-logs` |
|
||||
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `S3_WORKSPACE_BUCKET` | `workspaces` |
|
||||
| `version_artifact` | `S3_VERSION_BUCKET` | `versions` |
|
||||
| `run_log`, `run_result` | `S3_RUN_LOG_BUCKET` | `run-logs` |
|
||||
| (soft-delete target) | `S3_TRASH_BUCKET` | `trash` |
|
||||
|
||||
若 `Workspaces.artifact_bucket` 非空,优先用 per-workspace 桶(覆盖 usage_type 路由)。
|
||||
|
||||
桶在 `STORAGE_BACKEND=s3` 时是 4 个独立 S3 bucket,在
|
||||
`STORAGE_BACKEND=local` 时是 `LOCAL_STORAGE_BASE_DIR` 下的 4 个子目录。
|
||||
|
||||
---
|
||||
|
||||
## 十、健康检查
|
||||
@@ -751,7 +779,8 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断
|
||||
| `GET` | `/api/v1/health` | 公开健康检查(前端可访问) |
|
||||
|
||||
`/health/ready` 支持 `READINESS_TARGETS` 环境变量,逗号分隔的 `host:port`
|
||||
列表,例如 `mysql:3306,rustfs:9000`,全部 TCP 通则返回 200,否则 503。
|
||||
列表,例如 `mysql:3306,s3:9000`,全部 TCP 通则返回 200,否则 503。
|
||||
`STORAGE_BACKEND=local` 模式下不需要 S3 host,列表里删掉即可。
|
||||
|
||||
---
|
||||
|
||||
|
||||
+1
-1
@@ -24,7 +24,7 @@ Schedule Executor(APScheduler)
|
||||
|-- MySQL APSchedulerJobStore
|
||||
|-- MySQL Outbox 轮询兜底
|
||||
|-- DAG 节点执行与重试
|
||||
|-- RustFS 日志/结果
|
||||
|-- S3 日志/结果
|
||||
+-- Backend 内部 Storage API
|
||||
```
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
## Current architecture
|
||||
|
||||
- `frontend`: React Router SPA. Production files are built in `nginx/Dockerfile`.
|
||||
- `backend`: public FastAPI API and internal RustFS storage API in one process.
|
||||
- `backend`: public FastAPI API and internal S3 storage API in one process.
|
||||
- `runtime`: shared Jupyter lifecycle, MySQL edit leases and short-lived in-memory access tickets.
|
||||
- `schedule`: APScheduler, MySQL JobStore, MySQL Outbox polling and DAG execution.
|
||||
- `common`: SQLAlchemy models, database/session helpers, IDs and object-store helpers.
|
||||
@@ -51,7 +51,7 @@ pnpm build
|
||||
- Cron jobs are persisted by APScheduler in MySQL table `apscheduler_jobs`.
|
||||
- Runtime must stay single-replica while file leases and Jupyter tickets use the simplified implementation.
|
||||
- Never expose the internal Jupyter token to the browser.
|
||||
- Never delete Docker volumes when preserving MySQL or RustFS data is required.
|
||||
- Never delete Docker volumes when preserving MySQL or storage data is required.
|
||||
|
||||
## Main entrypoints
|
||||
|
||||
|
||||
+73
-43
@@ -12,7 +12,7 @@ common/ Pure-Python shared library
|
||||
db/ SQLAlchemy 2.0 async engine, session_scope, Base
|
||||
db/models/ 26 tables in 9 domain files (zero FK, zero relationship)
|
||||
scheduler/ build_sqlalchemy_jobstore (delayed import)
|
||||
storage/ RustFSObjectStore + StorageClient + Pydantic schemas
|
||||
storage/ AsyncStorageBackend abstraction (s3 + local impls) + Pydantic schemas
|
||||
eventing.py add_outbox_event / utcnow / event_time
|
||||
service_app.py /health/ready TCP probe, /api/v1/health
|
||||
schemas.py StrictModel base
|
||||
@@ -21,14 +21,14 @@ common/ Pure-Python shared library
|
||||
backend/ Public FastAPI service + internal /internal/v1/* sub-app
|
||||
main.py lifespan + route registration
|
||||
jupyter.py /api/v1/auth/jupyter — the ONLY auth entry
|
||||
scripts.py CRUD for scripts/notebooks (workspace_fs=rustfs)
|
||||
scripts.py CRUD for scripts/notebooks (object storage via AsyncStorageBackend)
|
||||
schedules.py DAG CRUD: schedules, nodes, edges
|
||||
schedule_runs.py Trigger / list / get runs
|
||||
schedule_schemas.py Pydantic request/response models
|
||||
admin.py Admin endpoints
|
||||
resources.py Misc data resources
|
||||
storage_api.py /internal/v1/* (sub-app merged into main)
|
||||
storage_client.py HTTP client for the storage sub-app
|
||||
storage_client.py Stub (HTTP client removed post-migration; rewrite pending)
|
||||
schedule_client.py Placeholder module (was the HTTP-push executor client)
|
||||
runtime_client.py Self-contained httpx wrapper for the runtime
|
||||
jupyter.py auth_request handler
|
||||
@@ -41,7 +41,7 @@ schedule/ Schedule Executor (DAG worker)
|
||||
worker.py NodeExecutor (notebook / python execution)
|
||||
service.py SchedulerService facade (composes the three)
|
||||
main.py Lifespan + FastAPI app
|
||||
storage_client.py SchedulerStorageClient (subclass of common StorageClient)
|
||||
storage_client.py Stub (SchedulerStorageClient rewrite pending — use AsyncStorageBackend directly)
|
||||
execution.py execute_artifact (notebook + python paths)
|
||||
notebook_runner.py Subprocess entry point (nbclient)
|
||||
|
||||
@@ -68,16 +68,17 @@ All env vars go through one place: `common/src/common/config.py`.
|
||||
from common.config import settings
|
||||
|
||||
settings.database_url # str
|
||||
settings.rustfs_endpoint # str (full URL, e.g. "http://rustfs:9000")
|
||||
settings.rustfs_access_key # str
|
||||
settings.rustfs_secret_key # str
|
||||
settings.rustfs_workspace_bucket
|
||||
settings.rustfs_version_bucket
|
||||
settings.rustfs_run_log_bucket
|
||||
settings.jwt_secret # HS256 secret for the auth_request handler
|
||||
settings.workspace_root # schedule subprocess cwd; backend ignores
|
||||
settings.workspaces_root # runtime rclone FUSE mount point
|
||||
settings.remote_bucket # rclone remote spec (e.g. "rustfs:workspaces")
|
||||
settings.storage_backend # str: "s3" (default) or "local"
|
||||
settings.local_storage_base_dir # str: root dir for storage data (default "/data"); see "Storage" below for per-mode derivation
|
||||
settings.s3_endpoint # str (full URL, e.g. "http://s3:9000"; s3 mode only)
|
||||
settings.s3_access_key # str (s3 mode only)
|
||||
settings.s3_secret_key # str (s3 mode only)
|
||||
settings.s3_workspace_bucket # str (s3 mode only)
|
||||
settings.s3_version_bucket # str (s3 mode only)
|
||||
settings.s3_run_log_bucket # str (s3 mode only)
|
||||
settings.s3_trash_bucket # str (s3 mode only)
|
||||
settings.s3_trash_retention_days # int (s3 mode only)
|
||||
settings.jwt_secret # HS256 secret for the auth_request handler
|
||||
settings.backend_api_url # schedule → backend HTTP base
|
||||
settings.runtime_api_url # backend → runtime HTTP base
|
||||
settings.public_base_url # runtime public base URL
|
||||
@@ -125,12 +126,31 @@ grep -rnE 'os\.(environ\[?["\x27][A-Z_]+|getenv\(["\x27][A-Z_]+)' --include="*.p
|
||||
|
||||
### Storage
|
||||
|
||||
- All object bytes go to **RustFS** via boto3.
|
||||
- Use `StorageClient` (HTTP) or `RustFSObjectStore` (direct) — never
|
||||
the local filesystem.
|
||||
- `bucket_name` is one of: `RUSTFS_WORKSPACE_BUCKET` (default
|
||||
`workspaces`), `RUSTFS_VERSION_BUCKET` (default `versions`,
|
||||
reserved), `RUSTFS_RUN_LOG_BUCKET` (default `run-logs`, reserved).
|
||||
- All object bytes go through `common.storage.AsyncStorageBackend`,
|
||||
created by `create_storage(config)` from `common.storage.factory`.
|
||||
- Two backends are registered: `local` (filesystem, local mode) and
|
||||
`s3` (S3-compatible service, s3 mode). Selection is per-deployment
|
||||
via `settings.storage_backend` (`"s3"` default, `"local"` for
|
||||
dev / single-node / air-gapped).
|
||||
- The factory helper `build_storage_config(bucket_name)` returns the
|
||||
right `create_storage` kwargs for each of the 4 purpose buckets
|
||||
(`workspace`, `version`, `run_log`, `trash`). Use it in lifespan code;
|
||||
route handlers don't see the difference.
|
||||
- Bucket resolution from `usage_type` is in **one place**
|
||||
(`backend/storage_api.py:resolve_bucket`); route handlers only know
|
||||
about `app.state.object_stores[bucket_name]`.
|
||||
- The runtime's view of the workspace bucket on disk is exposed by
|
||||
`common.storage.workspaces_root()`:
|
||||
- `s3` mode: `${settings.local_storage_base_dir}/workspaces`
|
||||
(default `/data/workspaces`, the rclone FUSE mount target).
|
||||
- `local` mode: `${settings.local_storage_base_dir}/workspace`
|
||||
(default `/data/workspace`, a subdir of the shared local-storage
|
||||
volume).
|
||||
`settings.local_storage_base_dir` is the **only** path setting; the
|
||||
helper handles the per-mode suffix. Don't read `settings.workspaces_root`
|
||||
or any other path setting directly in runtime code — use this helper.
|
||||
- The pre-2026 abstraction (`RustFSObjectStore` / `common.storage.client`
|
||||
/ `StorageClient` HTTP wrapper) is gone. Don't reintroduce it.
|
||||
|
||||
### Auth
|
||||
|
||||
@@ -202,9 +222,13 @@ cd frontend && pnpm install && cd ..
|
||||
```bash
|
||||
# Backend (terminal 1)
|
||||
export DATABASE_URL="mysql+asyncmy://model_platform:model_platform@127.0.0.1:3306/model_platform?charset=utf8mb4"
|
||||
export RUSTFS_ACCESS_KEY=modelplatform
|
||||
export RUSTFS_SECRET_KEY=modelplatformsecret
|
||||
export RUSTFS_ENDPOINT=http://127.0.0.1:9000
|
||||
export STORAGE_BACKEND=s3
|
||||
export S3_ACCESS_KEY=modelplatform
|
||||
export S3_SECRET_KEY=modelplatformsecret
|
||||
export S3_ENDPOINT=http://127.0.0.1:9000
|
||||
# Or for local mode:
|
||||
# export STORAGE_BACKEND=local
|
||||
# export LOCAL_STORAGE_BASE_DIR=/data
|
||||
uv run --frozen --package backend uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload
|
||||
|
||||
# Schedule Executor (terminal 2)
|
||||
@@ -246,7 +270,7 @@ PYTHONPATH="backend/src:common/src" uv run --frozen --package backend python -c
|
||||
from backend.main import app
|
||||
from common.config import settings
|
||||
print('backend:', len(app.routes), 'routes')
|
||||
print('settings ok:', settings.rustfs_endpoint)
|
||||
print('settings ok:', settings.s3_endpoint)
|
||||
"
|
||||
```
|
||||
|
||||
@@ -284,36 +308,42 @@ See "Adding a new env var" above.
|
||||
uv run --frozen --package backend alembic upgrade head
|
||||
```
|
||||
|
||||
### Wire a new RustFS bucket
|
||||
### Wire a new storage bucket
|
||||
|
||||
The current three buckets are wired in `backend/storage_api.py:resolve_bucket`:
|
||||
The current 4 buckets are wired in `backend/storage_api.py:resolve_bucket`:
|
||||
|
||||
```python
|
||||
BUCKET_FOR_USAGE: dict[str, str] = {
|
||||
"working_copy": settings.rustfs_workspace_bucket,
|
||||
"public_script": settings.rustfs_workspace_bucket,
|
||||
"data_resource": settings.rustfs_workspace_bucket,
|
||||
"snapshot": settings.rustfs_workspace_bucket,
|
||||
"version_artifact": settings.rustfs_version_bucket,
|
||||
"run_log": settings.rustfs_run_log_bucket,
|
||||
"run_result": settings.rustfs_run_log_bucket,
|
||||
"working_copy": settings.s3_workspace_bucket,
|
||||
"public_script": settings.s3_workspace_bucket,
|
||||
"data_resource": settings.s3_workspace_bucket,
|
||||
"snapshot": settings.s3_workspace_bucket,
|
||||
"version_artifact": settings.s3_version_bucket,
|
||||
"run_log": settings.s3_run_log_bucket,
|
||||
"run_result": settings.s3_run_log_bucket,
|
||||
}
|
||||
```
|
||||
|
||||
To add a fourth bucket:
|
||||
The constant `PURPOSE_BUCKETS = ("workspace", "version", "run_log", "trash")`
|
||||
in `common.storage.factory` enumerates the four backends built in the
|
||||
backend lifespan. To add a fifth bucket:
|
||||
|
||||
1. Add the env var to `Settings`:
|
||||
1. Add the env var to `Settings` (s3 mode only):
|
||||
```python
|
||||
rustfs_<feature>_bucket: str = Field(default="<feature>", description="...")
|
||||
s3_<feature>_bucket: str = Field(default="<feature>", description="...")
|
||||
```
|
||||
2. Add to `.env.example` with a one-line comment.
|
||||
3. Extend the `Literal` in `common/storage/schemas.py` (in
|
||||
`ServerObjectRequest.usage_type`, `CreateUploadRequest.usage_type`,
|
||||
`CompleteUploadRequest.usage_type`) to include the new value.
|
||||
4. Add an entry in `BUCKET_FOR_USAGE` mapping the new `usage_type` to
|
||||
3. Append `"<feature>"` to the `PURPOSE_BUCKETS` tuple in
|
||||
`common/storage/factory.py`. `build_storage_config("<feature>")`
|
||||
will then automatically read `settings.s3_<feature>_bucket` (s3
|
||||
mode) or use `<local_storage_base_dir>/<feature>` (local mode).
|
||||
4. Extend the `Literal` in `common/storage/schemas.py` (in
|
||||
`CreateUploadRequest.usage_type`, `ServerObjectRequest.usage_type`)
|
||||
to include the new value.
|
||||
5. Add an entry in `BUCKET_FOR_USAGE` mapping the new `usage_type` to
|
||||
the new bucket env var.
|
||||
5. Add the bucket to the `ensure_bucket` loop in
|
||||
`backend/main.py` lifespan.
|
||||
6. Pre-create the bucket (s3 mode) or subdirectory (local mode) in the
|
||||
deployment. The backend no longer auto-creates buckets.
|
||||
|
||||
A workspace's `artifact_bucket` column (when non-null) overrides the
|
||||
default for that workspace, regardless of `usage_type`.
|
||||
@@ -354,7 +384,7 @@ contexts. Add `greenlet>=3.0.0` to `common/pyproject.toml` and
|
||||
Either MySQL isn't running, or the network namespace doesn't allow
|
||||
`mysql:3306` resolution. Inside the Docker network, services reach
|
||||
each other by service name (`mysql`, `backend`, `runtime`,
|
||||
`schedule`, `rustfs`).
|
||||
`schedule`, `s3`).
|
||||
|
||||
### Jupyter routing 401s
|
||||
|
||||
|
||||
@@ -5,8 +5,9 @@ an interactive workspace, a DAG scheduler, an object-storage-backed artifact
|
||||
store, and per-workspace runtime isolation — all behind a single Nginx
|
||||
gateway.
|
||||
|
||||
> Stack: React Router SPA · FastAPI · APScheduler · MySQL · RustFS (S3)
|
||||
> · shared Jupyter · FUSE mount via rclone
|
||||
> Stack: React Router SPA · FastAPI · APScheduler · MySQL · S3-compatible
|
||||
> storage (or local filesystem via `STORAGE_BACKEND=local`) · shared
|
||||
> Jupyter · FUSE mount via rclone (s3 mode only)
|
||||
> Single ingress (Nginx :80); all other services are Docker-internal.
|
||||
|
||||
## What it does
|
||||
@@ -15,11 +16,11 @@ gateway.
|
||||
|---|---|
|
||||
| Workspace-scoped notebook editing with row-level lock | `backend/jupyter.py` + `scripts.is_locked` |
|
||||
| Authenticated Jupyter routing (browser never sees the runtime token) | `nginx/default.conf` + `auth_request` + `backend/jupyter.py` |
|
||||
| Object storage for notebooks / scripts / versions / run logs (RustFS, S3 API) | `common/storage/` + `backend/scripts.py` |
|
||||
| Object storage for notebooks / scripts / versions / run logs (s3 / local toggle) | `common/storage/` + `backend/scripts.py` |
|
||||
| DAG-style scheduling: nodes, edges, cron, manual trigger, retries, snapshots | `backend/schedules.py` + `backend/schedule_runs.py` + `schedule/` (5 modules) |
|
||||
| DAG execution via MySQL Outbox (no Redis, no in-process queues) | `schedule/orchestrator.py` + `schedule/worker.py` |
|
||||
| Per-workspace Jupyter sub-process pool with asyncio locks | `runtime/process.py` |
|
||||
| rclone FUSE mount of the workspace bucket into the runtime | `runtime/mount.py` |
|
||||
| rclone FUSE mount of the workspace bucket into the runtime (s3 mode) | `runtime/mount.py` |
|
||||
| MySQL-only persistence (26 tables, soft-delete, no foreign keys) | `common/db/models/` |
|
||||
|
||||
## Architecture at a glance
|
||||
@@ -42,13 +43,13 @@ gateway.
|
||||
│ (storage) ├──────────────►│ - subprocess pool │
|
||||
│ - DAG CRUD │ │ (per workspace) │
|
||||
│ - script CRUD │ └──────────┬───────────┘
|
||||
│ - auth_request │ │ FUSE
|
||||
│ - auth_request │ │ FUSE / shared vol
|
||||
│ - /api/v1/... │ ▼
|
||||
└────┬──────┬──────┘ ┌──────────────────────┐
|
||||
│ │ │ RustFS (S3) │
|
||||
│ └──────── HTTP ───────►│ bucket: workspaces │
|
||||
▼ │ bucket: versions │
|
||||
┌────────────┐ │ bucket: run-logs │
|
||||
│ │ │ Object storage │
|
||||
│ └──────── HTTP ───────►│ (s3: S3 service / │
|
||||
▼ │ local: shared vol) │
|
||||
┌────────────┐ │ 4 buckets per usage │
|
||||
│ MySQL │◄───────── poll ─────│ │
|
||||
│ - 26 tbls │ └──────────────────────┘
|
||||
│ - outbox │
|
||||
@@ -65,6 +66,11 @@ gateway.
|
||||
└───────────────────────────────┘
|
||||
```
|
||||
|
||||
Object storage is selectable via `STORAGE_BACKEND` (s3 | local). In s3 mode
|
||||
the 4 purpose-named buckets (`workspaces` / `versions` / `run-logs` / `trash`)
|
||||
are S3 buckets; in local mode they're subdirectories of `LOCAL_STORAGE_BASE_DIR`,
|
||||
shared via the `local-storage` Docker volume. See `DEVELOP.md` §Storage.
|
||||
|
||||
Detailed design lives in `ARCHITECTURE.md`. Implementation deviations and
|
||||
recent refactors are recorded in `HANDOVER.md`.
|
||||
|
||||
@@ -90,9 +96,9 @@ default.conf Nginx template (mounted, rendered at start)
|
||||
|
||||
| Service | Image | Exposed | Purpose |
|
||||
|---|---|---|---|
|
||||
| `web` | `nginx:alpine` | host `:8888` → `:80` | SPA, `/api/` reverse-proxy, `/jupyter/{ws}/` auth_request proxy, `/storage/` RustFS passthrough |
|
||||
| `web` | `nginx:alpine` | host `:8888` → `:80` | SPA, `/api/` reverse-proxy, `/jupyter/{ws}/` auth_request proxy, `/storage/` S3 passthrough (s3 mode only) |
|
||||
| `backend` | `Dockerfile` | internal only | DAG CRUD, script CRUD, schedule triggers, `/api/v1/auth/jupyter`, `/internal/v1/*` storage control plane |
|
||||
| `runtime` | `Dockerfile` | internal only | Per-workspace Jupyter sub-process pool, rclone FUSE mount of `workspaces` bucket |
|
||||
| `runtime` | `Dockerfile` | internal only | Per-workspace Jupyter sub-process pool, rclone FUSE mount of `workspaces` bucket (s3 mode) |
|
||||
| `schedule` | `Dockerfile` | internal only | Cron tick + DAG execution via MySQL Outbox polling |
|
||||
|
||||
The architecture **deliberately has only one host port** (the gateway);
|
||||
@@ -103,7 +109,7 @@ all other services are on the Docker internal network. This is enforced in
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Edit .env — at minimum change MYSQL password and RUSTFS credentials.
|
||||
# Edit .env — at minimum change MYSQL password and (in s3 mode) S3 credentials.
|
||||
|
||||
# Static check
|
||||
uv sync --all-packages
|
||||
@@ -130,7 +136,7 @@ docker compose logs -f schedule
|
||||
docker compose logs -f runtime
|
||||
```
|
||||
|
||||
### Tear down (keeps MySQL + RustFS volumes)
|
||||
### Tear down (keeps MySQL + S3 / local-storage volumes)
|
||||
|
||||
```bash
|
||||
docker compose down
|
||||
@@ -157,14 +163,28 @@ See `DEVELOP.md` for the full list of variables and their meanings.
|
||||
|
||||
## Storage layout
|
||||
|
||||
Three purpose-named RustFS buckets. The mapping from `StorageObjects.usage_type`
|
||||
to bucket is decided in **one place** (`storage_api.py:resolve_bucket`):
|
||||
Four purpose-named buckets. The mapping from `StorageObjects.usage_type`
|
||||
to bucket is decided in **one place** (`backend/storage_api.py:resolve_bucket`):
|
||||
|
||||
| `usage_type` | Bucket (env var) | Default name |
|
||||
|---|---|---|
|
||||
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `RUSTFS_WORKSPACE_BUCKET` | `workspaces` |
|
||||
| `version_artifact` | `RUSTFS_VERSION_BUCKET` | `versions` |
|
||||
| `run_log`, `run_result` | `RUSTFS_RUN_LOG_BUCKET` | `run-logs` |
|
||||
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `S3_WORKSPACE_BUCKET` | `workspaces` |
|
||||
| `version_artifact` | `S3_VERSION_BUCKET` | `versions` |
|
||||
| `run_log`, `run_result` | `S3_RUN_LOG_BUCKET` | `run-logs` |
|
||||
| (soft-delete target) | `S3_TRASH_BUCKET` | `trash` |
|
||||
|
||||
In `STORAGE_BACKEND=s3` mode these are 4 separate S3 buckets. In
|
||||
`STORAGE_BACKEND=local` mode they are 4 subdirectories under
|
||||
`LOCAL_STORAGE_BASE_DIR` (default `/data`), so the layout above
|
||||
becomes:
|
||||
|
||||
```
|
||||
/data/
|
||||
├── workspace/ # S3_WORKSPACE_BUCKET
|
||||
├── version/ # S3_VERSION_BUCKET
|
||||
├── run_log/ # S3_RUN_LOG_BUCKET
|
||||
└── trash/ # S3_TRASH_BUCKET
|
||||
```
|
||||
|
||||
A workspace's `artifact_bucket` column (when non-null) overrides the
|
||||
default for that workspace, regardless of `usage_type` — useful for
|
||||
@@ -174,24 +194,19 @@ The object key is a flat two-level path — `workspace_id` and a server-
|
||||
issued `ulid` for the object:
|
||||
|
||||
```
|
||||
s3://workspaces/
|
||||
└── <workspace_id>/
|
||||
├── <ulid-1> # working_copy / data_resource / snapshot / ...
|
||||
├── <ulid-2>
|
||||
└── ...
|
||||
|
||||
s3://versions/<workspace_id>/<ulid> # immutable script versions
|
||||
s3://run-logs/<workspace_id>/<ulid> # node run logs and results
|
||||
<workspace_bucket>/<workspace_id>/<ulid>{.<ext>}
|
||||
```
|
||||
|
||||
The file name, extension, content type, and logical path live in the
|
||||
`StorageObjects` and `Scripts` rows, not in the S3 key, so the bucket
|
||||
can be re-organised without a database rewrite.
|
||||
`StorageObjects` and `Scripts` rows, not in the object key, so the
|
||||
storage can be re-organised without a database rewrite.
|
||||
|
||||
Backend code never writes to the container's local filesystem. Schedule
|
||||
Executor stages node artifacts in `tempfile.TemporaryDirectory()` (auto-
|
||||
cleaned). Only the `runtime` container keeps a host volume — it is required
|
||||
by the rclone FUSE mount.
|
||||
Backend code never writes to the container's local filesystem (except
|
||||
in `STORAGE_BACKEND=local` mode, where the shared `local-storage` volume
|
||||
is the canonical store). Schedule Executor stages node artifacts in
|
||||
`tempfile.TemporaryDirectory()` (auto-cleaned). Only the `runtime`
|
||||
container keeps a host volume — required by the rclone FUSE mount in
|
||||
s3 mode, and a no-op pass-through in local mode.
|
||||
|
||||
## Documentation
|
||||
|
||||
|
||||
+7
-2
@@ -28,8 +28,8 @@
|
||||
|
||||
```text
|
||||
mysql
|
||||
rustfs
|
||||
jupyter
|
||||
s3 # 外部 S3-兼容服务(MinIO/RustFS/SeaweedFS/...),由运维在 compose 外启动
|
||||
jupyter # 注释保留;当前实现未在 compose 启此独立容器
|
||||
migrate(一次性)
|
||||
backend
|
||||
runtime
|
||||
@@ -37,6 +37,11 @@ schedule
|
||||
gateway
|
||||
```
|
||||
|
||||
> 补充说明:`STORAGE_BACKEND=local` 模式下不需要外部 S3 服务,backend
|
||||
> 与 runtime 共享 docker 卷 `local-storage`,挂载到
|
||||
> `LOCAL_STORAGE_BASE_DIR`(默认 `/data/storage`);runtime 也跳过
|
||||
> rclone FUSE 挂载(见 `runtime/src/runtime/mount.py`)。
|
||||
|
||||
## 已执行检查
|
||||
|
||||
- Python 全项目 `compileall` 通过;
|
||||
|
||||
+4
-1
@@ -1,5 +1,8 @@
|
||||
# Backend
|
||||
|
||||
统一 FastAPI 管理服务。包含用户、Workspace、脚本、稳定版本、调度定义、
|
||||
立即运行以及 RustFS 对象接口。原 `platform_api` 与 `storage_api` 已在此
|
||||
立即运行以及 S3 对象接口。原 `platform_api` 与 `storage_api` 已在此
|
||||
模块合并,外部 REST 契约保持不变。
|
||||
|
||||
底层走的是 `common.storage.AsyncStorageBackend` 抽象,按
|
||||
`settings.storage_backend` 切换 s3 / local 两种实现。
|
||||
@@ -65,5 +65,6 @@ python -m migrations.data.migrate_legacy_workspaces --source "<path>/server.py"
|
||||
## 第 11、12 小步数据处理决定
|
||||
|
||||
根据实施确认,第 11、12 小步不迁移旧版资源、脚本或稳定版本数据。新实现直接
|
||||
使用 MySQL、Workspace 文件目录和 RustFS,从空的 `data_resources`、`scripts`
|
||||
及 `versions` 表开始运行。功能验收产生的临时对象和数据库记录均已清理。
|
||||
使用 MySQL、Workspace 文件目录和对象存储(s3 模式连 S3-兼容服务,local 模式
|
||||
走 `/data/storage` 共享卷),从空的 `data_resources`、`scripts` 及
|
||||
`versions` 表开始运行。功能验收产生的临时对象和数据库记录均已清理。
|
||||
|
||||
Reference in New Issue
Block a user