docs: align with new storage architecture (s3 + local + server-proxied PUT)
All operator- and developer-facing docs updated to reflect:
- The unified AsyncStorageBackend abstraction (s3 + local backends).
- The STORAGE_BACKEND toggle ("s3" default, "local" for dev /
single-node / air-gapped deployments).
- The 4-purpose-bucket layout (workspace / version / run_log / trash)
in both modes — 4 separate S3 buckets in s3 mode, 4 subdirectories
of LOCAL_STORAGE_BASE_DIR in local mode.
- The S3_* env var naming (was RUSTFS_*).
- The server-proxied upload flow (was browser-direct presign-PUT):
POST /internal/v1/uploads → PUT /internal/v1/uploads/{id} with
raw bytes → server calls backend.put().
- The factory helpers workspaces_root() (runtime's view of the
workspace bucket on disk) and rclone_remote_spec() (s3-mode mount
source).
- The "two settings describing the same thing" cleanup: the deleted
settings.workspace_root, settings.workspaces_root, and
settings.remote_bucket fields.
Files touched:
- API.md (§5 data-resource upload flow, §9 storage control plane,
§10 readiness example)
- ARCHITECTURE.md (storage layer diagram)
- CLAUDE.md (architecture description + volume-preservation note)
- DEVELOP.md (settings list, Storage section, "Wire a new bucket"
how-to, dev-export example, troubleshooting network hint)
- README.md (architecture diagram, container table, quick-start
credentials note, tear-down note, Storage layout section)
- REFACTOR_NOTES.md (final container list with s3 explanation)
- backend/README.md (storage backend description)
- migrations/data/README.md (step 11/12 record mentioning object
storage)
A handful of historical "RustFS" mentions are intentionally retained
where they name a specific S3-compatible product (e.g. as an example
in REFACTOR_NOTES.md's container list) or document the pre-2026
abstraction name (DEVELOP.md Storage section).
This commit is contained in:
@@ -92,7 +92,7 @@
|
||||
|
||||
### 3.2 `POST /api/v1/workspace-directories`
|
||||
|
||||
创建一个**逻辑目录**(RustFS 上是隐式前缀,无需落对象)。
|
||||
创建一个**逻辑目录**(对象存储上是隐式前缀,无需落对象)。
|
||||
|
||||
- **请求体**:
|
||||
```json
|
||||
@@ -163,7 +163,7 @@
|
||||
|
||||
### 3.7 `POST /api/v1/scripts/upload?file_name=...&parent_path=...&visibility=...`
|
||||
|
||||
multipart/binary 形式上传大文件(走 presigned PUT)。
|
||||
multipart/binary 形式上传大文件(走 server-proxied PUT,详见 §九)。
|
||||
|
||||
- **查询参数**: `file_name`(必填)、`parent_path`、`visibility`
|
||||
- **请求体**: 原始文件字节(`Content-Type` 必须与脚本类型匹配)
|
||||
@@ -203,7 +203,7 @@ multipart/binary 形式上传大文件(走 presigned PUT)。
|
||||
|
||||
### 3.11 `POST /api/v1/scripts/{script_id}/versions`
|
||||
|
||||
发布一个**稳定版本**(immutable,绑定到 `RUSTFS_VERSION_BUCKET`)。门禁同 §3.8。
|
||||
发布一个**稳定版本**(immutable,绑定到 `S3_VERSION_BUCKET`)。门禁同 §3.8。
|
||||
|
||||
- **请求体**:
|
||||
```json
|
||||
@@ -243,11 +243,11 @@ multipart/binary 形式上传大文件(走 presigned PUT)。
|
||||
|
||||
### 3.14 `DELETE /api/v1/versions/{versions_id}`
|
||||
|
||||
从调度候选中**隐藏**此版本(不删除 RustFS 对象)。门禁:**owner 校验基于所属 `Scripts` 的 owner**——即"按整本 script 判定",而非"按版本发布者判定"。
|
||||
从调度候选中**隐藏**此版本(不删除对象存储里的对象)。门禁:**owner 校验基于所属 `Scripts` 的 owner**——即"按整本 script 判定",而非"按版本发布者判定"。
|
||||
|
||||
### 3.15 `POST /api/v1/versions/{versions_id}/download-url`
|
||||
|
||||
生成 RustFS 的 presigned download URL。
|
||||
生成对象存储的 presigned download URL(走 S3 兼容协议,local 模式下该 endpoint 在 s3 模式才生效)。
|
||||
|
||||
- **请求体**:
|
||||
```json
|
||||
@@ -399,8 +399,8 @@ queued ──→ running ──┬─→ succeeded
|
||||
|
||||
| 方法 | 路径 | 说明 |
|
||||
|---|---|---|
|
||||
| `POST` | `/api/v1/data-resources/uploads` | 创建上传会话,返回 presigned PUT URL + `upload_id` |
|
||||
| `POST` | `/api/v1/data-resources/uploads/{upload_id}/complete` | 完成上传(写 `StorageObjects` 行) |
|
||||
| `POST` | `/api/v1/data-resources/uploads` | 创建上传会话,返回 `upload_id` + `upload_path` |
|
||||
| `PUT` | `/api/v1/data-resources/uploads/{upload_id}` | 上传字节(请求体即文件内容) |
|
||||
| `GET` | `/api/v1/data-resources` | 列表(workspace 范围) |
|
||||
| `GET` | `/api/v1/data-resources/{id}` | 详情 |
|
||||
| `POST` | `/api/v1/data-resources/{id}/download-url` | 生成 presigned GET URL |
|
||||
@@ -420,15 +420,18 @@ queued ──→ running ──┬─→ succeeded
|
||||
|
||||
**完整上传流程(前端应实现的模式)**:
|
||||
```
|
||||
1. POST /uploads → {upload_id, presigned_url}
|
||||
2. PUT presigned_url with file bytes
|
||||
3. POST /uploads/{upload_id}/complete
|
||||
4. 服务器完成 → 200 {data: StorageObjectPayload}
|
||||
1. POST /uploads → {upload_id, upload_path, expires_at}
|
||||
2. PUT upload_path with raw file bytes (Content-Type: application/octet-stream)
|
||||
3. 服务器端走 backend.put() → 200 {data: StorageObjectPayload}
|
||||
```
|
||||
|
||||
字节经过 backend 进程(server-proxied upload),最大 100 MiB,由 backend
|
||||
直接调 `AsyncStorageBackend.put()` 写入存储(不再走 presigned PUT 直传)。
|
||||
前端无需关心 S3 协议或签名。
|
||||
|
||||
**小对象(<100 KiB)捷径**:直接调 `create_server_object` 把字节 base64 放进
|
||||
`content_base64` 字段(走 `StorageClient.create_server_object`),前端无需
|
||||
走 presigned PUT 三步。
|
||||
`content_base64` 字段(JSON 体里走),内部走同一条 `AsyncStorageBackend.put`
|
||||
路径,前端无需分两步。
|
||||
|
||||
---
|
||||
|
||||
@@ -672,9 +675,13 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断
|
||||
## 九、对象存储控制面
|
||||
|
||||
> 路径前缀 `/internal/v1/...`,**前端不要直接调用**。这是 backend 内部
|
||||
> 异步消息处理(Schedule worker)用的 RPC 端点,经 `StorageClient` HTTP
|
||||
> 客户端访问。Backend 通过 ASGI `auth_request_set` 路由转发,外部无法
|
||||
> 访问。
|
||||
> 异步消息处理(Schedule worker)用的 RPC 端点,经 in-process ASGI 直接
|
||||
> 转发(`storage_app` 路由被 `app.include_router` 进同一个 backend 进程),
|
||||
> 外部无法访问。
|
||||
|
||||
底层抽象:`common.storage.AsyncStorageBackend`(`put/get/delete/exists/stat/
|
||||
list/get_url/copy`)。按 `settings.storage_backend` 选实现:`"s3"` 走
|
||||
S3-兼容服务,`"local"` 走 `LOCAL_STORAGE_BASE_DIR` 子目录。
|
||||
|
||||
### 9.1 `POST /internal/v1/uploads`
|
||||
|
||||
@@ -689,23 +696,36 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断
|
||||
"content_type": "text/x-python",
|
||||
"expected_size_bytes": 1024,
|
||||
"expected_hash": "<optional sha256 hex>",
|
||||
"idempotency_key": "..."
|
||||
"idempotency_key": "...",
|
||||
"visibility": "private",
|
||||
"is_immutable": false
|
||||
}
|
||||
```
|
||||
|
||||
返回 `{upload_id, bucket_name, object_key, presigned_url, expires_in_seconds}`。
|
||||
返回 `{upload_id, status, upload_path, expires_at}`。`upload_path` 是
|
||||
第 9.2 步要 PUT 的端点(本进程内 `/internal/v1/uploads/{upload_id}`)。
|
||||
|
||||
### 9.2 `POST /internal/v1/uploads/{upload_id}/complete`
|
||||
### 9.2 `PUT /internal/v1/uploads/{upload_id}`
|
||||
|
||||
完成上传。从 RustFS 读 HEAD → 校验 hash → 写 `StorageObjects` 行。
|
||||
完成上传(server-proxied PUT)。**请求体即原始字节**,`Content-Type:
|
||||
application/octet-stream`。后端 `await request.body()` 读字节 → 校验
|
||||
size + sha256 → 调 `await backend.put(key, bytes, content_type=...,
|
||||
metadata={"sha256": ...})` → 写 `StorageObjects` 行 → 标 session 为
|
||||
completed。最大 100 MiB。
|
||||
|
||||
> 历史:旧版本这一步是 `POST /uploads/{id}/complete`,靠
|
||||
> presigned-PUT + head() 验证。已被 server-proxied PUT 取代。
|
||||
|
||||
### 9.3 `POST /internal/v1/uploads/{upload_id}/abort`
|
||||
|
||||
主动放弃。释放 `UploadSessions` 行,对象不入库。
|
||||
主动放弃。删除可能已经写了一半的对象字节,释放 `UploadSessions` 行。
|
||||
|
||||
### 9.4 `POST /internal/v1/objects`
|
||||
|
||||
**单步创建**(不走 presigned PUT,字节随请求体直传)。适用 < 100 KiB 对象。
|
||||
**单步创建**(不走两步上传,字节 base64 进 JSON 体)。适用 < 100 KiB
|
||||
对象(避免 multipart/大请求体的前端复杂度)。内部直接调
|
||||
`AsyncStorageBackend.put(key, content, content_type=...,
|
||||
metadata={"sha256": ...})`。
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -717,28 +737,36 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断
|
||||
"content_base64": "PHN0ZXAtY29udGVudD4=",
|
||||
"visibility": "private",
|
||||
"is_immutable": false,
|
||||
"idempotency_key": "..."
|
||||
"idempotency_key": "...",
|
||||
"relative_path": null
|
||||
}
|
||||
```
|
||||
|
||||
### 9.5 `POST /internal/v1/objects/{storage_object_id}/download-url`
|
||||
|
||||
生成 presigned GET URL。
|
||||
生成 presigned GET URL(s3 模式:`AsyncStorageBackend.get_url()`;
|
||||
local 模式:目前抛 `NotImplementedError`,需要 native FS serving 配合
|
||||
nginx 静态 location)。
|
||||
|
||||
### 9.6 `DELETE /internal/v1/objects/{storage_object_id}`
|
||||
|
||||
软删。`is_immutable == 1` 的对象拒绝删除。
|
||||
软删。`is_immutable == 1` 的对象拒绝删除。`move_to_trash` 走跨后端
|
||||
`copy + delete`(同一进程内的两个 backend 实例)。
|
||||
|
||||
### 9.7 usage_type → 桶路由(自动)
|
||||
|
||||
| usage_type | 实际桶(env var) | 默认桶名 |
|
||||
|---|---|---|
|
||||
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `RUSTFS_WORKSPACE_BUCKET` | `workspaces` |
|
||||
| `version_artifact` | `RUSTFS_VERSION_BUCKET` | `versions` |
|
||||
| `run_log`, `run_result` | `RUSTFS_RUN_LOG_BUCKET` | `run-logs` |
|
||||
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `S3_WORKSPACE_BUCKET` | `workspaces` |
|
||||
| `version_artifact` | `S3_VERSION_BUCKET` | `versions` |
|
||||
| `run_log`, `run_result` | `S3_RUN_LOG_BUCKET` | `run-logs` |
|
||||
| (soft-delete target) | `S3_TRASH_BUCKET` | `trash` |
|
||||
|
||||
若 `Workspaces.artifact_bucket` 非空,优先用 per-workspace 桶(覆盖 usage_type 路由)。
|
||||
|
||||
桶在 `STORAGE_BACKEND=s3` 时是 4 个独立 S3 bucket,在
|
||||
`STORAGE_BACKEND=local` 时是 `LOCAL_STORAGE_BASE_DIR` 下的 4 个子目录。
|
||||
|
||||
---
|
||||
|
||||
## 十、健康检查
|
||||
@@ -751,7 +779,8 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断
|
||||
| `GET` | `/api/v1/health` | 公开健康检查(前端可访问) |
|
||||
|
||||
`/health/ready` 支持 `READINESS_TARGETS` 环境变量,逗号分隔的 `host:port`
|
||||
列表,例如 `mysql:3306,rustfs:9000`,全部 TCP 通则返回 200,否则 503。
|
||||
列表,例如 `mysql:3306,s3:9000`,全部 TCP 通则返回 200,否则 503。
|
||||
`STORAGE_BACKEND=local` 模式下不需要 S3 host,列表里删掉即可。
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user