fix(security): P0-1 — port exposure + service-token auth on /internal/* + jupyter RPC
The fix lands in three concentric layers, all backed by a single
INTERNAL_SERVICE_TOKEN shared secret so we have one mechanism
instead of three:
1. docker-compose: drop the backend.ports: 8891:8000 and
runtime.ports: 8892:8000 mappings. Nginx is the only host
ingress again (architecture §2.2).
2. /internal/v1/*: the storage control plane had six endpoints, five
of which were dead code (frontend already migrated to
/api/v1/data-resources/* with JWT; schedule only ever called
POST /internal/v1/objects). Delete the dead routes, mount the
one survivor with Depends(require_internal_service) that
compares the X-Internal-Service-Token header against
settings.internal_service_token with secrets.compare_digest.
3. POST /api/v1/jupyter on the runtime container: previously open
inside the Docker network. Same token mechanism — backend's
runtime_http_client now carries the header, runtime's
handle_jupyter_action requires the same header. /api/v1/health
stays open for the Nginx and compose healthchecks.
The schedule worker was already configured to call
POST /internal/v1/objects; build_storage_http_client now
sets the token header so its existing call site keeps working
without changes.
Files touched:
backend/src/backend/storage_api.py # 5 dead routes deleted + token guard
backend/src/backend/main.py # runtime_http_client header
runtime/src/runtime/main.py # require_internal_service Depends
common/src/common/config.py # internal_service_token setting
schedule/src/schedule/service.py # httpx client header
docker-compose.yml # ports dropped, INTERNAL_SERVICE_TOKEN env
.env.example # INTERNAL_SERVICE_TOKEN placeholder
API.md / README.md / DEVELOP.md # §9 trimmed to 1 endpoint
Verified:
compileall -> 0 errors
pytest backend/tests -> 37 passed
in-process ASGI smoke:
POST /internal/v1/objects no/wrong/correct token -> 401/401/200
POST /api/v1/jupyter no/wrong/correct token -> 401/401/200
5 deleted internal routes -> 404
docker compose config (with env) -> OK
P0-1 still has one open sub-item (rclone RC --rc-no-auth) that
the user has explicitly deferred; not touched here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
tao.chen
co-authored by
Claude Fable 5
Claude Fable 5
parent
6258cf5d12
commit
dfe3f0b118
@@ -27,7 +27,7 @@
|
||||
6. [管理后台 (`/api/v1/admin/...`)](#六管理后台)
|
||||
7. [系统管理 (`/api/v1/platform/...`)](#七系统管理-apiv1platform)
|
||||
8. [Jupyter 路由 (Nginx `auth_request`)](#八jupyter-路由)
|
||||
9. [对象存储控制面 (`/internal/v1/...`,同进程 RPC)](#九对象存储控制面)
|
||||
9. [对象存储控制面 (`/internal/v1/objects`,服务间 RPC + Token 鉴权)](#九对象存储控制面)
|
||||
10. [健康检查](#十健康检查)
|
||||
|
||||
---
|
||||
@@ -1006,53 +1006,29 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断
|
||||
|
||||
## 九、对象存储控制面
|
||||
|
||||
> 路径前缀 `/internal/v1/...`,**前端不要直接调用**。这是 backend 内部
|
||||
> 异步消息处理(Schedule worker)用的 RPC 端点,经 in-process ASGI 直接
|
||||
> 转发(`storage_app` 路由被 `app.include_router` 进同一个 backend 进程),
|
||||
> 外部无法访问。
|
||||
> **范围**:Schedule worker 调用 backend 写 `run_log` / `run_result` 用的
|
||||
> 单端点 RPC。**前端不要直接调用,也不要把这个路径用于任何用户输入**。
|
||||
> 历史上有 6 个 `/internal/v1/*` 端点,经过 P0-1 修复后只剩这 1 个;
|
||||
> 删除的端点全部已迁移到 JWT 保护的 `/api/v1/data-resources/*` 与
|
||||
> `/api/v1/scripts/*` 路由(见 §五、§三)。
|
||||
|
||||
底层抽象:`common.storage.AsyncStorageBackend`(`put/get/delete/exists/stat/
|
||||
list/get_url/copy`)。按 `settings.storage_backend` 选实现:`"s3"` 走
|
||||
S3-兼容服务,`"local"` 走 `LOCAL_STORAGE_BASE_DIR` 子目录。
|
||||
|
||||
### 9.1 `POST /internal/v1/uploads`
|
||||
### 鉴权(P0-1 后)
|
||||
|
||||
创建上传会话。`Idempotency-Key` 必填,同 key + 同元数据 → 复用;同 key + 不同元数据 → 409。
|
||||
所有 `/internal/v1/*` 路由都会校验请求头里的 service token:
|
||||
|
||||
```json
|
||||
{
|
||||
"workspace_id": "...",
|
||||
"user_id": "...",
|
||||
"usage_type": "working_copy",
|
||||
"file_name": "train.py",
|
||||
"content_type": "text/x-python",
|
||||
"expected_size_bytes": 1024,
|
||||
"expected_hash": "<optional sha256 hex>",
|
||||
"idempotency_key": "...",
|
||||
"visibility": "private",
|
||||
"is_immutable": false
|
||||
}
|
||||
```
|
||||
X-Internal-Service-Token: <settings.internal_service_token>
|
||||
```
|
||||
|
||||
返回 `{upload_id, status, upload_path, expires_at}`。`upload_path` 是
|
||||
第 9.2 步要 PUT 的端点(本进程内 `/internal/v1/uploads/{upload_id}`)。
|
||||
- backend 与 schedule 必须把同一个值注入到 `INTERNAL_SERVICE_TOKEN` 环境变量。
|
||||
- 缺失或不相符 → `401`。
|
||||
- backend 配置为空 → `503`(`internal service token not configured`)。
|
||||
|
||||
### 9.2 `PUT /internal/v1/uploads/{upload_id}`
|
||||
|
||||
完成上传(server-proxied PUT)。**请求体即原始字节**,`Content-Type:
|
||||
application/octet-stream`。后端 `await request.body()` 读字节 → 校验
|
||||
size + sha256 → 调 `await backend.put(key, bytes, content_type=...,
|
||||
metadata={"sha256": ...})` → 写 `StorageObjects` 行 → 标 session 为
|
||||
completed。最大 100 MiB。
|
||||
|
||||
> 历史:旧版本这一步是 `POST /uploads/{id}/complete`,靠
|
||||
> presigned-PUT + head() 验证。已被 server-proxied PUT 取代。
|
||||
|
||||
### 9.3 `POST /internal/v1/uploads/{upload_id}/abort`
|
||||
|
||||
主动放弃。删除可能已经写了一半的对象字节,释放 `UploadSessions` 行。
|
||||
|
||||
### 9.4 `POST /internal/v1/objects`
|
||||
### 9.1 `POST /internal/v1/objects`
|
||||
|
||||
**单步创建**(不走两步上传,字节 base64 进 JSON 体)。适用 < 100 KiB
|
||||
对象(避免 multipart/大请求体的前端复杂度)。内部直接调
|
||||
@@ -1068,24 +1044,13 @@ metadata={"sha256": ...})`。
|
||||
"content_type": "text/plain",
|
||||
"content_base64": "PHN0ZXAtY29udGVudD4=",
|
||||
"visibility": "private",
|
||||
"is_immutable": false,
|
||||
"is_immutable": true,
|
||||
"idempotency_key": "...",
|
||||
"relative_path": null
|
||||
}
|
||||
```
|
||||
|
||||
### 9.5 `POST /internal/v1/objects/{storage_object_id}/download-url`
|
||||
|
||||
生成 presigned GET URL(s3 模式:`AsyncStorageBackend.get_url()`;
|
||||
local 模式:目前抛 `NotImplementedError`,需要 native FS serving 配合
|
||||
nginx 静态 location)。
|
||||
|
||||
### 9.6 `DELETE /internal/v1/objects/{storage_object_id}`
|
||||
|
||||
软删。`is_immutable == 1` 的对象拒绝删除。`move_to_trash` 走跨后端
|
||||
`copy + delete`(同一进程内的两个 backend 实例)。
|
||||
|
||||
### 9.7 usage_type → 桶路由(自动)
|
||||
### 9.2 usage_type → 桶路由(自动)
|
||||
|
||||
| usage_type | 实际桶(env var) | 默认桶名 |
|
||||
|---|---|---|
|
||||
@@ -1099,6 +1064,16 @@ nginx 静态 location)。
|
||||
桶在 `STORAGE_BACKEND=s3` 时是 4 个独立 S3 bucket,在
|
||||
`STORAGE_BACKEND=local` 时是 `LOCAL_STORAGE_BASE_DIR` 下的 4 个子目录。
|
||||
|
||||
### 9.3 已删除的端点(P0-1 收纳)
|
||||
|
||||
| 旧端点 | 替代路由 | 说明 |
|
||||
|---|---|---|
|
||||
| `POST /internal/v1/uploads` | `POST /api/v1/data-resources/uploads`(JWT) | 前端走公开路径 |
|
||||
| `PUT /internal/v1/uploads/{id}` | `PUT /api/v1/data-resources/uploads/{id}` | 同上 |
|
||||
| `POST /internal/v1/uploads/{id}/abort` | (客户端取消即可) | 无后端状态 |
|
||||
| `POST /internal/v1/objects/{id}/download-url` | 各自的公开路由生成 presigned URL | download URL 由公开路由返回 |
|
||||
| `DELETE /internal/v1/objects/{id}` | 公开路由的删除操作 | 与 data-resource 联动 |
|
||||
|
||||
---
|
||||
|
||||
## 十、健康检查
|
||||
|
||||
Reference in New Issue
Block a user