From 309b657d3583840012a13408d1228fae0959ca25 Mon Sep 17 00:00:00 2001 From: "tao.chen" <93983997+taochen-ct@users.noreply.github.com> Date: Wed, 5 Aug 2026 13:13:20 +0800 Subject: [PATCH] docs: align with new storage architecture (s3 + local + server-proxied PUT) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All operator- and developer-facing docs updated to reflect: - The unified AsyncStorageBackend abstraction (s3 + local backends). - The STORAGE_BACKEND toggle ("s3" default, "local" for dev / single-node / air-gapped deployments). - The 4-purpose-bucket layout (workspace / version / run_log / trash) in both modes — 4 separate S3 buckets in s3 mode, 4 subdirectories of LOCAL_STORAGE_BASE_DIR in local mode. - The S3_* env var naming (was RUSTFS_*). - The server-proxied upload flow (was browser-direct presign-PUT): POST /internal/v1/uploads → PUT /internal/v1/uploads/{id} with raw bytes → server calls backend.put(). - The factory helpers workspaces_root() (runtime's view of the workspace bucket on disk) and rclone_remote_spec() (s3-mode mount source). - The "two settings describing the same thing" cleanup: the deleted settings.workspace_root, settings.workspaces_root, and settings.remote_bucket fields. Files touched: - API.md (§5 data-resource upload flow, §9 storage control plane, §10 readiness example) - ARCHITECTURE.md (storage layer diagram) - CLAUDE.md (architecture description + volume-preservation note) - DEVELOP.md (settings list, Storage section, "Wire a new bucket" how-to, dev-export example, troubleshooting network hint) - README.md (architecture diagram, container table, quick-start credentials note, tear-down note, Storage layout section) - REFACTOR_NOTES.md (final container list with s3 explanation) - backend/README.md (storage backend description) - migrations/data/README.md (step 11/12 record mentioning object storage) A handful of historical "RustFS" mentions are intentionally retained where they name a specific S3-compatible product (e.g. as an example in REFACTOR_NOTES.md's container list) or document the pre-2026 abstraction name (DEVELOP.md Storage section). --- API.md | 87 ++++++++++++++++++---------- ARCHITECTURE.md | 2 +- CLAUDE.md | 4 +- DEVELOP.md | 116 ++++++++++++++++++++++++-------------- README.md | 79 +++++++++++++++----------- REFACTOR_NOTES.md | 9 ++- backend/README.md | 5 +- migrations/data/README.md | 5 +- 8 files changed, 195 insertions(+), 112 deletions(-) diff --git a/API.md b/API.md index 8902d0b..df27108 100644 --- a/API.md +++ b/API.md @@ -92,7 +92,7 @@ ### 3.2 `POST /api/v1/workspace-directories` -创建一个**逻辑目录**(RustFS 上是隐式前缀,无需落对象)。 +创建一个**逻辑目录**(对象存储上是隐式前缀,无需落对象)。 - **请求体**: ```json @@ -163,7 +163,7 @@ ### 3.7 `POST /api/v1/scripts/upload?file_name=...&parent_path=...&visibility=...` -multipart/binary 形式上传大文件(走 presigned PUT)。 +multipart/binary 形式上传大文件(走 server-proxied PUT,详见 §九)。 - **查询参数**: `file_name`(必填)、`parent_path`、`visibility` - **请求体**: 原始文件字节(`Content-Type` 必须与脚本类型匹配) @@ -203,7 +203,7 @@ multipart/binary 形式上传大文件(走 presigned PUT)。 ### 3.11 `POST /api/v1/scripts/{script_id}/versions` -发布一个**稳定版本**(immutable,绑定到 `RUSTFS_VERSION_BUCKET`)。门禁同 §3.8。 +发布一个**稳定版本**(immutable,绑定到 `S3_VERSION_BUCKET`)。门禁同 §3.8。 - **请求体**: ```json @@ -243,11 +243,11 @@ multipart/binary 形式上传大文件(走 presigned PUT)。 ### 3.14 `DELETE /api/v1/versions/{versions_id}` -从调度候选中**隐藏**此版本(不删除 RustFS 对象)。门禁:**owner 校验基于所属 `Scripts` 的 owner**——即"按整本 script 判定",而非"按版本发布者判定"。 +从调度候选中**隐藏**此版本(不删除对象存储里的对象)。门禁:**owner 校验基于所属 `Scripts` 的 owner**——即"按整本 script 判定",而非"按版本发布者判定"。 ### 3.15 `POST /api/v1/versions/{versions_id}/download-url` -生成 RustFS 的 presigned download URL。 +生成对象存储的 presigned download URL(走 S3 兼容协议,local 模式下该 endpoint 在 s3 模式才生效)。 - **请求体**: ```json @@ -399,8 +399,8 @@ queued ──→ running ──┬─→ succeeded | 方法 | 路径 | 说明 | |---|---|---| -| `POST` | `/api/v1/data-resources/uploads` | 创建上传会话,返回 presigned PUT URL + `upload_id` | -| `POST` | `/api/v1/data-resources/uploads/{upload_id}/complete` | 完成上传(写 `StorageObjects` 行) | +| `POST` | `/api/v1/data-resources/uploads` | 创建上传会话,返回 `upload_id` + `upload_path` | +| `PUT` | `/api/v1/data-resources/uploads/{upload_id}` | 上传字节(请求体即文件内容) | | `GET` | `/api/v1/data-resources` | 列表(workspace 范围) | | `GET` | `/api/v1/data-resources/{id}` | 详情 | | `POST` | `/api/v1/data-resources/{id}/download-url` | 生成 presigned GET URL | @@ -420,15 +420,18 @@ queued ──→ running ──┬─→ succeeded **完整上传流程(前端应实现的模式)**: ``` -1. POST /uploads → {upload_id, presigned_url} -2. PUT presigned_url with file bytes -3. POST /uploads/{upload_id}/complete -4. 服务器完成 → 200 {data: StorageObjectPayload} +1. POST /uploads → {upload_id, upload_path, expires_at} +2. PUT upload_path with raw file bytes (Content-Type: application/octet-stream) +3. 服务器端走 backend.put() → 200 {data: StorageObjectPayload} ``` +字节经过 backend 进程(server-proxied upload),最大 100 MiB,由 backend +直接调 `AsyncStorageBackend.put()` 写入存储(不再走 presigned PUT 直传)。 +前端无需关心 S3 协议或签名。 + **小对象(<100 KiB)捷径**:直接调 `create_server_object` 把字节 base64 放进 -`content_base64` 字段(走 `StorageClient.create_server_object`),前端无需 -走 presigned PUT 三步。 +`content_base64` 字段(JSON 体里走),内部走同一条 `AsyncStorageBackend.put` +路径,前端无需分两步。 --- @@ -672,9 +675,13 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断 ## 九、对象存储控制面 > 路径前缀 `/internal/v1/...`,**前端不要直接调用**。这是 backend 内部 -> 异步消息处理(Schedule worker)用的 RPC 端点,经 `StorageClient` HTTP -> 客户端访问。Backend 通过 ASGI `auth_request_set` 路由转发,外部无法 -> 访问。 +> 异步消息处理(Schedule worker)用的 RPC 端点,经 in-process ASGI 直接 +> 转发(`storage_app` 路由被 `app.include_router` 进同一个 backend 进程), +> 外部无法访问。 + +底层抽象:`common.storage.AsyncStorageBackend`(`put/get/delete/exists/stat/ +list/get_url/copy`)。按 `settings.storage_backend` 选实现:`"s3"` 走 +S3-兼容服务,`"local"` 走 `LOCAL_STORAGE_BASE_DIR` 子目录。 ### 9.1 `POST /internal/v1/uploads` @@ -689,23 +696,36 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断 "content_type": "text/x-python", "expected_size_bytes": 1024, "expected_hash": "", - "idempotency_key": "..." + "idempotency_key": "...", + "visibility": "private", + "is_immutable": false } ``` -返回 `{upload_id, bucket_name, object_key, presigned_url, expires_in_seconds}`。 +返回 `{upload_id, status, upload_path, expires_at}`。`upload_path` 是 +第 9.2 步要 PUT 的端点(本进程内 `/internal/v1/uploads/{upload_id}`)。 -### 9.2 `POST /internal/v1/uploads/{upload_id}/complete` +### 9.2 `PUT /internal/v1/uploads/{upload_id}` -完成上传。从 RustFS 读 HEAD → 校验 hash → 写 `StorageObjects` 行。 +完成上传(server-proxied PUT)。**请求体即原始字节**,`Content-Type: +application/octet-stream`。后端 `await request.body()` 读字节 → 校验 +size + sha256 → 调 `await backend.put(key, bytes, content_type=..., +metadata={"sha256": ...})` → 写 `StorageObjects` 行 → 标 session 为 +completed。最大 100 MiB。 + +> 历史:旧版本这一步是 `POST /uploads/{id}/complete`,靠 +> presigned-PUT + head() 验证。已被 server-proxied PUT 取代。 ### 9.3 `POST /internal/v1/uploads/{upload_id}/abort` -主动放弃。释放 `UploadSessions` 行,对象不入库。 +主动放弃。删除可能已经写了一半的对象字节,释放 `UploadSessions` 行。 ### 9.4 `POST /internal/v1/objects` -**单步创建**(不走 presigned PUT,字节随请求体直传)。适用 < 100 KiB 对象。 +**单步创建**(不走两步上传,字节 base64 进 JSON 体)。适用 < 100 KiB +对象(避免 multipart/大请求体的前端复杂度)。内部直接调 +`AsyncStorageBackend.put(key, content, content_type=..., +metadata={"sha256": ...})`。 ```json { @@ -717,28 +737,36 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断 "content_base64": "PHN0ZXAtY29udGVudD4=", "visibility": "private", "is_immutable": false, - "idempotency_key": "..." + "idempotency_key": "...", + "relative_path": null } ``` ### 9.5 `POST /internal/v1/objects/{storage_object_id}/download-url` -生成 presigned GET URL。 +生成 presigned GET URL(s3 模式:`AsyncStorageBackend.get_url()`; +local 模式:目前抛 `NotImplementedError`,需要 native FS serving 配合 +nginx 静态 location)。 ### 9.6 `DELETE /internal/v1/objects/{storage_object_id}` -软删。`is_immutable == 1` 的对象拒绝删除。 +软删。`is_immutable == 1` 的对象拒绝删除。`move_to_trash` 走跨后端 +`copy + delete`(同一进程内的两个 backend 实例)。 ### 9.7 usage_type → 桶路由(自动) | usage_type | 实际桶(env var) | 默认桶名 | |---|---|---| -| `working_copy`, `public_script`, `data_resource`, `snapshot` | `RUSTFS_WORKSPACE_BUCKET` | `workspaces` | -| `version_artifact` | `RUSTFS_VERSION_BUCKET` | `versions` | -| `run_log`, `run_result` | `RUSTFS_RUN_LOG_BUCKET` | `run-logs` | +| `working_copy`, `public_script`, `data_resource`, `snapshot` | `S3_WORKSPACE_BUCKET` | `workspaces` | +| `version_artifact` | `S3_VERSION_BUCKET` | `versions` | +| `run_log`, `run_result` | `S3_RUN_LOG_BUCKET` | `run-logs` | +| (soft-delete target) | `S3_TRASH_BUCKET` | `trash` | 若 `Workspaces.artifact_bucket` 非空,优先用 per-workspace 桶(覆盖 usage_type 路由)。 +桶在 `STORAGE_BACKEND=s3` 时是 4 个独立 S3 bucket,在 +`STORAGE_BACKEND=local` 时是 `LOCAL_STORAGE_BASE_DIR` 下的 4 个子目录。 + --- ## 十、健康检查 @@ -751,7 +779,8 @@ Nginx 把这些状态原样透传给浏览器,前端可在 `onerror` 里判断 | `GET` | `/api/v1/health` | 公开健康检查(前端可访问) | `/health/ready` 支持 `READINESS_TARGETS` 环境变量,逗号分隔的 `host:port` -列表,例如 `mysql:3306,rustfs:9000`,全部 TCP 通则返回 200,否则 503。 +列表,例如 `mysql:3306,s3:9000`,全部 TCP 通则返回 200,否则 503。 +`STORAGE_BACKEND=local` 模式下不需要 S3 host,列表里删掉即可。 --- diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 633687e..141ceec 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -24,7 +24,7 @@ Schedule Executor（APScheduler） |-- MySQL APSchedulerJobStore |-- MySQL Outbox 轮询兜底 |-- DAG 节点执行与重试 - |-- RustFS 日志/结果 + |-- S3 日志/结果 +-- Backend 内部 Storage API ``` diff --git a/CLAUDE.md b/CLAUDE.md index 32cb110..e65c5ca 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -3,7 +3,7 @@ ## Current architecture - `frontend`: React Router SPA. Production files are built in `nginx/Dockerfile`. -- `backend`: public FastAPI API and internal RustFS storage API in one process. +- `backend`: public FastAPI API and internal S3 storage API in one process. - `runtime`: shared Jupyter lifecycle, MySQL edit leases and short-lived in-memory access tickets. - `schedule`: APScheduler, MySQL JobStore, MySQL Outbox polling and DAG execution. - `common`: SQLAlchemy models, database/session helpers, IDs and object-store helpers. @@ -51,7 +51,7 @@ pnpm build - Cron jobs are persisted by APScheduler in MySQL table `apscheduler_jobs`. - Runtime must stay single-replica while file leases and Jupyter tickets use the simplified implementation. - Never expose the internal Jupyter token to the browser. -- Never delete Docker volumes when preserving MySQL or RustFS data is required. +- Never delete Docker volumes when preserving MySQL or storage data is required. ## Main entrypoints diff --git a/DEVELOP.md b/DEVELOP.md index 32e1d95..c06f437 100644 --- a/DEVELOP.md +++ b/DEVELOP.md @@ -12,7 +12,7 @@ common/ Pure-Python shared library db/ SQLAlchemy 2.0 async engine, session_scope, Base db/models/ 26 tables in 9 domain files (zero FK, zero relationship) scheduler/ build_sqlalchemy_jobstore (delayed import) - storage/ RustFSObjectStore + StorageClient + Pydantic schemas + storage/ AsyncStorageBackend abstraction (s3 + local impls) + Pydantic schemas eventing.py add_outbox_event / utcnow / event_time service_app.py /health/ready TCP probe, /api/v1/health schemas.py StrictModel base @@ -21,14 +21,14 @@ common/ Pure-Python shared library backend/ Public FastAPI service + internal /internal/v1/* sub-app main.py lifespan + route registration jupyter.py /api/v1/auth/jupyter — the ONLY auth entry - scripts.py CRUD for scripts/notebooks (workspace_fs=rustfs) + scripts.py CRUD for scripts/notebooks (object storage via AsyncStorageBackend) schedules.py DAG CRUD: schedules, nodes, edges schedule_runs.py Trigger / list / get runs schedule_schemas.py Pydantic request/response models admin.py Admin endpoints resources.py Misc data resources storage_api.py /internal/v1/* (sub-app merged into main) - storage_client.py HTTP client for the storage sub-app + storage_client.py Stub (HTTP client removed post-migration; rewrite pending) schedule_client.py Placeholder module (was the HTTP-push executor client) runtime_client.py Self-contained httpx wrapper for the runtime jupyter.py auth_request handler @@ -41,7 +41,7 @@ schedule/ Schedule Executor (DAG worker) worker.py NodeExecutor (notebook / python execution) service.py SchedulerService facade (composes the three) main.py Lifespan + FastAPI app - storage_client.py SchedulerStorageClient (subclass of common StorageClient) + storage_client.py Stub (SchedulerStorageClient rewrite pending — use AsyncStorageBackend directly) execution.py execute_artifact (notebook + python paths) notebook_runner.py Subprocess entry point (nbclient) @@ -68,16 +68,17 @@ All env vars go through one place: `common/src/common/config.py`. from common.config import settings settings.database_url # str -settings.rustfs_endpoint # str (full URL, e.g. "http://rustfs:9000") -settings.rustfs_access_key # str -settings.rustfs_secret_key # str -settings.rustfs_workspace_bucket -settings.rustfs_version_bucket -settings.rustfs_run_log_bucket -settings.jwt_secret # HS256 secret for the auth_request handler -settings.workspace_root # schedule subprocess cwd; backend ignores -settings.workspaces_root # runtime rclone FUSE mount point -settings.remote_bucket # rclone remote spec (e.g. "rustfs:workspaces") +settings.storage_backend # str: "s3" (default) or "local" +settings.local_storage_base_dir # str: root dir for storage data (default "/data"); see "Storage" below for per-mode derivation +settings.s3_endpoint # str (full URL, e.g. "http://s3:9000"; s3 mode only) +settings.s3_access_key # str (s3 mode only) +settings.s3_secret_key # str (s3 mode only) +settings.s3_workspace_bucket # str (s3 mode only) +settings.s3_version_bucket # str (s3 mode only) +settings.s3_run_log_bucket # str (s3 mode only) +settings.s3_trash_bucket # str (s3 mode only) +settings.s3_trash_retention_days # int (s3 mode only) +settings.jwt_secret # HS256 secret for the auth_request handler settings.backend_api_url # schedule → backend HTTP base settings.runtime_api_url # backend → runtime HTTP base settings.public_base_url # runtime public base URL @@ -125,12 +126,31 @@ grep -rnE 'os\.(environ\[?["\x27][A-Z_]+|getenv\(["\x27][A-Z_]+)' --include="*.p ### Storage -- All object bytes go to **RustFS** via boto3. -- Use `StorageClient` (HTTP) or `RustFSObjectStore` (direct) — never - the local filesystem. -- `bucket_name` is one of: `RUSTFS_WORKSPACE_BUCKET` (default - `workspaces`), `RUSTFS_VERSION_BUCKET` (default `versions`, - reserved), `RUSTFS_RUN_LOG_BUCKET` (default `run-logs`, reserved). +- All object bytes go through `common.storage.AsyncStorageBackend`, + created by `create_storage(config)` from `common.storage.factory`. +- Two backends are registered: `local` (filesystem, local mode) and + `s3` (S3-compatible service, s3 mode). Selection is per-deployment + via `settings.storage_backend` (`"s3"` default, `"local"` for + dev / single-node / air-gapped). +- The factory helper `build_storage_config(bucket_name)` returns the + right `create_storage` kwargs for each of the 4 purpose buckets + (`workspace`, `version`, `run_log`, `trash`). Use it in lifespan code; + route handlers don't see the difference. +- Bucket resolution from `usage_type` is in **one place** + (`backend/storage_api.py:resolve_bucket`); route handlers only know + about `app.state.object_stores[bucket_name]`. +- The runtime's view of the workspace bucket on disk is exposed by + `common.storage.workspaces_root()`: + - `s3` mode: `${settings.local_storage_base_dir}/workspaces` + (default `/data/workspaces`, the rclone FUSE mount target). + - `local` mode: `${settings.local_storage_base_dir}/workspace` + (default `/data/workspace`, a subdir of the shared local-storage + volume). + `settings.local_storage_base_dir` is the **only** path setting; the + helper handles the per-mode suffix. Don't read `settings.workspaces_root` + or any other path setting directly in runtime code — use this helper. +- The pre-2026 abstraction (`RustFSObjectStore` / `common.storage.client` + / `StorageClient` HTTP wrapper) is gone. Don't reintroduce it. ### Auth @@ -202,9 +222,13 @@ cd frontend && pnpm install && cd .. ```bash # Backend (terminal 1) export DATABASE_URL="mysql+asyncmy://model_platform:model_platform@127.0.0.1:3306/model_platform?charset=utf8mb4" -export RUSTFS_ACCESS_KEY=modelplatform -export RUSTFS_SECRET_KEY=modelplatformsecret -export RUSTFS_ENDPOINT=http://127.0.0.1:9000 +export STORAGE_BACKEND=s3 +export S3_ACCESS_KEY=modelplatform +export S3_SECRET_KEY=modelplatformsecret +export S3_ENDPOINT=http://127.0.0.1:9000 +# Or for local mode: +# export STORAGE_BACKEND=local +# export LOCAL_STORAGE_BASE_DIR=/data uv run --frozen --package backend uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload # Schedule Executor (terminal 2) @@ -246,7 +270,7 @@ PYTHONPATH="backend/src:common/src" uv run --frozen --package backend python -c from backend.main import app from common.config import settings print('backend:', len(app.routes), 'routes') -print('settings ok:', settings.rustfs_endpoint) +print('settings ok:', settings.s3_endpoint) " ``` @@ -284,36 +308,42 @@ See "Adding a new env var" above. uv run --frozen --package backend alembic upgrade head ``` -### Wire a new RustFS bucket +### Wire a new storage bucket -The current three buckets are wired in `backend/storage_api.py:resolve_bucket`: +The current 4 buckets are wired in `backend/storage_api.py:resolve_bucket`: ```python BUCKET_FOR_USAGE: dict[str, str] = { - "working_copy": settings.rustfs_workspace_bucket, - "public_script": settings.rustfs_workspace_bucket, - "data_resource": settings.rustfs_workspace_bucket, - "snapshot": settings.rustfs_workspace_bucket, - "version_artifact": settings.rustfs_version_bucket, - "run_log": settings.rustfs_run_log_bucket, - "run_result": settings.rustfs_run_log_bucket, + "working_copy": settings.s3_workspace_bucket, + "public_script": settings.s3_workspace_bucket, + "data_resource": settings.s3_workspace_bucket, + "snapshot": settings.s3_workspace_bucket, + "version_artifact": settings.s3_version_bucket, + "run_log": settings.s3_run_log_bucket, + "run_result": settings.s3_run_log_bucket, } ``` -To add a fourth bucket: +The constant `PURPOSE_BUCKETS = ("workspace", "version", "run_log", "trash")` +in `common.storage.factory` enumerates the four backends built in the +backend lifespan. To add a fifth bucket: -1. Add the env var to `Settings`: +1. Add the env var to `Settings` (s3 mode only): ```python - rustfs__bucket: str = Field(default="", description="...") + s3__bucket: str = Field(default="", description="...") ``` 2. Add to `.env.example` with a one-line comment. -3. Extend the `Literal` in `common/storage/schemas.py` (in - `ServerObjectRequest.usage_type`, `CreateUploadRequest.usage_type`, - `CompleteUploadRequest.usage_type`) to include the new value. -4. Add an entry in `BUCKET_FOR_USAGE` mapping the new `usage_type` to +3. Append `""` to the `PURPOSE_BUCKETS` tuple in + `common/storage/factory.py`. `build_storage_config("")` + will then automatically read `settings.s3__bucket` (s3 + mode) or use `/` (local mode). +4. Extend the `Literal` in `common/storage/schemas.py` (in + `CreateUploadRequest.usage_type`, `ServerObjectRequest.usage_type`) + to include the new value. +5. Add an entry in `BUCKET_FOR_USAGE` mapping the new `usage_type` to the new bucket env var. -5. Add the bucket to the `ensure_bucket` loop in - `backend/main.py` lifespan. +6. Pre-create the bucket (s3 mode) or subdirectory (local mode) in the + deployment. The backend no longer auto-creates buckets. A workspace's `artifact_bucket` column (when non-null) overrides the default for that workspace, regardless of `usage_type`. @@ -354,7 +384,7 @@ contexts. Add `greenlet>=3.0.0` to `common/pyproject.toml` and Either MySQL isn't running, or the network namespace doesn't allow `mysql:3306` resolution. Inside the Docker network, services reach each other by service name (`mysql`, `backend`, `runtime`, -`schedule`, `rustfs`). +`schedule`, `s3`). ### Jupyter routing 401s diff --git a/README.md b/README.md index b1c0fc8..f955d7f 100644 --- a/README.md +++ b/README.md @@ -5,8 +5,9 @@ an interactive workspace, a DAG scheduler, an object-storage-backed artifact store, and per-workspace runtime isolation — all behind a single Nginx gateway. -> Stack: React Router SPA · FastAPI · APScheduler · MySQL · RustFS (S3) -> · shared Jupyter · FUSE mount via rclone +> Stack: React Router SPA · FastAPI · APScheduler · MySQL · S3-compatible +> storage (or local filesystem via `STORAGE_BACKEND=local`) · shared +> Jupyter · FUSE mount via rclone (s3 mode only) > Single ingress (Nginx :80); all other services are Docker-internal. ## What it does @@ -15,11 +16,11 @@ gateway. |---|---| | Workspace-scoped notebook editing with row-level lock | `backend/jupyter.py` + `scripts.is_locked` | | Authenticated Jupyter routing (browser never sees the runtime token) | `nginx/default.conf` + `auth_request` + `backend/jupyter.py` | -| Object storage for notebooks / scripts / versions / run logs (RustFS, S3 API) | `common/storage/` + `backend/scripts.py` | +| Object storage for notebooks / scripts / versions / run logs (s3 / local toggle) | `common/storage/` + `backend/scripts.py` | | DAG-style scheduling: nodes, edges, cron, manual trigger, retries, snapshots | `backend/schedules.py` + `backend/schedule_runs.py` + `schedule/` (5 modules) | | DAG execution via MySQL Outbox (no Redis, no in-process queues) | `schedule/orchestrator.py` + `schedule/worker.py` | | Per-workspace Jupyter sub-process pool with asyncio locks | `runtime/process.py` | -| rclone FUSE mount of the workspace bucket into the runtime | `runtime/mount.py` | +| rclone FUSE mount of the workspace bucket into the runtime (s3 mode) | `runtime/mount.py` | | MySQL-only persistence (26 tables, soft-delete, no foreign keys) | `common/db/models/` | ## Architecture at a glance @@ -42,13 +43,13 @@ gateway. │ (storage) ├──────────────►│ - subprocess pool │ │ - DAG CRUD │ │ (per workspace) │ │ - script CRUD │ └──────────┬───────────┘ - │ - auth_request │ │ FUSE + │ - auth_request │ │ FUSE / shared vol │ - /api/v1/... │ ▼ └────┬──────┬──────┘ ┌──────────────────────┐ - │ │ │ RustFS (S3) │ - │ └──────── HTTP ───────►│ bucket: workspaces │ - ▼ │ bucket: versions │ - ┌────────────┐ │ bucket: run-logs │ + │ │ │ Object storage │ + │ └──────── HTTP ───────►│ (s3: S3 service / │ + ▼ │ local: shared vol) │ + ┌────────────┐ │ 4 buckets per usage │ │ MySQL │◄───────── poll ─────│ │ │ - 26 tbls │ └──────────────────────┘ │ - outbox │ @@ -65,6 +66,11 @@ gateway. └───────────────────────────────┘ ``` +Object storage is selectable via `STORAGE_BACKEND` (s3 | local). In s3 mode +the 4 purpose-named buckets (`workspaces` / `versions` / `run-logs` / `trash`) +are S3 buckets; in local mode they're subdirectories of `LOCAL_STORAGE_BASE_DIR`, +shared via the `local-storage` Docker volume. See `DEVELOP.md` §Storage. + Detailed design lives in `ARCHITECTURE.md`. Implementation deviations and recent refactors are recorded in `HANDOVER.md`. @@ -90,9 +96,9 @@ default.conf Nginx template (mounted, rendered at start) | Service | Image | Exposed | Purpose | |---|---|---|---| -| `web` | `nginx:alpine` | host `:8888` → `:80` | SPA, `/api/` reverse-proxy, `/jupyter/{ws}/` auth_request proxy, `/storage/` RustFS passthrough | +| `web` | `nginx:alpine` | host `:8888` → `:80` | SPA, `/api/` reverse-proxy, `/jupyter/{ws}/` auth_request proxy, `/storage/` S3 passthrough (s3 mode only) | | `backend` | `Dockerfile` | internal only | DAG CRUD, script CRUD, schedule triggers, `/api/v1/auth/jupyter`, `/internal/v1/*` storage control plane | -| `runtime` | `Dockerfile` | internal only | Per-workspace Jupyter sub-process pool, rclone FUSE mount of `workspaces` bucket | +| `runtime` | `Dockerfile` | internal only | Per-workspace Jupyter sub-process pool, rclone FUSE mount of `workspaces` bucket (s3 mode) | | `schedule` | `Dockerfile` | internal only | Cron tick + DAG execution via MySQL Outbox polling | The architecture **deliberately has only one host port** (the gateway); @@ -103,7 +109,7 @@ all other services are on the Docker internal network. This is enforced in ```bash cp .env.example .env -# Edit .env — at minimum change MYSQL password and RUSTFS credentials. +# Edit .env — at minimum change MYSQL password and (in s3 mode) S3 credentials. # Static check uv sync --all-packages @@ -130,7 +136,7 @@ docker compose logs -f schedule docker compose logs -f runtime ``` -### Tear down (keeps MySQL + RustFS volumes) +### Tear down (keeps MySQL + S3 / local-storage volumes) ```bash docker compose down @@ -157,14 +163,28 @@ See `DEVELOP.md` for the full list of variables and their meanings. ## Storage layout -Three purpose-named RustFS buckets. The mapping from `StorageObjects.usage_type` -to bucket is decided in **one place** (`storage_api.py:resolve_bucket`): +Four purpose-named buckets. The mapping from `StorageObjects.usage_type` +to bucket is decided in **one place** (`backend/storage_api.py:resolve_bucket`): | `usage_type` | Bucket (env var) | Default name | |---|---|---| -| `working_copy`, `public_script`, `data_resource`, `snapshot` | `RUSTFS_WORKSPACE_BUCKET` | `workspaces` | -| `version_artifact` | `RUSTFS_VERSION_BUCKET` | `versions` | -| `run_log`, `run_result` | `RUSTFS_RUN_LOG_BUCKET` | `run-logs` | +| `working_copy`, `public_script`, `data_resource`, `snapshot` | `S3_WORKSPACE_BUCKET` | `workspaces` | +| `version_artifact` | `S3_VERSION_BUCKET` | `versions` | +| `run_log`, `run_result` | `S3_RUN_LOG_BUCKET` | `run-logs` | +| (soft-delete target) | `S3_TRASH_BUCKET` | `trash` | + +In `STORAGE_BACKEND=s3` mode these are 4 separate S3 buckets. In +`STORAGE_BACKEND=local` mode they are 4 subdirectories under +`LOCAL_STORAGE_BASE_DIR` (default `/data`), so the layout above +becomes: + +``` +/data/ +├── workspace/ # S3_WORKSPACE_BUCKET +├── version/ # S3_VERSION_BUCKET +├── run_log/ # S3_RUN_LOG_BUCKET +└── trash/ # S3_TRASH_BUCKET +``` A workspace's `artifact_bucket` column (when non-null) overrides the default for that workspace, regardless of `usage_type` — useful for @@ -174,24 +194,19 @@ The object key is a flat two-level path — `workspace_id` and a server- issued `ulid` for the object: ``` -s3://workspaces/ -└── / - ├── # working_copy / data_resource / snapshot / ... - ├── - └── ... - -s3://versions// # immutable script versions -s3://run-logs// # node run logs and results +//{.} ``` The file name, extension, content type, and logical path live in the -`StorageObjects` and `Scripts` rows, not in the S3 key, so the bucket -can be re-organised without a database rewrite. +`StorageObjects` and `Scripts` rows, not in the object key, so the +storage can be re-organised without a database rewrite. -Backend code never writes to the container's local filesystem. Schedule -Executor stages node artifacts in `tempfile.TemporaryDirectory()` (auto- -cleaned). Only the `runtime` container keeps a host volume — it is required -by the rclone FUSE mount. +Backend code never writes to the container's local filesystem (except +in `STORAGE_BACKEND=local` mode, where the shared `local-storage` volume +is the canonical store). Schedule Executor stages node artifacts in +`tempfile.TemporaryDirectory()` (auto-cleaned). Only the `runtime` +container keeps a host volume — required by the rclone FUSE mount in +s3 mode, and a no-op pass-through in local mode. ## Documentation diff --git a/REFACTOR_NOTES.md b/REFACTOR_NOTES.md index 4b17b93..9eb69f2 100644 --- a/REFACTOR_NOTES.md +++ b/REFACTOR_NOTES.md @@ -28,8 +28,8 @@ ```text mysql -rustfs -jupyter +s3 # 外部 S3-兼容服务（MinIO/RustFS/SeaweedFS/...），由运维在 compose 外启动 +jupyter # 注释保留；当前实现未在 compose 启此独立容器 migrate（一次性） backend runtime @@ -37,6 +37,11 @@ schedule gateway ``` +> 补充说明：`STORAGE_BACKEND=local` 模式下不需要外部 S3 服务，backend +> 与 runtime 共享 docker 卷 `local-storage`，挂载到 +> `LOCAL_STORAGE_BASE_DIR`（默认 `/data/storage`）；runtime 也跳过 +> rclone FUSE 挂载（见 `runtime/src/runtime/mount.py`）。 + ## 已执行检查 - Python 全项目 `compileall` 通过； diff --git a/backend/README.md b/backend/README.md index eea5476..5e943a5 100644 --- a/backend/README.md +++ b/backend/README.md @@ -1,5 +1,8 @@ # Backend 统一 FastAPI 管理服务。包含用户、Workspace、脚本、稳定版本、调度定义、 -立即运行以及 RustFS 对象接口。原 `platform_api` 与 `storage_api` 已在此 +立即运行以及 S3 对象接口。原 `platform_api` 与 `storage_api` 已在此 模块合并，外部 REST 契约保持不变。 + +底层走的是 `common.storage.AsyncStorageBackend` 抽象，按 +`settings.storage_backend` 切换 s3 / local 两种实现。 \ No newline at end of file diff --git a/migrations/data/README.md b/migrations/data/README.md index e9a0a48..352e00b 100644 --- a/migrations/data/README.md +++ b/migrations/data/README.md @@ -65,5 +65,6 @@ python -m migrations.data.migrate_legacy_workspaces --source "/server.py" ## 第 11、12 小步数据处理决定 根据实施确认，第 11、12 小步不迁移旧版资源、脚本或稳定版本数据。新实现直接 -使用 MySQL、Workspace 文件目录和 RustFS，从空的 `data_resources`、`scripts` -及 `versions` 表开始运行。功能验收产生的临时对象和数据库记录均已清理。 +使用 MySQL、Workspace 文件目录和对象存储（s3 模式连 S3-兼容服务，local 模式 +走 `/data/storage` 共享卷），从空的 `data_resources`、`scripts` 及 +`versions` 表开始运行。功能验收产生的临时对象和数据库记录均已清理。