update: README.md

2026-08-14 19:30:25 +08:00
parent 139f2c02c1
commit 76af0f763c
2 changed files with 328 additions and 100 deletions
@@ -1,29 +1,26 @@
 # Model Platform

-A self-hosted **Jupyter-based model development platform** that combines
-an interactive workspace, a DAG scheduler, an object-storage-backed artifact
-store, and per-workspace runtime isolation — all behind a single Nginx
-gateway.
+自托管的 **Jupyter 模型开发平台**：交互式 workspace、DAG 调度、对象存储工件、
+按 workspace 隔离的运行时 — 全部经一个 Nginx 网关对外。

-> Stack: React Router SPA · FastAPI · APScheduler · MySQL · S3-compatible
-> storage (or local filesystem via `STORAGE_BACKEND=local`) · shared
-> Jupyter · FUSE mount via rclone (s3 mode only)
-> Single ingress (Nginx :80); all other services are Docker-internal.
+> 技术栈：React Router SPA · FastAPI · APScheduler · MySQL · S3 兼容存储
+> （或本地文件系统，`STORAGE_BACKEND=local`）· 共享 Jupyter · rclone FUSE 挂载（仅 s3 模式）
+> 单一入口（Nginx :80）；其他服务只在 Docker 内网互通。

-## What it does
+## 它做什么

-| Capability | Where |
+| 能力 | 位置 |
 |---|---|
-| Workspace-scoped notebook editing with row-level lock | `backend/jupyter.py` + `scripts.is_locked` |
-| Authenticated Jupyter routing (browser never sees the runtime token) | `nginx/default.conf` + `auth_request` + `backend/jupyter.py` |
-| Object storage for notebooks / scripts / versions / run logs (s3 / local toggle) | `common/storage/` + `backend/scripts.py` |
-| DAG-style scheduling: nodes, edges, cron, manual trigger, retries, snapshots | `backend/schedules.py` + `backend/schedule_runs.py` + `schedule/` (5 modules) |
-| DAG execution via MySQL Outbox (no Redis, no in-process queues) | `schedule/orchestrator.py` + `schedule/worker.py` |
-| Per-workspace Jupyter sub-process pool with asyncio locks | `runtime/process.py` |
-| rclone FUSE mount of the workspace bucket into the runtime (s3 mode) | `runtime/mount.py` |
-| MySQL-only persistence (26 tables, soft-delete, no foreign keys) | `common/db/models/` |
+| workspace 内 notebook 编辑，行级锁 | `backend/jupyter.py` + `scripts.is_locked` |
+| Jupyter 鉴权路由（浏览器永远拿不到 runtime token） | `nginx/default.conf` + `auth_request` + `backend/jupyter.py` |
+| notebook / script / version / run_log 的对象存储（s3 / local 二选一） | `common/storage/` + `backend/scripts.py` |
+| DAG 调度：节点、边、cron、手动触发、重试、快照 | `backend/schedules.py` + `backend/schedule_runs.py` + `schedule/`（5 个模块） |
+| DAG 执行走 MySQL Outbox（无 Redis，无进程内队列） | `schedule/orchestrator.py` + `schedule/worker.py` |
+| 每个 workspace 一个 Jupyter 子进程池，配 asyncio 锁 | `runtime/process.py` |
+| runtime 内 rclone FUSE 把 workspace 桶挂上来（s3 模式） | `runtime/mount.py` |
+| 仅 MySQL 持久化（26 张表，软删除，无外键） | `common/db/models/` |

-## Architecture at a glance
+## 架构一览

 ```
                        ┌────────────────────┐
@@ -66,69 +63,67 @@ gateway.
   └───────────────────────────────┘
 ```

-Object storage is selectable via `STORAGE_BACKEND` (s3 | local). In s3 mode
-the 4 purpose-named buckets (`workspaces` / `versions` / `run-logs` / `trash`)
-are S3 buckets; in local mode they're subdirectories of `LOCAL_STORAGE_BASE_DIR`,
-shared via the `local-storage` Docker volume. See `DEVELOP.md` §Storage.
+对象存储通过 `STORAGE_BACKEND`（s3 | local）二选一。s3 模式下 4 个 purpose 命名桶
+（`workspace` / `version` / `run-log` / `trash`）是独立的 S3 bucket；local 模式下
+是 `LOCAL_STORAGE_BASE_DIR` 的子目录，通过 Docker volume `local-storage` 共享。
+详见 `DEVELOP.md` §存储。

-Detailed design lives in `ARCHITECTURE.md`. Implementation deviations and
-recent refactors are recorded in `HANDOVER.md`.
+详细设计见 `ARCHITECTURE.md`。实现的偏离和近期重构记录在 `HANDOVER.md`。

-## Repository layout
+## 目录结构

 ```text
 frontend/                    React Router SPA
-backend/                     FastAPI: public API + internal storage API
-runtime/                     Jupyter subprocess manager + rclone FUSE
-schedule/                    DAG scheduler (5 modules: context/scheduler/
-                             orchestrator/worker/service)
-common/                      Settings, SQLAlchemy models, storage SDK,
-                             outbox events, jobstore
-migrations/                  Alembic baseline + per-feature revisions
-nginx/                       (concept only — see "Container" below)
-scripts/                     nginx-entrypoint.sh (template renderer)
-docker-compose.yml           4 services — web / backend / runtime / schedule
-default.conf                 Nginx template (mounted, rendered at start)
-.env.example                 All env vars consumed by common.config.Settings
+backend/                     FastAPI：公开 API + 内部存储 API
+runtime/                     Jupyter 子进程管理 + rclone FUSE
+schedule/                    DAG 调度器（5 模块：context/scheduler/
+                             orchestrator/worker/service）
+common/                      配置、SQLAlchemy 模型、存储 SDK、
+                             outbox 事件、jobstore
+migrations/                  Alembic 基线 + 各特性 migration
+nginx/                       （仅概念 — 见下方「容器」一节）
+scripts/                     nginx-entrypoint.sh（模板渲染）
+docker-compose.yml           4 服务 — web / backend / runtime / schedule
+default.conf                 Nginx 模板（挂载，启动时渲染）
+.env.example                 common.config.Settings 消费的所有环境变量
 ```

-## Containers
+## 容器

-| Service | Image | Exposed | Purpose |
+| 服务 | 镜像 | 暴露 | 用途 |
 |---|---|---|---|
-| `web` | `nginx:alpine` | host `:8888` → `:80` | SPA, `/api/` reverse-proxy, `/jupyter/{ws}/` auth_request proxy, `/storage/` S3 passthrough (s3 mode only) |
-| `backend` | `Dockerfile` | internal only | DAG CRUD, script CRUD, schedule triggers, `/api/v1/auth/jupyter`, `/internal/v1/*` storage control plane |
-| `runtime` | `Dockerfile` | internal only | Per-workspace Jupyter sub-process pool, rclone FUSE mount of `workspaces` bucket (s3 mode) |
-| `schedule` | `Dockerfile` | internal only | Cron tick + DAG execution via MySQL Outbox polling |
+| `web` | `nginx:alpine` | 宿主机 `:8888` → `:80` | SPA、`/api/` 反向代理、`/jupyter/{ws}/` auth_request 代理、`/storage/` S3 直通（仅 s3 模式） |
+| `backend` | `Dockerfile` | 仅内网 | DAG CRUD、script CRUD、schedule 触发、`/api/v1/auth/jupyter`、`/internal/v1/*` 存储控制面 |
+| `runtime` | `Dockerfile` | 仅内网 | 每个 workspace 一个 Jupyter 子进程池、rclone FUSE 挂载 `workspace` 桶（s3 模式） |
+| `schedule` | `Dockerfile` | 仅内网 | cron tick + DAG 执行（轮询 MySQL Outbox） |

-The architecture **deliberately has only one host port** (the gateway);
-all other services are on the Docker internal network. This is enforced in
-`docker-compose.yml` — no `ports:` on backend / runtime / schedule.
+架构**故意只暴露一个宿主机端口**（网关）；其他服务都在 Docker 内网。
+这一点在 `docker-compose.yml` 里强制执行 — backend / runtime / schedule 都没有 `ports:`。

-## Quick start
+## 快速启动

 ```bash
 cp .env.example .env
-# Edit .env — at minimum change MYSQL password and (in s3 mode) S3 credentials.
+# 编辑 .env — 至少改 MYSQL 密码，以及（s3 模式下）S3 凭据。

-# Static check
+# 静态检查
 uv sync --all-packages
 uv run --frozen --package backend python -m compileall -q backend/src common/src
 uv run --frozen --package schedule python -m compileall -q schedule/src
 uv run --frozen --package runtime python -m compileall -q runtime/src

-# Apply schema
+# 应用 schema
 uv run --frozen --package backend alembic upgrade head

-# Bring up the stack
-docker compose config          # validate
+# 启动整套服务
+docker compose config          # 校验
 docker compose up -d --build
 docker compose ps
 ```

-Visit `http://localhost:8888`.
+访问 `http://localhost:8888`。

-### Logs
+### 日志

 ```bash
 docker compose logs -f backend
@@ -136,47 +131,43 @@ docker compose logs -f schedule
 docker compose logs -f runtime
 ```

-### Tear down (keeps MySQL + S3 / local-storage volumes)
+### 停服（保留 MySQL + S3 / local-storage 数据卷）

 ```bash
 docker compose down
 ```

-### Wipe data
+### 抹数据

 ```bash
 docker compose down -v
 ```

-## Configuration
+## 配置

-All environment variables are declared once in `common/src/common/config.py`
-as a pydantic-settings `Settings` class, with a `@lru_cache` singleton.
-Adding a new env var:
+所有环境变量在 `common/src/common/config.py` 里用 pydantic-settings 的 `Settings`
+类一次性声明，外面套一层 `@lru_cache` 单例。新增环境变量：

-1. Add the field to `Settings` in `common/src/common/config.py` (with a
-   sensible default so dev-env "just works").
-2. Add the line to `.env.example` with a comment.
-3. Use `settings.<name>` at the call site. Never `os.environ["..."]`.
+1. 在 `common/src/common/config.py` 的 `Settings` 里加字段（带合理 default，使 dev 启动不需要设）
+2. 在 `.env.example` 加一行带注释
+3. 调用点用 `settings.<name>`，永远不要用 `os.environ["..."]`

-See `DEVELOP.md` for the full list of variables and their meanings.
+完整环境变量列表和含义见 `DEVELOP.md`。

-## Storage layout
+## 存储布局

-Four purpose-named buckets. The mapping from `StorageObjects.usage_type`
-to bucket is decided in **one place** (`backend/storage_api.py:resolve_bucket`):
+4 个 purpose 命名桶。从 `StorageObjects.usage_type` 到桶的映射由
+**单一入口**（`backend/storage_api.py:resolve_bucket`）决定：

-| `usage_type` | Bucket (env var) | Default name |
+| `usage_type` | 桶（环境变量） | 默认名 |
 |---|---|---|
-| `working_copy`, `public_script`, `data_resource`, `snapshot` | `S3_WORKSPACE_BUCKET` | `workspaces` |
-| `version_artifact` | `S3_VERSION_BUCKET` | `versions` |
-| `run_log`, `run_result` | `S3_RUN_LOG_BUCKET` | `run-logs` |
-| (soft-delete target) | `S3_TRASH_BUCKET` | `trash` |
+| `working_copy`、`public_script`、`data_resource`、`snapshot` | `S3_WORKSPACE_BUCKET` | `workspace` |
+| `version_artifact` | `S3_VERSION_BUCKET` | `version` |
+| `run_log`、`run_result` | `S3_RUN_LOG_BUCKET` | `run-log` |
+| （软删除目标） | `S3_TRASH_BUCKET` | `trash` |

-In `STORAGE_BACKEND=s3` mode these are 4 separate S3 buckets. In
-`STORAGE_BACKEND=local` mode they are 4 subdirectories under
-`LOCAL_STORAGE_BASE_DIR` (default `/data`), so the layout above
-becomes:
+`STORAGE_BACKEND=s3` 模式下是 4 个独立 S3 桶。`STORAGE_BACKEND=local` 模式下
+是 `LOCAL_STORAGE_BASE_DIR`（默认 `/data`）下的 4 个子目录：

 ```
 /data/
@@ -186,36 +177,31 @@ becomes:
 └── trash/       # S3_TRASH_BUCKET
 ```

-A workspace's `artifact_bucket` column (when non-null) overrides the
-default for that workspace, regardless of `usage_type` — useful for
-isolating a paying customer to their own bucket.
+某个 workspace 的 `artifact_bucket` 列（非 NULL 时）覆盖该 workspace 的默认桶，
+无视 `usage_type` — 适合把付费客户隔离到专属桶。

-The object key is a flat two-level path — `workspace_id` and a server-
-issued `ulid` for the object:
+对象 key 是两层扁平路径 — `workspace_id` 加服务端签发的 `ulid`：

 ```
 <workspace_bucket>/<workspace_id>/<ulid>{.<ext>}
 ```

-The file name, extension, content type, and logical path live in the
-`StorageObjects` and `Scripts` rows, not in the object key, so the
-storage can be re-organised without a database rewrite.
+文件名、扩展名、MIME、逻辑路径都放在 `StorageObjects` 和 `Scripts` 行里，不进
+object key — 重新组织存储不需要重写数据库。

-Backend code never writes to the container's local filesystem (except
-in `STORAGE_BACKEND=local` mode, where the shared `local-storage` volume
-is the canonical store). Schedule Executor stages node artifacts in
-`tempfile.TemporaryDirectory()` (auto-cleaned). Only the `runtime`
-container keeps a host volume — required by the rclone FUSE mount in
-s3 mode, and a no-op pass-through in local mode.
+Backend 代码从不写容器本地文件系统（`STORAGE_BACKEND=local` 模式除外，那里共享
+`local-storage` volume 就是规范存储）。Schedule Executor 在 `tempfile.TemporaryDirectory()`
+里暂存节点工件（自动清理）。只有 `runtime` 容器保留宿主 volume — s3 模式下 rclone FUSE
+挂载需要；local 模式下是 no-op 透传。

-## Documentation
+## 文档

- `README.md` (this file) — quick orientation
- `ARCHITECTURE.md` — design diagrams + simplification history
- `HANDOVER.md` — implementation deviations, recent refactors, pending work
- `DEVELOP.md` — developer guide (env vars, code conventions, common tasks)
- `CLAUDE.md` — agent-facing conventions for the repo
+- `README.md`（本文）— 快速导读
+- `ARCHITECTURE.md` — 设计图 + 简化历史
+- `HANDOVER.md` — 实现偏离、近期重构、待办事项
+- `DEVELOP.md` — 开发指南（环境变量、代码规约、常用操作）
+- `CLAUDE.md` — agent 面向的本仓库规约

-## License
+## 许可

-Internal.
+内部。