feat(scripts): 跨 owner 懒加载目录树 + 跨用户可见 workspace/public
修两个后端接口问题:
1) /api/v1/workspace-directories 返回为空,目录树结构消失
2) 同 workspace 内脚本/数据互相可见但默认排除 private
后端改动
--------
* list_scripts / list_resources / list_workspace_directories 新增
owner_user_id 可选 query 参数;缺省 = 当前请求者本人(scope 到
workspace/{me}/...),传值时 scope 到该 owner 的子树。前端根加载
默认只见自己一级,其他成员以折叠分组呈现。
* visibility 过滤统一:非 admin 请求者只返回 owner==me 或
visibility ∈ {workspace, public};admin 跳过。owner=me 含自己
的 private,owner=other 只剩其 workspace/public,排除他人 private。
* create_workspace_directory 两个分支 visibility 默认 'public'
(非 private),使跨 owner 目录树可见;响应新增 owner_user_id 字段。
* platform.list_members 鉴权从 system_admin_context 放宽为
系统管理员或该 workspace 活跃成员(让普通用户也能渲染同
workspace 成员名册,用于跨 owner 分组)。
* main.py 注册 platform 模块(随 list_members 改动补齐导入)。
* .env.example 同步 common/config.py 26 个字段。
前端改动
--------
* ScriptExplorer.memberScriptGroups 改由 members 列表播种分组,
display_name 取 members.display_name;inferredDirectories 现在按
owner_user_id 标记,统一跨 owner 目录渲染。删除脚本目录页头与
树分组标题的工作副本数量角标。
* WorkspaceTree 新增 ownerUserId 透传到 store.toggleExpanded;
仅"我"的分组 mount 时 auto-expand,他人分组默认折叠,展开才
调 loadOwnerGroup / owner-scoped loadScripts / loadChildren。
* scriptWorkspaceStore 引入 namespaced cache key
(ownerCacheKey = `${ownerUserId ?? me}:${path}`),loadedScriptPaths
/ loadedChildPaths / loadedOwnerGroups 全部按 owner 隔离;
toggleExpanded 用 loadPath === undefined 区分 group 头与真实
目录,修"他人子目录点击不触发接口"的 loadPath 前缀误判 bug。
* api.ts / AuthContext 透传 ownerUserId 给 listScripts /
listResources / listWorkspaceDirectories。
文档
----
* API.md: §3.2 创建目录 visibility 默认 public + 响应加 owner_user_id;
§3.3.1 GET directories 加 owner_user_id 参数 + 响应字段;
§3.4 GET scripts 改写为 owner 作用域 + visibility 过滤语义;
§五.1 GET data-resources 新增,同一套统一语义;
§7 intro 例外 — GET members 对系统管理员或 workspace 活跃成员开放。
* DEVELOP.md: Code layout 重写以反映 backend api/services/clients/
schemas 拆分 + schedule domain/scheduling/application/execution/
infrastructure 拆分 + common 子包(auth/storage/backends);
Configuration 系统补全 26 个 settings 字段;新增
"Owner-scoping + visibility (cross-owner browsing)" 小节;
Per-service dev 注释用 uv run 的源布局要求;Add a new DAG endpoint /
storage bucket 路径改为 backend/src/backend/api/* 与 services/*。
测试
----
* test_list_scripts_parent_path.py /
test_resources.py 补充 owner_user_id 参数化直接调用 + LIKE
前缀断言(workspace/{owner}/... 前缀)。
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
tao.chen
+137
-61
@@ -6,58 +6,89 @@ refactors see `HANDOVER.md`.
|
||||
|
||||
## Code layout
|
||||
|
||||
All Python packages use the `src/<pkg>/` layout; `uv` workspace glues them into
|
||||
one `.venv`. Always invoke via `uv run [--package <pkg>] <cmd>` (see "Local
|
||||
development" for the gotcha).
|
||||
|
||||
```
|
||||
common/ Pure-Python shared library
|
||||
config.py Settings (pydantic-settings, lru_cache singleton)
|
||||
db/ SQLAlchemy 2.0 async engine, session_scope, Base
|
||||
db/models/ 26 tables in 9 domain files (zero FK, zero relationship)
|
||||
scheduler/ build_sqlalchemy_jobstore (delayed import)
|
||||
storage/ AsyncStorageBackend abstraction (s3 + local impls) + Pydantic schemas
|
||||
eventing.py add_outbox_event / utcnow / event_time
|
||||
service_app.py /health/ready TCP probe, /api/v1/health
|
||||
schemas.py StrictModel base
|
||||
utils.py get_free_port, start_process
|
||||
common/src/common/ Pure-Python shared library
|
||||
config.py Settings (pydantic-settings, lru_cache singleton)
|
||||
db/ SQLAlchemy 2.0 async engine, session_scope, Base
|
||||
db/models/ 26 tables in 9 domain files (zero FK, zero relationship)
|
||||
auth/ JWT / bcrypt / workspace membership helpers
|
||||
scheduler/ APScheduler trigger helpers (delayed import)
|
||||
storage/ AsyncStorageBackend abstraction + Pydantic schemas
|
||||
base.py Abstract interface
|
||||
factory.py create_storage + build_storage_config + PURPOSE_BUCKETS
|
||||
schemas.py CreateUploadRequest / ServerObjectRequest
|
||||
backends/local.py Local filesystem impl
|
||||
backends/s3.py S3-compatible impl (boto3)
|
||||
registry.py Bucket registry
|
||||
eventing.py add_outbox_event / utcnow / event_time
|
||||
service_app.py /health/ready TCP probe, /api/v1/health
|
||||
logging.py loguru config (LOG_LEVEL)
|
||||
schemas.py StrictModel base
|
||||
ids.py ULID generation helpers
|
||||
utils.py get_free_port, start_process
|
||||
|
||||
backend/ Public FastAPI service + tiny /internal/v1/objects RPC
|
||||
main.py lifespan + route registration
|
||||
jupyter.py /api/v1/auth/jupyter — the ONLY auth entry
|
||||
scripts.py CRUD for scripts/notebooks (object storage via AsyncStorageBackend)
|
||||
schedules.py DAG CRUD: schedules, nodes, edges
|
||||
schedule_runs.py Trigger / list / get runs
|
||||
schedule_schemas.py Pydantic request/response models
|
||||
admin.py Admin endpoints
|
||||
resources.py Misc data resources
|
||||
storage_api.py /internal/v1/objects — single token-guarded endpoint (P0-1)
|
||||
storage_client.py Stub (HTTP client removed post-migration; rewrite pending)
|
||||
schedule_client.py Placeholder module (was the HTTP-push executor client)
|
||||
runtime_client.py Self-contained httpx wrapper for the runtime
|
||||
jupyter.py auth_request handler
|
||||
dependencies.py request_context, database_session
|
||||
backend/src/backend/ Public FastAPI service
|
||||
main.py lifespan + route registration
|
||||
audit.py HTTP access log middleware (loguru sink)
|
||||
api/ HTTP route handlers (one module per bounded context)
|
||||
auth.py /api/v1/auth/* (login / me / jupyter)
|
||||
jupyter.py /api/v1/auth/jupyter — the ONLY auth entry
|
||||
dependencies.py request_context, database_session
|
||||
platform.py /api/v1/platform/* (system admin)
|
||||
admin.py /api/v1/admin/* (workspace-internal admin)
|
||||
scripts.py /api/v1/scripts/* + /api/v1/workspace-directories
|
||||
resources.py /api/v1/data-resources/*
|
||||
schedules/schedules.py DAG CRUD
|
||||
schedules/runs.py Run lifecycle
|
||||
storage.py /internal/v1/objects — single token-guarded endpoint (P0-1)
|
||||
services/ Pure-Python business logic (no HTTP / no DI)
|
||||
scripts.py create_workspace_directory, visibility-filtered queries
|
||||
resources.py owner-scoped resource listing helpers
|
||||
jupyter.py jupyter_path / lock helpers
|
||||
storage.py object store helpers
|
||||
schedules.py DAG validation (cycle / orphan detection)
|
||||
schemas/ Pydantic request / response models
|
||||
auth.py / common.py / jupyter.py / platform.py / resources.py / schedules.py / scripts.py
|
||||
clients/ Outbound HTTP / RPC clients
|
||||
runtime.py Self-contained httpx wrapper for the runtime
|
||||
scheduler.py Backend → Schedule HTTP client (callback / dispatch)
|
||||
rclone.py rclone RC API client (FUSE cache invalidation)
|
||||
|
||||
schedule/ Schedule Executor (DAG worker)
|
||||
context.py Constants + naive_utc
|
||||
scheduler.py CronScheduler (APScheduler + 5s sync loop)
|
||||
orchestrator.py DispatchOrchestrator (Outbox poll + DAG advance)
|
||||
worker.py NodeExecutor (notebook / python execution)
|
||||
service.py SchedulerService facade (composes the three)
|
||||
main.py Lifespan + FastAPI app
|
||||
storage_client.py SchedulerStorageClient — talks to backend /internal/v1/objects
|
||||
execution.py execute_artifact (notebook + python paths)
|
||||
notebook_runner.py Subprocess entry point (nbclient)
|
||||
schedule/src/schedule/ Schedule Executor (DAG worker)
|
||||
main.py Lifespan + FastAPI app
|
||||
notebook_runner.py Subprocess entry point (nbclient) — DO NOT RENAME
|
||||
domain/ Pure-Python domain types
|
||||
execution.py ExecutionResult (frozen dataclass) + state enums
|
||||
context.py Constants + naive_utc
|
||||
scheduling/ Time-based trigger
|
||||
scheduler.py CronScheduler (APScheduler + 5s sync loop)
|
||||
application/ Facades / orchestrators
|
||||
service.py SchedulerService (composes the three)
|
||||
orchestrator.py DispatchOrchestrator (Outbox poll + DAG advance)
|
||||
execution/ DAG node execution
|
||||
executor.py NodeExecutor (notebook / python dispatch)
|
||||
worker.py asyncio entry, schedule-spawned task boundary
|
||||
runners/notebook.py nbclient subprocess path (6-line `notebook_runner` shim re-exports `main`)
|
||||
infrastructure/ External-system adapters
|
||||
storage/client.py SchedulerStorageClient — talks to backend /internal/v1/objects
|
||||
|
||||
runtime/ Jupyter Runtime
|
||||
main.py FastAPI entry: jupyter action endpoints
|
||||
process.py Per-workspace subprocess pool + asyncio locks
|
||||
mount.py rclone FUSE mount lifecycle
|
||||
runtime/src/runtime/ Jupyter Runtime
|
||||
main.py FastAPI entry: jupyter action endpoints
|
||||
process.py Per-workspace subprocess pool + asyncio locks
|
||||
mount.py rclone FUSE mount lifecycle
|
||||
|
||||
frontend/ React Router SPA (vite build → nginx)
|
||||
app/ features/ routes/ services/ components/
|
||||
frontend/ React Router SPA (vite build → nginx)
|
||||
app/ features/ routes/ services/ components/
|
||||
|
||||
migrations/ Alembic schema versions
|
||||
docker-compose.yml 4 services
|
||||
default.conf Nginx template
|
||||
migrations/ Alembic schema versions
|
||||
docker-compose.yml 4 services (gateway / backend / schedule / runtime)
|
||||
default.conf Nginx template
|
||||
scripts/nginx-entrypoint.sh
|
||||
.env.example
|
||||
.env.example All 26 config.py keys documented
|
||||
```
|
||||
|
||||
## Configuration system
|
||||
@@ -67,10 +98,31 @@ All env vars go through one place: `common/src/common/config.py`.
|
||||
```python
|
||||
from common.config import settings
|
||||
|
||||
settings.database_url # str
|
||||
settings.storage_backend # str: "s3" (default) or "local"
|
||||
settings.local_storage_base_dir # str: root dir for storage data (default "/data"); see "Storage" below for per-mode derivation
|
||||
settings.s3_endpoint # str (full URL, e.g. "http://s3:9000"; s3 mode only)
|
||||
# Auth / runtime
|
||||
settings.database_url # str — SQLAlchemy async URL (mysql+asyncmy, charset utf8mb4)
|
||||
settings.jwt_secret # HS256 secret for the auth_request handler
|
||||
settings.cookie_force_secure # bool — write Secure flag even on plain HTTP (TLS-terminating proxy)
|
||||
settings.service_name # surfaced in /health
|
||||
settings.schedule_event_namespace # APScheduler JobStore namespace + Outbox scope prefix
|
||||
settings.readiness_targets # CSV host:port list for /health/ready
|
||||
|
||||
# HTTP clients (intra-cluster URLs)
|
||||
settings.runtime_api_url # backend → runtime HTTP base
|
||||
settings.public_base_url # runtime public base URL (browser-facing /jupyter/)
|
||||
settings.backend_api_url # schedule → backend HTTP base
|
||||
settings.rclone_rc_url # backend → rclone RC control API
|
||||
settings.internal_service_token # Backend ↔ Schedule shared secret (X-Internal-Service-Token)
|
||||
|
||||
# Logging / audit
|
||||
settings.log_level # DEBUG / INFO / WARNING / ERROR / CRITICAL (lowercase → fallback INFO)
|
||||
settings.audit_log_dir # dir for daily audit logs (relative to cwd; "" disables file sink)
|
||||
settings.audit_log_retention_days # 0 disables cleanup
|
||||
settings.audit_excluded_paths # list[str] — paths skipped from audit (health probes, etc.)
|
||||
|
||||
# Storage
|
||||
settings.storage_backend # "s3" (default) or "local"
|
||||
settings.local_storage_base_dir # root dir for storage data (default "/data")
|
||||
settings.s3_endpoint # str (s3 mode only)
|
||||
settings.s3_access_key # str (s3 mode only)
|
||||
settings.s3_secret_key # str (s3 mode only)
|
||||
settings.s3_workspace_bucket # str (s3 mode only)
|
||||
@@ -78,17 +130,15 @@ settings.s3_version_bucket # str (s3 mode only)
|
||||
settings.s3_run_log_bucket # str (s3 mode only)
|
||||
settings.s3_trash_bucket # str (s3 mode only)
|
||||
settings.s3_trash_retention_days # int (s3 mode only)
|
||||
settings.jwt_secret # HS256 secret for the auth_request handler
|
||||
settings.backend_api_url # schedule → backend HTTP base
|
||||
settings.runtime_api_url # backend → runtime HTTP base
|
||||
settings.public_base_url # runtime public base URL
|
||||
settings.service_name # surfaced in /health
|
||||
settings.readiness_targets # CSV host:port list for /health/ready
|
||||
|
||||
# Schedule
|
||||
settings.schedule_execution_concurrency # int — max concurrent notebook subprocesses
|
||||
```
|
||||
|
||||
`Settings` reads from process env first, then from a `.env` file at CWD
|
||||
if present. `pydantic-settings` auto-loads. `case_sensitive=False` so
|
||||
`DATABASE_URL` / `database_url` both work.
|
||||
`DATABASE_URL` / `database_url` both work. The full list of 26 fields
|
||||
is in `common/src/common/config.py`.
|
||||
|
||||
### Adding a new env var
|
||||
|
||||
@@ -96,7 +146,8 @@ if present. `pydantic-settings` auto-loads. `case_sensitive=False` so
|
||||
```python
|
||||
new_var: str = Field(default="x", description="...")
|
||||
```
|
||||
2. Add the line to `.env.example` with a comment.
|
||||
2. Add the line to `.env.example` with a comment (keep it synced — every
|
||||
field in config.py must have a matching `.env.example` entry).
|
||||
3. Use `settings.new_var` at the call site.
|
||||
|
||||
Do **not** call `os.environ["NEW_VAR"]` or `os.getenv("NEW_VAR")` in
|
||||
@@ -190,13 +241,34 @@ grep -rnE 'os\.(environ\[?["\x27][A-Z_]+|getenv\(["\x27][A-Z_]+)' --include="*.p
|
||||
|
||||
- For write operations on a script/notebook, call
|
||||
`require_script_modify_access(script, user_id=..., is_admin=...)`
|
||||
from `backend/scripts.py`. It enforces:
|
||||
from `backend/src/backend/services/scripts.py`. It enforces:
|
||||
- admin or owner → allow
|
||||
- non-owner, `is_locked == 0` → allow
|
||||
- non-owner, `is_locked == 1` → 403
|
||||
- Read endpoints (`list_scripts`, `get_script`) intentionally do **not**
|
||||
check `is_locked` — workspace members can see the script list.
|
||||
|
||||
### Owner-scoping + visibility (cross-owner browsing)
|
||||
|
||||
`GET /api/v1/scripts`, `GET /api/v1/data-resources`, and
|
||||
`GET /api/v1/workspace-directories` all accept an optional
|
||||
`owner_user_id` query param and follow the same model:
|
||||
|
||||
- `owner_user_id` 缺省 = 当前请求者本人(scope to `workspace/{me}/...`).
|
||||
- 传值时 scope 到 `workspace/{owner_user_id}/...`,用于前端"点开其他成员
|
||||
分组"的懒加载(见 §3.4 of `API.md`).
|
||||
- `visibility` 过滤(非 admin):`owner_user_id == me OR visibility IN
|
||||
(workspace, public)` — 自己可见自己全部(含 private),他人只见其
|
||||
workspace/public,排除他人 private.
|
||||
- 系统管理员跳过 visibility 过滤.
|
||||
- 目录行默认 `visibility='public'`(由 `create_workspace_directory` 写入),
|
||||
不施加 visibility 过滤,使跨 owner 目录树可见.
|
||||
|
||||
实现 helper 在 `backend/src/backend/services/scripts.py`
|
||||
(`_build_list_scripts_owner_descendant_prefix`) 和
|
||||
`backend/src/backend/api/resources.py` 中按相同模式分别构造 owner-scoped
|
||||
LIKE 前缀。
|
||||
|
||||
### Outbox events
|
||||
|
||||
- The platform's only async-messaging fabric is the MySQL
|
||||
@@ -238,6 +310,10 @@ cd frontend && pnpm install && cd ..
|
||||
|
||||
### Per-service dev
|
||||
|
||||
Always go through `uv run` so the workspace `.venv` is used — bare
|
||||
`uvicorn` / `python` resolves to system Python and `from backend.X`
|
||||
imports fail with ModuleNotFoundError.
|
||||
|
||||
```bash
|
||||
# Backend (terminal 1)
|
||||
export DATABASE_URL="mysql+asyncmy://model_platform:model_platform@127.0.0.1:3306/model_platform?charset=utf8mb4"
|
||||
@@ -248,13 +324,13 @@ export S3_ENDPOINT=http://127.0.0.1:9000
|
||||
# Or for local mode:
|
||||
# export STORAGE_BACKEND=local
|
||||
# export LOCAL_STORAGE_BASE_DIR=/data
|
||||
uv run --frozen --package backend uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload
|
||||
uv run --package backend uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload
|
||||
|
||||
# Schedule Executor (terminal 2)
|
||||
uv run --frozen --package schedule uvicorn schedule.main:app --host 0.0.0.0 --port 8001 --reload
|
||||
uv run --package schedule uvicorn schedule.main:app --host 0.0.0.0 --port 8001 --reload
|
||||
|
||||
# Runtime (terminal 3 — needs SYS_ADMIN, FUSE, devmode)
|
||||
uv run --frozen --package runtime python -m runtime.main
|
||||
uv run --package runtime python -m runtime.main
|
||||
```
|
||||
|
||||
### Frontend dev
|
||||
|
||||
Reference in New Issue
Block a user