Develop #42
Assignees
tao.chen (tao chen)
Clear assignees
tao.chen (tao chen)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: tao.chen/model-platform#42
Reference in New Issue
Block a user
Backend — backend/src/backend/scripts.py - list_scripts accepts optional parent_path Query (default ""). - Extracts _build_list_scripts_descendant_prefix helper for the "direct children of parent_path" prefix. - WHERE clause now adds: StorageObjects.relative_path LIKE '<prefix>/%' AND NOT LIKE '<prefix>/%/%' so deeper descendants and prefix-siblings (foo/bar vs foo/bar2) are excluded. Empty parent_path filters to root-level only — this is the symmetric, intent-aligned behavior the lazy-load frontend relies on. - EXPLAIN confirms idx_scripts_workspace drives the scripts table; storage_objects PK lookup applies the LIKE filter per row. Frontend — frontend/app/services/api.ts + scriptWorkspaceStore.ts - listScripts accepts optional parentPath; built URL preserves the new filter param. - Store gains loadedScriptPaths / loadingScriptPaths Sets and a loadScripts(parentPath) action: idempotent, in-flight dedupe, dedup-by-id when merging into the flat scripts array so existing find() callers keep working. - load() now uses listScripts("") instead of bulk fetch — root only on first paint. - toggleExpanded() triggers loadScripts(parentPath) in parallel with loadChildren(parentPath) so folder expansion loads both sub-directories and direct-child scripts. Tests — backend/tests/test_list_scripts_parent_path.py - 7 unit tests: prefix construction, normalization, traversal rejection, and SQL compilation contract (LIKE prefix + NOT LIKE prefix + scripts.status filter). Verified: - pytest backend/tests: 50 passed (43 existing + 7 new) - pnpm typecheck: clean - alembic: no schema changes (migration-less feature) - EXPLAIN with real workspace: idx_scripts_workspace → PK lookupCodex (deepseek-v4-flash) review of feat(scripts) parent_path filter surfaced four problems: 1. Critical — AuthContext listScripts binding silently dropped parentPath - frontend/app/context/AuthContext.tsx:224 had: listScripts: () => rawApi.listScripts(workspaceId) so loadScripts("foo/bar") hit GET /api/v1/scripts with no query and always received root-level scripts. Typecheck passed because `() => ...` is assignable to `(parentPath?: string) => ...`. - Fix: forward the parentPath argument. 2. High — load() cache invalidation gap on toolbar refresh / create / delete - load() replaced `scripts` with root-only items but never invalidated `loadedScriptPaths` for subfolders, so previously-expanded folders rendered empty (cached no-op on re-expand) and open tabs pointing into subfolders were dropped by the validIds filter. - Fix: load() now re-fetches every path currently in loadedScriptPaths (and loadedChildPaths for directories), then dedups by id/path. On initial mount the cache is empty so this degrades to a single root fetch. 3. Low — test docstring overclaimed coverage - The "actual ORM roundtrip is covered by the existing integration tests" line is false (no other list_scripts test exists). - Fix: honest docstring noting the repo has no endpoint integration test layer; SQL assertions are brittle to SQLAlchemy/dialect formatting. 4. Low — backend docstring overclaimed index role - Claimed `idx_storage_workspace_relative_path` "avoided全表扫", but the index isn't declared in the ORM model, only exists via the baseline migration's upgrade path, and EXPLAIN doesn't drive through it (scripts-first plan via idx_scripts_workspace). - Fix: accurate description — MySQL drives via idx_scripts_workspace, storage_objects PK lookup applies LIKE per row. Notes where to optimize if 10万-scale perf becomes a real problem. Skipped findings (out of scope): - Medium LIKE escape for `_` / `%` (pre-existing in list_workspace_directories; not introduced here). - Low dashboard `scripts.length` count underreport (separate UI bug, pre-existing assumption that broke under the new semantics). Verified: pytest 50 passed, pnpm typecheck clean.ffec234follow-up) a9b682c1a1- Backend: GET /api/v1/data-resources accepts parent_path; LIKE '{ws_id}/%/{escaped}/%' AND NOT LIKE '{ws_id}/%/{escaped}/%/%' on StorageObjects.object_key (workspace-wide, escapes _ and %, mirrors list_scripts parent_path semantics). 13 new tests in test_resources.py (helper unit / SQL compile / SQLite behavioral). - Frontend: listResources gains parentPath arg, propagated through WorkspaceBoundApi + AuthContext binding. WorkspaceTreeGroup title count and ScriptExplorer header count now include dataResources. memberScriptGroups backfills data-only owners so users with only data resources still render a group. loadDataResources accepts an optional parentPath, default empty preserves prior behavior.list_resources 已放开成 workspace-wide，但 can_view 仍要求 visibility in {workspace, public} 或 admin，导致 A 的 private 资源出现在 B 的列表里、但 B 调 GET /{id}、POST /{id}/download-url 与 POST /{id}/jupyter-relative-path 全部 404。 can_view 改为恒真，访问边界全部交给 get_visible_resource 的 workspace_id + status=active SQL 限定；visibility 字段保留为 上传/绑定时的语义标签。delete_resource 仍按 owner/admin 校验 403，不受影响。补 4 个 unit test。按用户复查意见把审计折进 main.py 已有的 access_log 中间件， 少一个 middleware、让诊断与合规共用一个出口。 * 删 backend.audit.AuditMiddleware（user_id 解析搬到 main.py 的 _audit_user_id 模块级 helper，cookie/Bearer 头 + JWT 验签，失败/缺失一律 '-'，全异常捕获不让审计拖死业务）。 * access_log 在 success 路径补 logger.bind(user_id, method, path, status).info('audit')；exception 路径同样补一条 status=500 的审计行，然后 re-raise 让全局 handler 转 500。 * 删 app.add_middleware(AuditMiddleware)。 * audit.py 只剩 _DailyFileSink / configure_audit_logging / AUDIT_LOG_FORMAT，文件 sink 与 retention 清理逻辑不变。 * 测试改用 _AccessLogReplica 复制 access_log 审计契约（不 import main.py 避免触发 lifespan 里的 MySQL/engine 初始化）， 删 3 个 middleware 单独 case，加 access_log 端到端 success / 5xx / 无 JWT 三个 case。Stage 3 of the layered refactor. Relocate the two scheduling components into their own package so that domain / application / scheduling / execution / infrastructure boundaries actually exist on disk. - Add schedule/src/schedule/scheduling/__init__.py - Move scheduler.py (232 lines) -> scheduling/scheduler.py (byte-identical via diff; CronScheduler class name unchanged) - Move orchestrator.py (946 lines) -> scheduling/orchestrator.py (byte-identical via diff; DispatchOrchestrator + event constants unchanged; NOT further split this round, per plan) - service.py lines 39-40: import paths rewritten to the new module - tests/test_janitor.py: rewrite the import + 5 patch() string targets The 5 patch() targets ("schedule.orchestrator.session_scope" x3, "schedule.orchestrator.asyncio.sleep" x2) were NOT caught by the import-line grep — they patch module attributes at runtime and would have become dead no-ops after the move (and would hard-raise once the old module is deleted in stage 6). Rewriting them to "schedule.scheduling.orchestrator.*" keeps the janitor tests meaningfully exercising the new module. - old flat scheduler.py / orchestrator.py left on disk; stage 6 deletes them once all layers are extracted. Validation: - uv run --package schedule pytest schedule/tests -q: 18 passed - uv run python -m compileall schedule/src: zero errors - grep 'from schedule.(scheduler|orchestrator)\\b' (old paths): 0 matches - grep '"schedule.orchestrator.' (old patch targets): 0 matches - main.py / worker.py / domain/ / infrastructure/ / pyproject.toml byte-identical to HEAD Co-Authored-By: Claude <noreply@anthropic.com>4-phase restructuring of the previously flat backend/ package. Each phase lands as a single squash commit so future bisects stay readable per phase if needed. ## Phase 1 — move + shim (location-only, zero behavior change) * git mv 14 files into api/ schemas/ services/ clients/ subpackages (history preserved via RM/R renames) * New files: api/{admin,auth,dependencies,jupyter,platform,resources, scripts,storage}.py + api/schedules/{schedules,runs}.py * New files: schemas/{auth,common,jupyter,platform,resources, schedules,scripts}.py * New files: clients/{rclone,runtime,scheduler}.py * Old paths kept as 1-line `from backend.<new> import *` shims so tests/main.py/importers kept working untouched * schemas/__init__.py now re-exports from backend.schemas.<domain> ## Phase 2 — APIRouter prefix consolidation * Every APIRouter() now carries its prefix (e.g. prefix="/api/v1/auth") and decorators are stripped of the redundant path prefix * URL paths exposed to the frontend are byte-identical to before * Affected: api/{auth,jupyter,admin,platform,resources,scripts, storage}.py + api/schedules/{schedules,runs}.py ## Phase 3 — first service-layer extraction * backend.services.schedules.validate_dag moved out of api/ (pure DAG validator, no Request/BackgroundTasks/DB) * api/schedules/schedules.py now re-exports the symbol so existing 4 callsites keep working unchanged * Added backend/tests/test_validate_dag.py: 8 unit tests covering DAG_EMPTY, linear chain, diamond, cycle, self-edge, duplicate edge, orphan edge, multi-root ordering ## Phase 4 — delete shims + unify test imports * Removed 14 flat shim files + schemas/__init__.py * Migrated 5 test files (32 import sites) to new paths: backend.scripts.* → backend.api.scripts.* backend.resources.* → backend.api.resources.* backend.jupyter.* → backend.api.jupyter.* backend.runtime_client.* → backend.clients.runtime.* backend.schemas.UpdateScriptRequest → backend.schemas.scripts.* * audit.py kept at backend.audit (main.py references it; not a shim, real code) ## Final structure backend/src/backend/ main.py, audit.py, __init__.py api/ (10 files: routes + 2 subpackage) schemas/ (7 files: Pydantic contracts) services/ (storage + schedules) clients/ (rclone, runtime, scheduler) ## Verification * uv run python -m compileall backend/src backend/tests — clean * uv run --package backend pytest backend/tests -q — 122 passed (114 → 114 → 122 → 122 across phases) * grep -r 'from backend\.\(scripts\|resources\|...\)' backend/ — 0 hits * git blame --follow still traces file origins through the renames 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>1. 同 workspace 互相可见(排除 private): - list_scripts / count_scripts 已 workspace-wide + visibility 过滤,但 单条读取(get/content/latest-version/versions)不校验 visibility,非 owner 猜 id 即可读他人 private 脚本。新增 script_can_view(与 data resources 的 can_view 对称)并在 get_script_row / latest_version 强制, private 对非 owner 返回 404。 2. parent_path 为空默认拉根路径文件: - 物理存储为 workspace/{user_id}/...,根 prefix 原来是 workspace/ + NOT LIKE workspace/%/%,所有文件都在两层被整体排除,list_scripts("") 恒空。改为 workspace/%/,配合 LIKE workspace/%/% AND NOT LIKE workspace/%/%/% 返回各 owner 根级文件。 3. 非空 parent_path 跨 owner 查询: - 原来 workspace/foo/ 永远匹配不到 workspace/{uid}/foo/...,子目录 懒加载返回空,其他用户目录点击无内容。改为 workspace/%/foo/(owner 段通配,与 list_resources 一致),_ / % 仍按字面转义。 测试:更新前缀契约断言,新增 SQLite 行为测试(跨 owner 根/子目录、转义) 与 script_can_view / get_script 权限测试,133 passed。修两个后端接口问题: 1) /api/v1/workspace-directories 返回为空,目录树结构消失 2) 同 workspace 内脚本/数据互相可见但默认排除 private 后端改动 -------- * list_scripts / list_resources / list_workspace_directories 新增 owner_user_id 可选 query 参数;缺省 = 当前请求者本人(scope 到 workspace/{me}/...),传值时 scope 到该 owner 的子树。前端根加载 默认只见自己一级,其他成员以折叠分组呈现。 * visibility 过滤统一:非 admin 请求者只返回 owner==me 或 visibility ∈ {workspace, public};admin 跳过。owner=me 含自己 的 private,owner=other 只剩其 workspace/public,排除他人 private。 * create_workspace_directory 两个分支 visibility 默认 'public' (非 private),使跨 owner 目录树可见;响应新增 owner_user_id 字段。 * platform.list_members 鉴权从 system_admin_context 放宽为 系统管理员或该 workspace 活跃成员(让普通用户也能渲染同 workspace 成员名册,用于跨 owner 分组)。 * main.py 注册 platform 模块(随 list_members 改动补齐导入)。 * .env.example 同步 common/config.py 26 个字段。 前端改动 -------- * ScriptExplorer.memberScriptGroups 改由 members 列表播种分组, display_name 取 members.display_name;inferredDirectories 现在按 owner_user_id 标记,统一跨 owner 目录渲染。删除脚本目录页头与 树分组标题的工作副本数量角标。 * WorkspaceTree 新增 ownerUserId 透传到 store.toggleExpanded; 仅"我"的分组 mount 时 auto-expand,他人分组默认折叠,展开才 调 loadOwnerGroup / owner-scoped loadScripts / loadChildren。 * scriptWorkspaceStore 引入 namespaced cache key (ownerCacheKey = `${ownerUserId ?? me}:${path}`),loadedScriptPaths / loadedChildPaths / loadedOwnerGroups 全部按 owner 隔离; toggleExpanded 用 loadPath === undefined 区分 group 头与真实 目录,修"他人子目录点击不触发接口"的 loadPath 前缀误判 bug。 * api.ts / AuthContext 透传 ownerUserId 给 listScripts / listResources / listWorkspaceDirectories。 文档 ---- * API.md: §3.2 创建目录 visibility 默认 public + 响应加 owner_user_id; §3.3.1 GET directories 加 owner_user_id 参数 + 响应字段; §3.4 GET scripts 改写为 owner 作用域 + visibility 过滤语义; §五.1 GET data-resources 新增,同一套统一语义; §7 intro 例外 — GET members 对系统管理员或 workspace 活跃成员开放。 * DEVELOP.md: Code layout 重写以反映 backend api/services/clients/ schemas 拆分 + schedule domain/scheduling/application/execution/ infrastructure 拆分 + common 子包(auth/storage/backends); Configuration 系统补全 26 个 settings 字段;新增 "Owner-scoping + visibility (cross-owner browsing)" 小节; Per-service dev 注释用 uv run 的源布局要求;Add a new DAG endpoint / storage bucket 路径改为 backend/src/backend/api/* 与 services/*。 测试 ---- * test_list_scripts_parent_path.py / test_resources.py 补充 owner_user_id 参数化直接调用 + LIKE 前缀断言(workspace/{owner}/... 前缀)。 Co-Authored-By: Claude <noreply@anthropic.com>The drop-sync refactor is now contractually enforced: * create_storage({"mode": "sync"|"async"|<any>}) raises StorageConfigError -- the regression we most want to catch is someone "restoring" the sync shape by re-introducing mode= handling. * create_storage() preserves caller's dict, returns an AsyncStorageBackend subclass, and wraps constructor TypeError into a useful StorageConfigError. * build_storage_config emits no mode key on either local or s3 branch. * register_backend() refuses to silently overwrite a different class on the same name (collision guard), and accepts same-class re-registration. * registry surfaces the built-in local + s3 classes after import; the conflict-test cleanup pattern avoids leaking global state across tests. Adds [tool.pytest.ini_options] (asyncio_mode=auto, testpaths=tests) so uv run --package common pytest common/tests works from the workspace root. 17 new tests, all green. Backend suite still 136 passed.把 components/admin/* 和 components/platform/* 共 11 个文件 移到对应 features/<name>/ 下,components/ 只保留跨特性共享的 common/ 子目录(features/admin/* 之前和 components/admin/* 同名 共存,迫使 features/admin/AdminPages.tsx 出现 barrel,顺带消除了 barrel 副作用 CSS 丢失的隐患)。 新的目录约定: - features/<name>/ = 特性模块,所有页面 / Modal / state / hooks / routes 都在内,无 components/<name>/ 并列 - components/common/ = 仅放 ≥2 个特性共用的 widgets (Icon、Sidebar、Toast、Topbar、WelcomePanel) 改动: - 11 文件 git mv (components/{admin,platform}/* → features/*) - 8 文件的 Icon 相对路径 '../common/Icon' → '../../components/common/Icon' - 2 文件补 admin.css side-effect import(UserManagementPage / ProjectManagementPage 之前依赖 AdminPages barrel) - 3 文件(引用方)更新 import:SystemAdminPage、DashboardRoute、 ScriptsPage - 删除 features/admin/AdminPages.tsx barrel(无消费者) - 删除空目录 components/admin、components/platform 验证: - pnpm typecheck 通过 - pnpm build 通过,CSS 体积 35.11 kB 与重构前一致(无样式增减) - routes.ts / routes/platform.tsx 路径未动,route id 不变 踩坑: 第一轮把 '../common/Icon' 错改成 '../../../components/common/Icon', typecheck 报 11 个 Cannot find module。原因:features/admin/ 和 components/admin/ 都是 depth 2,'../../X' 在两边都解析到 app/X, 只有 '../X' 才需要多一层。正确改法是 '../../components/common/Icon'。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>Pull request closed