10 Commits
Author SHA1 Message Date
tao.chenandClaude b493907775 feat(scripts): 跨 owner 懒加载目录树 + 跨用户可见 workspace/public
修两个后端接口问题:
1) /api/v1/workspace-directories 返回为空,目录树结构消失
2) 同 workspace 内脚本/数据互相可见但默认排除 private

后端改动
--------
* list_scripts / list_resources / list_workspace_directories 新增
  owner_user_id 可选 query 参数;缺省 = 当前请求者本人(scope 到
  workspace/{me}/...),传值时 scope 到该 owner 的子树。前端根加载
  默认只见自己一级,其他成员以折叠分组呈现。
* visibility 过滤统一:非 admin 请求者只返回 owner==me 或
  visibility ∈ {workspace, public};admin 跳过。owner=me 含自己
  的 private,owner=other 只剩其 workspace/public,排除他人 private。
* create_workspace_directory 两个分支 visibility 默认 'public'
  (非 private),使跨 owner 目录树可见;响应新增 owner_user_id 字段。
* platform.list_members 鉴权从 system_admin_context 放宽为
  系统管理员或该 workspace 活跃成员(让普通用户也能渲染同
  workspace 成员名册,用于跨 owner 分组)。
* main.py 注册 platform 模块(随 list_members 改动补齐导入)。
* .env.example 同步 common/config.py 26 个字段。

前端改动
--------
* ScriptExplorer.memberScriptGroups 改由 members 列表播种分组,
  display_name 取 members.display_name;inferredDirectories 现在按
  owner_user_id 标记,统一跨 owner 目录渲染。删除脚本目录页头与
  树分组标题的工作副本数量角标。
* WorkspaceTree 新增 ownerUserId 透传到 store.toggleExpanded;
  仅"我"的分组 mount 时 auto-expand,他人分组默认折叠,展开才
  调 loadOwnerGroup / owner-scoped loadScripts / loadChildren。
* scriptWorkspaceStore 引入 namespaced cache key
  (ownerCacheKey = `${ownerUserId ?? me}:${path}`),loadedScriptPaths
  / loadedChildPaths / loadedOwnerGroups 全部按 owner 隔离;
  toggleExpanded 用 loadPath === undefined 区分 group 头与真实
  目录,修"他人子目录点击不触发接口"的 loadPath 前缀误判 bug。
* api.ts / AuthContext 透传 ownerUserId 给 listScripts /
  listResources / listWorkspaceDirectories。

文档
----
* API.md: §3.2 创建目录 visibility 默认 public + 响应加 owner_user_id;
  §3.3.1 GET directories 加 owner_user_id 参数 + 响应字段;
  §3.4 GET scripts 改写为 owner 作用域 + visibility 过滤语义;
  §五.1 GET data-resources 新增,同一套统一语义;
  §7 intro 例外 — GET members 对系统管理员或 workspace 活跃成员开放。
* DEVELOP.md: Code layout 重写以反映 backend api/services/clients/
  schemas 拆分 + schedule domain/scheduling/application/execution/
  infrastructure 拆分 + common 子包(auth/storage/backends);
  Configuration 系统补全 26 个 settings 字段;新增
  "Owner-scoping + visibility (cross-owner browsing)" 小节;
  Per-service dev 注释用 uv run 的源布局要求;Add a new DAG endpoint /
  storage bucket 路径改为 backend/src/backend/api/* 与 services/*。

测试
----
* test_list_scripts_parent_path.py /
  test_resources.py 补充 owner_user_id 参数化直接调用 + LIKE
  前缀断言(workspace/{owner}/... 前缀)。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-21 19:26:53 +08:00
tao.chenandClaude bca239ed4b refactor(backend): split into api/ schemas/ services/ clients/ layers
4-phase restructuring of the previously flat backend/ package. Each
phase lands as a single squash commit so future bisects stay readable
per phase if needed.

## Phase 1 — move + shim (location-only, zero behavior change)
* git mv 14 files into api/ schemas/ services/ clients/ subpackages
  (history preserved via RM/R renames)
* New files: api/{admin,auth,dependencies,jupyter,platform,resources,
  scripts,storage}.py + api/schedules/{schedules,runs}.py
* New files: schemas/{auth,common,jupyter,platform,resources,
  schedules,scripts}.py
* New files: clients/{rclone,runtime,scheduler}.py
* Old paths kept as 1-line `from backend.<new> import *` shims so
  tests/main.py/importers kept working untouched
* schemas/__init__.py now re-exports from backend.schemas.<domain>

## Phase 2 — APIRouter prefix consolidation
* Every APIRouter() now carries its prefix (e.g. prefix="/api/v1/auth")
  and decorators are stripped of the redundant path prefix
* URL paths exposed to the frontend are byte-identical to before
* Affected: api/{auth,jupyter,admin,platform,resources,scripts,
  storage}.py + api/schedules/{schedules,runs}.py

## Phase 3 — first service-layer extraction
* backend.services.schedules.validate_dag moved out of api/
  (pure DAG validator, no Request/BackgroundTasks/DB)
* api/schedules/schedules.py now re-exports the symbol so existing
  4 callsites keep working unchanged
* Added backend/tests/test_validate_dag.py: 8 unit tests covering
  DAG_EMPTY, linear chain, diamond, cycle, self-edge, duplicate
  edge, orphan edge, multi-root ordering

## Phase 4 — delete shims + unify test imports
* Removed 14 flat shim files + schemas/__init__.py
* Migrated 5 test files (32 import sites) to new paths:
  backend.scripts.* → backend.api.scripts.*
  backend.resources.* → backend.api.resources.*
  backend.jupyter.* → backend.api.jupyter.*
  backend.runtime_client.* → backend.clients.runtime.*
  backend.schemas.UpdateScriptRequest → backend.schemas.scripts.*
* audit.py kept at backend.audit (main.py references it; not a
  shim, real code)

## Final structure
backend/src/backend/
  main.py, audit.py, __init__.py
  api/            (10 files: routes + 2 subpackage)
  schemas/        (7 files: Pydantic contracts)
  services/       (storage + schedules)
  clients/        (rclone, runtime, scheduler)

## Verification
* uv run python -m compileall backend/src backend/tests — clean
* uv run --package backend pytest backend/tests -q — 122 passed
  (114 → 114 → 122 → 122 across phases)
* grep -r 'from backend\.\(scripts\|resources\|...\)' backend/ — 0 hits
* git blame --follow still traces file origins through the renames

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-21 15:32:04 +08:00
tao.chen d68055a96c feat(scripts/resources): align list_scripts visibility with list_resources
69a9a48 把 data resources 的可见性从 list 恒真改成
workspace-wide + visibility 过滤 + admin 短路,但 scripts 端
没动。两端不对称,导致:

* 非 admin 调 list_scripts 走 user_relative_path →
  workspace/{当前用户ID}/...,永远拿不到别人的脚本
* count_scripts 同样 user-scoped,dashboard "全部脚本"
  只统计自己
* list_resources 响应没带 owner_display_name,data-only
  owner 的目录名回退到 userId.slice(-6),显示不友好

修复:
* list_scripts / count_scripts 改 workspace-wide(新建
  _build_list_scripts_workspace_descendant_prefix helper;
  旧 _build_list_scripts_descendant_prefix 保留标 deprecated
  避免破坏其它调用方);非 admin 追加
  or_(owner_user_id = me, visibility in {workspace, public})
  与 list_resources 完全对称;admin 短路
* resource_payload 加 owner_display_name 字段(与
  script_payload 对称);list_resources SELECT 加
  Users.display_name + outerjoin
* 前端 ScriptExplorer displayName 回退链:scripts 的
  owner_display_name → data resources 的 owner_display_name
  → 本人 user.display_name → userId.slice(-6) 占位
  ;ResourceItem 类型同步加 owner_display_name?: string|null

存储物理布局仍是 workspace/{user_id}/...,仅读取侧
listing/count 跨 owner。
2026-08-21 14:06:45 +08:00
tao.chen 69a9a48a0b fix(resources): enforce visibility on list + get (private invisible to non-owner)
回退 5483d19 的 workspace-wide 改动。用户真实语义:
- owner 永远可见自己的资源(含 private)
- 非 owner 只见别人 visibility in {workspace, public} 的资源
- admin 全部可见

can_view 与 list_resources 共享同一谓词。list 端点恢复
owner == me OR visibility in {workspace, public} 的过滤,
admin 跳过。get_resource / download_url / jupyter-relative-path
/ delete_resource 全部经 get_visible_resource → can_view,自然
收敛到同一语义。补 1 个 owner-看自己-private 测试,4 个
visibility 测试保持。
2026-08-21 12:46:09 +08:00
tao.chen 5483d19dbd fix(resources): align can_view with list_resources (workspace-wide)
list_resources 已放开成 workspace-wide,但 can_view 仍要求
visibility in {workspace, public} 或 admin,导致 A 的 private
资源出现在 B 的列表里、但 B 调 GET /{id}、POST /{id}/download-url
与 POST /{id}/jupyter-relative-path 全部 404。

can_view 改为恒真,访问边界全部交给 get_visible_resource 的
workspace_id + status=active SQL 限定;visibility 字段保留为
上传/绑定时的语义标签。delete_resource 仍按 owner/admin 校验
403,不受影响。补 4 个 unit test。
2026-08-21 12:40:27 +08:00
tao.chen 48fe49df3b feat(scripts/data-resources): merge tree + add parent_path filter
- Backend: GET /api/v1/data-resources accepts parent_path; LIKE
  '{ws_id}/%/{escaped}/%' AND NOT LIKE '{ws_id}/%/{escaped}/%/%' on
  StorageObjects.object_key (workspace-wide, escapes _ and %, mirrors
  list_scripts parent_path semantics). 13 new tests in
  test_resources.py (helper unit / SQL compile / SQLite behavioral).
- Frontend: listResources gains parentPath arg, propagated through
  WorkspaceBoundApi + AuthContext binding. WorkspaceTreeGroup title
  count and ScriptExplorer header count now include dataResources.
  memberScriptGroups backfills data-only owners so users with only
  data resources still render a group. loadDataResources accepts an
  optional parentPath, default empty preserves prior behavior.
2026-08-21 12:36:43 +08:00
tao.chen b6eb069849 fix(backend): enhance storage/resource concurrency, idempotency, and deletion safety
Summary of changes:

- Resources Bind Idempotency (High 1+2):
  - Check existing active storage object bindings before duplicate check.
  - Return existing binding (`reused: true`) on retry with same upload_id.
  - Filter by `status == "active"` to bypass dead/deleted rows during reuse check.

- Storage Concurrent Overwrite (High 3):
  - Add `acquire_named_lock` and `release_named_lock` helpers using MySQL `GET_LOCK`/`RELEASE_LOCK` hashed to <= 64 chars.
  - Wrap `upload_bytes_to_session` PUT+INSERT critical section with named lock on `object_key`.
  - Re-check key collision inside lock; append ULID suffix on collision.

- Shared Reference Deletion Protection (Medium 4):
  - Check active references before deleting storage objects in `delete_resource`.
  - Delete only `DataResources` record if storage object is still referenced elsewhere.

- Robust Usage Type Fallback (Medium 5):
  - Replace direct dict lookup for `USAGE_TYPE_TO_PURPOSE[item.usage_type]` with `.get(..., "workspace")` default.

- Idempotency Key Path Matching (Medium 6):
  - Move `file_name`/`target_path` validation forward and include path dimension in comparison.
  - Strip uniqueness suffix via `_strip_uniqueness_suffix` before key comparison to avoid false 409s on valid retries.

- Usage Type & Bind Concurrency Control (Low 7 & 8):
  - Reject bind requests with 409 if upload session purpose is not `data_resource`.
  - Wrap resource duplicate check and creation in named lock using `(owner, directory, name)`.

- Trash Key Uniqueness & Restore Compatibility (Low 9):
  - Update `trash_key` format to `{purpose}/{object_key}-{storage_object_id}` to prevent collisions.
  - Update `object_key_hash` on trash move.
  - Update restore logic in `storage_api.py` to strip suffix while maintaining backward compatibility with legacy keys.

- Dead Code Removal (Low 10):
  - Remove unreachable `upload_status = "failed"` and redundant `session.rollback()` in `IntegrityError` block.

- Tests & Mocks:
  - Add/update 5 test cases covering non-data_resource bind rejection, suffix stripping, and path recovery.
  - Add named lock statement mocks for DB testing.
2026-08-14 20:53:07 +08:00
tao.chen f72dfd10e8 fix: delete bug 2026-08-14 20:37:25 +08:00
tao.chen 25e563dcaa update: ruff check --fix 2026-08-14 19:51:58 +08:00
tao.chen 5d49ff5e34 fix: file upload error 2026-08-14 18:22:46 +08:00