Commit Graph
9 Commits
Author SHA1 Message Date
tao.chenandClaude bca239ed4b refactor(backend): split into api/ schemas/ services/ clients/ layers
4-phase restructuring of the previously flat backend/ package. Each
phase lands as a single squash commit so future bisects stay readable
per phase if needed.

## Phase 1 — move + shim (location-only, zero behavior change)
* git mv 14 files into api/ schemas/ services/ clients/ subpackages
  (history preserved via RM/R renames)
* New files: api/{admin,auth,dependencies,jupyter,platform,resources,
  scripts,storage}.py + api/schedules/{schedules,runs}.py
* New files: schemas/{auth,common,jupyter,platform,resources,
  schedules,scripts}.py
* New files: clients/{rclone,runtime,scheduler}.py
* Old paths kept as 1-line `from backend.<new> import *` shims so
  tests/main.py/importers kept working untouched
* schemas/__init__.py now re-exports from backend.schemas.<domain>

## Phase 2 — APIRouter prefix consolidation
* Every APIRouter() now carries its prefix (e.g. prefix="/api/v1/auth")
  and decorators are stripped of the redundant path prefix
* URL paths exposed to the frontend are byte-identical to before
* Affected: api/{auth,jupyter,admin,platform,resources,scripts,
  storage}.py + api/schedules/{schedules,runs}.py

## Phase 3 — first service-layer extraction
* backend.services.schedules.validate_dag moved out of api/
  (pure DAG validator, no Request/BackgroundTasks/DB)
* api/schedules/schedules.py now re-exports the symbol so existing
  4 callsites keep working unchanged
* Added backend/tests/test_validate_dag.py: 8 unit tests covering
  DAG_EMPTY, linear chain, diamond, cycle, self-edge, duplicate
  edge, orphan edge, multi-root ordering

## Phase 4 — delete shims + unify test imports
* Removed 14 flat shim files + schemas/__init__.py
* Migrated 5 test files (32 import sites) to new paths:
  backend.scripts.* → backend.api.scripts.*
  backend.resources.* → backend.api.resources.*
  backend.jupyter.* → backend.api.jupyter.*
  backend.runtime_client.* → backend.clients.runtime.*
  backend.schemas.UpdateScriptRequest → backend.schemas.scripts.*
* audit.py kept at backend.audit (main.py references it; not a
  shim, real code)

## Final structure
backend/src/backend/
  main.py, audit.py, __init__.py
  api/            (10 files: routes + 2 subpackage)
  schemas/        (7 files: Pydantic contracts)
  services/       (storage + schedules)
  clients/        (rclone, runtime, scheduler)

## Verification
* uv run python -m compileall backend/src backend/tests — clean
* uv run --package backend pytest backend/tests -q — 122 passed
  (114 → 114 → 122 → 122 across phases)
* grep -r 'from backend\.\(scripts\|resources\|...\)' backend/ — 0 hits
* git blame --follow still traces file origins through the renames

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-21 15:32:04 +08:00
tao.chen d68055a96c feat(scripts/resources): align list_scripts visibility with list_resources
69a9a48 把 data resources 的可见性从 list 恒真改成
workspace-wide + visibility 过滤 + admin 短路,但 scripts 端
没动。两端不对称,导致:

* 非 admin 调 list_scripts 走 user_relative_path →
  workspace/{当前用户ID}/...,永远拿不到别人的脚本
* count_scripts 同样 user-scoped,dashboard "全部脚本"
  只统计自己
* list_resources 响应没带 owner_display_name,data-only
  owner 的目录名回退到 userId.slice(-6),显示不友好

修复:
* list_scripts / count_scripts 改 workspace-wide(新建
  _build_list_scripts_workspace_descendant_prefix helper;
  旧 _build_list_scripts_descendant_prefix 保留标 deprecated
  避免破坏其它调用方);非 admin 追加
  or_(owner_user_id = me, visibility in {workspace, public})
  与 list_resources 完全对称;admin 短路
* resource_payload 加 owner_display_name 字段(与
  script_payload 对称);list_resources SELECT 加
  Users.display_name + outerjoin
* 前端 ScriptExplorer displayName 回退链:scripts 的
  owner_display_name → data resources 的 owner_display_name
  → 本人 user.display_name → userId.slice(-6) 占位
  ;ResourceItem 类型同步加 owner_display_name?: string|null

存储物理布局仍是 workspace/{user_id}/...,仅读取侧
listing/count 跨 owner。
2026-08-21 14:06:45 +08:00
tao.chen 69a9a48a0b fix(resources): enforce visibility on list + get (private invisible to non-owner)
回退 5483d19 的 workspace-wide 改动。用户真实语义:
- owner 永远可见自己的资源(含 private)
- 非 owner 只见别人 visibility in {workspace, public} 的资源
- admin 全部可见

can_view 与 list_resources 共享同一谓词。list 端点恢复
owner == me OR visibility in {workspace, public} 的过滤,
admin 跳过。get_resource / download_url / jupyter-relative-path
/ delete_resource 全部经 get_visible_resource → can_view,自然
收敛到同一语义。补 1 个 owner-看自己-private 测试,4 个
visibility 测试保持。
2026-08-21 12:46:09 +08:00
tao.chen 5483d19dbd fix(resources): align can_view with list_resources (workspace-wide)
list_resources 已放开成 workspace-wide,但 can_view 仍要求
visibility in {workspace, public} 或 admin,导致 A 的 private
资源出现在 B 的列表里、但 B 调 GET /{id}、POST /{id}/download-url
与 POST /{id}/jupyter-relative-path 全部 404。

can_view 改为恒真,访问边界全部交给 get_visible_resource 的
workspace_id + status=active SQL 限定;visibility 字段保留为
上传/绑定时的语义标签。delete_resource 仍按 owner/admin 校验
403,不受影响。补 4 个 unit test。
2026-08-21 12:40:27 +08:00
tao.chen 48fe49df3b feat(scripts/data-resources): merge tree + add parent_path filter
- Backend: GET /api/v1/data-resources accepts parent_path; LIKE
  '{ws_id}/%/{escaped}/%' AND NOT LIKE '{ws_id}/%/{escaped}/%/%' on
  StorageObjects.object_key (workspace-wide, escapes _ and %, mirrors
  list_scripts parent_path semantics). 13 new tests in
  test_resources.py (helper unit / SQL compile / SQLite behavioral).
- Frontend: listResources gains parentPath arg, propagated through
  WorkspaceBoundApi + AuthContext binding. WorkspaceTreeGroup title
  count and ScriptExplorer header count now include dataResources.
  memberScriptGroups backfills data-only owners so users with only
  data resources still render a group. loadDataResources accepts an
  optional parentPath, default empty preserves prior behavior.
2026-08-21 12:36:43 +08:00
tao.chen b6eb069849 fix(backend): enhance storage/resource concurrency, idempotency, and deletion safety
Summary of changes:

- Resources Bind Idempotency (High 1+2):
  - Check existing active storage object bindings before duplicate check.
  - Return existing binding (`reused: true`) on retry with same upload_id.
  - Filter by `status == "active"` to bypass dead/deleted rows during reuse check.

- Storage Concurrent Overwrite (High 3):
  - Add `acquire_named_lock` and `release_named_lock` helpers using MySQL `GET_LOCK`/`RELEASE_LOCK` hashed to <= 64 chars.
  - Wrap `upload_bytes_to_session` PUT+INSERT critical section with named lock on `object_key`.
  - Re-check key collision inside lock; append ULID suffix on collision.

- Shared Reference Deletion Protection (Medium 4):
  - Check active references before deleting storage objects in `delete_resource`.
  - Delete only `DataResources` record if storage object is still referenced elsewhere.

- Robust Usage Type Fallback (Medium 5):
  - Replace direct dict lookup for `USAGE_TYPE_TO_PURPOSE[item.usage_type]` with `.get(..., "workspace")` default.

- Idempotency Key Path Matching (Medium 6):
  - Move `file_name`/`target_path` validation forward and include path dimension in comparison.
  - Strip uniqueness suffix via `_strip_uniqueness_suffix` before key comparison to avoid false 409s on valid retries.

- Usage Type & Bind Concurrency Control (Low 7 & 8):
  - Reject bind requests with 409 if upload session purpose is not `data_resource`.
  - Wrap resource duplicate check and creation in named lock using `(owner, directory, name)`.

- Trash Key Uniqueness & Restore Compatibility (Low 9):
  - Update `trash_key` format to `{purpose}/{object_key}-{storage_object_id}` to prevent collisions.
  - Update `object_key_hash` on trash move.
  - Update restore logic in `storage_api.py` to strip suffix while maintaining backward compatibility with legacy keys.

- Dead Code Removal (Low 10):
  - Remove unreachable `upload_status = "failed"` and redundant `session.rollback()` in `IntegrityError` block.

- Tests & Mocks:
  - Add/update 5 test cases covering non-data_resource bind rejection, suffix stripping, and path recovery.
  - Add named lock statement mocks for DB testing.
2026-08-14 20:53:07 +08:00
tao.chen f72dfd10e8 fix: delete bug 2026-08-14 20:37:25 +08:00
tao.chen 25e563dcaa update: ruff check --fix 2026-08-14 19:51:58 +08:00
tao.chen 5d49ff5e34 fix: file upload error 2026-08-14 18:22:46 +08:00