Commit Graph
104 Commits
Author SHA1 Message Date
tao.chen 6258cf5d12 fix: offline deploy 2026-08-17 15:59:19 +08:00
tao.chen 0974881fbb fix: offline deploy 2026-08-17 15:30:21 +08:00
tao.chen b6eb069849 fix(backend): enhance storage/resource concurrency, idempotency, and deletion safety
Summary of changes:

- Resources Bind Idempotency (High 1+2):
  - Check existing active storage object bindings before duplicate check.
  - Return existing binding (`reused: true`) on retry with same upload_id.
  - Filter by `status == "active"` to bypass dead/deleted rows during reuse check.

- Storage Concurrent Overwrite (High 3):
  - Add `acquire_named_lock` and `release_named_lock` helpers using MySQL `GET_LOCK`/`RELEASE_LOCK` hashed to <= 64 chars.
  - Wrap `upload_bytes_to_session` PUT+INSERT critical section with named lock on `object_key`.
  - Re-check key collision inside lock; append ULID suffix on collision.

- Shared Reference Deletion Protection (Medium 4):
  - Check active references before deleting storage objects in `delete_resource`.
  - Delete only `DataResources` record if storage object is still referenced elsewhere.

- Robust Usage Type Fallback (Medium 5):
  - Replace direct dict lookup for `USAGE_TYPE_TO_PURPOSE[item.usage_type]` with `.get(..., "workspace")` default.

- Idempotency Key Path Matching (Medium 6):
  - Move `file_name`/`target_path` validation forward and include path dimension in comparison.
  - Strip uniqueness suffix via `_strip_uniqueness_suffix` before key comparison to avoid false 409s on valid retries.

- Usage Type & Bind Concurrency Control (Low 7 & 8):
  - Reject bind requests with 409 if upload session purpose is not `data_resource`.
  - Wrap resource duplicate check and creation in named lock using `(owner, directory, name)`.

- Trash Key Uniqueness & Restore Compatibility (Low 9):
  - Update `trash_key` format to `{purpose}/{object_key}-{storage_object_id}` to prevent collisions.
  - Update `object_key_hash` on trash move.
  - Update restore logic in `storage_api.py` to strip suffix while maintaining backward compatibility with legacy keys.

- Dead Code Removal (Low 10):
  - Remove unreachable `upload_status = "failed"` and redundant `session.rollback()` in `IntegrityError` block.

- Tests & Mocks:
  - Add/update 5 test cases covering non-data_resource bind rejection, suffix stripping, and path recovery.
  - Add named lock statement mocks for DB testing.
2026-08-14 20:53:07 +08:00
tao.chen f72dfd10e8 fix: delete bug 2026-08-14 20:37:25 +08:00
tao.chen 25e563dcaa update: ruff check --fix 2026-08-14 19:51:58 +08:00
tao.chen d855912791 fix: soft delete helper 2026-08-14 19:39:09 +08:00
tao.chen 139f2c02c1 fix: delete error 2026-08-14 19:19:41 +08:00
tao.chen 5c5dec9a4a fix: allows_same_name_different_parent 2026-08-14 18:49:27 +08:00
tao.chen 5d49ff5e34 fix: file upload error 2026-08-14 18:22:46 +08:00
tao.chen c65d6dc684 fix: build error 2026-08-14 17:19:07 +08:00
tao.chen 45cb2f8409 fix: build error 2026-08-14 17:16:53 +08:00
tao.chen 341cc77a79 update: Dockerfile 2026-08-14 16:02:19 +08:00
tao.chen 2014d59681 update: Dockerfile 2026-08-14 15:55:00 +08:00
tao.chen af9466c4e6 update: UV_HTTP_TIMEOUT 2026-08-14 15:27:16 +08:00
Winnie be475dd0d9 fix: support local storage runtime and schedule logs 2026-08-14 15:06:36 +08:00
tao.chen c45687ef18 fix: update unique key 2026-08-14 13:44:57 +08:00
tao.chen b9becffab2 update: 数据资源文件上传 2026-08-14 11:31:48 +08:00
tao.chen 462617b66d feat: datasource 2026-08-13 20:13:37 +08:00
tao.chen 0b5b50edb9 update: rel path 2026-08-13 19:51:20 +08:00
tao.chen 63a14b1884 fix: params error
NameError: name 'session' is not defined
2026-08-13 19:20:36 +08:00
tao.chen f5cb22e8cd feat: upload resource 2026-08-13 19:09:13 +08:00
tao.chen 19157ad20b test 2026-08-13 18:08:12 +08:00
tao.chen 3094a47298 fix: path error 2026-08-13 11:48:24 +08:00
tao.chen 49c8a27841 update: update storage struct 2026-08-13 11:01:37 +08:00
xiaozhu 7b10f08484 Merge branch 'develop' of http://8.153.151.51:8888/team_group/model-develop into develop 2026-08-13 10:06:46 +08:00
tao.chen da9a1ec472 update: storage path 2026-08-13 09:35:27 +08:00
tao.chen b1ceaeb173 fix: storage and directories bug 2026-08-12 20:18:29 +08:00
tao.chen 495018de34 update: workspace lazy load 2026-08-12 18:36:17 +08:00
tao.chen 6b9f851810 update: scripts.py api 2026-08-12 17:14:18 +08:00
tao.chen 2fcdf51cfc update: PythonEditor.tsx 2026-08-12 16:49:21 +08:00
xiaozhu 04447abfb0 fix: Dockerfile 移除 --frozen 参数 2026-08-12 16:38:31 +08:00
xiaozhu 55e37b5a69 fix: Dockerfile 中先更新 uv.lock 再安装 2026-08-12 16:32:35 +08:00
xiaozhu 8d83adf1aa fix:路由优化 2026-08-12 15:27:10 +08:00
xiaozhu fe65034aa8 feat:脚本只读 2026-08-12 14:34:31 +08:00
tao.chen 416ff4d06a feat: add logger 2026-08-12 12:43:31 +08:00
tao.chen 3981c32781 refactor: directory 2026-08-12 12:23:19 +08:00
tao.chen a378ea5352 feat: add pytest in dev 2026-08-12 12:21:48 +08:00
tao.chen e23df83e0d fix: bugs 2026-08-11 22:03:01 +08:00
tao.chen bff7b7f9a6 fix: bugs 2026-08-11 21:53:52 +08:00
tao.chen dd6f176ca8 fix: file lock 2026-08-11 21:26:55 +08:00
tao.chen d4013a07a9 rollback 2026-08-11 20:41:15 +08:00
tao.chen e1360c5c38 test: backend 2026-08-11 20:09:44 +08:00
tao.chen 05091c425a update: filter by owner / visibility 2026-08-11 18:56:47 +08:00
tao.chen fe7f1a744e feat: schedule node add python version 2026-08-11 15:56:05 +08:00
tao.chen 4269eed018 fix: 422 error 2026-08-11 15:13:13 +08:00
tao.chen 084de5b2d6 fix: storage.py 2026-08-07 20:14:29 +08:00
tao.chen e7bce03794 fix: storage_api.py 2026-08-07 19:58:42 +08:00
tao.chen 7b4b752109 fix: import workspace member 2026-08-07 19:25:04 +08:00
tao.chen 873a464629 fix: auth error 2026-08-07 16:14:13 +08:00
tao.chen 2ee36b48ec update: auth
update auth api
2026-08-07 14:41:20 +08:00