The fix lands in three concentric layers, all backed by a single
INTERNAL_SERVICE_TOKEN shared secret so we have one mechanism
instead of three:
1. docker-compose: drop the backend.ports: 8891:8000 and
runtime.ports: 8892:8000 mappings. Nginx is the only host
ingress again (architecture §2.2).
2. /internal/v1/*: the storage control plane had six endpoints, five
of which were dead code (frontend already migrated to
/api/v1/data-resources/* with JWT; schedule only ever called
POST /internal/v1/objects). Delete the dead routes, mount the
one survivor with Depends(require_internal_service) that
compares the X-Internal-Service-Token header against
settings.internal_service_token with secrets.compare_digest.
3. POST /api/v1/jupyter on the runtime container: previously open
inside the Docker network. Same token mechanism — backend's
runtime_http_client now carries the header, runtime's
handle_jupyter_action requires the same header. /api/v1/health
stays open for the Nginx and compose healthchecks.
The schedule worker was already configured to call
POST /internal/v1/objects; build_storage_http_client now
sets the token header so its existing call site keeps working
without changes.
Files touched:
backend/src/backend/storage_api.py # 5 dead routes deleted + token guard
backend/src/backend/main.py # runtime_http_client header
runtime/src/runtime/main.py # require_internal_service Depends
common/src/common/config.py # internal_service_token setting
schedule/src/schedule/service.py # httpx client header
docker-compose.yml # ports dropped, INTERNAL_SERVICE_TOKEN env
.env.example # INTERNAL_SERVICE_TOKEN placeholder
API.md / README.md / DEVELOP.md # §9 trimmed to 1 endpoint
Verified:
compileall -> 0 errors
pytest backend/tests -> 37 passed
in-process ASGI smoke:
POST /internal/v1/objects no/wrong/correct token -> 401/401/200
POST /api/v1/jupyter no/wrong/correct token -> 401/401/200
5 deleted internal routes -> 404
docker compose config (with env) -> OK
P0-1 still has one open sub-item (rclone RC --rc-no-auth) that
the user has explicitly deferred; not touched here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Summary of changes:
- Resources Bind Idempotency (High 1+2):
- Check existing active storage object bindings before duplicate check.
- Return existing binding (`reused: true`) on retry with same upload_id.
- Filter by `status == "active"` to bypass dead/deleted rows during reuse check.
- Storage Concurrent Overwrite (High 3):
- Add `acquire_named_lock` and `release_named_lock` helpers using MySQL `GET_LOCK`/`RELEASE_LOCK` hashed to <= 64 chars.
- Wrap `upload_bytes_to_session` PUT+INSERT critical section with named lock on `object_key`.
- Re-check key collision inside lock; append ULID suffix on collision.
- Shared Reference Deletion Protection (Medium 4):
- Check active references before deleting storage objects in `delete_resource`.
- Delete only `DataResources` record if storage object is still referenced elsewhere.
- Robust Usage Type Fallback (Medium 5):
- Replace direct dict lookup for `USAGE_TYPE_TO_PURPOSE[item.usage_type]` with `.get(..., "workspace")` default.
- Idempotency Key Path Matching (Medium 6):
- Move `file_name`/`target_path` validation forward and include path dimension in comparison.
- Strip uniqueness suffix via `_strip_uniqueness_suffix` before key comparison to avoid false 409s on valid retries.
- Usage Type & Bind Concurrency Control (Low 7 & 8):
- Reject bind requests with 409 if upload session purpose is not `data_resource`.
- Wrap resource duplicate check and creation in named lock using `(owner, directory, name)`.
- Trash Key Uniqueness & Restore Compatibility (Low 9):
- Update `trash_key` format to `{purpose}/{object_key}-{storage_object_id}` to prevent collisions.
- Update `object_key_hash` on trash move.
- Update restore logic in `storage_api.py` to strip suffix while maintaining backward compatibility with legacy keys.
- Dead Code Removal (Low 10):
- Remove unreachable `upload_status = "failed"` and redundant `session.rollback()` in `IntegrityError` block.
- Tests & Mocks:
- Add/update 5 test cases covering non-data_resource bind rejection, suffix stripping, and path recovery.
- Add named lock statement mocks for DB testing.
tao.chen