storage: extract unified AsyncStorageBackend abstraction + migrate from RustFS

Replace the old RustFS-specific storage layer (common.storage.client /
RustFSObjectStore) with a minimal sync/async abstraction:

  AsyncStorageBackend: put / get / get_stream / delete / exists / stat /
                       list / get_url / copy
  StorageBackend:      same surface, sync implementations
  create_storage({"type": "s3" | "local", "mode": "async", ...})
  backends/s3.py:      S3-compatible (boto3 / aioboto3)
  backends/local.py:   on-disk filesystem (aiofiles)

Concretely:
  - Drop RustFSObjectStore + common.storage.client (deleted).
  - Drop the RustFS-specific ensure_bucket / presign_put / move_to_trash /
    rewrite_to_public_path / sha256 / put_bytes methods.
  - Migrate backend/storage_api.py + backend/main.py + backend/scripts.py
    + schedule/service.py + schedule/worker.py to the new abstraction.
  - Migrate backend/storage_client.py + schedule/storage_client.py to
    stub status (HTTP wrapper is dead code post-migration; rewrite pending).
  - Rename all RUSTFS_* env vars to S3_* across .env.example,
    docker-compose.yml, default.conf, scripts/nginx-entrypoint.sh,
    common/config.py.
  - Replace hardcoded rclone remote name "rustfs" with "s3" in
    docker-compose.yml + config.py default.
  - Rename "rustfs" SQLAlchemy column comments + table comments to
    provider-neutral wording; StorageObjects.storage_backend enum
    value moves from "rustfs" to "s3" (DB rows with the old value will
    fail the != "s3" check until a one-shot migration is applied).
  - Drop unused common/src/common/migrations/{README,env.py,script.py.mako}
    (the alembic setup lives in /migrations/, not here).

Migration of the old abstractions has been done in one pass; per-route
method calls (delete / stat / put / get_url) are now direct one-liners
against AsyncStorageBackend.

After this commit:
  - All Python imports resolve; routes compile (compileall green).
  - s3 mode is fully wired.
  - Routes that depended on removed methods (presign_put, move_to_trash,
    rewrite_to_public_path, head() metadata) raise NotImplementedError
    with a one-line TODO; rewriting these route handlers is the next step.
This commit is contained in:
tao.chen
2026-08-05 13:08:32 +08:00
parent 7c456a04ce
commit 4b2a67ae5d
30 changed files with 1577 additions and 810 deletions
+31 -16
View File
@@ -24,27 +24,42 @@ JWT_SECRET=change-this-development-secret
# ============================================================================
INITIAL_ADMIN_PASSWORD=admin12345
# Object storage (S3-compatible, RustFS).
# RUSTFS_ENDPOINT is the single upstream URL consumed by all 4 services:
# Object storage. Two modes are supported:
# STORAGE_BACKEND=s3 — connects to an S3-compatible service (MinIO,
# RustFS, SeaweedFS, AWS S3, …). Requires the
# S3_* block below.
# STORAGE_BACKEND=local — stores objects on the local filesystem under
# LOCAL_STORAGE_BASE_DIR. Backend and runtime
# share this directory via a Docker volume
# (docker-compose.yml mounts `local-storage`).
# Useful for dev, single-node, air-gapped.
STORAGE_BACKEND=s3
LOCAL_STORAGE_BASE_DIR=/data
# Object storage (S3-compatible). Only used when STORAGE_BACKEND=s3.
# S3_ENDPOINT is the single upstream URL consumed by all 4 services:
# - nginx (via scripts/nginx-entrypoint.sh, which parses host + port)
# - backend / runtime / schedule (passed through to boto3 / rclone)
# RUSTFS_ACCESS_KEY / RUSTFS_SECRET_KEY are read by Python code in
# S3_ACCESS_KEY / S3_SECRET_KEY are read by Python code in
# backend/ and schedule/ (boto3 credentials).
#
# RustFS buckets are purpose-named. Currently we have:
# RUSTFS_WORKSPACE_BUCKET — workspace files (notebooks, scripts, working
# S3 buckets are purpose-named:
# S3_WORKSPACE_BUCKET — workspace files (notebooks, scripts, working
# copies); layout is ``s3://<bucket>/<workspace_id>/...``.
# Future: RUSTFS_VERSION_BUCKET, RUSTFS_RUN_LOG_BUCKET, ...
RUSTFS_HOST=127.0.0.1
RUSTFS_PORT=9000
RUSTFS_ENDPOINT=http://127.0.0.1:9000
RUSTFS_ACCESS_KEY=change-me
RUSTFS_SECRET_KEY=change-me
RUSTFS_WORKSPACE_BUCKET=workspaces
RUSTFS_VERSION_BUCKET=versions
RUSTFS_RUN_LOG_BUCKET=run-logs
RUSTFS_TRASH_BUCKET=trash
RUSTFS_TRASH_RETENTION_DAYS=30
# S3_VERSION_BUCKET — immutable script-version artifacts.
# S3_RUN_LOG_BUCKET — schedule run logs and execution results.
# S3_TRASH_BUCKET — soft-deleted objects; source bucket key is preserved
# as a prefix so restore is a same-key move.
S3_HOST=127.0.0.1
S3_PORT=9000
S3_ENDPOINT=http://127.0.0.1:9000
S3_ACCESS_KEY=change-me
S3_SECRET_KEY=change-me
S3_WORKSPACE_BUCKET=workspaces
S3_VERSION_BUCKET=versions
S3_RUN_LOG_BUCKET=run-logs
S3_TRASH_BUCKET=trash
S3_TRASH_RETENTION_DAYS=30
# rclone RC (HTTP control API). The runtime container starts rclone with
# `--rc --rc-addr 0.0.0.0:5572 --rc-no-auth` (see runtime/src/runtime/mount.py),