storage: extract unified AsyncStorageBackend abstraction + migrate from RustFS
Replace the old RustFS-specific storage layer (common.storage.client /
RustFSObjectStore) with a minimal sync/async abstraction:
AsyncStorageBackend: put / get / get_stream / delete / exists / stat /
list / get_url / copy
StorageBackend: same surface, sync implementations
create_storage({"type": "s3" | "local", "mode": "async", ...})
backends/s3.py: S3-compatible (boto3 / aioboto3)
backends/local.py: on-disk filesystem (aiofiles)
Concretely:
- Drop RustFSObjectStore + common.storage.client (deleted).
- Drop the RustFS-specific ensure_bucket / presign_put / move_to_trash /
rewrite_to_public_path / sha256 / put_bytes methods.
- Migrate backend/storage_api.py + backend/main.py + backend/scripts.py
+ schedule/service.py + schedule/worker.py to the new abstraction.
- Migrate backend/storage_client.py + schedule/storage_client.py to
stub status (HTTP wrapper is dead code post-migration; rewrite pending).
- Rename all RUSTFS_* env vars to S3_* across .env.example,
docker-compose.yml, default.conf, scripts/nginx-entrypoint.sh,
common/config.py.
- Replace hardcoded rclone remote name "rustfs" with "s3" in
docker-compose.yml + config.py default.
- Rename "rustfs" SQLAlchemy column comments + table comments to
provider-neutral wording; StorageObjects.storage_backend enum
value moves from "rustfs" to "s3" (DB rows with the old value will
fail the != "s3" check until a one-shot migration is applied).
- Drop unused common/src/common/migrations/{README,env.py,script.py.mako}
(the alembic setup lives in /migrations/, not here).
Migration of the old abstractions has been done in one pass; per-route
method calls (delete / stat / put / get_url) are now direct one-liners
against AsyncStorageBackend.
After this commit:
- All Python imports resolve; routes compile (compileall green).
- s3 mode is fully wired.
- Routes that depended on removed methods (presign_put, move_to_trash,
rewrite_to_public_path, head() metadata) raise NotImplementedError
with a one-line TODO; rewriting these route handlers is the next step.
This commit is contained in:
+31
-16
@@ -24,27 +24,42 @@ JWT_SECRET=change-this-development-secret
|
||||
# ============================================================================
|
||||
INITIAL_ADMIN_PASSWORD=admin12345
|
||||
|
||||
# Object storage (S3-compatible, RustFS).
|
||||
# RUSTFS_ENDPOINT is the single upstream URL consumed by all 4 services:
|
||||
# Object storage. Two modes are supported:
|
||||
# STORAGE_BACKEND=s3 — connects to an S3-compatible service (MinIO,
|
||||
# RustFS, SeaweedFS, AWS S3, …). Requires the
|
||||
# S3_* block below.
|
||||
# STORAGE_BACKEND=local — stores objects on the local filesystem under
|
||||
# LOCAL_STORAGE_BASE_DIR. Backend and runtime
|
||||
# share this directory via a Docker volume
|
||||
# (docker-compose.yml mounts `local-storage`).
|
||||
# Useful for dev, single-node, air-gapped.
|
||||
STORAGE_BACKEND=s3
|
||||
LOCAL_STORAGE_BASE_DIR=/data
|
||||
|
||||
# Object storage (S3-compatible). Only used when STORAGE_BACKEND=s3.
|
||||
# S3_ENDPOINT is the single upstream URL consumed by all 4 services:
|
||||
# - nginx (via scripts/nginx-entrypoint.sh, which parses host + port)
|
||||
# - backend / runtime / schedule (passed through to boto3 / rclone)
|
||||
# RUSTFS_ACCESS_KEY / RUSTFS_SECRET_KEY are read by Python code in
|
||||
# S3_ACCESS_KEY / S3_SECRET_KEY are read by Python code in
|
||||
# backend/ and schedule/ (boto3 credentials).
|
||||
#
|
||||
# RustFS buckets are purpose-named. Currently we have:
|
||||
# RUSTFS_WORKSPACE_BUCKET — workspace files (notebooks, scripts, working
|
||||
# S3 buckets are purpose-named:
|
||||
# S3_WORKSPACE_BUCKET — workspace files (notebooks, scripts, working
|
||||
# copies); layout is ``s3://<bucket>/<workspace_id>/...``.
|
||||
# Future: RUSTFS_VERSION_BUCKET, RUSTFS_RUN_LOG_BUCKET, ...
|
||||
RUSTFS_HOST=127.0.0.1
|
||||
RUSTFS_PORT=9000
|
||||
RUSTFS_ENDPOINT=http://127.0.0.1:9000
|
||||
RUSTFS_ACCESS_KEY=change-me
|
||||
RUSTFS_SECRET_KEY=change-me
|
||||
RUSTFS_WORKSPACE_BUCKET=workspaces
|
||||
RUSTFS_VERSION_BUCKET=versions
|
||||
RUSTFS_RUN_LOG_BUCKET=run-logs
|
||||
RUSTFS_TRASH_BUCKET=trash
|
||||
RUSTFS_TRASH_RETENTION_DAYS=30
|
||||
# S3_VERSION_BUCKET — immutable script-version artifacts.
|
||||
# S3_RUN_LOG_BUCKET — schedule run logs and execution results.
|
||||
# S3_TRASH_BUCKET — soft-deleted objects; source bucket key is preserved
|
||||
# as a prefix so restore is a same-key move.
|
||||
S3_HOST=127.0.0.1
|
||||
S3_PORT=9000
|
||||
S3_ENDPOINT=http://127.0.0.1:9000
|
||||
S3_ACCESS_KEY=change-me
|
||||
S3_SECRET_KEY=change-me
|
||||
S3_WORKSPACE_BUCKET=workspaces
|
||||
S3_VERSION_BUCKET=versions
|
||||
S3_RUN_LOG_BUCKET=run-logs
|
||||
S3_TRASH_BUCKET=trash
|
||||
S3_TRASH_RETENTION_DAYS=30
|
||||
|
||||
# rclone RC (HTTP control API). The runtime container starts rclone with
|
||||
# `--rc --rc-addr 0.0.0.0:5572 --rc-no-auth` (see runtime/src/runtime/mount.py),
|
||||
|
||||
Reference in New Issue
Block a user