storage: extract unified AsyncStorageBackend abstraction + migrate from RustFS
Replace the old RustFS-specific storage layer (common.storage.client /
RustFSObjectStore) with a minimal sync/async abstraction:
AsyncStorageBackend: put / get / get_stream / delete / exists / stat /
list / get_url / copy
StorageBackend: same surface, sync implementations
create_storage({"type": "s3" | "local", "mode": "async", ...})
backends/s3.py: S3-compatible (boto3 / aioboto3)
backends/local.py: on-disk filesystem (aiofiles)
Concretely:
- Drop RustFSObjectStore + common.storage.client (deleted).
- Drop the RustFS-specific ensure_bucket / presign_put / move_to_trash /
rewrite_to_public_path / sha256 / put_bytes methods.
- Migrate backend/storage_api.py + backend/main.py + backend/scripts.py
+ schedule/service.py + schedule/worker.py to the new abstraction.
- Migrate backend/storage_client.py + schedule/storage_client.py to
stub status (HTTP wrapper is dead code post-migration; rewrite pending).
- Rename all RUSTFS_* env vars to S3_* across .env.example,
docker-compose.yml, default.conf, scripts/nginx-entrypoint.sh,
common/config.py.
- Replace hardcoded rclone remote name "rustfs" with "s3" in
docker-compose.yml + config.py default.
- Rename "rustfs" SQLAlchemy column comments + table comments to
provider-neutral wording; StorageObjects.storage_backend enum
value moves from "rustfs" to "s3" (DB rows with the old value will
fail the != "s3" check until a one-shot migration is applied).
- Drop unused common/src/common/migrations/{README,env.py,script.py.mako}
(the alembic setup lives in /migrations/, not here).
Migration of the old abstractions has been done in one pass; per-route
method calls (delete / stat / put / get_url) are now direct one-liners
against AsyncStorageBackend.
After this commit:
- All Python imports resolve; routes compile (compileall green).
- s3 mode is fully wired.
- Routes that depended on removed methods (presign_put, move_to_trash,
rewrite_to_public_path, head() metadata) raise NotImplementedError
with a one-line TODO; rewriting these route handlers is the next step.
This commit is contained in:
tao.chen
+35
-33
@@ -59,35 +59,57 @@ class Settings(BaseSettings):
|
||||
description="Backend → rclone RC HTTP endpoint (VFS cache invalidation).",
|
||||
)
|
||||
|
||||
# ── RustFS object storage ────────────────────────────────────
|
||||
rustfs_endpoint: str = Field(
|
||||
default="http://rustfs:9000",
|
||||
description="S3 endpoint for the RustFS upstream.",
|
||||
# ── object storage backend selection ─────────────────────────
|
||||
storage_backend: str = Field(
|
||||
default="s3",
|
||||
description=(
|
||||
"Which storage backend the deployment uses. ``s3`` (default) "
|
||||
"reads the ``s3_*`` settings and connects to an S3-compatible "
|
||||
"service. ``local`` uses on-disk filesystems under "
|
||||
"``local_storage_base_dir`` — useful for dev / single-node / "
|
||||
"air-gapped deployments."
|
||||
),
|
||||
)
|
||||
rustfs_access_key: str = Field(
|
||||
local_storage_base_dir: str = Field(
|
||||
default="/data",
|
||||
description=(
|
||||
"Root directory for the local-filesystem storage backend. The 4 "
|
||||
"buckets become subdirectories: ``<root>/workspace``, "
|
||||
"``<root>/version``, ``<root>/run_log``, ``<root>/trash``. "
|
||||
"Default ``/data``; this directory must be a shared Docker "
|
||||
"volume between the backend and runtime containers in local mode."
|
||||
),
|
||||
)
|
||||
|
||||
# ── S3-compatible object storage ─────────────────────────────
|
||||
s3_endpoint: str = Field(
|
||||
default="http://s3:9000",
|
||||
description="S3 endpoint for the object-storage upstream.",
|
||||
)
|
||||
s3_access_key: str = Field(
|
||||
default="modelplatform",
|
||||
description="boto3 access key for RustFS.",
|
||||
description="boto3 access key for S3-compatible storage.",
|
||||
)
|
||||
rustfs_secret_key: str = Field(
|
||||
s3_secret_key: str = Field(
|
||||
default="modelplatformsecret",
|
||||
description="boto3 secret key for RustFS.",
|
||||
description="boto3 secret key for S3-compatible storage.",
|
||||
)
|
||||
rustfs_workspace_bucket: str = Field(
|
||||
s3_workspace_bucket: str = Field(
|
||||
default="workspaces",
|
||||
description=(
|
||||
"Bucket for workspace files (notebooks / scripts / working "
|
||||
"copies). Layout: s3://<bucket>/<workspace_id>/..."
|
||||
),
|
||||
)
|
||||
rustfs_version_bucket: str = Field(
|
||||
s3_version_bucket: str = Field(
|
||||
default="versions",
|
||||
description="Bucket for immutable script-version artifacts.",
|
||||
)
|
||||
rustfs_run_log_bucket: str = Field(
|
||||
s3_run_log_bucket: str = Field(
|
||||
default="run-logs",
|
||||
description="Bucket for schedule run logs.",
|
||||
)
|
||||
rustfs_trash_bucket: str = Field(
|
||||
s3_trash_bucket: str = Field(
|
||||
default="trash",
|
||||
description=(
|
||||
"Bucket for soft-deleted objects. The source bucket key is "
|
||||
@@ -95,7 +117,7 @@ class Settings(BaseSettings):
|
||||
"Trash is reaped on a schedule out of band."
|
||||
),
|
||||
)
|
||||
rustfs_trash_retention_days: int = Field(
|
||||
s3_trash_retention_days: int = Field(
|
||||
default=30,
|
||||
description=(
|
||||
"How long a trashed object is retained before reaping. "
|
||||
@@ -104,26 +126,6 @@ class Settings(BaseSettings):
|
||||
),
|
||||
)
|
||||
|
||||
# ── local FS roots ────────────────────────────────────────────
|
||||
workspace_root: str = Field(
|
||||
default="/app/workspaces",
|
||||
description=(
|
||||
"Schedule subprocess cwd; staging area for notebook_runner. "
|
||||
"Backend no longer writes here — workspace files live in "
|
||||
"RustFS via rustfs_workspace_bucket."
|
||||
),
|
||||
)
|
||||
workspaces_root: str = Field(
|
||||
default="/app/workspaces",
|
||||
description="Runtime rclone FUSE mount point for the workspace bucket.",
|
||||
)
|
||||
|
||||
# ── rclone remote spec ────────────────────────────────────────
|
||||
remote_bucket: str = Field(
|
||||
default="rustfs:workspaces",
|
||||
description="rclone remote spec for the workspace bucket.",
|
||||
)
|
||||
|
||||
# ── schedule → backend API ────────────────────────────────────
|
||||
backend_api_url: str = Field(
|
||||
default="http://backend:8000",
|
||||
|
||||
Reference in New Issue
Block a user