Replace the old RustFS-specific storage layer (common.storage.client /
RustFSObjectStore) with a minimal sync/async abstraction:
AsyncStorageBackend: put / get / get_stream / delete / exists / stat /
list / get_url / copy
StorageBackend: same surface, sync implementations
create_storage({"type": "s3" | "local", "mode": "async", ...})
backends/s3.py: S3-compatible (boto3 / aioboto3)
backends/local.py: on-disk filesystem (aiofiles)
Concretely:
- Drop RustFSObjectStore + common.storage.client (deleted).
- Drop the RustFS-specific ensure_bucket / presign_put / move_to_trash /
rewrite_to_public_path / sha256 / put_bytes methods.
- Migrate backend/storage_api.py + backend/main.py + backend/scripts.py
+ schedule/service.py + schedule/worker.py to the new abstraction.
- Migrate backend/storage_client.py + schedule/storage_client.py to
stub status (HTTP wrapper is dead code post-migration; rewrite pending).
- Rename all RUSTFS_* env vars to S3_* across .env.example,
docker-compose.yml, default.conf, scripts/nginx-entrypoint.sh,
common/config.py.
- Replace hardcoded rclone remote name "rustfs" with "s3" in
docker-compose.yml + config.py default.
- Rename "rustfs" SQLAlchemy column comments + table comments to
provider-neutral wording; StorageObjects.storage_backend enum
value moves from "rustfs" to "s3" (DB rows with the old value will
fail the != "s3" check until a one-shot migration is applied).
- Drop unused common/src/common/migrations/{README,env.py,script.py.mako}
(the alembic setup lives in /migrations/, not here).
Migration of the old abstractions has been done in one pass; per-route
method calls (delete / stat / put / get_url) are now direct one-liners
against AsyncStorageBackend.
After this commit:
- All Python imports resolve; routes compile (compileall green).
- s3 mode is fully wired.
- Routes that depended on removed methods (presign_put, move_to_trash,
rewrite_to_public_path, head() metadata) raise NotImplementedError
with a one-line TODO; rewriting these route handlers is the next step.
- scripts.update_script: repoint script.current_object_id to the
newly uploaded StorageObject and best-effort delete the old
working copy. Previously the old row was mutated with the new
content_hash while the script still pointed at it, silently
losing user edits on the next publish_version.
- scripts.publish_version: read object bytes via the new
RustFSObjectStore.get_bytes() instead of the non-existent
.get_object(); publish 500'd on every call.
- scripts.upload_script / storage_api.complete_upload_record:
Path(file_name) raised NameError (only PurePosixPath imported),
crashing every upload and every upload finalization. Use
PurePosixPath.
- RustFSObjectStore: add get_bytes() helper (sync, body.close in
finally) for in-process callers that need raw bytes.
Co-Authored-By: Claude <noreply@anthropic.com>
- common.config.Settings: pydantic-settings with @lru_cache singleton;
all env vars now declared in one place (database / JWT / RUSTFS_*
credentials + 3 purpose-named buckets / workspace FS roots / etc.).
Replaces os.environ / os.getenv in backend / schedule / runtime /
common modules.
- storage_api: object_key layout flattens from
"{ws}/{usage_type}/{ulid}/{name}" to "{ws}/{ulid}". File name, type,
and logical path live in the StorageObjects / Scripts row, not in
the S3 key, so the bucket can be re-organised without a DB rewrite.
- storage_api: new BUCKET_FOR_USAGE map and resolve_bucket() helper
route uploads by usage_type to the right purpose-named bucket:
working_copy / public_script / data_resource / snapshot
→ RUSTFS_WORKSPACE_BUCKET (workspaces)
version_artifact
→ RUSTFS_VERSION_BUCKET (versions)
run_log / run_result
→ RUSTFS_RUN_LOG_BUCKET (run-logs)
workspace.artifact_bucket override wins over the default for that
workspace. Unknown usage_type falls through to the workspace bucket
so uploads are never silently dropped.
- backend.main lifespan: ensure_bucket loops over all three buckets at
startup.
- common.storage.schemas: extend usage_type Literal to include
working_copy / public_script (consumed by scripts.py after the local
FS removal).
- common.storage.client: raise StorageClientError / StorageUnavailable /
StorageRequestFailed instead of FastAPI HTTPException, so the client
is usable from non-FastAPI contexts (e.g. schedule worker). The
register_workspace_object method is removed (the local-FS path it
routed to no longer exists).
- common.pyproject.toml: add greenlet>=3.0.0 (SQLAlchemy 2.0 async
engine.dispose() requires it) and pydantic-settings>=2.14.2.
Verified: backend.main 57 routes; docker compose config; 20 SQLAlchemy
tables, 0 ForeignKey; grep os.environ / os.getenv in
backend|schedule|runtime|common = 0.
tao.chen