storage: extract unified AsyncStorageBackend abstraction + migrate from RustFS

Replace the old RustFS-specific storage layer (common.storage.client /
RustFSObjectStore) with a minimal sync/async abstraction:

  AsyncStorageBackend: put / get / get_stream / delete / exists / stat /
                       list / get_url / copy
  StorageBackend:      same surface, sync implementations
  create_storage({"type": "s3" | "local", "mode": "async", ...})
  backends/s3.py:      S3-compatible (boto3 / aioboto3)
  backends/local.py:   on-disk filesystem (aiofiles)

Concretely:
  - Drop RustFSObjectStore + common.storage.client (deleted).
  - Drop the RustFS-specific ensure_bucket / presign_put / move_to_trash /
    rewrite_to_public_path / sha256 / put_bytes methods.
  - Migrate backend/storage_api.py + backend/main.py + backend/scripts.py
    + schedule/service.py + schedule/worker.py to the new abstraction.
  - Migrate backend/storage_client.py + schedule/storage_client.py to
    stub status (HTTP wrapper is dead code post-migration; rewrite pending).
  - Rename all RUSTFS_* env vars to S3_* across .env.example,
    docker-compose.yml, default.conf, scripts/nginx-entrypoint.sh,
    common/config.py.
  - Replace hardcoded rclone remote name "rustfs" with "s3" in
    docker-compose.yml + config.py default.
  - Rename "rustfs" SQLAlchemy column comments + table comments to
    provider-neutral wording; StorageObjects.storage_backend enum
    value moves from "rustfs" to "s3" (DB rows with the old value will
    fail the != "s3" check until a one-shot migration is applied).
  - Drop unused common/src/common/migrations/{README,env.py,script.py.mako}
    (the alembic setup lives in /migrations/, not here).

Migration of the old abstractions has been done in one pass; per-route
method calls (delete / stat / put / get_url) are now direct one-liners
against AsyncStorageBackend.

After this commit:
  - All Python imports resolve; routes compile (compileall green).
  - s3 mode is fully wired.
  - Routes that depended on removed methods (presign_put, move_to_trash,
    rewrite_to_public_path, head() metadata) raise NotImplementedError
    with a one-line TODO; rewriting these route handlers is the next step.
This commit is contained in:
tao.chen
2026-08-05 13:08:32 +08:00
parent 7c456a04ce
commit 4b2a67ae5d
30 changed files with 1577 additions and 810 deletions
+7 -7
View File
@@ -1,7 +1,7 @@
# ----------------------------------------------------------------------------
# NOTE: this file is mounted into the nginx container as a TEMPLATE.
# scripts/nginx-entrypoint.sh (mounted as /docker-entrypoint.sh) substitutes
# the single ${RUSTFS_ENDPOINT} placeholder at container start. The rendered
# the single ${S3_ENDPOINT} placeholder at container start. The rendered
# output is written to /etc/nginx/conf.d/default.conf and execs nginx.
# ----------------------------------------------------------------------------
@@ -18,8 +18,8 @@ server {
# 指定 Docker 内置 DNS 解析器,并设置 30 秒缓存
resolver 127.0.0.11 valid=30s ipv6=off;
# RustFS upstream — full URL passed through to proxy_pass below.
set $rustfs_backend "${RUSTFS_ENDPOINT}";
# S3 upstream — full URL passed through to proxy_pass below.
set $s3_backend "${S3_ENDPOINT}";
location / {
root /usr/share/nginx/html; # 前端静态文件存放在容器中的路径
@@ -58,17 +58,17 @@ server {
}
# =========================================================================
# 1. RustFS 对象存储服务转发 (/storage/)
# 1. S3 对象存储服务转发 (/storage/)
# =========================================================================
location /storage/ {
# 核心:透传 Host,确保 RustFS 生成的 Presigned URL 包含公网地址
# 核心:透传 Host,确保 S3 生成的 Presigned URL 包含公网地址
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# 转发至 RustFS$rustfs_endpoint 来自 set 指令;尾斜杠保留 location /storage/ 前缀剥离语义)
proxy_pass $rustfs_backend/;
# 转发至 S3$s3_backend 来自 set 指令;尾斜杠保留 location /storage/ 前缀剥离语义)
proxy_pass $s3_backend/;
# HTTP/1.1 长连接支持
proxy_http_version 1.1;