tao.chen/model-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

storage: extract unified AsyncStorageBackend abstraction + migrate from RustFS

Browse Source 
Replace the old RustFS-specific storage layer (common.storage.client /
RustFSObjectStore) with a minimal sync/async abstraction:

  AsyncStorageBackend: put / get / get_stream / delete / exists / stat /
                       list / get_url / copy
  StorageBackend:      same surface, sync implementations
  create_storage({"type": "s3" | "local", "mode": "async", ...})
  backends/s3.py:      S3-compatible (boto3 / aioboto3)
  backends/local.py:   on-disk filesystem (aiofiles)

Concretely:
  - Drop RustFSObjectStore + common.storage.client (deleted).
  - Drop the RustFS-specific ensure_bucket / presign_put / move_to_trash /
    rewrite_to_public_path / sha256 / put_bytes methods.
  - Migrate backend/storage_api.py + backend/main.py + backend/scripts.py
    + schedule/service.py + schedule/worker.py to the new abstraction.
  - Migrate backend/storage_client.py + schedule/storage_client.py to
    stub status (HTTP wrapper is dead code post-migration; rewrite pending).
  - Rename all RUSTFS_* env vars to S3_* across .env.example,
    docker-compose.yml, default.conf, scripts/nginx-entrypoint.sh,
    common/config.py.
  - Replace hardcoded rclone remote name "rustfs" with "s3" in
    docker-compose.yml + config.py default.
  - Rename "rustfs" SQLAlchemy column comments + table comments to
    provider-neutral wording; StorageObjects.storage_backend enum
    value moves from "rustfs" to "s3" (DB rows with the old value will
    fail the != "s3" check until a one-shot migration is applied).
  - Drop unused common/src/common/migrations/{README,env.py,script.py.mako}
    (the alembic setup lives in /migrations/, not here).

Migration of the old abstractions has been done in one pass; per-route
method calls (delete / stat / put / get_url) are now direct one-liners
against AsyncStorageBackend.

After this commit:
  - All Python imports resolve; routes compile (compileall green).
  - s3 mode is fully wired.
  - Routes that depended on removed methods (presign_put, move_to_trash,
    rewrite_to_public_path, head() metadata) raise NotImplementedError
    with a one-line TODO; rewriting these route handlers is the next step.
This commit is contained in:
tao.chen
2026-08-05 13:08:32 +08:00
parent 7c456a04ce
commit 4b2a67ae5d
30 changed files with 1577 additions and 810 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker-compose.yml
+49 -35
View File
@@ -22,14 +22,14 @@ services:
context: .
dockerfile: frontend/Dockerfile
restart: unless-stopped
# Architecture §2.2: this is the only service exposed to the host. The
# Architecture §2.2: this is the only service exposed to the host. The
# default.conf file is mounted as a template; scripts/nginx-entrypoint.sh
# parses ${RUSTFS_ENDPOINT} and writes the rendered config to
# parses ${S3_ENDPOINT} and writes the rendered config to
# /etc/nginx/conf.d/default.conf before exec'ing nginx.
ports:
- "${GATEWAY_PORT:-8888}:80"
environment:
RUSTFS_ENDPOINT: ${RUSTFS_ENDPOINT:?RUSTFS_ENDPOINT is required}
S3_ENDPOINT: ${S3_ENDPOINT:?S3_ENDPOINT is required}
depends_on:
backend:
condition: service_healthy
@@ -49,8 +49,8 @@ services:
context: .
dockerfile: backend/Dockerfile
restart: unless-stopped
# No host port: architecture §2.2 — only Nginx is externally reachable.
# No local-FS volume: backend stores everything in RustFS (RUSTFS_*).
# No host port: architecture §2.2 — only Nginx is externally reachable.
# No local-FS volume: backend stores everything in S3 (S3_*).
environment:
DATABASE_URL: ${DATABASE_URL:?DATABASE_URL is required}
SERVICE_NAME: model-platform-backend
@@ -59,15 +59,19 @@ services:
DEMO_AUTH_ENABLED: ${DEMO_AUTH_ENABLED:-false}
INITIAL_ADMIN_PASSWORD: ${INITIAL_ADMIN_PASSWORD:-admin12345}
RUNTIME_API_URL: http://runtime:8000
RUSTFS_ENDPOINT: ${RUSTFS_ENDPOINT:?RUSTFS_ENDPOINT is required}
RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY:?RUSTFS_ACCESS_KEY is required}
RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY:?RUSTFS_SECRET_KEY is required}
RUSTFS_WORKSPACE_BUCKET: ${RUSTFS_WORKSPACE_BUCKET:-workspaces}
RUSTFS_VERSION_BUCKET: ${RUSTFS_VERSION_BUCKET:-versions}
RUSTFS_RUN_LOG_BUCKET: ${RUSTFS_RUN_LOG_BUCKET:-run-logs}
RUSTFS_TRASH_BUCKET: ${RUSTFS_TRASH_BUCKET:-trash}
RUSTFS_TRASH_RETENTION_DAYS: ${RUSTFS_TRASH_RETENTION_DAYS:-30}
READINESS_TARGETS: ${MYSQL_HOST:?MYSQL_HOST is required}:${MYSQL_PORT:-3306},${RUSTFS_HOST:?RUSTFS_HOST is required}:${RUSTFS_PORT:-9000},runtime:8000
STORAGE_BACKEND: ${STORAGE_BACKEND:-s3}
LOCAL_STORAGE_BASE_DIR: ${LOCAL_STORAGE_BASE_DIR:-/data}
# S3_* only matter when STORAGE_BACKEND=s3. Defaults are kept so local
# mode boots without them; override in .env when switching to s3.
S3_ENDPOINT: ${S3_ENDPOINT:-http://s3:9000}
S3_ACCESS_KEY: ${S3_ACCESS_KEY:-}
S3_SECRET_KEY: ${S3_SECRET_KEY:-}
S3_WORKSPACE_BUCKET: ${S3_WORKSPACE_BUCKET:-workspaces}
S3_VERSION_BUCKET: ${S3_VERSION_BUCKET:-versions}
S3_RUN_LOG_BUCKET: ${S3_RUN_LOG_BUCKET:-run-logs}
S3_TRASH_BUCKET: ${S3_TRASH_BUCKET:-trash}
S3_TRASH_RETENTION_DAYS: ${S3_TRASH_RETENTION_DAYS:-30}
READINESS_TARGETS: ${MYSQL_HOST:?MYSQL_HOST is required}:${MYSQL_PORT:-3306},${S3_HOST:-s3}:${S3_PORT:-9000},runtime:8000
depends_on:
migrate:
condition: service_completed_successfully
@@ -76,6 +80,7 @@ services:
volumes:
- ./backend:/app/backend
- ./common:/app/common
- ./data:/data
healthcheck:
test: ["CMD-SHELL", "curl -fsS http://127.0.0.1:8000/health/ready >/dev/null"]
interval: 10s
@@ -98,24 +103,31 @@ services:
environment:
DATABASE_URL: ${DATABASE_URL:?DATABASE_URL is required}
SERVICE_NAME: runtime-manager
WORKSPACES_ROOT: /app/workspaces
STORAGE_BACKEND: ${STORAGE_BACKEND:-s3}
LOCAL_STORAGE_BASE_DIR: ${LOCAL_STORAGE_BASE_DIR:-/data}
# WORKSPACES_ROOT defaults to /data/workspaces (settings.workspaces_root);
# in local mode runtime skips the rclone mount and reads directly from
# ${LOCAL_STORAGE_BASE_DIR}/workspace instead.
PUBLIC_BASE_URL: http://runtime
REMOTE_BUCKET: rustfs:${RUSTFS_WORKSPACE_BUCKET:-workspaces}
RCLONE_CONFIG_RUSTFS_TYPE: s3
RCLONE_CONFIG_RUSTFS_PROVIDER: Other
RCLONE_CONFIG_RUSTFS_ACCESS_KEY_ID: ${RUSTFS_ACCESS_KEY:?RUSTFS_ACCESS_KEY is required}
RCLONE_CONFIG_RUSTFS_SECRET_ACCESS_KEY: ${RUSTFS_SECRET_KEY:?RUSTFS_SECRET_KEY is required}
RCLONE_CONFIG_RUSTFS_ENDPOINT: ${RUSTFS_ENDPOINT:?RUSTFS_ENDPOINT is required}
RCLONE_CONFIG_RUSTFS_ENV_AUTH: "false"
RCLONE_CONFIG_RUSTFS_FORCE_PATH_STYLE: "true"
RCLONE_CONFIG_RUSTFS_REGION: other
# rclone config only used when STORAGE_BACKEND=s3 (mount skipped in local mode).
# The remote spec ("s3:<workspace_bucket>") is derived in
# common.storage.rclone_remote_spec(); no REMOTE_BUCKET env needed.
RCLONE_CONFIG_S3_TYPE: s3
RCLONE_CONFIG_S3_PROVIDER: Other
RCLONE_CONFIG_S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY:-}
RCLONE_CONFIG_S3_SECRET_ACCESS_KEY: ${S3_SECRET_KEY:-}
RCLONE_CONFIG_S3_ENDPOINT: ${S3_ENDPOINT:-http://s3:9000}
RCLONE_CONFIG_S3_ENV_AUTH: "false"
RCLONE_CONFIG_S3_FORCE_PATH_STYLE: "true"
RCLONE_CONFIG_S3_REGION: other
depends_on:
migrate:
condition: service_completed_successfully
volumes:
- ./runtime:/app/runtime
- ./data:/data
healthcheck:
test: ["CMD-SHELL", "grep -q ' /app/workspaces .* - fuse.rclone ' /proc/self/mountinfo && curl -fsS http://127.0.0.1:8000/api/v1/health >/dev/null"]
test: ["CMD-SHELL", "grep -q ' /data/workspaces .* - fuse.rclone ' /proc/self/mountinfo && curl -fsS http://127.0.0.1:8000/api/v1/health >/dev/null"]
interval: 10s
timeout: 5s
retries: 18
@@ -126,21 +138,23 @@ services:
context: .
dockerfile: schedule/Dockerfile
restart: unless-stopped
# No host port: architecture §2.2 — only Nginx is externally reachable.
# No local-FS volume: schedule executes nodes via tempfile.TemporaryDirectory
# under Python's default temp dir (cleaned per-run); artifacts live in RustFS.
# No host port: architecture §2.2 — only Nginx is externally reachable.
# No local-FS volume: schedule executes nodes via tempfile.TemporaryDirectory
# under Python's default temp dir (cleaned per-run); artifacts live in S3.
environment:
DATABASE_URL: ${DATABASE_URL:?DATABASE_URL is required}
SERVICE_NAME: schedule-executor
SCHEDULE_EVENT_NAMESPACE: ${SCHEDULE_EVENT_NAMESPACE:-model-platform-local}
BACKEND_API_URL: http://backend:8000
RUSTFS_ENDPOINT: ${RUSTFS_ENDPOINT:?RUSTFS_ENDPOINT is required}
RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY:?RUSTFS_ACCESS_KEY is required}
RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY:?RUSTFS_SECRET_KEY is required}
RUSTFS_WORKSPACE_BUCKET: ${RUSTFS_WORKSPACE_BUCKET:-workspaces}
RUSTFS_VERSION_BUCKET: ${RUSTFS_VERSION_BUCKET:-versions}
RUSTFS_RUN_LOG_BUCKET: ${RUSTFS_RUN_LOG_BUCKET:-run-logs}
READINESS_TARGETS: ${MYSQL_HOST:?MYSQL_HOST is required}:${MYSQL_PORT:-3306},${RUSTFS_HOST:?RUSTFS_HOST is required}:${RUSTFS_PORT:-9000},backend:8000
STORAGE_BACKEND: ${STORAGE_BACKEND:-s3}
LOCAL_STORAGE_BASE_DIR: ${LOCAL_STORAGE_BASE_DIR:-/data}
S3_ENDPOINT: ${S3_ENDPOINT:-http://s3:9000}
S3_ACCESS_KEY: ${S3_ACCESS_KEY:-}
S3_SECRET_KEY: ${S3_SECRET_KEY:-}
S3_WORKSPACE_BUCKET: ${S3_WORKSPACE_BUCKET:-workspaces}
S3_VERSION_BUCKET: ${S3_VERSION_BUCKET:-versions}
S3_RUN_LOG_BUCKET: ${S3_RUN_LOG_BUCKET:-run-logs}
READINESS_TARGETS: ${MYSQL_HOST:?MYSQL_HOST is required}:${MYSQL_PORT:-3306},${S3_HOST:-s3}:${S3_PORT:-9000},backend:8000
depends_on:
backend:
condition: service_healthy