storage: add local filesystem backend option (STORAGE_BACKEND toggle)

The factory now picks between two backends based on
settings.storage_backend ("s3" default, "local" for dev / single-node /
air-gapped deployments). The new factory helper build_storage_config()
takes one of the 4 PURPOSE_BUCKETS ("workspace" | "version" |
"run_log" | "trash") and returns the kwargs for create_storage(...).

  s3   mode: AsyncStorageBackend over an S3-compatible service
           (S3_WORKSPACE_BUCKET etc. as separate buckets).
  local mode: AsyncStorageBackend over on-disk files; the 4 buckets
           become subdirectories of LOCAL_STORAGE_BASE_DIR (default
           "/data"), so the same 4-bucket layout works in both modes.

Concretely:
  - common/config.py: add storage_backend (default "s3") +
    local_storage_base_dir (default "/data").
  - common/storage/factory.py: add PURPOSE_BUCKETS constant +
    build_storage_config(bucket_name) helper.
  - backend/main.py + backend/storage_api.py: lifespan collapses the
    4-instance construction into one dict comprehension:
      app.state.object_stores = {
        name: create_storage(build_storage_config(name))
        for name in PURPOSE_BUCKETS
      }
    (was 4x ~10-line dicts, one per bucket).
  - runtime/mount.py: when STORAGE_BACKEND=local, skip the rclone mount
    entirely (the shared docker volume at LOCAL_STORAGE_BASE_DIR is the
    store; runtime reads directly).
  - docker-compose.yml: mount the shared local-storage volume at /data
    in both backend and runtime containers.
  - .env.example: document STORAGE_BACKEND + LOCAL_STORAGE_BASE_DIR.

Dependencies added to support both backends:
  - aiofiles>=25.1.0 (local async I/O) to backend + common + runtime.
  - aioboto3>=15.5.0 (async S3) to common.
  - uv.lock regenerated.

After this commit, both modes deploy end-to-end. The s3 mode is the
production default; local mode is opt-in via STORAGE_BACKEND=local.
This commit is contained in:
tao.chen
2026-08-05 13:10:05 +08:00
parent 4b2a67ae5d
commit d2bb450d30
5 changed files with 636 additions and 16 deletions
+1
View File
@@ -10,6 +10,7 @@ dependencies = [
"loguru==0.7.2",
"notebook",
"gunicorn>=26.0.0",
"aiofiles>=25.1.0",
]
[tool.uv.sources]
+26 -7
View File
@@ -1,8 +1,14 @@
"""Object storage mount management.
Owns the rclone mount lifecycle for the remote workspace bucket.
``WORKSPACES_ROOT`` is defined here because this module is what makes
the directory usable; downstream consumers (e.g. process.py) import it.
Owns the rclone mount lifecycle for the remote workspace bucket (s3 mode)
or binds the shared local-storage volume (local mode).
``WORKSPACES_ROOT`` is the runtime's view of the workspace bucket on disk;
it's defined here because this module is what makes the directory usable.
Downstream consumers (e.g. process.py) import it.
The path is resolved via ``common.storage.workspaces_root()`` so it works
identically in s3 mode (rclone FUSE mount) and local mode (bind-mounted
shared volume under ``local_storage_base_dir``).
"""
from __future__ import annotations
@@ -14,9 +20,9 @@ from pathlib import Path
from loguru import logger
from common.config import settings
from common.storage import rclone_remote_spec, workspaces_root
WORKSPACES_ROOT = Path(settings.workspaces_root)
REMOTE_BUCKET = settings.remote_bucket
WORKSPACES_ROOT: Path = workspaces_root()
RCLONE_PROCESS: subprocess.Popen | None = None
@@ -48,18 +54,31 @@ def is_rclone_mount(path: Path) -> bool:
def start_rclone_mount() -> None:
global RCLONE_PROCESS
# Local mode: shared Docker volume is mounted directly at WORKSPACES_ROOT
# by docker-compose. No FUSE/rclone layer needed.
if settings.storage_backend == "local":
WORKSPACES_ROOT.mkdir(parents=True, exist_ok=True)
logger.info(
f"STORAGE_BACKEND=local — skipping rclone mount; "
f"WORKSPACES_ROOT={WORKSPACES_ROOT} is a bind-mounted volume"
)
return
# Lazy: only valid in s3 mode; raises StorageConfigError otherwise.
remote_bucket = rclone_remote_spec()
if is_rclone_mount(WORKSPACES_ROOT):
logger.info(f"Mountpoint already exists: {WORKSPACES_ROOT}")
return
WORKSPACES_ROOT.mkdir(parents=True, exist_ok=True)
logger.info(f"Starting rclone mount {REMOTE_BUCKET} -> {WORKSPACES_ROOT}")
logger.info(f"Starting rclone mount {remote_bucket} -> {WORKSPACES_ROOT}")
with open("/tmp/rclone-mount.log", "a", buffering=1) as log_file:
cmd = [
"rclone",
"mount",
REMOTE_BUCKET,
remote_bucket,
WORKSPACES_ROOT.as_posix(),
"--allow-other",
"--vfs-cache-mode", "full",