storage: add local filesystem backend option (STORAGE_BACKEND toggle)
The factory now picks between two backends based on
settings.storage_backend ("s3" default, "local" for dev / single-node /
air-gapped deployments). The new factory helper build_storage_config()
takes one of the 4 PURPOSE_BUCKETS ("workspace" | "version" |
"run_log" | "trash") and returns the kwargs for create_storage(...).
s3 mode: AsyncStorageBackend over an S3-compatible service
(S3_WORKSPACE_BUCKET etc. as separate buckets).
local mode: AsyncStorageBackend over on-disk files; the 4 buckets
become subdirectories of LOCAL_STORAGE_BASE_DIR (default
"/data"), so the same 4-bucket layout works in both modes.
Concretely:
- common/config.py: add storage_backend (default "s3") +
local_storage_base_dir (default "/data").
- common/storage/factory.py: add PURPOSE_BUCKETS constant +
build_storage_config(bucket_name) helper.
- backend/main.py + backend/storage_api.py: lifespan collapses the
4-instance construction into one dict comprehension:
app.state.object_stores = {
name: create_storage(build_storage_config(name))
for name in PURPOSE_BUCKETS
}
(was 4x ~10-line dicts, one per bucket).
- runtime/mount.py: when STORAGE_BACKEND=local, skip the rclone mount
entirely (the shared docker volume at LOCAL_STORAGE_BASE_DIR is the
store; runtime reads directly).
- docker-compose.yml: mount the shared local-storage volume at /data
in both backend and runtime containers.
- .env.example: document STORAGE_BACKEND + LOCAL_STORAGE_BASE_DIR.
Dependencies added to support both backends:
- aiofiles>=25.1.0 (local async I/O) to backend + common + runtime.
- aioboto3>=15.5.0 (async S3) to common.
- uv.lock regenerated.
After this commit, both modes deploy end-to-end. The s3 mode is the
production default; local mode is opt-in via STORAGE_BACKEND=local.
This commit is contained in:
@@ -10,6 +10,7 @@ dependencies = [
|
||||
"loguru==0.7.2",
|
||||
"notebook",
|
||||
"gunicorn>=26.0.0",
|
||||
"aiofiles>=25.1.0",
|
||||
]
|
||||
|
||||
[tool.uv.sources]
|
||||
|
||||
@@ -1,8 +1,14 @@
|
||||
"""Object storage mount management.
|
||||
|
||||
Owns the rclone mount lifecycle for the remote workspace bucket.
|
||||
``WORKSPACES_ROOT`` is defined here because this module is what makes
|
||||
the directory usable; downstream consumers (e.g. process.py) import it.
|
||||
Owns the rclone mount lifecycle for the remote workspace bucket (s3 mode)
|
||||
or binds the shared local-storage volume (local mode).
|
||||
``WORKSPACES_ROOT`` is the runtime's view of the workspace bucket on disk;
|
||||
it's defined here because this module is what makes the directory usable.
|
||||
Downstream consumers (e.g. process.py) import it.
|
||||
|
||||
The path is resolved via ``common.storage.workspaces_root()`` so it works
|
||||
identically in s3 mode (rclone FUSE mount) and local mode (bind-mounted
|
||||
shared volume under ``local_storage_base_dir``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -14,9 +20,9 @@ from pathlib import Path
|
||||
from loguru import logger
|
||||
|
||||
from common.config import settings
|
||||
from common.storage import rclone_remote_spec, workspaces_root
|
||||
|
||||
WORKSPACES_ROOT = Path(settings.workspaces_root)
|
||||
REMOTE_BUCKET = settings.remote_bucket
|
||||
WORKSPACES_ROOT: Path = workspaces_root()
|
||||
|
||||
RCLONE_PROCESS: subprocess.Popen | None = None
|
||||
|
||||
@@ -48,18 +54,31 @@ def is_rclone_mount(path: Path) -> bool:
|
||||
|
||||
def start_rclone_mount() -> None:
|
||||
global RCLONE_PROCESS
|
||||
# Local mode: shared Docker volume is mounted directly at WORKSPACES_ROOT
|
||||
# by docker-compose. No FUSE/rclone layer needed.
|
||||
if settings.storage_backend == "local":
|
||||
WORKSPACES_ROOT.mkdir(parents=True, exist_ok=True)
|
||||
logger.info(
|
||||
f"STORAGE_BACKEND=local — skipping rclone mount; "
|
||||
f"WORKSPACES_ROOT={WORKSPACES_ROOT} is a bind-mounted volume"
|
||||
)
|
||||
return
|
||||
|
||||
# Lazy: only valid in s3 mode; raises StorageConfigError otherwise.
|
||||
remote_bucket = rclone_remote_spec()
|
||||
|
||||
if is_rclone_mount(WORKSPACES_ROOT):
|
||||
logger.info(f"Mountpoint already exists: {WORKSPACES_ROOT}")
|
||||
return
|
||||
|
||||
WORKSPACES_ROOT.mkdir(parents=True, exist_ok=True)
|
||||
logger.info(f"Starting rclone mount {REMOTE_BUCKET} -> {WORKSPACES_ROOT}")
|
||||
logger.info(f"Starting rclone mount {remote_bucket} -> {WORKSPACES_ROOT}")
|
||||
|
||||
with open("/tmp/rclone-mount.log", "a", buffering=1) as log_file:
|
||||
cmd = [
|
||||
"rclone",
|
||||
"mount",
|
||||
REMOTE_BUCKET,
|
||||
remote_bucket,
|
||||
WORKSPACES_ROOT.as_posix(),
|
||||
"--allow-other",
|
||||
"--vfs-cache-mode", "full",
|
||||
|
||||
Reference in New Issue
Block a user