tao.chen
d2bb450d30
The factory now picks between two backends based on
settings.storage_backend ("s3" default, "local" for dev / single-node /
air-gapped deployments). The new factory helper build_storage_config()
takes one of the 4 PURPOSE_BUCKETS ("workspace" | "version" |
"run_log" | "trash") and returns the kwargs for create_storage(...).
s3 mode: AsyncStorageBackend over an S3-compatible service
(S3_WORKSPACE_BUCKET etc. as separate buckets).
local mode: AsyncStorageBackend over on-disk files; the 4 buckets
become subdirectories of LOCAL_STORAGE_BASE_DIR (default
"/data"), so the same 4-bucket layout works in both modes.
Concretely:
- common/config.py: add storage_backend (default "s3") +
local_storage_base_dir (default "/data").
- common/storage/factory.py: add PURPOSE_BUCKETS constant +
build_storage_config(bucket_name) helper.
- backend/main.py + backend/storage_api.py: lifespan collapses the
4-instance construction into one dict comprehension:
app.state.object_stores = {
name: create_storage(build_storage_config(name))
for name in PURPOSE_BUCKETS
}
(was 4x ~10-line dicts, one per bucket).
- runtime/mount.py: when STORAGE_BACKEND=local, skip the rclone mount
entirely (the shared docker volume at LOCAL_STORAGE_BASE_DIR is the
store; runtime reads directly).
- docker-compose.yml: mount the shared local-storage volume at /data
in both backend and runtime containers.
- .env.example: document STORAGE_BACKEND + LOCAL_STORAGE_BASE_DIR.
Dependencies added to support both backends:
- aiofiles>=25.1.0 (local async I/O) to backend + common + runtime.
- aioboto3>=15.5.0 (async S3) to common.
- uv.lock regenerated.
After this commit, both modes deploy end-to-end. The s3 mode is the
production default; local mode is opt-in via STORAGE_BACKEND=local.
29 lines
537 B
TOML
29 lines
537 B
TOML
[project]
|
|
name = "runtime"
|
|
version = "0.2.0"
|
|
requires-python = ">=3.12"
|
|
dependencies = [
|
|
"common",
|
|
"fastapi==0.116.1",
|
|
"uvicorn[standard]==0.35.0",
|
|
"httpx==0.28.1",
|
|
"loguru==0.7.2",
|
|
"notebook",
|
|
"gunicorn>=26.0.0",
|
|
"aiofiles>=25.1.0",
|
|
]
|
|
|
|
[tool.uv.sources]
|
|
common = { workspace = true }
|
|
|
|
[[tool.uv.index]]
|
|
url = "https://pypi.tuna.tsinghua.edu.cn/simple/"
|
|
default = true
|
|
|
|
[build-system]
|
|
requires = ["hatchling"]
|
|
build-backend = "hatchling.build"
|
|
|
|
[tool.hatch.build.targets.wheel]
|
|
packages = ["src/runtime"]
|