The factory now picks between two backends based on
settings.storage_backend ("s3" default, "local" for dev / single-node /
air-gapped deployments). The new factory helper build_storage_config()
takes one of the 4 PURPOSE_BUCKETS ("workspace" | "version" |
"run_log" | "trash") and returns the kwargs for create_storage(...).
s3 mode: AsyncStorageBackend over an S3-compatible service
(S3_WORKSPACE_BUCKET etc. as separate buckets).
local mode: AsyncStorageBackend over on-disk files; the 4 buckets
become subdirectories of LOCAL_STORAGE_BASE_DIR (default
"/data"), so the same 4-bucket layout works in both modes.
Concretely:
- common/config.py: add storage_backend (default "s3") +
local_storage_base_dir (default "/data").
- common/storage/factory.py: add PURPOSE_BUCKETS constant +
build_storage_config(bucket_name) helper.
- backend/main.py + backend/storage_api.py: lifespan collapses the
4-instance construction into one dict comprehension:
app.state.object_stores = {
name: create_storage(build_storage_config(name))
for name in PURPOSE_BUCKETS
}
(was 4x ~10-line dicts, one per bucket).
- runtime/mount.py: when STORAGE_BACKEND=local, skip the rclone mount
entirely (the shared docker volume at LOCAL_STORAGE_BASE_DIR is the
store; runtime reads directly).
- docker-compose.yml: mount the shared local-storage volume at /data
in both backend and runtime containers.
- .env.example: document STORAGE_BACKEND + LOCAL_STORAGE_BASE_DIR.
Dependencies added to support both backends:
- aiofiles>=25.1.0 (local async I/O) to backend + common + runtime.
- aioboto3>=15.5.0 (async S3) to common.
- uv.lock regenerated.
After this commit, both modes deploy end-to-end. The s3 mode is the
production default; local mode is opt-in via STORAGE_BACKEND=local.
Replace uvicorn with gunicorn + UvicornWorker in backend, runtime,
and schedule Dockerfiles. New gunicorn.conf.py per service exposes
bind/workers/timeout/graceful_timeout as GUNICORN_* env knobs.
Notable details:
- schedule: timeout=0 (disables gunicorn worker heartbeat) so long
notebook execution isn't killed by gunicorn's silent-worker kill.
- All three configs: drop dead threads=4 setting; UvicornWorker is
async and ignores threads.
- Pin gunicorn>=26.0.0 in each pyproject; uv.lock regenerated.
- Drop stale 'COPY contracts ./contracts' from runtime and schedule
Dockerfiles (contracts dir was deleted in an earlier refactor;
builds would have failed).
- backend Dockerfile: switch to uv sync layout matching runtime/
schedule; add build deps for native wheels.
Co-Authored-By: Claude <noreply@anthropic.com>
tao.chen