The drop-sync refactor is now contractually enforced:
* create_storage({"mode": "sync"|"async"|<any>}) raises StorageConfigError
-- the regression we most want to catch is someone "restoring" the
sync shape by re-introducing mode= handling.
* create_storage() preserves caller's dict, returns an AsyncStorageBackend
subclass, and wraps constructor TypeError into a useful StorageConfigError.
* build_storage_config emits no mode key on either local or s3 branch.
* register_backend() refuses to silently overwrite a different class on
the same name (collision guard), and accepts same-class re-registration.
* registry surfaces the built-in local + s3 classes after import; the
conflict-test cleanup pattern avoids leaking global state across tests.
Adds [tool.pytest.ini_options] (asyncio_mode=auto, testpaths=tests) so
uv run --package common pytest common/tests works from the workspace root.
17 new tests, all green. Backend suite still 136 passed.
The async-only direction was already the only one used in production:
* create_storage never accepted mode=sync; build_storage_config always
emitted mode=async; zero callers referenced StorageBackend / SyncData
/ S3StorageBackend.sync / LocalStorageBackend.sync anywhere.
* Drop the parallel sync base class, the sync concrete classes in
backends/local.py and backends/s3.py, and the boto3 dependency.
* Drop example_usage.py (zero importers; demonstration code, not part
of the public surface).
* Rename LocalAsyncStorageBackend -> LocalStorageBackend,
S3AsyncStorageBackend -> S3StorageBackend to reflect the single
remaining class per type.
* Tighten create_storage: any mode=... key now raises StorageConfigError
with the new pointer (settings.storage_backend controls behavior).
* Cleanup call sites: schedule.application.service.build_object_store
no longer passes mode=async to create_storage.
* Cosmetic touch-ups in backend/services/storage.py and
common/config.py docstrings where they still said "boto3" instead of
"S3 client".
Public API surface preserved: AsyncStorageBackend / ObjectMeta /
create_storage / build_storage_config / register_backend all keep
their names and call signatures. backend tests: 136 passed.
The factory now picks between two backends based on
settings.storage_backend ("s3" default, "local" for dev / single-node /
air-gapped deployments). The new factory helper build_storage_config()
takes one of the 4 PURPOSE_BUCKETS ("workspace" | "version" |
"run_log" | "trash") and returns the kwargs for create_storage(...).
s3 mode: AsyncStorageBackend over an S3-compatible service
(S3_WORKSPACE_BUCKET etc. as separate buckets).
local mode: AsyncStorageBackend over on-disk files; the 4 buckets
become subdirectories of LOCAL_STORAGE_BASE_DIR (default
"/data"), so the same 4-bucket layout works in both modes.
Concretely:
- common/config.py: add storage_backend (default "s3") +
local_storage_base_dir (default "/data").
- common/storage/factory.py: add PURPOSE_BUCKETS constant +
build_storage_config(bucket_name) helper.
- backend/main.py + backend/storage_api.py: lifespan collapses the
4-instance construction into one dict comprehension:
app.state.object_stores = {
name: create_storage(build_storage_config(name))
for name in PURPOSE_BUCKETS
}
(was 4x ~10-line dicts, one per bucket).
- runtime/mount.py: when STORAGE_BACKEND=local, skip the rclone mount
entirely (the shared docker volume at LOCAL_STORAGE_BASE_DIR is the
store; runtime reads directly).
- docker-compose.yml: mount the shared local-storage volume at /data
in both backend and runtime containers.
- .env.example: document STORAGE_BACKEND + LOCAL_STORAGE_BASE_DIR.
Dependencies added to support both backends:
- aiofiles>=25.1.0 (local async I/O) to backend + common + runtime.
- aioboto3>=15.5.0 (async S3) to common.
- uv.lock regenerated.
After this commit, both modes deploy end-to-end. The s3 mode is the
production default; local mode is opt-in via STORAGE_BACKEND=local.
tao.chenandtao.chen