refactor: object_key flat layout + usage_type→bucket routing + Settings singleton
- common.config.Settings: pydantic-settings with @lru_cache singleton;
all env vars now declared in one place (database / JWT / RUSTFS_*
credentials + 3 purpose-named buckets / workspace FS roots / etc.).
Replaces os.environ / os.getenv in backend / schedule / runtime /
common modules.
- storage_api: object_key layout flattens from
"{ws}/{usage_type}/{ulid}/{name}" to "{ws}/{ulid}". File name, type,
and logical path live in the StorageObjects / Scripts row, not in
the S3 key, so the bucket can be re-organised without a DB rewrite.
- storage_api: new BUCKET_FOR_USAGE map and resolve_bucket() helper
route uploads by usage_type to the right purpose-named bucket:
working_copy / public_script / data_resource / snapshot
→ RUSTFS_WORKSPACE_BUCKET (workspaces)
version_artifact
→ RUSTFS_VERSION_BUCKET (versions)
run_log / run_result
→ RUSTFS_RUN_LOG_BUCKET (run-logs)
workspace.artifact_bucket override wins over the default for that
workspace. Unknown usage_type falls through to the workspace bucket
so uploads are never silently dropped.
- backend.main lifespan: ensure_bucket loops over all three buckets at
startup.
- common.storage.schemas: extend usage_type Literal to include
working_copy / public_script (consumed by scripts.py after the local
FS removal).
- common.storage.client: raise StorageClientError / StorageUnavailable /
StorageRequestFailed instead of FastAPI HTTPException, so the client
is usable from non-FastAPI contexts (e.g. schedule worker). The
register_workspace_object method is removed (the local-FS path it
routed to no longer exists).
- common.pyproject.toml: add greenlet>=3.0.0 (SQLAlchemy 2.0 async
engine.dispose() requires it) and pydantic-settings>=2.14.2.
Verified: backend.main 57 routes; docker compose config; 20 SQLAlchemy
tables, 0 ForeignKey; grep os.environ / os.getenv in
backend|schedule|runtime|common = 0.
This commit is contained in:
@@ -34,11 +34,18 @@ async def lifespan(app: Any) -> AsyncIterator[None]:
|
||||
access_key=settings.rustfs_access_key,
|
||||
secret_key=settings.rustfs_secret_key,
|
||||
)
|
||||
# Ensure all three purpose-named buckets exist; the storage edge picks
|
||||
# the right one per upload (see resolve_bucket in storage_api.py).
|
||||
for bucket in (
|
||||
settings.rustfs_workspace_bucket,
|
||||
settings.rustfs_version_bucket,
|
||||
settings.rustfs_run_log_bucket,
|
||||
):
|
||||
await asyncio.to_thread(
|
||||
app.state.object_store.ensure_bucket,
|
||||
bucket,
|
||||
)
|
||||
app.state.default_bucket = settings.rustfs_workspace_bucket
|
||||
await asyncio.to_thread(
|
||||
app.state.object_store.ensure_bucket,
|
||||
app.state.default_bucket,
|
||||
)
|
||||
storage_http_client = httpx.AsyncClient(
|
||||
transport=httpx.ASGITransport(app=app),
|
||||
base_url="http://backend.internal",
|
||||
|
||||
@@ -58,6 +58,39 @@ def safe_file_name(value: str) -> str:
|
||||
return name
|
||||
|
||||
|
||||
# Map an upload's usage_type to the RustFS bucket that should hold the
|
||||
# resulting object. ``usage_type`` is the only signal available at the
|
||||
# storage edge (the request comes from either the public API or the
|
||||
# internal schedule worker), so we make the routing decision in one place
|
||||
# here and let every other layer — server-object create, multipart upload,
|
||||
# direct put — inherit the mapping.
|
||||
BUCKET_FOR_USAGE: dict[str, str] = {
|
||||
"working_copy": settings.rustfs_workspace_bucket,
|
||||
"public_script": settings.rustfs_workspace_bucket,
|
||||
"data_resource": settings.rustfs_workspace_bucket,
|
||||
"snapshot": settings.rustfs_workspace_bucket,
|
||||
"version_artifact": settings.rustfs_version_bucket,
|
||||
"run_log": settings.rustfs_run_log_bucket,
|
||||
"run_result": settings.rustfs_run_log_bucket,
|
||||
}
|
||||
|
||||
|
||||
def resolve_bucket(
|
||||
usage_type: str,
|
||||
*,
|
||||
workspace: Workspaces,
|
||||
) -> str:
|
||||
"""Pick the bucket for ``usage_type``.
|
||||
|
||||
``workspace.artifact_bucket`` (per-workspace override) wins over the
|
||||
usage-type default. An unknown ``usage_type`` falls through to the
|
||||
workspace bucket so we never silently drop an object.
|
||||
"""
|
||||
if workspace.artifact_bucket:
|
||||
return workspace.artifact_bucket
|
||||
return BUCKET_FOR_USAGE.get(usage_type, settings.rustfs_workspace_bucket)
|
||||
|
||||
|
||||
def storage_payload(item: StorageObjects) -> dict[str, Any]:
|
||||
return {
|
||||
"storage_object_id": item.storage_object_id,
|
||||
@@ -165,9 +198,7 @@ async def create_upload_record(
|
||||
upload = existing
|
||||
else:
|
||||
upload_id = new_ulid()
|
||||
bucket_name = (
|
||||
workspace.artifact_bucket or request.app.state.default_bucket
|
||||
)
|
||||
bucket_name = resolve_bucket(payload.usage_type, workspace=workspace)
|
||||
# Object key is a flat two-level path: workspace id + upload id. The
|
||||
# original file name and content type live in the StorageObjects row
|
||||
# (file_name / mime_type / object_key) — they are not part of the
|
||||
|
||||
Reference in New Issue
Block a user