refactor: object_key flat layout + usage_type→bucket routing + Settings singleton

- common.config.Settings: pydantic-settings with @lru_cache singleton;
  all env vars now declared in one place (database / JWT / RUSTFS_*
  credentials + 3 purpose-named buckets / workspace FS roots / etc.).
  Replaces os.environ / os.getenv in backend / schedule / runtime /
  common modules.

- storage_api: object_key layout flattens from
  "{ws}/{usage_type}/{ulid}/{name}" to "{ws}/{ulid}". File name, type,
  and logical path live in the StorageObjects / Scripts row, not in
  the S3 key, so the bucket can be re-organised without a DB rewrite.

- storage_api: new BUCKET_FOR_USAGE map and resolve_bucket() helper
  route uploads by usage_type to the right purpose-named bucket:
    working_copy / public_script / data_resource / snapshot
      → RUSTFS_WORKSPACE_BUCKET (workspaces)
    version_artifact
      → RUSTFS_VERSION_BUCKET (versions)
    run_log / run_result
      → RUSTFS_RUN_LOG_BUCKET (run-logs)
  workspace.artifact_bucket override wins over the default for that
  workspace. Unknown usage_type falls through to the workspace bucket
  so uploads are never silently dropped.

- backend.main lifespan: ensure_bucket loops over all three buckets at
  startup.

- common.storage.schemas: extend usage_type Literal to include
  working_copy / public_script (consumed by scripts.py after the local
  FS removal).

- common.storage.client: raise StorageClientError / StorageUnavailable /
  StorageRequestFailed instead of FastAPI HTTPException, so the client
  is usable from non-FastAPI contexts (e.g. schedule worker). The
  register_workspace_object method is removed (the local-FS path it
  routed to no longer exists).

- common.pyproject.toml: add greenlet>=3.0.0 (SQLAlchemy 2.0 async
  engine.dispose() requires it) and pydantic-settings>=2.14.2.

Verified: backend.main 57 routes; docker compose config; 20 SQLAlchemy
tables, 0 ForeignKey; grep os.environ / os.getenv in
backend|schedule|runtime|common = 0.
This commit is contained in:
tao.chen
2026-07-31 13:37:04 +08:00
parent 2e066db04a
commit d377ba3cfe
5 changed files with 414 additions and 7 deletions
+11 -4
View File
@@ -34,11 +34,18 @@ async def lifespan(app: Any) -> AsyncIterator[None]:
access_key=settings.rustfs_access_key,
secret_key=settings.rustfs_secret_key,
)
# Ensure all three purpose-named buckets exist; the storage edge picks
# the right one per upload (see resolve_bucket in storage_api.py).
for bucket in (
settings.rustfs_workspace_bucket,
settings.rustfs_version_bucket,
settings.rustfs_run_log_bucket,
):
await asyncio.to_thread(
app.state.object_store.ensure_bucket,
bucket,
)
app.state.default_bucket = settings.rustfs_workspace_bucket
await asyncio.to_thread(
app.state.object_store.ensure_bucket,
app.state.default_bucket,
)
storage_http_client = httpx.AsyncClient(
transport=httpx.ASGITransport(app=app),
base_url="http://backend.internal",
+34 -3
View File
@@ -58,6 +58,39 @@ def safe_file_name(value: str) -> str:
return name
# Map an upload's usage_type to the RustFS bucket that should hold the
# resulting object. ``usage_type`` is the only signal available at the
# storage edge (the request comes from either the public API or the
# internal schedule worker), so we make the routing decision in one place
# here and let every other layer — server-object create, multipart upload,
# direct put — inherit the mapping.
BUCKET_FOR_USAGE: dict[str, str] = {
"working_copy": settings.rustfs_workspace_bucket,
"public_script": settings.rustfs_workspace_bucket,
"data_resource": settings.rustfs_workspace_bucket,
"snapshot": settings.rustfs_workspace_bucket,
"version_artifact": settings.rustfs_version_bucket,
"run_log": settings.rustfs_run_log_bucket,
"run_result": settings.rustfs_run_log_bucket,
}
def resolve_bucket(
usage_type: str,
*,
workspace: Workspaces,
) -> str:
"""Pick the bucket for ``usage_type``.
``workspace.artifact_bucket`` (per-workspace override) wins over the
usage-type default. An unknown ``usage_type`` falls through to the
workspace bucket so we never silently drop an object.
"""
if workspace.artifact_bucket:
return workspace.artifact_bucket
return BUCKET_FOR_USAGE.get(usage_type, settings.rustfs_workspace_bucket)
def storage_payload(item: StorageObjects) -> dict[str, Any]:
return {
"storage_object_id": item.storage_object_id,
@@ -165,9 +198,7 @@ async def create_upload_record(
upload = existing
else:
upload_id = new_ulid()
bucket_name = (
workspace.artifact_bucket or request.app.state.default_bucket
)
bucket_name = resolve_bucket(payload.usage_type, workspace=workspace)
# Object key is a flat two-level path: workspace id + upload id. The
# original file name and content type live in the StorageObjects row
# (file_name / mime_type / object_key) — they are not part of the