refactor: object_key flat layout + usage_type→bucket routing + Settings singleton

- common.config.Settings: pydantic-settings with @lru_cache singleton; all env vars now declared in one place (database / JWT / RUSTFS_* credentials + 3 purpose-named buckets / workspace FS roots / etc.). Replaces os.environ / os.getenv in backend / schedule / runtime / common modules. - storage_api: object_key layout flattens from "{ws}/{usage_type}/{ulid}/{name}" to "{ws}/{ulid}". File name, type, and logical path live in the StorageObjects / Scripts row, not in the S3 key, so the bucket can be re-organised without a DB rewrite. - storage_api: new BUCKET_FOR_USAGE map and resolve_bucket() helper route uploads by usage_type to the right purpose-named bucket: working_copy / public_script / data_resource / snapshot → RUSTFS_WORKSPACE_BUCKET (workspaces) version_artifact → RUSTFS_VERSION_BUCKET (versions) run_log / run_result → RUSTFS_RUN_LOG_BUCKET (run-logs) workspace.artifact_bucket override wins over the default for that workspace. Unknown usage_type falls through to the workspace bucket so uploads are never silently dropped. - backend.main lifespan: ensure_bucket loops over all three buckets at startup. - common.storage.schemas: extend usage_type Literal to include working_copy / public_script (consumed by scripts.py after the local FS removal). - common.storage.client: raise StorageClientError / StorageUnavailable / StorageRequestFailed instead of FastAPI HTTPException, so the client is usable from non-FastAPI contexts (e.g. schedule worker). The register_workspace_object method is removed (the local-FS path it routed to no longer exists). - common.pyproject.toml: add greenlet>=3.0.0 (SQLAlchemy 2.0 async engine.dispose() requires it) and pydantic-settings>=2.14.2. Verified: backend.main 57 routes; docker compose config; 20 SQLAlchemy tables, 0 ForeignKey; grep os.environ / os.getenv in backend|schedule|runtime|common = 0.
2026-07-31 13:37:04 +08:00
parent 2e066db04a
commit d377ba3cfe
5 changed files with 414 additions and 7 deletions
@@ -0,0 +1,141 @@
+"""Centralised configuration via pydantic-settings.
+
+All Python services (backend / schedule / runtime) import ``settings`` from
+this module. The instance is a process-wide singleton (lru_cache wrapped),
+so each field is parsed from the environment exactly once at first access.
+
+Rules for adding a new variable:
+
+  1. Add the field here with a sensible default that lets local dev
+     boot without the env var set.
+  2. Use ``settings.<name>`` at the call site. Never re-introduce
+     ``os.environ`` / ``os.getenv`` for application config — they
+     bypass this central registry.
+  3. Document the env var name in ``.env.example`` so operators know
+     it exists.
+"""
+
+from __future__ import annotations
+
+from functools import lru_cache
+
+from pydantic import Field
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
+
+class Settings(BaseSettings):
+    # ── database ──────────────────────────────────────────────────
+    database_url: str = Field(
+        default=(
+            "mysql+asyncmy://model_platform:model_platform@mysql:3306/"
+            "model_platform?charset=utf8mb4"
+        ),
+        description="SQLAlchemy async URI for the platform MySQL.",
+    )
+
+    # ── JWT ───────────────────────────────────────────────────────
+    jwt_secret: str = Field(
+        default="dev-only-not-for-production",
+        description="HS256 secret used by backend's jupyter auth_request.",
+    )
+
+    # ── service identity ──────────────────────────────────────────
+    service_name: str = Field(
+        default="service",
+        description="Service label surfaced in lifespan / health checks.",
+    )
+
+    # ── runtime container endpoint ───────────────────────────────
+    runtime_api_url: str = Field(
+        default="http://runtime:8000",
+        description="Backend → Runtime HTTP endpoint.",
+    )
+
+    # ── RustFS object storage ────────────────────────────────────
+    rustfs_endpoint: str = Field(
+        default="http://rustfs:9000",
+        description="S3 endpoint for the RustFS upstream.",
+    )
+    rustfs_access_key: str = Field(
+        default="modelplatform",
+        description="boto3 access key for RustFS.",
+    )
+    rustfs_secret_key: str = Field(
+        default="modelplatformsecret",
+        description="boto3 secret key for RustFS.",
+    )
+    rustfs_workspace_bucket: str = Field(
+        default="workspaces",
+        description=(
+            "Bucket for workspace files (notebooks / scripts / working "
+            "copies). Layout: s3://<bucket>/<workspace_id>/..."
+        ),
+    )
+    rustfs_version_bucket: str = Field(
+        default="versions",
+        description="Bucket for immutable script-version artifacts.",
+    )
+    rustfs_run_log_bucket: str = Field(
+        default="run-logs",
+        description="Bucket for schedule run logs.",
+    )
+
+    # ── local FS roots ────────────────────────────────────────────
+    workspace_root: str = Field(
+        default="/workspace/workspaces",
+        description=(
+            "Schedule subprocess cwd; staging area for notebook_runner. "
+            "Backend no longer writes here — workspace files live in "
+            "RustFS via rustfs_workspace_bucket."
+        ),
+    )
+    workspaces_root: str = Field(
+        default="/app/workspaces",
+        description="Runtime rclone FUSE mount point for the workspace bucket.",
+    )
+
+    # ── rclone remote spec ────────────────────────────────────────
+    remote_bucket: str = Field(
+        default="rustfs:workspaces",
+        description="rclone remote spec for the workspace bucket.",
+    )
+
+    # ── schedule → backend API ────────────────────────────────────
+    backend_api_url: str = Field(
+        default="http://backend:8000",
+        description="Schedule → Backend HTTP base URL (cron post-back).",
+    )
+
+    # ── runtime public base URL ──────────────────────────────────
+    public_base_url: str = Field(
+        default="http://runtime",
+        description="Public base URL for the runtime container.",
+    )
+
+    # ── readiness probes ──────────────────────────────────────────
+    readiness_targets: str = Field(
+        default="",
+        description=(
+            "Comma-separated host:port list checked by /health/ready. "
+            "Empty disables the check."
+        ),
+    )
+
+    model_config = SettingsConfigDict(
+        env_file=".env",
+        env_file_encoding="utf-8",
+        case_sensitive=False,
+        extra="ignore",
+    )
+
+
+@lru_cache
+def get_settings() -> Settings:
+    """Construct (and cache) the singleton Settings instance."""
+    return Settings()
+
+
+settings: Settings = get_settings()
+
+
+__all__ = ["Settings", "get_settings", "settings"]