diff --git a/DEVELOP.md b/DEVELOP.md index e7330f6..32e1d95 100644 --- a/DEVELOP.md +++ b/DEVELOP.md @@ -286,16 +286,37 @@ See "Adding a new env var" above. ### Wire a new RustFS bucket +The current three buckets are wired in `backend/storage_api.py:resolve_bucket`: + +```python +BUCKET_FOR_USAGE: dict[str, str] = { + "working_copy": settings.rustfs_workspace_bucket, + "public_script": settings.rustfs_workspace_bucket, + "data_resource": settings.rustfs_workspace_bucket, + "snapshot": settings.rustfs_workspace_bucket, + "version_artifact": settings.rustfs_version_bucket, + "run_log": settings.rustfs_run_log_bucket, + "run_result": settings.rustfs_run_log_bucket, +} +``` + +To add a fourth bucket: + 1. Add the env var to `Settings`: ```python rustfs__bucket: str = Field(default="", description="...") ``` -2. Add to `.env.example`. -3. Extend the `Literal` in `ServerObjectRequest.usage_type` to include - the new `usage_type` value, if applicable. -4. In the consumer of the bucket, branch on `usage_type` (or the - `bucket_name` argument) and pick the right bucket via - `settings.rustfs__bucket`. +2. Add to `.env.example` with a one-line comment. +3. Extend the `Literal` in `common/storage/schemas.py` (in + `ServerObjectRequest.usage_type`, `CreateUploadRequest.usage_type`, + `CompleteUploadRequest.usage_type`) to include the new value. +4. Add an entry in `BUCKET_FOR_USAGE` mapping the new `usage_type` to + the new bucket env var. +5. Add the bucket to the `ensure_bucket` loop in + `backend/main.py` lifespan. + +A workspace's `artifact_bucket` column (when non-null) overrides the +default for that workspace, regardless of `usage_type`. ### Add a new schedule node type diff --git a/README.md b/README.md index 746eefe..b1c0fc8 100644 --- a/README.md +++ b/README.md @@ -157,26 +157,36 @@ See `DEVELOP.md` for the full list of variables and their meanings. ## Storage layout -Single bucket `workspaces` (configurable via `RUSTFS_WORKSPACE_BUCKET`). +Three purpose-named RustFS buckets. The mapping from `StorageObjects.usage_type` +to bucket is decided in **one place** (`storage_api.py:resolve_bucket`): + +| `usage_type` | Bucket (env var) | Default name | +|---|---|---| +| `working_copy`, `public_script`, `data_resource`, `snapshot` | `RUSTFS_WORKSPACE_BUCKET` | `workspaces` | +| `version_artifact` | `RUSTFS_VERSION_BUCKET` | `versions` | +| `run_log`, `run_result` | `RUSTFS_RUN_LOG_BUCKET` | `run-logs` | + +A workspace's `artifact_bucket` column (when non-null) overrides the +default for that workspace, regardless of `usage_type` — useful for +isolating a paying customer to their own bucket. + The object key is a flat two-level path — `workspace_id` and a server- -issued `ulid` for the object. The file name, extension, content type, -and logical path live in the `StorageObjects` and `Scripts` rows, not in -the S3 key, so the bucket can be re-organised without a database -rewrite. +issued `ulid` for the object: ``` s3://workspaces/ └── / - ├── # script / notebook / data resource + ├── # working_copy / data_resource / snapshot / ... ├── └── ... -s3://versions/ (RUSTFS_VERSION_BUCKET — reserved, used by publish_version) -s3://run-logs/ (RUSTFS_RUN_LOG_BUCKET — reserved, used by node executor) +s3://versions// # immutable script versions +s3://run-logs// # node run logs and results ``` -To find the original file name and its logical path for a given bucket -object, join `StorageObjects.bucket_name + object_key` to the row. +The file name, extension, content type, and logical path live in the +`StorageObjects` and `Scripts` rows, not in the S3 key, so the bucket +can be re-organised without a database rewrite. Backend code never writes to the container's local filesystem. Schedule Executor stages node artifacts in `tempfile.TemporaryDirectory()` (auto-