Develop #16
+27
-6
@@ -286,16 +286,37 @@ See "Adding a new env var" above.
|
||||
|
||||
### Wire a new RustFS bucket
|
||||
|
||||
The current three buckets are wired in `backend/storage_api.py:resolve_bucket`:
|
||||
|
||||
```python
|
||||
BUCKET_FOR_USAGE: dict[str, str] = {
|
||||
"working_copy": settings.rustfs_workspace_bucket,
|
||||
"public_script": settings.rustfs_workspace_bucket,
|
||||
"data_resource": settings.rustfs_workspace_bucket,
|
||||
"snapshot": settings.rustfs_workspace_bucket,
|
||||
"version_artifact": settings.rustfs_version_bucket,
|
||||
"run_log": settings.rustfs_run_log_bucket,
|
||||
"run_result": settings.rustfs_run_log_bucket,
|
||||
}
|
||||
```
|
||||
|
||||
To add a fourth bucket:
|
||||
|
||||
1. Add the env var to `Settings`:
|
||||
```python
|
||||
rustfs_<feature>_bucket: str = Field(default="<feature>", description="...")
|
||||
```
|
||||
2. Add to `.env.example`.
|
||||
3. Extend the `Literal` in `ServerObjectRequest.usage_type` to include
|
||||
the new `usage_type` value, if applicable.
|
||||
4. In the consumer of the bucket, branch on `usage_type` (or the
|
||||
`bucket_name` argument) and pick the right bucket via
|
||||
`settings.rustfs_<feature>_bucket`.
|
||||
2. Add to `.env.example` with a one-line comment.
|
||||
3. Extend the `Literal` in `common/storage/schemas.py` (in
|
||||
`ServerObjectRequest.usage_type`, `CreateUploadRequest.usage_type`,
|
||||
`CompleteUploadRequest.usage_type`) to include the new value.
|
||||
4. Add an entry in `BUCKET_FOR_USAGE` mapping the new `usage_type` to
|
||||
the new bucket env var.
|
||||
5. Add the bucket to the `ensure_bucket` loop in
|
||||
`backend/main.py` lifespan.
|
||||
|
||||
A workspace's `artifact_bucket` column (when non-null) overrides the
|
||||
default for that workspace, regardless of `usage_type`.
|
||||
|
||||
### Add a new schedule node type
|
||||
|
||||
|
||||
@@ -157,26 +157,36 @@ See `DEVELOP.md` for the full list of variables and their meanings.
|
||||
|
||||
## Storage layout
|
||||
|
||||
Single bucket `workspaces` (configurable via `RUSTFS_WORKSPACE_BUCKET`).
|
||||
Three purpose-named RustFS buckets. The mapping from `StorageObjects.usage_type`
|
||||
to bucket is decided in **one place** (`storage_api.py:resolve_bucket`):
|
||||
|
||||
| `usage_type` | Bucket (env var) | Default name |
|
||||
|---|---|---|
|
||||
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `RUSTFS_WORKSPACE_BUCKET` | `workspaces` |
|
||||
| `version_artifact` | `RUSTFS_VERSION_BUCKET` | `versions` |
|
||||
| `run_log`, `run_result` | `RUSTFS_RUN_LOG_BUCKET` | `run-logs` |
|
||||
|
||||
A workspace's `artifact_bucket` column (when non-null) overrides the
|
||||
default for that workspace, regardless of `usage_type` — useful for
|
||||
isolating a paying customer to their own bucket.
|
||||
|
||||
The object key is a flat two-level path — `workspace_id` and a server-
|
||||
issued `ulid` for the object. The file name, extension, content type,
|
||||
and logical path live in the `StorageObjects` and `Scripts` rows, not in
|
||||
the S3 key, so the bucket can be re-organised without a database
|
||||
rewrite.
|
||||
issued `ulid` for the object:
|
||||
|
||||
```
|
||||
s3://workspaces/
|
||||
└── <workspace_id>/
|
||||
├── <ulid-1> # script / notebook / data resource
|
||||
├── <ulid-1> # working_copy / data_resource / snapshot / ...
|
||||
├── <ulid-2>
|
||||
└── ...
|
||||
|
||||
s3://versions/ (RUSTFS_VERSION_BUCKET — reserved, used by publish_version)
|
||||
s3://run-logs/ (RUSTFS_RUN_LOG_BUCKET — reserved, used by node executor)
|
||||
s3://versions/<workspace_id>/<ulid> # immutable script versions
|
||||
s3://run-logs/<workspace_id>/<ulid> # node run logs and results
|
||||
```
|
||||
|
||||
To find the original file name and its logical path for a given bucket
|
||||
object, join `StorageObjects.bucket_name + object_key` to the row.
|
||||
The file name, extension, content type, and logical path live in the
|
||||
`StorageObjects` and `Scripts` rows, not in the S3 key, so the bucket
|
||||
can be re-organised without a database rewrite.
|
||||
|
||||
Backend code never writes to the container's local filesystem. Schedule
|
||||
Executor stages node artifacts in `tempfile.TemporaryDirectory()` (auto-
|
||||
|
||||
Reference in New Issue
Block a user