Develop #16
+27
-6
@@ -286,16 +286,37 @@ See "Adding a new env var" above.
|
|||||||
|
|
||||||
### Wire a new RustFS bucket
|
### Wire a new RustFS bucket
|
||||||
|
|
||||||
|
The current three buckets are wired in `backend/storage_api.py:resolve_bucket`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
BUCKET_FOR_USAGE: dict[str, str] = {
|
||||||
|
"working_copy": settings.rustfs_workspace_bucket,
|
||||||
|
"public_script": settings.rustfs_workspace_bucket,
|
||||||
|
"data_resource": settings.rustfs_workspace_bucket,
|
||||||
|
"snapshot": settings.rustfs_workspace_bucket,
|
||||||
|
"version_artifact": settings.rustfs_version_bucket,
|
||||||
|
"run_log": settings.rustfs_run_log_bucket,
|
||||||
|
"run_result": settings.rustfs_run_log_bucket,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
To add a fourth bucket:
|
||||||
|
|
||||||
1. Add the env var to `Settings`:
|
1. Add the env var to `Settings`:
|
||||||
```python
|
```python
|
||||||
rustfs_<feature>_bucket: str = Field(default="<feature>", description="...")
|
rustfs_<feature>_bucket: str = Field(default="<feature>", description="...")
|
||||||
```
|
```
|
||||||
2. Add to `.env.example`.
|
2. Add to `.env.example` with a one-line comment.
|
||||||
3. Extend the `Literal` in `ServerObjectRequest.usage_type` to include
|
3. Extend the `Literal` in `common/storage/schemas.py` (in
|
||||||
the new `usage_type` value, if applicable.
|
`ServerObjectRequest.usage_type`, `CreateUploadRequest.usage_type`,
|
||||||
4. In the consumer of the bucket, branch on `usage_type` (or the
|
`CompleteUploadRequest.usage_type`) to include the new value.
|
||||||
`bucket_name` argument) and pick the right bucket via
|
4. Add an entry in `BUCKET_FOR_USAGE` mapping the new `usage_type` to
|
||||||
`settings.rustfs_<feature>_bucket`.
|
the new bucket env var.
|
||||||
|
5. Add the bucket to the `ensure_bucket` loop in
|
||||||
|
`backend/main.py` lifespan.
|
||||||
|
|
||||||
|
A workspace's `artifact_bucket` column (when non-null) overrides the
|
||||||
|
default for that workspace, regardless of `usage_type`.
|
||||||
|
|
||||||
### Add a new schedule node type
|
### Add a new schedule node type
|
||||||
|
|
||||||
|
|||||||
@@ -157,26 +157,36 @@ See `DEVELOP.md` for the full list of variables and their meanings.
|
|||||||
|
|
||||||
## Storage layout
|
## Storage layout
|
||||||
|
|
||||||
Single bucket `workspaces` (configurable via `RUSTFS_WORKSPACE_BUCKET`).
|
Three purpose-named RustFS buckets. The mapping from `StorageObjects.usage_type`
|
||||||
|
to bucket is decided in **one place** (`storage_api.py:resolve_bucket`):
|
||||||
|
|
||||||
|
| `usage_type` | Bucket (env var) | Default name |
|
||||||
|
|---|---|---|
|
||||||
|
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `RUSTFS_WORKSPACE_BUCKET` | `workspaces` |
|
||||||
|
| `version_artifact` | `RUSTFS_VERSION_BUCKET` | `versions` |
|
||||||
|
| `run_log`, `run_result` | `RUSTFS_RUN_LOG_BUCKET` | `run-logs` |
|
||||||
|
|
||||||
|
A workspace's `artifact_bucket` column (when non-null) overrides the
|
||||||
|
default for that workspace, regardless of `usage_type` — useful for
|
||||||
|
isolating a paying customer to their own bucket.
|
||||||
|
|
||||||
The object key is a flat two-level path — `workspace_id` and a server-
|
The object key is a flat two-level path — `workspace_id` and a server-
|
||||||
issued `ulid` for the object. The file name, extension, content type,
|
issued `ulid` for the object:
|
||||||
and logical path live in the `StorageObjects` and `Scripts` rows, not in
|
|
||||||
the S3 key, so the bucket can be re-organised without a database
|
|
||||||
rewrite.
|
|
||||||
|
|
||||||
```
|
```
|
||||||
s3://workspaces/
|
s3://workspaces/
|
||||||
└── <workspace_id>/
|
└── <workspace_id>/
|
||||||
├── <ulid-1> # script / notebook / data resource
|
├── <ulid-1> # working_copy / data_resource / snapshot / ...
|
||||||
├── <ulid-2>
|
├── <ulid-2>
|
||||||
└── ...
|
└── ...
|
||||||
|
|
||||||
s3://versions/ (RUSTFS_VERSION_BUCKET — reserved, used by publish_version)
|
s3://versions/<workspace_id>/<ulid> # immutable script versions
|
||||||
s3://run-logs/ (RUSTFS_RUN_LOG_BUCKET — reserved, used by node executor)
|
s3://run-logs/<workspace_id>/<ulid> # node run logs and results
|
||||||
```
|
```
|
||||||
|
|
||||||
To find the original file name and its logical path for a given bucket
|
The file name, extension, content type, and logical path live in the
|
||||||
object, join `StorageObjects.bucket_name + object_key` to the row.
|
`StorageObjects` and `Scripts` rows, not in the S3 key, so the bucket
|
||||||
|
can be re-organised without a database rewrite.
|
||||||
|
|
||||||
Backend code never writes to the container's local filesystem. Schedule
|
Backend code never writes to the container's local filesystem. Schedule
|
||||||
Executor stages node artifacts in `tempfile.TemporaryDirectory()` (auto-
|
Executor stages node artifacts in `tempfile.TemporaryDirectory()` (auto-
|
||||||
|
|||||||
Reference in New Issue
Block a user