Develop #16

Merged
tao.chen merged 273 commits from develop into main 2026-08-21 10:42:09 +08:00
2 changed files with 47 additions and 16 deletions
Showing only changes of commit 2874ccdce3 - Show all commits
+27 -6
View File
@@ -286,16 +286,37 @@ See "Adding a new env var" above.
### Wire a new RustFS bucket
The current three buckets are wired in `backend/storage_api.py:resolve_bucket`:
```python
BUCKET_FOR_USAGE: dict[str, str] = {
"working_copy": settings.rustfs_workspace_bucket,
"public_script": settings.rustfs_workspace_bucket,
"data_resource": settings.rustfs_workspace_bucket,
"snapshot": settings.rustfs_workspace_bucket,
"version_artifact": settings.rustfs_version_bucket,
"run_log": settings.rustfs_run_log_bucket,
"run_result": settings.rustfs_run_log_bucket,
}
```
To add a fourth bucket:
1. Add the env var to `Settings`:
```python
rustfs_<feature>_bucket: str = Field(default="<feature>", description="...")
```
2. Add to `.env.example`.
3. Extend the `Literal` in `ServerObjectRequest.usage_type` to include
the new `usage_type` value, if applicable.
4. In the consumer of the bucket, branch on `usage_type` (or the
`bucket_name` argument) and pick the right bucket via
`settings.rustfs_<feature>_bucket`.
2. Add to `.env.example` with a one-line comment.
3. Extend the `Literal` in `common/storage/schemas.py` (in
`ServerObjectRequest.usage_type`, `CreateUploadRequest.usage_type`,
`CompleteUploadRequest.usage_type`) to include the new value.
4. Add an entry in `BUCKET_FOR_USAGE` mapping the new `usage_type` to
the new bucket env var.
5. Add the bucket to the `ensure_bucket` loop in
`backend/main.py` lifespan.
A workspace's `artifact_bucket` column (when non-null) overrides the
default for that workspace, regardless of `usage_type`.
### Add a new schedule node type
+20 -10
View File
@@ -157,26 +157,36 @@ See `DEVELOP.md` for the full list of variables and their meanings.
## Storage layout
Single bucket `workspaces` (configurable via `RUSTFS_WORKSPACE_BUCKET`).
Three purpose-named RustFS buckets. The mapping from `StorageObjects.usage_type`
to bucket is decided in **one place** (`storage_api.py:resolve_bucket`):
| `usage_type` | Bucket (env var) | Default name |
|---|---|---|
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `RUSTFS_WORKSPACE_BUCKET` | `workspaces` |
| `version_artifact` | `RUSTFS_VERSION_BUCKET` | `versions` |
| `run_log`, `run_result` | `RUSTFS_RUN_LOG_BUCKET` | `run-logs` |
A workspace's `artifact_bucket` column (when non-null) overrides the
default for that workspace, regardless of `usage_type` — useful for
isolating a paying customer to their own bucket.
The object key is a flat two-level path — `workspace_id` and a server-
issued `ulid` for the object. The file name, extension, content type,
and logical path live in the `StorageObjects` and `Scripts` rows, not in
the S3 key, so the bucket can be re-organised without a database
rewrite.
issued `ulid` for the object:
```
s3://workspaces/
└── <workspace_id>/
├── <ulid-1> # script / notebook / data resource
├── <ulid-1> # working_copy / data_resource / snapshot / ...
├── <ulid-2>
└── ...
s3://versions/ (RUSTFS_VERSION_BUCKET — reserved, used by publish_version)
s3://run-logs/ (RUSTFS_RUN_LOG_BUCKET — reserved, used by node executor)
s3://versions/<workspace_id>/<ulid> # immutable script versions
s3://run-logs/<workspace_id>/<ulid> # node run logs and results
```
To find the original file name and its logical path for a given bucket
object, join `StorageObjects.bucket_name + object_key` to the row.
The file name, extension, content type, and logical path live in the
`StorageObjects` and `Scripts` rows, not in the S3 key, so the bucket
can be re-organised without a database rewrite.
Backend code never writes to the container's local filesystem. Schedule
Executor stages node artifacts in `tempfile.TemporaryDirectory()` (auto-