docs: README + DEVELOP — code map, Settings, bucket routing, troubleshooting
- README: refresh "what it does", "architecture at a glance", "containers" table, "configuration", "storage layout" (with usage_type→bucket routing table), and add "documentation" section pointing to ARCHITECTURE / HANDOVER / DEVELOP / CLAUDE. - DEVELOP: add the developer-facing guide that was previously only in CLAUDE.md. Covers code layout, the Settings singleton, conventions (DB / Storage / Auth / Permission gates / Outbox / async-sync signatures), local dev workflow, common tasks (adding a DAG endpoint, env var, MySQL table, RustFS bucket, schedule node type), tests status, and a troubleshooting section with the four real bugs hit this session (greenlet, MySQL, Jupyter 401, Schedule not advancing).
This commit is contained in:
@@ -157,26 +157,36 @@ See `DEVELOP.md` for the full list of variables and their meanings.
|
||||
|
||||
## Storage layout
|
||||
|
||||
Single bucket `workspaces` (configurable via `RUSTFS_WORKSPACE_BUCKET`).
|
||||
Three purpose-named RustFS buckets. The mapping from `StorageObjects.usage_type`
|
||||
to bucket is decided in **one place** (`storage_api.py:resolve_bucket`):
|
||||
|
||||
| `usage_type` | Bucket (env var) | Default name |
|
||||
|---|---|---|
|
||||
| `working_copy`, `public_script`, `data_resource`, `snapshot` | `RUSTFS_WORKSPACE_BUCKET` | `workspaces` |
|
||||
| `version_artifact` | `RUSTFS_VERSION_BUCKET` | `versions` |
|
||||
| `run_log`, `run_result` | `RUSTFS_RUN_LOG_BUCKET` | `run-logs` |
|
||||
|
||||
A workspace's `artifact_bucket` column (when non-null) overrides the
|
||||
default for that workspace, regardless of `usage_type` — useful for
|
||||
isolating a paying customer to their own bucket.
|
||||
|
||||
The object key is a flat two-level path — `workspace_id` and a server-
|
||||
issued `ulid` for the object. The file name, extension, content type,
|
||||
and logical path live in the `StorageObjects` and `Scripts` rows, not in
|
||||
the S3 key, so the bucket can be re-organised without a database
|
||||
rewrite.
|
||||
issued `ulid` for the object:
|
||||
|
||||
```
|
||||
s3://workspaces/
|
||||
└── <workspace_id>/
|
||||
├── <ulid-1> # script / notebook / data resource
|
||||
├── <ulid-1> # working_copy / data_resource / snapshot / ...
|
||||
├── <ulid-2>
|
||||
└── ...
|
||||
|
||||
s3://versions/ (RUSTFS_VERSION_BUCKET — reserved, used by publish_version)
|
||||
s3://run-logs/ (RUSTFS_RUN_LOG_BUCKET — reserved, used by node executor)
|
||||
s3://versions/<workspace_id>/<ulid> # immutable script versions
|
||||
s3://run-logs/<workspace_id>/<ulid> # node run logs and results
|
||||
```
|
||||
|
||||
To find the original file name and its logical path for a given bucket
|
||||
object, join `StorageObjects.bucket_name + object_key` to the row.
|
||||
The file name, extension, content type, and logical path live in the
|
||||
`StorageObjects` and `Scripts` rows, not in the S3 key, so the bucket
|
||||
can be re-organised without a database rewrite.
|
||||
|
||||
Backend code never writes to the container's local filesystem. Schedule
|
||||
Executor stages node artifacts in `tempfile.TemporaryDirectory()` (auto-
|
||||
|
||||
Reference in New Issue
Block a user