tao.chen
309b657d35
All operator- and developer-facing docs updated to reflect:
- The unified AsyncStorageBackend abstraction (s3 + local backends).
- The STORAGE_BACKEND toggle ("s3" default, "local" for dev /
single-node / air-gapped deployments).
- The 4-purpose-bucket layout (workspace / version / run_log / trash)
in both modes — 4 separate S3 buckets in s3 mode, 4 subdirectories
of LOCAL_STORAGE_BASE_DIR in local mode.
- The S3_* env var naming (was RUSTFS_*).
- The server-proxied upload flow (was browser-direct presign-PUT):
POST /internal/v1/uploads → PUT /internal/v1/uploads/{id} with
raw bytes → server calls backend.put().
- The factory helpers workspaces_root() (runtime's view of the
workspace bucket on disk) and rclone_remote_spec() (s3-mode mount
source).
- The "two settings describing the same thing" cleanup: the deleted
settings.workspace_root, settings.workspaces_root, and
settings.remote_bucket fields.
Files touched:
- API.md (§5 data-resource upload flow, §9 storage control plane,
§10 readiness example)
- ARCHITECTURE.md (storage layer diagram)
- CLAUDE.md (architecture description + volume-preservation note)
- DEVELOP.md (settings list, Storage section, "Wire a new bucket"
how-to, dev-export example, troubleshooting network hint)
- README.md (architecture diagram, container table, quick-start
credentials note, tear-down note, Storage layout section)
- REFACTOR_NOTES.md (final container list with s3 explanation)
- backend/README.md (storage backend description)
- migrations/data/README.md (step 11/12 record mentioning object
storage)
A handful of historical "RustFS" mentions are intentionally retained
where they name a specific S3-compatible product (e.g. as an example
in REFACTOR_NOTES.md's container list) or document the pre-2026
abstraction name (DEVELOP.md Storage section).
65 lines
1.9 KiB
Markdown
65 lines
1.9 KiB
Markdown
# Repository Guide
|
|
|
|
## Current architecture
|
|
|
|
- `frontend`: React Router SPA. Production files are built in `nginx/Dockerfile`.
|
|
- `backend`: public FastAPI API and internal S3 storage API in one process.
|
|
- `runtime`: shared Jupyter lifecycle, MySQL edit leases and short-lived in-memory access tickets.
|
|
- `schedule`: APScheduler, MySQL JobStore, MySQL Outbox polling and DAG execution.
|
|
- `common`: SQLAlchemy models, database/session helpers, IDs and object-store helpers.
|
|
- `migrations`: Alembic schema and seed migrations.
|
|
- `nginx`: static frontend, `/api/` proxy and authenticated `/jupyter/` proxy.
|
|
|
|
Redis and the former separate Storage API container are intentionally removed.
|
|
|
|
## Commands
|
|
|
|
From the repository root:
|
|
|
|
```bash
|
|
# Local Python workspace
|
|
uv sync --all-packages
|
|
|
|
# Static Python check
|
|
python -m compileall common/src backend/src runtime/src schedule/src
|
|
|
|
# Database migration
|
|
uv run --package backend alembic upgrade head
|
|
|
|
# Full Docker stack
|
|
cp .env.example .env
|
|
docker compose config
|
|
docker compose up -d --build
|
|
```
|
|
|
|
Frontend development:
|
|
|
|
```bash
|
|
cd frontend
|
|
pnpm install
|
|
pnpm dev
|
|
pnpm typecheck
|
|
pnpm build
|
|
```
|
|
|
|
## Service rules
|
|
|
|
- Browser traffic enters through Gateway only.
|
|
- Frontend API calls use same-origin `/api/v1/...` paths.
|
|
- Backend writes `schedule_runs` and `outbox_events`, then performs best-effort HTTP dispatch to Schedule Executor.
|
|
- Schedule Executor always polls pending MySQL Outbox rows, so HTTP dispatch failure does not lose a task.
|
|
- Cron jobs are persisted by APScheduler in MySQL table `apscheduler_jobs`.
|
|
- Runtime must stay single-replica while file leases and Jupyter tickets use the simplified implementation.
|
|
- Never expose the internal Jupyter token to the browser.
|
|
- Never delete Docker volumes when preserving MySQL or storage data is required.
|
|
|
|
## Main entrypoints
|
|
|
|
```text
|
|
frontend/app/routes/platform.tsx
|
|
backend/src/backend/main.py
|
|
runtime/src/runtime/main.py
|
|
schedule/src/schedule/main.py
|
|
nginx/default.conf.template
|
|
```
|