tao.chen
309b657d35
All operator- and developer-facing docs updated to reflect:
- The unified AsyncStorageBackend abstraction (s3 + local backends).
- The STORAGE_BACKEND toggle ("s3" default, "local" for dev /
single-node / air-gapped deployments).
- The 4-purpose-bucket layout (workspace / version / run_log / trash)
in both modes — 4 separate S3 buckets in s3 mode, 4 subdirectories
of LOCAL_STORAGE_BASE_DIR in local mode.
- The S3_* env var naming (was RUSTFS_*).
- The server-proxied upload flow (was browser-direct presign-PUT):
POST /internal/v1/uploads → PUT /internal/v1/uploads/{id} with
raw bytes → server calls backend.put().
- The factory helpers workspaces_root() (runtime's view of the
workspace bucket on disk) and rclone_remote_spec() (s3-mode mount
source).
- The "two settings describing the same thing" cleanup: the deleted
settings.workspace_root, settings.workspaces_root, and
settings.remote_bucket fields.
Files touched:
- API.md (§5 data-resource upload flow, §9 storage control plane,
§10 readiness example)
- ARCHITECTURE.md (storage layer diagram)
- CLAUDE.md (architecture description + volume-preservation note)
- DEVELOP.md (settings list, Storage section, "Wire a new bucket"
how-to, dev-export example, troubleshooting network hint)
- README.md (architecture diagram, container table, quick-start
credentials note, tear-down note, Storage layout section)
- REFACTOR_NOTES.md (final container list with s3 explanation)
- backend/README.md (storage backend description)
- migrations/data/README.md (step 11/12 record mentioning object
storage)
A handful of historical "RustFS" mentions are intentionally retained
where they name a specific S3-compatible product (e.g. as an example
in REFACTOR_NOTES.md's container list) or document the pre-2026
abstraction name (DEVELOP.md Storage section).
1.9 KiB
1.9 KiB
Repository Guide
Current architecture
frontend: React Router SPA. Production files are built innginx/Dockerfile.backend: public FastAPI API and internal S3 storage API in one process.runtime: shared Jupyter lifecycle, MySQL edit leases and short-lived in-memory access tickets.schedule: APScheduler, MySQL JobStore, MySQL Outbox polling and DAG execution.common: SQLAlchemy models, database/session helpers, IDs and object-store helpers.migrations: Alembic schema and seed migrations.nginx: static frontend,/api/proxy and authenticated/jupyter/proxy.
Redis and the former separate Storage API container are intentionally removed.
Commands
From the repository root:
# Local Python workspace
uv sync --all-packages
# Static Python check
python -m compileall common/src backend/src runtime/src schedule/src
# Database migration
uv run --package backend alembic upgrade head
# Full Docker stack
cp .env.example .env
docker compose config
docker compose up -d --build
Frontend development:
cd frontend
pnpm install
pnpm dev
pnpm typecheck
pnpm build
Service rules
- Browser traffic enters through Gateway only.
- Frontend API calls use same-origin
/api/v1/...paths. - Backend writes
schedule_runsandoutbox_events, then performs best-effort HTTP dispatch to Schedule Executor. - Schedule Executor always polls pending MySQL Outbox rows, so HTTP dispatch failure does not lose a task.
- Cron jobs are persisted by APScheduler in MySQL table
apscheduler_jobs. - Runtime must stay single-replica while file leases and Jupyter tickets use the simplified implementation.
- Never expose the internal Jupyter token to the browser.
- Never delete Docker volumes when preserving MySQL or storage data is required.
Main entrypoints
frontend/app/routes/platform.tsx
backend/src/backend/main.py
runtime/src/runtime/main.py
schedule/src/schedule/main.py
nginx/default.conf.template