Files

T

tao.chen 309b657d35 docs: align with new storage architecture (s3 + local + server-proxied PUT)

All operator- and developer-facing docs updated to reflect:

  - The unified AsyncStorageBackend abstraction (s3 + local backends).
  - The STORAGE_BACKEND toggle ("s3" default, "local" for dev /
    single-node / air-gapped deployments).
  - The 4-purpose-bucket layout (workspace / version / run_log / trash)
    in both modes — 4 separate S3 buckets in s3 mode, 4 subdirectories
    of LOCAL_STORAGE_BASE_DIR in local mode.
  - The S3_* env var naming (was RUSTFS_*).
  - The server-proxied upload flow (was browser-direct presign-PUT):
    POST /internal/v1/uploads → PUT /internal/v1/uploads/{id} with
    raw bytes → server calls backend.put().
  - The factory helpers workspaces_root() (runtime's view of the
    workspace bucket on disk) and rclone_remote_spec() (s3-mode mount
    source).
  - The "two settings describing the same thing" cleanup: the deleted
    settings.workspace_root, settings.workspaces_root, and
    settings.remote_bucket fields.

Files touched:
  - API.md (§5 data-resource upload flow, §9 storage control plane,
    §10 readiness example)
  - ARCHITECTURE.md (storage layer diagram)
  - CLAUDE.md (architecture description + volume-preservation note)
  - DEVELOP.md (settings list, Storage section, "Wire a new bucket"
    how-to, dev-export example, troubleshooting network hint)
  - README.md (architecture diagram, container table, quick-start
    credentials note, tear-down note, Storage layout section)
  - REFACTOR_NOTES.md (final container list with s3 explanation)
  - backend/README.md (storage backend description)
  - migrations/data/README.md (step 11/12 record mentioning object
    storage)

A handful of historical "RustFS" mentions are intentionally retained
where they name a specific S3-compatible product (e.g. as an example
in REFACTOR_NOTES.md's container list) or document the pre-2026
abstraction name (DEVELOP.md Storage section).

2026-08-05 13:13:20 +08:00

1.9 KiB

Raw Blame History

Repository Guide

Current architecture

frontend: React Router SPA. Production files are built in nginx/Dockerfile.
backend: public FastAPI API and internal S3 storage API in one process.
runtime: shared Jupyter lifecycle, MySQL edit leases and short-lived in-memory access tickets.
schedule: APScheduler, MySQL JobStore, MySQL Outbox polling and DAG execution.
common: SQLAlchemy models, database/session helpers, IDs and object-store helpers.
migrations: Alembic schema and seed migrations.
nginx: static frontend, /api/ proxy and authenticated /jupyter/ proxy.

Redis and the former separate Storage API container are intentionally removed.

Commands

From the repository root:

# Local Python workspace
uv sync --all-packages

# Static Python check
python -m compileall common/src backend/src runtime/src schedule/src

# Database migration
uv run --package backend alembic upgrade head

# Full Docker stack
cp .env.example .env
docker compose config
docker compose up -d --build

Frontend development:

cd frontend
pnpm install
pnpm dev
pnpm typecheck
pnpm build

Service rules

Browser traffic enters through Gateway only.
Frontend API calls use same-origin /api/v1/... paths.
Backend writes schedule_runs and outbox_events, then performs best-effort HTTP dispatch to Schedule Executor.
Schedule Executor always polls pending MySQL Outbox rows, so HTTP dispatch failure does not lose a task.
Cron jobs are persisted by APScheduler in MySQL table apscheduler_jobs.
Runtime must stay single-replica while file leases and Jupyter tickets use the simplified implementation.
Never expose the internal Jupyter token to the browser.
Never delete Docker volumes when preserving MySQL or storage data is required.

Main entrypoints

frontend/app/routes/platform.tsx
backend/src/backend/main.py
runtime/src/runtime/main.py
schedule/src/schedule/main.py
nginx/default.conf.template

1.9 KiB Raw Blame History

Repository Guide

Current architecture

Commands

Service rules

Main entrypoints

1.9 KiB

Raw Blame History