tao.chen
4b2a67ae5d
Replace the old RustFS-specific storage layer (common.storage.client /
RustFSObjectStore) with a minimal sync/async abstraction:
AsyncStorageBackend: put / get / get_stream / delete / exists / stat /
list / get_url / copy
StorageBackend: same surface, sync implementations
create_storage({"type": "s3" | "local", "mode": "async", ...})
backends/s3.py: S3-compatible (boto3 / aioboto3)
backends/local.py: on-disk filesystem (aiofiles)
Concretely:
- Drop RustFSObjectStore + common.storage.client (deleted).
- Drop the RustFS-specific ensure_bucket / presign_put / move_to_trash /
rewrite_to_public_path / sha256 / put_bytes methods.
- Migrate backend/storage_api.py + backend/main.py + backend/scripts.py
+ schedule/service.py + schedule/worker.py to the new abstraction.
- Migrate backend/storage_client.py + schedule/storage_client.py to
stub status (HTTP wrapper is dead code post-migration; rewrite pending).
- Rename all RUSTFS_* env vars to S3_* across .env.example,
docker-compose.yml, default.conf, scripts/nginx-entrypoint.sh,
common/config.py.
- Replace hardcoded rclone remote name "rustfs" with "s3" in
docker-compose.yml + config.py default.
- Rename "rustfs" SQLAlchemy column comments + table comments to
provider-neutral wording; StorageObjects.storage_backend enum
value moves from "rustfs" to "s3" (DB rows with the old value will
fail the != "s3" check until a one-shot migration is applied).
- Drop unused common/src/common/migrations/{README,env.py,script.py.mako}
(the alembic setup lives in /migrations/, not here).
Migration of the old abstractions has been done in one pass; per-route
method calls (delete / stat / put / get_url) are now direct one-liners
against AsyncStorageBackend.
After this commit:
- All Python imports resolve; routes compile (compileall green).
- s3 mode is fully wired.
- Routes that depended on removed methods (presign_put, move_to_trash,
rewrite_to_public_path, head() metadata) raise NotImplementedError
with a one-line TODO; rewriting these route handlers is the next step.
70 lines
3.1 KiB
Bash
70 lines
3.1 KiB
Bash
COMPOSE_PROJECT_NAME=model-platform-develop
|
|
SCHEDULE_EVENT_NAMESPACE=model-platform-develop
|
|
|
|
# External port of the Nginx gateway. Only Nginx is exposed to the host
|
|
# (architecture §2.2); backend/runtime/schedule stay on the Docker internal
|
|
# network. Override here to expose Nginx on a different host port.
|
|
GATEWAY_PORT=8890
|
|
|
|
# External MySQL. URL-encode reserved characters in DATABASE_URL.
|
|
MYSQL_HOST=127.0.0.1
|
|
MYSQL_PORT=3306
|
|
MYSQL_USER=root
|
|
MYSQL_PASSWORD=change-me
|
|
MYSQL_DATABASE=model_platform
|
|
DATABASE_URL=mysql+asyncmy://root:change-me@127.0.0.1:3306/model_platform?charset=utf8mb4
|
|
|
|
# Demo login is only intended for this self-hosted development UI.
|
|
DEMO_AUTH_ENABLED=true
|
|
JWT_SECRET=change-this-development-secret
|
|
|
|
# ============================================================================
|
|
# CRITICAL: must set BEFORE first run. The initial admin user is seeded by
|
|
# the deployment bootstrap. Never keep the development default in production.
|
|
# ============================================================================
|
|
INITIAL_ADMIN_PASSWORD=admin12345
|
|
|
|
# Object storage. Two modes are supported:
|
|
# STORAGE_BACKEND=s3 — connects to an S3-compatible service (MinIO,
|
|
# RustFS, SeaweedFS, AWS S3, …). Requires the
|
|
# S3_* block below.
|
|
# STORAGE_BACKEND=local — stores objects on the local filesystem under
|
|
# LOCAL_STORAGE_BASE_DIR. Backend and runtime
|
|
# share this directory via a Docker volume
|
|
# (docker-compose.yml mounts `local-storage`).
|
|
# Useful for dev, single-node, air-gapped.
|
|
STORAGE_BACKEND=s3
|
|
LOCAL_STORAGE_BASE_DIR=/data
|
|
|
|
# Object storage (S3-compatible). Only used when STORAGE_BACKEND=s3.
|
|
# S3_ENDPOINT is the single upstream URL consumed by all 4 services:
|
|
# - nginx (via scripts/nginx-entrypoint.sh, which parses host + port)
|
|
# - backend / runtime / schedule (passed through to boto3 / rclone)
|
|
# S3_ACCESS_KEY / S3_SECRET_KEY are read by Python code in
|
|
# backend/ and schedule/ (boto3 credentials).
|
|
#
|
|
# S3 buckets are purpose-named:
|
|
# S3_WORKSPACE_BUCKET — workspace files (notebooks, scripts, working
|
|
# copies); layout is ``s3://<bucket>/<workspace_id>/...``.
|
|
# S3_VERSION_BUCKET — immutable script-version artifacts.
|
|
# S3_RUN_LOG_BUCKET — schedule run logs and execution results.
|
|
# S3_TRASH_BUCKET — soft-deleted objects; source bucket key is preserved
|
|
# as a prefix so restore is a same-key move.
|
|
S3_HOST=127.0.0.1
|
|
S3_PORT=9000
|
|
S3_ENDPOINT=http://127.0.0.1:9000
|
|
S3_ACCESS_KEY=change-me
|
|
S3_SECRET_KEY=change-me
|
|
S3_WORKSPACE_BUCKET=workspaces
|
|
S3_VERSION_BUCKET=versions
|
|
S3_RUN_LOG_BUCKET=run-logs
|
|
S3_TRASH_BUCKET=trash
|
|
S3_TRASH_RETENTION_DAYS=30
|
|
|
|
# rclone RC (HTTP control API). The runtime container starts rclone with
|
|
# `--rc --rc-addr 0.0.0.0:5572 --rc-no-auth` (see runtime/src/runtime/mount.py),
|
|
# so the backend can POST /vfs/refresh here to invalidate the FUSE dir-cache
|
|
# after writing new workspace files. Default points at the runtime service
|
|
# over the compose network.
|
|
RCLONE_RC_URL=http://runtime:5572
|