fix(security): P0-1 — port exposure + service-token auth on /internal/* + jupyter RPC

The fix lands in three concentric layers, all backed by a single
INTERNAL_SERVICE_TOKEN shared secret so we have one mechanism
instead of three:

1. docker-compose: drop the backend.ports: 8891:8000 and
   runtime.ports: 8892:8000 mappings. Nginx is the only host
   ingress again (architecture §2.2).
2. /internal/v1/*: the storage control plane had six endpoints, five
   of which were dead code (frontend already migrated to
   /api/v1/data-resources/* with JWT; schedule only ever called
   POST /internal/v1/objects). Delete the dead routes, mount the
   one survivor with Depends(require_internal_service) that
   compares the X-Internal-Service-Token header against
   settings.internal_service_token with secrets.compare_digest.
3. POST /api/v1/jupyter on the runtime container: previously open
   inside the Docker network. Same token mechanism — backend's
   runtime_http_client now carries the header, runtime's
   handle_jupyter_action requires the same header. /api/v1/health
   stays open for the Nginx and compose healthchecks.

The schedule worker was already configured to call
POST /internal/v1/objects; build_storage_http_client now
sets the token header so its existing call site keeps working
without changes.

Files touched:
  backend/src/backend/storage_api.py   # 5 dead routes deleted + token guard
  backend/src/backend/main.py          # runtime_http_client header
  runtime/src/runtime/main.py          # require_internal_service Depends
  common/src/common/config.py          # internal_service_token setting
  schedule/src/schedule/service.py     # httpx client header
  docker-compose.yml                   # ports dropped, INTERNAL_SERVICE_TOKEN env
  .env.example                         # INTERNAL_SERVICE_TOKEN placeholder
  API.md / README.md / DEVELOP.md      # §9 trimmed to 1 endpoint

Verified:
  compileall -> 0 errors
  pytest backend/tests -> 37 passed
  in-process ASGI smoke:
    POST /internal/v1/objects no/wrong/correct token -> 401/401/200
    POST /api/v1/jupyter   no/wrong/correct token -> 401/401/200
    5 deleted internal routes -> 404
  docker compose config (with env) -> OK

P0-1 still has one open sub-item (rclone RC --rc-no-auth) that
the user has explicitly deferred; not touched here.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
tao.chen
2026-08-17 16:48:46 +08:00
co-authored by Claude Fable 5
parent 6258cf5d12
commit dfe3f0b118
10 changed files with 195 additions and 138 deletions
+22 -3
View File
@@ -18,7 +18,7 @@ common/ Pure-Python shared library
schemas.py StrictModel base
utils.py get_free_port, start_process
backend/ Public FastAPI service + internal /internal/v1/* sub-app
backend/ Public FastAPI service + tiny /internal/v1/objects RPC
main.py lifespan + route registration
jupyter.py /api/v1/auth/jupyter — the ONLY auth entry
scripts.py CRUD for scripts/notebooks (object storage via AsyncStorageBackend)
@@ -27,7 +27,7 @@ backend/ Public FastAPI service + internal /internal/v1/* sub-app
schedule_schemas.py Pydantic request/response models
admin.py Admin endpoints
resources.py Misc data resources
storage_api.py /internal/v1/* (sub-app merged into main)
storage_api.py /internal/v1/objects — single token-guarded endpoint (P0-1)
storage_client.py Stub (HTTP client removed post-migration; rewrite pending)
schedule_client.py Placeholder module (was the HTTP-push executor client)
runtime_client.py Self-contained httpx wrapper for the runtime
@@ -41,7 +41,7 @@ schedule/ Schedule Executor (DAG worker)
worker.py NodeExecutor (notebook / python execution)
service.py SchedulerService facade (composes the three)
main.py Lifespan + FastAPI app
storage_client.py Stub (SchedulerStorageClient rewrite pending — use AsyncStorageBackend directly)
storage_client.py SchedulerStorageClient — talks to backend /internal/v1/objects
execution.py execute_artifact (notebook + python paths)
notebook_runner.py Subprocess entry point (nbclient)
@@ -164,6 +164,25 @@ grep -rnE 'os\.(environ\[?["\x27][A-Z_]+|getenv\(["\x27][A-Z_]+)' --include="*.p
4. Calls `RuntimeClient.get_workspace` / `start_workspace`.
5. Returns `x-upstream-addr` + `x-jupyter-internal-token` response
headers. **Browser never holds the runtime token.**
### Service-to-service auth (P0-1 fix)
- Schedule → Backend single endpoint ``POST /internal/v1/objects`` is
guarded by ``require_internal_service`` in ``backend.storage_api``.
- The token header is ``X-Internal-Service-Token`` (case-insensitive
on the wire because FastAPI ``Header`` lowercase-matches the name
``x-internal-service-token``); the secret value comes from
``settings.internal_service_token`` / env ``INTERNAL_SERVICE_TOKEN``.
- Comparison uses ``secrets.compare_digest`` — never equality.
- Backend and schedule must be configured with the same value; a
mismatch fails fast at the first notebook run (``401``) which is
intentional. ``.env.example`` ships a placeholder
``change-me-internal-service-token`` and the docker-compose
``${INTERNAL_SERVICE_TOKEN:?...}`` reference forces production
deployments to set a real value.
- Removing the legacy backend / runtime host-port mappings
(``8891:8000`` / ``8892:8000``) is part of the same fix — no service
is reachable from the host except Nginx anymore.
- Nginx captures the headers via `auth_request_set` and proxies to the
upstream sub-process with `Authorization: token $jupyter_token`.