Phase 6: 部署文档 + 启动脚本 + 配置样例

- README.md (英文): 项目概览 + 11 Tool 表格 + 12 env 变量全表 + 6 admin 端点 + 安全模型 (3 道防线 + 4 项额外保证) + dev 命令 - .env.example: 12 env 变量 + 默认值 + 必填标注 - scripts/dev.sh: 一键 build + 后台启动 (nohup + PID 文件) + healthz 验证 - scripts/seed.sh: admin POST 一个示例 cluster (从 .env 读 admin token) - ARCHITECTURE.md (中文): 包依赖图 + 数据流 + 11 Tool 分类 + 关键设计决策 - docs/runbook.md (中文): 部署 + 升级 + 故障排查 (401/404/422 等) - docs/systemd/spark-mcp-go.service: systemd unit 模板 README 安全模型小修正: 第 3 道防线从"spark_submit flag 白名单" (实际不存在) 改为 "upload_file 路径白名单", 加 Additional guarantees 小节列 slice form / json:"-" / 跨主机 redirect / per-tool log 权限。 Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-10 17:24:19 +08:00
parent be4ca460c9
commit d6dd2b846f
7 changed files with 419 additions and 0 deletions
@@ -0,0 +1,139 @@
+# spark-mcp-go
+
+MCP (Model Context Protocol) server for Apache Spark on YARN. Lets an LLM agent
+discover Spark/YARN endpoints, submit jobs, fetch logs, and analyze them via
+11 typed Tools. Streamable HTTP transport, SQLite-backed configuration, and
+`log/slog` structured logging with per-Tool call files.
+
+## What it does
+
+- Exposes 11 MCP Tools over Streamable HTTP at `/mcp`
+- Admin API at `/admin/*` for cluster and audit configuration
+- HTTP Basic and YARN SimpleAuth for Spark/YARN endpoints
+- SSRF protection with cluster URL allowlist plus DNS rebinding guard
+- Per-Tool independent audit log (file mode `0600`)
+- `spark-submit` via local `exec.Command` (no shell, no injection)
+
+## Quick start (3 steps)
+
+1. Copy and edit the environment file:
+   ```bash
+   cp .env.example .env
+   # Edit ADMIN_TOKENS and AGENT_TOKEN
+   ```
+
+2. Build and start the server:
+   ```bash
+   go build ./...
+   ./spark-mcp-go
+   # Or use the helper:
+   # ./scripts/dev.sh
+   ```
+
+3. Initialize an MCP session, then call `list_clusters` to discover endpoints.
+
+## 11 Tools
+
+| Tool | Type | Purpose |
+|---|---|---|
+| `list_clusters` | discovery | Return all active clusters (LLM entry point) |
+| `spark_submit` | exec | Local `spark-submit` process, extracts `app_id` |
+| `list_applications` | RM | `GET /ws/v1/cluster/apps[?state=&user=]` |
+| `get_application_status` | RM | `GET /ws/v1/cluster/apps/{id}` |
+| `get_application_logs` | RM | Fallback chain: `amContainerLogs` -> aggregated-logs -> logs |
+| `kill_application` | RM | `PUT` state `KILLED` |
+| `fetch_spark_metrics` | SHS | Executor metrics + summary mode triggers analyzer |
+| `fetch_cluster_env` | RM | Aggregate `/cluster/info` and `/cluster/metrics` |
+| `analyze_spark_log` | analyzer | RM logs + heuristic rules + LLM-ready prompt |
+| `fetch_url` | HTTP | Generic primitive, uses cluster allowlist and auth |
+| `upload_file` | FS | Write to `./data/uploads/`, reference from `spark_submit` |
+
+## Environment variables
+
+| Variable | Default | Required | Description |
+|---|---|---|---|
+| `LISTEN_ADDR` | `:8080` | no | HTTP listen address |
+| `DATA_DIR` | `./data` | no | SQLite, uploads, and log root |
+| `ADMIN_TOKENS` | — | **yes** | Comma-separated tokens for `/admin/*` |
+| `AGENT_TOKEN` | — | **yes** | Token for `/mcp` requests |
+| `HTTP_CLIENT_TIMEOUT` | `30s` | no | Timeout for `fetch_url`/RM Tool HTTP calls |
+| `MAX_RESPONSE_BYTES` | `1048576` (1 MiB) | no | HTTP response truncation limit |
+| `SPARK_SUBMIT_TIMEOUT` | `60s` | no | `spark-submit` process timeout |
+| `LOG_DIR` | `./data/logs` | no | Log root directory |
+| `LOG_LEVEL` | `info` | no | `debug`, `info`, `warn`, or `error` |
+| `LOG_FORMAT` | `text` | no | `text` for terminal, `json` for log files |
+| `ANALYZER_DATA_SKEW_RATIO` | `3.0` | no | Data-skew rule threshold (max/min ratio) |
+| `ANALYZER_GC_PRESSURE_RATIO` | `0.1` | no | GC-pressure rule threshold (GC/CPU ratio) |
+| `ANALYZER_BOTTLENECK_SHUFFLE_GB` | `50` | no | Bottleneck rule threshold (shuffle GB) |
+
+## Admin API
+
+All endpoints require `Authorization: Bearer <admin_token>`.
+
+| Method | Path | Description |
+|---|---|---|
+| `GET` | `/admin/clusters` | List all clusters |
+| `POST` | `/admin/clusters` | Create a cluster |
+| `GET` | `/admin/clusters/:id` | Get one cluster |
+| `PUT` | `/admin/clusters/:id` | Update a cluster |
+| `DELETE` | `/admin/clusters/:id` | Delete a cluster |
+| `GET` | `/admin/audit?limit=100` | Query audit log |
+
+Create a cluster:
+
+```bash
+curl -sS -X POST \\
+  -H "Authorization: Bearer ${ADMIN_TOKEN}" \\
+  -H "Content-Type: application/json" \\
+  -d '{
+    "id": "prod",
+    "name": "prod",
+    "rm_url": "http://rm.example.com:8088",
+    "shs_url": "http://shs.example.com:18080",
+    "spark_submit_execute_bin": "/opt/spark/bin/spark-submit",
+    "is_active": true,
+    "auth_type": "simple",
+    "auth_username": "yarn",
+    "rate_limit_per_min": 10,
+    "url_allowlist": ["rm.example.com:8088", "shs.example.com:18080"]
+  }' \\
+  "http://127.0.0.1:${LISTEN_ADDR:-:8080}/admin/clusters"
+```
+
+Note: `auth_password` is not accepted via JSON (`json:"-"`). Set it through a
+dedicated password endpoint or seed the database directly.
+
+## Security model
+
+Three lines of defense:
+
+1. **Token auth** - `ADMIN_TOKENS` protects `/admin/*`; `AGENT_TOKEN` protects `/mcp`.
+2. **SSRF guard** - Every outbound URL must match the cluster's `url_allowlist`,
+   with private-IP CIDR blacklist (incl. `169.254.0.0/16` AWS/GCP metadata).
+3. **Path allowlist** - `upload_file` writes only to `./data/uploads/`, rejects
+   absolute paths, `..`, and non-`[a-zA-Z0-9._-]` filenames.
+
+Additional guarantees:
+
+- `spark_submit` runs the cluster's binary via `exec.Command(name, args...)` —
+  slice form, never `sh -c`. No shell metacharacter interpretation.
+- `auth_password` is tagged `json:"-"`; never serialized in responses, never
+  accepted from JSON request bodies.
+- Cross-host redirects (RM → NM 307) preserve the `Authorization` header
+  through `DoWithRedirect` only when the destination host is in
+  `url_allowlist`.
+- Per-Tool log files are created with mode `0600` and live under
+  `./data/logs/tools/`.
+
+## Development
+
+```bash
+go build ./...
+go test ./...
+go vet ./...
+```
+
+## More docs
+
+- [ARCHITECTURE.md](ARCHITECTURE.md) - Design and package layout
+- [docs/runbook.md](docs/runbook.md) - Deployment, upgrade, and troubleshooting