From a4e2472716fac52a49598a5e3d69c94c5a10df38 Mon Sep 17 00:00:00 2001 From: "tao.chen" Date: Fri, 10 Jul 2026 12:12:53 +0800 Subject: [PATCH] =?UTF-8?q?Phase=200+1+2=20=E7=BB=AD:=20=E9=A1=B9=E7=9B=AE?= =?UTF-8?q?=E9=AA=A8=E6=9E=B6=20+=20=E9=85=8D=E7=BD=AE=20+=20=E6=97=A5?= =?UTF-8?q?=E5=BF=97=20+=20=E5=AD=98=E5=82=A8=E5=B1=82?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 0 (项目骨架): - main.go 改为最小骨架 (config + logging + gin + /healthz + 优雅退出) - internal/{config,logging,storage,cluster,...}/ 目录占位 Phase 1 (配置 + 日志): - internal/config: env 解析 + 必填校验 + token 隐藏的 String() - internal/logging: slog multi-handler 双输出 (终端 text + 主文件 JSON) + StartToolCall per-tool 独立文件 (0600), tools 目录 0700 Phase 2 续 (存储层): - internal/cluster: 16 字段 Cluster struct (AuthPassword json:"-") - internal/storage: modernc.org/sqlite 接入, WAL 模式, schema 自动迁移 - ClusterRepo CRUD: Create/Get/List/Update/Delete + ErrNotFound + AuthPassword 空字符串 = 保留旧密码 (核心约定) - 7 个单元测试全绿 (:memory: DB) - created_at/updated_at 改纳秒精度, 消除 List 测试的 sleep 特殊 case Co-Authored-By: Claude --- CLAUDE.md | 58 +++ findings.md | 253 ++++++++++ go.mod | 16 +- go.sum | 135 +++++ internal/cluster/types.go | 56 +++ internal/config/config.go | 225 +++++++++ internal/logging/logger.go | 135 +++++ internal/logging/tool_call.go | 191 +++++++ internal/storage/cluster_repo.go | 252 ++++++++++ internal/storage/cluster_repo_test.go | 306 +++++++++++ internal/storage/migrate.go | 16 + internal/storage/sqlite.go | 85 ++++ main.go | 92 +++- progress.md | 143 ++++++ task_plan.md | 700 ++++++++++++++++++++++++++ 15 files changed, 2648 insertions(+), 15 deletions(-) create mode 100644 CLAUDE.md create mode 100644 findings.md create mode 100644 go.sum create mode 100644 internal/cluster/types.go create mode 100644 internal/config/config.go create mode 100644 internal/logging/logger.go create mode 100644 internal/logging/tool_call.go create mode 100644 internal/storage/cluster_repo.go create mode 100644 internal/storage/cluster_repo_test.go create mode 100644 internal/storage/migrate.go create mode 100644 internal/storage/sqlite.go create mode 100644 progress.md create mode 100644 task_plan.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..fe60c5c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,58 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Status + +**This repository is freshly initialized.** The only Go source file is `main.go`, which is still the GoLand "Hello, gopher" template. The `go.mod` and `go.sum` already pin the intended runtime dependencies, but no application code has been written yet. + +Intended purpose (inferred from module name and pinned dependencies): an **MCP (Model Context Protocol) server for Apache Spark**, written in Go, with HTTP transport, MongoDB-backed persistence, and a local `data/` directory for Spark connection state and pending job state (per `.gitignore`). + +When working here, expect to be **building from scratch**, not modifying existing logic. + +## Build, Test, Run + +Standard Go toolchain (Go 1.25.5 per `go.mod`). All commands run from the repo root. + +| Task | Command | +|---|---| +| Build | `go build ./...` | +| Run | `go run .` (currently just prints the template `main`) | +| Test (all) | `go test ./...` | +| Test (single test by name regex) | `go test -run TestName ./...` | +| Test (single test verbose) | `go test -v -run TestName ./path/to/pkg` | +| Test (with coverage) | `go test -cover ./...` | +| Vet (static analysis) | `go vet ./...` | +| Format | `gofmt -w .` | +| Tidy modules | `go mod tidy` | +| Download deps | `go mod download` | + +There is no `Makefile`, no `golangci-lint` config, and no test files yet — do not assume they exist. + +## Key Pinned Dependencies + +`go.mod` declares these as `// indirect` (they will become direct once code is added that imports them): + +- **`github.com/mark3labs/mcp-go v0.56.0`** — MCP server framework. This is the core library; all MCP tool/resource/prompt definitions will be built on top of it. +- **`github.com/gin-gonic/gin v1.12.0`** — HTTP router. Likely the transport layer that exposes the MCP server over HTTP (SSE / streamable HTTP). +- **`go.mongodb.org/mongo-driver/v2 v2.5.0`** — MongoDB client. Likely for persisting connection profiles, session state, and job history. +- **`github.com/bytedance/sonic v1.15.0`** — High-performance JSON encoder used internally by Gin. + +## Layout Conventions + +- Module path: `spark-mcp-go` (see `go.mod:1`). Local package import paths use this prefix. +- Runtime data directory: `data/` at the repo root, **gitignored**. Used for "persisted Spark connections / pending jobs" (per `.gitignore` comment). Anything that needs to survive across process restarts but should not go to MongoDB belongs here. +- IDE config: `.idea/` is gitignored — JetBrains (GoLand / IntelliJ) workspace files. Safe to delete; not part of the project. + +## Things To Confirm Before Coding + +The codebase is empty enough that a few decisions are not yet pinned down and will affect every subsequent file. Confirm with the user before assuming: + +1. **MCP transport** — stdio, HTTP+SSE, or streamable HTTP? The Gin dependency suggests HTTP, but `mark3labs/mcp-go` supports both. +2. **Spark connectivity** — does this server submit jobs to an existing Spark cluster (REST / Livy / Spark Connect), or does it embed a Spark session (likely not — Spark itself is JVM/Scala and won't run in a Go binary)? +3. **Persistence split** — what goes in MongoDB vs. the local `data/` directory? The `.gitignore` implies both will be used. +4. **Configuration** — env vars? config file (YAML/JSON)? CLI flags? Nothing is wired up yet. + +## Memory + +This is a fresh repo with no `MEMORY.md` or project-level memory file. Decisions made during early development (transport choice, Spark backend interface, persistence schema) are worth recording once they are made — see the `claude-md-management` skill conventions. diff --git a/findings.md b/findings.md new file mode 100644 index 0000000..bd90950 --- /dev/null +++ b/findings.md @@ -0,0 +1,253 @@ +# Findings & Research + +> 调研发现、风险分析、决策依据 + +--- + +## 1. 仓库现状 (2026-07-10 调研) + +**提交历史**: `1dfef0f init` (单提交) +**Go 版本**: 1.25.5 +**模块名**: `spark-mcp-go` + +### 1.1 文件清单 + +| 文件 | 状态 | 备注 | +|---|---|---| +| `.gitignore` | 存在 | 包含 Python 模板 + `data/` 目录(Spark 相关) | +| `.idea/` | gitignored | JetBrains IDE 配置 | +| `go.mod` | 存在 | 所有依赖 `// indirect` 状态 | +| `go.sum` | 存在 | — | +| `main.go` | GoLand 模板 | "Hello gopher",非业务代码 | +| `CLAUDE.md` | 已创建 | 项目元信息 + 待确认决策 | +| `task_plan.md` | v0.3 | 7 阶段计划 | +| `findings.md` | (本文件) | 调研与决策依据 | +| `progress.md` | (会话日志) | — | + +### 1.2 关键发现 + +- `go.mod` 里所有依赖都是 `// indirect` — 还没有任何代码 import 它们 +- `mongo-driver/v2` 已 pin 但用户决定改用 SQLite → **Phase 0.1 需要移除** +- `.gitignore` 里的 `data/` 注释"persisted Spark connections / pending jobs"暗示原意是用作本地状态存储 → 现统一改用 SQLite +- 没有任何 README、文档、测试 → 从零开始 + +--- + +## 2. 架构评审结论 + +### 2.1 用户原始方案的关键问题 (v0.1 评审) + +用户在初版架构文档里提出的 5 个 Tool + Gin + MCP + MongoDB 方案,经过 Linus 视角评审,发现以下问题: + +#### 问题 1: 存储选型自我矛盾 +- 文档说 "SQLite or GORM" +- go.mod pin 了 `mongo-driver/v2` +- .gitignore 保留 `data/` 目录 +- **结论**: 三选一 → 选 SQLite(用户已确认),`data/` 目录保留作为上传文件存储 + +#### 问题 2: "元工具" 归属不清 +- 文档提到 `fetch_url` / `exec_shell` / `upload_file` 是"底层元工具" +- **结论**: 用户确认只暴露 `fetch_url` + `upload_file`(不暴露 `exec_shell`) + - `fetch_url` 是 LLM 的 HTTP 出入口 + - `upload_file` 是 LLM 的本地文件存储入口 + +#### 问题 3: Tool 粒度问题 +- 5 个 Tool 里 3 个本质都是 `GET ` +- **结论**: v0.2 收敛到 4 个 Tool;v0.3 因加入 spark_submit 回到 5 个 + +#### 问题 4: 示例代码 API 不存在 +- `server.NewStreamableHTTPServer` 等在 mcp-go v0.56.0 不存在 +- **结论**: Phase 6 实施时**先查 mcp-go 真实 API 再写代码** + +### 2.2 v0.3.x 范围变更汇总 + +| 版本 | 变更 | 触发原因 | +|---|---|---| +| v0.3 | 加 `spark_submit` Tool,`upload_file` 改本地存储,Cluster 加 binary 路径 | 用户从"纯读"改"读+提交" | +| v0.3.1 | 合并 `spark_submit_path` + `spark2_submit_path` → `spark_submit_execute_bin` | 消除 special case | +| v0.3.2 | `list_clusters` 标为强制入口,Cluster 加 `is_active` 字段 | LLM 必须能拿到 RM/SHS 地址 | +| v0.3.3 | 加回 `fetch_spark_metrics` + `fetch_cluster_env` (高层 Tool,内部调 `fetch_url`) | 业务端点封装,LLM 不必拼 URL | +| v0.3.4 | 加回 `analyze_spark_log` + 服务端启发式过滤 (data_skew / gc_pressure / bottleneck) + LLM-Ready Prompt | 用户原 spec 的 [Log Analyzer + Regex Rules + LLM-Ready Prompt] 组件。规则是 first-pass filter,LLM 拿 findings 后做最终判断 | +| v0.3.5 | 拆 `yarn_job_manage` → 4 个 RM Tool (list_applications / get_application_status / get_application_logs / kill_application) | 用户参考实现是 4 个独立 Python 入口,1 Tool = 1 端点 | +| v0.3.5 | 加鉴权层 (none / basic / kerberos) + SSL + manual redirect | Spark on YARN + Hadoop 2.7,生产环境需要 SPNEGO;`get_application_logs` RM→NM 跨主机 307 必须保留 Authorization | +| v0.3.5 | `analyze_spark_log` 日志改走 RM 降级链 | SHS logs 端点不稳定,RM aggregated-logs + amContainerLogs + NM 是生产链路 | +| v0.3.6 | 鉴权层简化成 `none` / `simple` / `basic` | 用户环境 simple auth,kerberos 推到 v2 | +| v0.3.6 | Hadoop 2.7 兼容性"已验证" | 用户实测 endpoint 通,风险解除 | +| v0.3.7 | 加 per-cluster `default_submit_args` + `rate_limit_per_min` | 减 LLM 负担 + 防刷 | +| v0.3.7 | 删 `applications` 表,加 `audit_log` 表 | 纯读模式不需要 app 缓存,audit 必需 | +| v0.3.7 | 多 `AdminTokens` + analyzer 阈值 env 可配 | 多用户 + 集群特定阈值 | +| v0.3.8 | 用 `log/slog` 替换默认 log,加双输出 + per-Tool 独立日志 | 用户要求结构化日志 + 调试 LLM 行为 | + +**v0.3.3 设计模式确立**: +- **通用 HTTP 原语** (`fetch_url`): LLM 自主拼任意 URL +- **业务高层封装** (`fetch_spark_metrics`, `fetch_cluster_env`): 内部调 `fetch_url`,处理 URL 拼装、cluster 校验 +- 两者并存,LLM 按场景选:常用端点用高层,长尾查询用原语 + +### 2.3 待确认项 + +- (已清空 — v0.3.7 用户接受所有 6 项建议,无未决项) +- **环境约束** (用户告知 + 验证): Spark 脚本, YARN-on-Spark, **Hadoop 2.7 (RM/SHS endpoint 已验证工作)** → 无需额外兼容代码 +- **v1 鉴权范围**: `none` / `simple` / `basic` (YARN SimpleAuth 用 `user.name` query param)。**Kerberos 不在 v1**(用户环境不需要) + +用户中途修正了 "纯 RM/SHS 不支持提交" 决策,改为 "读+本地 spark-submit 提交": + +- **新增** `spark_submit` Tool,使用 Server 本地 `exec.Command(binary, args...)` +- **重新定义** `upload_file`:从"通用 HTTP 上传"改为"本地文件存储" (LLM 把脚本内容传给 Server,Server 写到 `./data/`) +- **Cluster 配置新增** 单一 `spark_submit_execute_bin` 字段 (v0.3.1 进一步合并 `spark_submit_path` + `spark2_submit_path`) + +**重新引入的风险**:命令注入 — 通过 `exec.Command` 切片传参 + Tool 参数 schema 强约束来缓解 + +--- + +## 3. 技术选型依据 + +### 3.1 SQLite 驱动对比 + +| 驱动 | 优点 | 缺点 | +|---|---|---| +| `modernc.org/sqlite` | 纯 Go,无 CGO,跨平台编译简单 | 性能略低于原生 | +| `mattn/go-sqlite3` | CGO 绑定,性能最优 | 需要 C 编译器,跨平台编译复杂 | + +**选择**: `modernc.org/sqlite` (本项目数据量小,QPS 极低,纯 Go 部署友好) + +### 3.2 MCP 传输对比 + +| 传输 | 标准 | 优点 | 缺点 | +|---|---|---|---| +| stdio | MCP 基础 | 简单,本地 Agent 首选 | 不能与 HTTP 框架共存 | +| HTTP + SSE | MCP legacy | 经典 HTTP 传输 | 仅 server → client 单向 | +| **Streamable HTTP** | MCP 2025-03+ | POST + 可选 SSE,现代标准 | 实现稍复杂 | + +**选择**: Streamable HTTP(用户已确认),与 Gin 路由共存 + +### 3.3 HTTP 框架: Gin + +- 用户原始方案已选 +- 适合: admin API + 挂载 MCP HTTP handler +- 生态成熟,中间件丰富 +- 与 `mark3labs/mcp-go` 的 Streamable HTTP handler 通过 `gin.WrapH` 集成 + +### 3.4 共享 HTTP 客户端 + +**为什么独立包**: +- SSRF 防护集中实现,所有 Tool 复用 +- Timeout / Size Limit 统一配置 +- DNS rebinding 防护只在 client 层做一次 + +--- + +## 4. 安全考量 + +### 4.1 SSRF 防护 (核心) + +**威胁**: `fetch_url` / `upload_file` 被恶意利用访问内网服务或元数据 endpoint + +**防护策略**: +1. **IP 黑名单**: `127.0.0.0/8`, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `169.254.0.0/16`, `::1/128`, `fc00::/7`, `fe80::/10` +2. **Hostname allowlist**: 来自 cluster 配置的 `rm_url` / `shs_url` 的 host 走豁免 +3. **DNS rebinding 防护**: 解析 hostname 拿 IP,然后 dial 用 IP(不用 hostname 二次解析) +4. **Scheme 校验**: 仅允许 `http` / `https` + +### 4.2 鉴权 + +| 端点 | Token | 来源 | +|---|---|---| +| `/admin/*` | `ADMIN_TOKEN` | 环境变量,必填 | +| `/mcp` | `AGENT_TOKEN` | 环境变量,必填 | +| `/healthz` | 公开 | 探活用 | + +**实现**: `Authorization: Bearer ` Header,中间件统一校验 + +**未来演进**: 多租户 / OAuth / mTLS(当前计划不包含) + +### 4.3 输入校验 + +| 输入 | 校验 | +|---|---| +| `cluster.rm_url` / `shs_url` | 必须 `http://` 或 `https://` 开头 | +| `cluster.id` | 非空,长度限制 | +| `upload_file.file_path` | 必须在 `./data/` 允许目录内 | +| `fetch_url.url` | 必须绝对 URL,scheme 校验 | +| `yarn_job_manage.action` | enum: `kill` | `status` | + +### 4.4 资源限制 + +| 资源 | 限制 | 默认值 | +|---|---|---| +| HTTP response body | `MaxResponseBytes` | 1 MB | +| HTTP request timeout | `HTTPClientTimeout` | 30s | +| Token 大小 | 启动时校验 | < 1 KB | +| 集群数量 | 业务上无限制,DB 层建议 1000 | — | + +--- + +## 5. 已知不确定项 + +### 5.1 mcp-go v0.56.0 真实 API (Phase 6.1 需要查) + +- `server.NewMCPServer(name, version string, opts ...ServerOption) *MCPServer` +- `mcp.NewTool(name string, opts ...ToolOption) Tool` +- `mcp.WithString(name string, opts ...PropertyOption) ToolOption` +- `mcp.WithArray(name string, opts ...PropertyOption) ToolOption` (用于 spark_submit 的 args) +- `server.NewStreamableHTTPServer(s *MCPServer, opts ...StreamableHTTPOption) *StreamableHTTPServer` +- `mcp.NewServer(...)` / `mcp.Server` / `mcp.NewTool` 是 alias? + +**行动**: Phase 6 实施时先看 GitHub tag 源码或官方 docs + +### 5.2 upload_file 的具体业务场景 (已解决) + +- **确认**: 本地文件存储,供 spark_submit 引用 +- **输入**: 文本或 base64 编码的 content +- **输出**: 相对 `data_dir` 的路径 + +### 5.3 applications 表是否需要 + +- 当前为"后续阶段"预留(缓存 RM/SHS 返回的 app 状态) +- spark_submit 返回的 app_id 是否要入库?本阶段暂不入库,LLM 自行跟踪 +- **行动**: Phase 2 末根据实际需求决定保留或删除 + +### 5.4 Gin 与 mcp-go Streamable HTTP 的集成方式 + +- `gin.WrapH(mcpHandler)` 把 http.Handler 适配为 gin.HandlerFunc +- 需要确认: mcp-go 的 handler 是否能直接 Wrap,还是需要中间件处理 CORS / Content-Type +- **行动**: Phase 6.4 实施时验证 + +### 5.5 spark-submit 的 binary 路径在不同环境下的差异 + +- 开发环境: `/usr/local/bin/spark-submit` +- 生产环境: `/opt/cloudera/parcels/SPARK2/bin/spark-submit` 或 `/opt/spark/bin/spark-submit` +- 由 admin 在 cluster 配置中指定 — 已经处理 + +### 5.6 app_id 解析的 YARN 输出格式 + +- YARN 提交成功时输出:`Submitted application application_12345_0001` +- 不同 YARN 版本可能略有差异(如 Spark Standalone 模式输出不同) +- 方案: 用 regex `application_\d+_\d+` 提取,失败则 app_id 为空字符串 +- 保留 regex 可配置(可选,本阶段不强制) + +--- + +## 6. 风险矩阵 + +| 风险 | 概率 | 影响 | 缓解 | +|---|---|---|---| +| mcp-go v0.56.0 API 与文档不一致 | 中 | 中 | Phase 6.1 先查源码 | +| **命令注入 (spark_submit 引入)** | **中** | **高** | **`exec.Command(name, args...)` 切片形式 + submitter enum + path 校验** | +| 私网 IP 漏判 (IPv6 段不全) | 低 | 高 | 写测试覆盖 IPv6 段 | +| DNS rebinding TOCTOU | 中 | 高 | 解析时锁定 IP,后续 dial 用 IP | +| upload_file 路径逃逸 | 中 | 高 | filepath.Clean + 必须在 data_dir 之下 | +| Token 通过 env 泄露到日志 | 低 | 高 | 启动时打印配置时隐藏 token 字段 | +| spark-submit binary 被替换 | 低 | 高 | 启动时 + runtime 都校验 binary 存在 + 可执行 + 绝对路径 | +| SQLite 并发写冲突 | 低 | 中 | 配置 `journal_mode=WAL`, 单写多读 | +| YARN 输出格式变化导致 app_id 解析失败 | 中 | 低 | regex 解析失败则 app_id 为空,不影响主流程 | +| 内存爆炸 (大响应/大 stdout) | 中 | 中 | HTTP body 1MB 截断,spark stdout 10MB 截断 | + +--- + +## 7. 参考资料 + +- [mark3labs/mcp-go GitHub](https://github.com/mark3labs/mcp-go) — MCP Server 库 +- [Model Context Protocol 规范](https://modelcontextprotocol.io/) — MCP 2025-03+ 标准 +- [YARN RM REST API](https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html) — YARN RM 接口 +- [Spark History Server REST API](https://spark.apache.org/docs/latest/monitoring.html#rest-api) — SHS 接口 +- [modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite) — 纯 Go SQLite 驱动 diff --git a/go.mod b/go.mod index da03dc8..e62ef92 100644 --- a/go.mod +++ b/go.mod @@ -2,36 +2,46 @@ module spark-mcp-go go 1.25.5 +require ( + github.com/gin-gonic/gin v1.12.0 + modernc.org/sqlite v1.53.0 +) + require ( github.com/bytedance/gopkg v0.1.3 // indirect github.com/bytedance/sonic v1.15.0 // indirect github.com/bytedance/sonic/loader v0.5.0 // indirect github.com/cloudwego/base64x v0.1.6 // indirect + github.com/dustin/go-humanize v1.0.1 // indirect github.com/gabriel-vasile/mimetype v1.4.12 // indirect github.com/gin-contrib/sse v1.1.0 // indirect - github.com/gin-gonic/gin v1.12.0 // indirect github.com/go-playground/locales v0.14.1 // indirect github.com/go-playground/universal-translator v0.18.1 // indirect github.com/go-playground/validator/v10 v10.30.1 // indirect github.com/goccy/go-json v0.10.5 // indirect github.com/goccy/go-yaml v1.19.2 // indirect + github.com/google/uuid v1.6.0 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/klauspost/cpuid/v2 v2.3.0 // indirect github.com/leodido/go-urn v1.4.0 // indirect - github.com/mark3labs/mcp-go v0.56.0 // indirect github.com/mattn/go-isatty v0.0.20 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/ncruces/go-strftime v1.0.0 // indirect github.com/pelletier/go-toml/v2 v2.2.4 // indirect github.com/quic-go/qpack v0.6.0 // indirect github.com/quic-go/quic-go v0.59.0 // indirect + github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect github.com/twitchyliquid64/golang-asm v0.15.1 // indirect github.com/ugorji/go/codec v1.3.1 // indirect go.mongodb.org/mongo-driver/v2 v2.5.0 // indirect golang.org/x/arch v0.22.0 // indirect golang.org/x/crypto v0.48.0 // indirect golang.org/x/net v0.51.0 // indirect - golang.org/x/sys v0.41.0 // indirect + golang.org/x/sys v0.44.0 // indirect golang.org/x/text v0.34.0 // indirect google.golang.org/protobuf v1.36.10 // indirect + modernc.org/libc v1.73.4 // indirect + modernc.org/mathutil v1.7.1 // indirect + modernc.org/memory v1.11.0 // indirect ) diff --git a/go.sum b/go.sum new file mode 100644 index 0000000..f861ada --- /dev/null +++ b/go.sum @@ -0,0 +1,135 @@ +github.com/bytedance/gopkg v0.1.3 h1:TPBSwH8RsouGCBcMBktLt1AymVo2TVsBVCY4b6TnZ/M= +github.com/bytedance/gopkg v0.1.3/go.mod h1:576VvJ+eJgyCzdjS+c4+77QF3p7ubbtiKARP3TxducM= +github.com/bytedance/sonic v1.15.0 h1:/PXeWFaR5ElNcVE84U0dOHjiMHQOwNIx3K4ymzh/uSE= +github.com/bytedance/sonic v1.15.0/go.mod h1:tFkWrPz0/CUCLEF4ri4UkHekCIcdnkqXw9VduqpJh0k= +github.com/bytedance/sonic/loader v0.5.0 h1:gXH3KVnatgY7loH5/TkeVyXPfESoqSBSBEiDd5VjlgE= +github.com/bytedance/sonic/loader v0.5.0/go.mod h1:AR4NYCk5DdzZizZ5djGqQ92eEhCCcdf5x77udYiSJRo= +github.com/cloudwego/base64x v0.1.6 h1:t11wG9AECkCDk5fMSoxmufanudBtJ+/HemLstXDLI2M= +github.com/cloudwego/base64x v0.1.6/go.mod h1:OFcloc187FXDaYHvrNIjxSe8ncn0OOM8gEHfghB2IPU= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= +github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= +github.com/gabriel-vasile/mimetype v1.4.12 h1:e9hWvmLYvtp846tLHam2o++qitpguFiYCKbn0w9jyqw= +github.com/gabriel-vasile/mimetype v1.4.12/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s= +github.com/gin-contrib/sse v1.1.0 h1:n0w2GMuUpWDVp7qSpvze6fAu9iRxJY4Hmj6AmBOU05w= +github.com/gin-contrib/sse v1.1.0/go.mod h1:hxRZ5gVpWMT7Z0B0gSNYqqsSCNIJMjzvm6fqCz9vjwM= +github.com/gin-gonic/gin v1.12.0 h1:b3YAbrZtnf8N//yjKeU2+MQsh2mY5htkZidOM7O0wG8= +github.com/gin-gonic/gin v1.12.0/go.mod h1:VxccKfsSllpKshkBWgVgRniFFAzFb9csfngsqANjnLc= +github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s= +github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4= +github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA= +github.com/go-playground/locales v0.14.1/go.mod h1:hxrqLVvrK65+Rwrd5Fc6F2O76J/NuW9t0sjnWqG1slY= +github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJnYK9S473LQFuzCbDbfSFY= +github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY= +github.com/go-playground/validator/v10 v10.30.1 h1:f3zDSN/zOma+w6+1Wswgd9fLkdwy06ntQJp0BBvFG0w= +github.com/go-playground/validator/v10 v10.30.1/go.mod h1:oSuBIQzuJxL//3MelwSLD5hc2Tu889bF0Idm9Dg26cM= +github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4= +github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M= +github.com/goccy/go-yaml v1.19.2 h1:PmFC1S6h8ljIz6gMRBopkjP1TVT7xuwrButHID66PoM= +github.com/goccy/go-yaml v1.19.2/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs= +github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= +github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= +github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= +github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y= +github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0= +github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ= +github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI= +github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= +github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= +github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= +github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= +github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w= +github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls= +github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4= +github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8= +github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII= +github.com/quic-go/quic-go v0.59.0 h1:OLJkp1Mlm/aS7dpKgTc6cnpynnD2Xg7C1pwL6vy/SAw= +github.com/quic-go/quic-go v0.59.0/go.mod h1:upnsH4Ju1YkqpLXC305eW3yDZ4NfnNbmQRCMWS58IKU= +github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE= +github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= +github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= +github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= +github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= +github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= +github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS4MhqMhdFk5YI= +github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08= +github.com/ugorji/go/codec v1.3.1 h1:waO7eEiFDwidsBN6agj1vJQ4AG7lh2yqXyOXqhgQuyY= +github.com/ugorji/go/codec v1.3.1/go.mod h1:pRBVtBSKl77K30Bv8R2P+cLSGaTtex6fsA2Wjqmfxj4= +go.mongodb.org/mongo-driver/v2 v2.5.0 h1:yXUhImUjjAInNcpTcAlPHiT7bIXhshCTL3jVBkF3xaE= +go.mongodb.org/mongo-driver/v2 v2.5.0/go.mod h1:yOI9kBsufol30iFsl1slpdq1I0eHPzybRWdyYUs8K/0= +go.uber.org/mock v0.6.0 h1:hyF9dfmbgIX5EfOdasqLsWD6xqpNZlXblLB/Dbnwv3Y= +go.uber.org/mock v0.6.0/go.mod h1:KiVJ4BqZJaMj4svdfmHM0AUx4NJYO8ZNpPnZn1Z+BBU= +golang.org/x/arch v0.22.0 h1:c/Zle32i5ttqRXjdLyyHZESLD/bB90DCU1g9l/0YBDI= +golang.org/x/arch v0.22.0/go.mod h1:dNHoOeKiyja7GTvF9NJS1l3Z2yntpQNzgrjh1cU103A= +golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts= +golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos= +golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4= +golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ= +golang.org/x/net v0.51.0 h1:94R/GTO7mt3/4wIKpcR5gkGmRLOuE/2hNGeWq/GBIFo= +golang.org/x/net v0.51.0/go.mod h1:aamm+2QF5ogm02fjy5Bb7CQ0WMt1/WVM7FtyaTLlA9Y= +golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4= +golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.44.0 h1:ildZl3J4uzeKP07r2F++Op7E9B29JRUy+a27EibtBTQ= +golang.org/x/sys v0.44.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk= +golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA= +golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8= +golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0= +google.golang.org/protobuf v1.36.10 h1:AYd7cD/uASjIL6Q9LiTjz8JLcrh/88q5UObnmY3aOOE= +google.golang.org/protobuf v1.36.10/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +modernc.org/cc/v4 v4.28.4 h1:Hd/4Es+MBj+/7hSdZaisNyu6bv3V0Dp2MdllyfqaH+c= +modernc.org/cc/v4 v4.28.4/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI= +modernc.org/ccgo/v4 v4.34.4 h1:OVnSOWQjVKOYkFxoHYB+qQmSHK5gqMqARM+K9DpR/Ws= +modernc.org/ccgo/v4 v4.34.4/go.mod h1:qdKqE8FNIYyysougB1RX9MxCzp5oJOcQXSobANJ4TuE= +modernc.org/fileutil v1.4.0 h1:j6ZzNTftVS054gi281TyLjHPp6CPHr2KCxEXjEbD6SM= +modernc.org/fileutil v1.4.0/go.mod h1:EqdKFDxiByqxLk8ozOxObDSfcVOv/54xDs/DUHdvCUU= +modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI= +modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito= +modernc.org/gc/v3 v3.1.3 h1:6QAplYyVO+KdPW3pGnqmJDUxtkec8ooEWvks/hhU3lc= +modernc.org/gc/v3 v3.1.3/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY= +modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks= +modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI= +modernc.org/libc v1.73.4 h1:+ra4Ui8ngyt8HDcO1FTDPWlkAh6yOdaO2yAoh8MddQA= +modernc.org/libc v1.73.4/go.mod h1:DXZ3eO8qMCNn2SnmTNCiC71nJ9Rcq3PsnpU6Vc4rWK8= +modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU= +modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg= +modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI= +modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw= +modernc.org/opt v0.2.0 h1:tGyef5ApycA7FSEOMraay9SaTk5zmbx7Tu+cJs4QKZg= +modernc.org/opt v0.2.0/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns= +modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w= +modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE= +modernc.org/sqlite v1.53.0 h1:20WG8N9q4ji/dEqGk4uiI0c6OPjSeLTNYGFCc3+7c1M= +modernc.org/sqlite v1.53.0/go.mod h1:xoEpOIpGrgT48H5iiyt/YXPCZPEzlfmfFwtk8Lklw8s= +modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0= +modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A= +modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y= +modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM= diff --git a/internal/cluster/types.go b/internal/cluster/types.go new file mode 100644 index 0000000..7a46bc1 --- /dev/null +++ b/internal/cluster/types.go @@ -0,0 +1,56 @@ +// Package cluster defines the Cluster domain type — a configured Spark/YARN +// endpoint pair plus authentication, SSL, rate-limit, and Spark-submit metadata. +// +// Cluster is the only entity persisted in this project besides the audit log. +// It is the discovery root for the LLM agent: list_clusters returns the full +// struct (minus the password) so the agent can resolve RM/SHS URLs, choose +// the correct auth flavor, and respect the URL allowlist. +package cluster + +import "time" + +// AuthType enumerates the YARN/Hadoop authentication flavors supported in v1. +// +// "none" — no credentials; public or pre-trusted clusters +// "simple" — YARN SimpleAuth: appends ?user.name= to every request +// "basic" — HTTP Basic; AuthUsername + AuthPassword (encrypted at rest) +// +// Kerberos is intentionally absent in v1; the deployment uses SimpleAuth. +type AuthType string + +const ( + AuthNone AuthType = "none" + AuthSimple AuthType = "simple" + AuthBasic AuthType = "basic" +) + +// Cluster is one configured Spark-on-YARN endpoint pair. +// +// Field semantics: +// - SparkSubmitExecuteBin: absolute path to spark-submit (or spark2-submit); +// exec.Command uses this binary directly, never the shell. +// - IsActive: only active clusters are returned by LLM-facing list_clusters. +// - AuthPassword: zero value ("") on Update means "do not touch the stored +// password" — keeps the existing encrypted blob intact. +// - URLAllowlist: extra glob patterns beyond the RM/SHS hosts; fetch_url +// uses this to permit long-tail SHS endpoints the agent may construct. +// - DefaultSubmitArgs: prepended to every spark_submit Tool call's args. +// - RateLimitPerMin: 0 disables the per-cluster limiter. +type Cluster struct { + ID string `json:"id"` + Name string `json:"name"` + RMURL string `json:"rm_url"` + SHSURL string `json:"shs_url"` + SparkSubmitExecuteBin string `json:"spark_submit_execute_bin"` + IsActive bool `json:"is_active"` + AuthType AuthType `json:"auth_type"` + AuthUsername string `json:"auth_username,omitempty"` + AuthPassword string `json:"-"` // never serialized; encrypted at rest in auth_password_enc + SSLVerify bool `json:"ssl_verify"` + SSLCABundle string `json:"ssl_ca_bundle,omitempty"` + URLAllowlist []string `json:"url_allowlist,omitempty"` + DefaultSubmitArgs []string `json:"default_submit_args,omitempty"` + RateLimitPerMin int `json:"rate_limit_per_min"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` +} diff --git a/internal/config/config.go b/internal/config/config.go new file mode 100644 index 0000000..6319fc1 --- /dev/null +++ b/internal/config/config.go @@ -0,0 +1,225 @@ +// Package config loads server configuration from environment variables. +package config + +import ( + "fmt" + "os" + "path/filepath" + "strconv" + "strings" + "time" +) + +// Config holds runtime configuration for the Spark MCP server. +type Config struct { + ListenAddr string + DataDir string + SQLitePath string + AdminTokens []string + AgentToken string + HTTPClientTimeout time.Duration + MaxResponseBytes int64 + SparkSubmitTimeout time.Duration + LogDir string + LogLevel string + LogFormat string + AnalyzerDataSkewRatio float64 + AnalyzerGCPressureRatio float64 + AnalyzerBottleneckShuffleGB float64 +} + +// Load reads configuration from environment variables and returns a populated Config. +// Required variables (ADMIN_TOKENS, AGENT_TOKEN) produce clear errors when missing. +func Load() (*Config, error) { + cfg := &Config{ + ListenAddr: ":8080", + DataDir: "./data", + HTTPClientTimeout: 30 * time.Second, + MaxResponseBytes: 1 << 20, + SparkSubmitTimeout: 60 * time.Second, + LogDir: "./data/logs", + LogLevel: "info", + LogFormat: "text", + AnalyzerDataSkewRatio: 3.0, + AnalyzerGCPressureRatio: 0.1, + AnalyzerBottleneckShuffleGB: 50.0, + } + + cfg.ListenAddr = envString("LISTEN_ADDR", cfg.ListenAddr) + cfg.DataDir = envString("DATA_DIR", cfg.DataDir) + + if err := loadAdminTokens(cfg); err != nil { + return nil, err + } + if err := loadAgentToken(cfg); err != nil { + return nil, err + } + + var err error + + cfg.HTTPClientTimeout, err = parseDuration("HTTP_CLIENT_TIMEOUT", cfg.HTTPClientTimeout) + if err != nil { + return nil, err + } + + cfg.MaxResponseBytes, err = parseInt64("MAX_RESPONSE_BYTES", cfg.MaxResponseBytes) + if err != nil { + return nil, err + } + + cfg.SparkSubmitTimeout, err = parseDuration("SPARK_SUBMIT_TIMEOUT", cfg.SparkSubmitTimeout) + if err != nil { + return nil, err + } + + cfg.LogDir = envString("LOG_DIR", cfg.LogDir) + cfg.LogLevel = envString("LOG_LEVEL", cfg.LogLevel) + cfg.LogFormat = envString("LOG_FORMAT", cfg.LogFormat) + + cfg.AnalyzerDataSkewRatio, err = parseFloat64("ANALYZER_DATA_SKEW_RATIO", cfg.AnalyzerDataSkewRatio) + if err != nil { + return nil, err + } + + cfg.AnalyzerGCPressureRatio, err = parseFloat64("ANALYZER_GC_PRESSURE_RATIO", cfg.AnalyzerGCPressureRatio) + if err != nil { + return nil, err + } + + cfg.AnalyzerBottleneckShuffleGB, err = parseFloat64("ANALYZER_BOTTLENECK_SHUFFLE_GB", cfg.AnalyzerBottleneckShuffleGB) + if err != nil { + return nil, err + } + + if err := validateLogLevel(cfg.LogLevel); err != nil { + return nil, err + } + if err := validateLogFormat(cfg.LogFormat); err != nil { + return nil, err + } + + cfg.SQLitePath = filepath.Join(cfg.DataDir, "spark-mcp.db") + + return cfg, nil +} + +func loadAdminTokens(cfg *Config) error { + v := os.Getenv("ADMIN_TOKENS") + if v == "" { + return fmt.Errorf("config: required environment variable ADMIN_TOKENS is not set") + } + + parts := strings.Split(v, ",") + cfg.AdminTokens = make([]string, 0, len(parts)) + for _, p := range parts { + p = strings.TrimSpace(p) + if p == "" { + continue + } + cfg.AdminTokens = append(cfg.AdminTokens, p) + } + + if len(cfg.AdminTokens) == 0 { + return fmt.Errorf("config: environment variable ADMIN_TOKENS contains no valid tokens") + } + return nil +} + +func loadAgentToken(cfg *Config) error { + cfg.AgentToken = os.Getenv("AGENT_TOKEN") + if cfg.AgentToken == "" { + return fmt.Errorf("config: required environment variable AGENT_TOKEN is not set") + } + return nil +} + +func envString(key, defaultValue string) string { + if v := os.Getenv(key); v != "" { + return v + } + return defaultValue +} + +func parseDuration(key string, defaultValue time.Duration) (time.Duration, error) { + v := os.Getenv(key) + if v == "" { + return defaultValue, nil + } + d, err := time.ParseDuration(v) + if err != nil { + return 0, fmt.Errorf("config: invalid duration for %s: %w", key, err) + } + return d, nil +} + +func parseInt64(key string, defaultValue int64) (int64, error) { + v := os.Getenv(key) + if v == "" { + return defaultValue, nil + } + n, err := strconv.ParseInt(v, 10, 64) + if err != nil { + return 0, fmt.Errorf("config: invalid integer for %s: %w", key, err) + } + return n, nil +} + +func parseFloat64(key string, defaultValue float64) (float64, error) { + v := os.Getenv(key) + if v == "" { + return defaultValue, nil + } + f, err := strconv.ParseFloat(v, 64) + if err != nil { + return 0, fmt.Errorf("config: invalid float for %s: %w", key, err) + } + return f, nil +} + +func validateLogLevel(level string) error { + switch level { + case "debug", "info", "warn", "error": + return nil + } + return fmt.Errorf("config: invalid LOG_LEVEL %q, want debug/info/warn/error", level) +} + +func validateLogFormat(format string) error { + switch format { + case "text", "json": + return nil + } + return fmt.Errorf("config: invalid LOG_FORMAT %q, want text/json", format) +} + +// String returns a human-readable representation of the configuration. +// Sensitive values (AdminTokens and AgentToken) are summarized, not printed. +func (c *Config) String() string { + adminTotalLen := 0 + for _, t := range c.AdminTokens { + adminTotalLen += len(t) + } + adminSummary := fmt.Sprintf("%d token(s), total_len=%d", len(c.AdminTokens), adminTotalLen) + agentSummary := fmt.Sprintf("len=%d", len(c.AgentToken)) + + return fmt.Sprintf( + "ListenAddr=%s DataDir=%s SQLitePath=%s AdminTokens=%s AgentToken=%s "+ + "HTTPClientTimeout=%s MaxResponseBytes=%d SparkSubmitTimeout=%s "+ + "LogDir=%s LogLevel=%s LogFormat=%s AnalyzerDataSkewRatio=%.1f "+ + "AnalyzerGCPressureRatio=%.1f AnalyzerBottleneckShuffleGB=%.1f", + c.ListenAddr, + c.DataDir, + c.SQLitePath, + adminSummary, + agentSummary, + c.HTTPClientTimeout, + c.MaxResponseBytes, + c.SparkSubmitTimeout, + c.LogDir, + c.LogLevel, + c.LogFormat, + c.AnalyzerDataSkewRatio, + c.AnalyzerGCPressureRatio, + c.AnalyzerBottleneckShuffleGB, + ) +} diff --git a/internal/logging/logger.go b/internal/logging/logger.go new file mode 100644 index 0000000..d0ede87 --- /dev/null +++ b/internal/logging/logger.go @@ -0,0 +1,135 @@ +package logging + +import ( + "context" + "errors" + "fmt" + "log/slog" + "os" + "path/filepath" + "strings" + "sync" +) + +// Config configures the application logger. +type Config struct { + Level string // "debug"|"info"|"warn"|"error" + Dir string // e.g. "./data/logs" + Format string // "text"|"json" — controls terminal output; files are always JSON + LogToStd bool // true: enable terminal output +} + +// ToolCallLogger records the lifecycle of a single tool call. +type ToolCallLogger interface { + WithResult(result any) + WithError(err error) + End() +} + +// logDir is shared with StartToolCall so the per-tool-call logger knows where +// to create files. It is set by New and read by StartToolCall. +var ( + logDir string + logDirMu sync.RWMutex +) + +func parseLevel(s string) slog.Level { + switch strings.ToLower(s) { + case "debug": + return slog.LevelDebug + case "warn", "warning": + return slog.LevelWarn + case "error": + return slog.LevelError + default: + return slog.LevelInfo + } +} + +// multiHandler fans out records to multiple underlying handlers. +type multiHandler struct { + handlers []slog.Handler +} + +func (m *multiHandler) Enabled(ctx context.Context, l slog.Level) bool { + for _, h := range m.handlers { + if h.Enabled(ctx, l) { + return true + } + } + return false +} + +func (m *multiHandler) Handle(ctx context.Context, r slog.Record) error { + var errs []error + for _, h := range m.handlers { + if h.Enabled(ctx, r.Level) { + if err := h.Handle(ctx, r); err != nil { + errs = append(errs, err) + } + } + } + return errors.Join(errs...) +} + +func (m *multiHandler) WithAttrs(attrs []slog.Attr) slog.Handler { + handlers := make([]slog.Handler, len(m.handlers)) + for i, h := range m.handlers { + handlers[i] = h.WithAttrs(attrs) + } + return &multiHandler{handlers: handlers} +} + +func (m *multiHandler) WithGroup(name string) slog.Handler { + handlers := make([]slog.Handler, len(m.handlers)) + for i, h := range m.handlers { + handlers[i] = h.WithGroup(name) + } + return &multiHandler{handlers: handlers} +} + +// New creates the main application logger and a shutdown closure. +func New(cfg Config) (*slog.Logger, func(), error) { + if err := os.MkdirAll(cfg.Dir, 0o755); err != nil { + return nil, nil, fmt.Errorf("logging: create log dir: %w", err) + } + + toolsDir := filepath.Join(cfg.Dir, "tools") + if err := os.MkdirAll(toolsDir, 0o700); err != nil { + return nil, nil, fmt.Errorf("logging: create tools log dir: %w", err) + } + + logDirMu.Lock() + logDir = cfg.Dir + logDirMu.Unlock() + + level := parseLevel(cfg.Level) + opts := &slog.HandlerOptions{Level: level} + + mainPath := filepath.Join(cfg.Dir, "spark-mcp.log") + f, err := os.OpenFile(mainPath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600) + if err != nil { + return nil, nil, fmt.Errorf("logging: open main log file: %w", err) + } + + fileHandler := slog.NewJSONHandler(f, opts) + + var handler slog.Handler = fileHandler + if cfg.LogToStd { + var stdHandler slog.Handler + switch strings.ToLower(cfg.Format) { + case "json": + stdHandler = slog.NewJSONHandler(os.Stdout, opts) + default: + stdHandler = slog.NewTextHandler(os.Stdout, opts) + } + handler = &multiHandler{handlers: []slog.Handler{fileHandler, stdHandler}} + } + + shutdown := func() { + _ = f.Close() + } + + logger := slog.New(handler).With("component", "main") + return logger, shutdown, nil +} diff --git a/internal/logging/tool_call.go b/internal/logging/tool_call.go new file mode 100644 index 0000000..d396736 --- /dev/null +++ b/internal/logging/tool_call.go @@ -0,0 +1,191 @@ +package logging + +import ( + "log/slog" + "context" + "crypto/rand" + "encoding/hex" + "encoding/json" + "fmt" + "os" + "path/filepath" + "regexp" + "strings" + "sync" + "time" +) + +const toolCallResultMaxBytes = 10 * 1024 + +var toolNameSanitizer = regexp.MustCompile(`[^a-zA-Z0-9_]+`) + +func sanitizeToolName(name string) string { + s := toolNameSanitizer.ReplaceAllString(name, "_") + if len(s) > 32 { + s = s[:32] + } + s = strings.Trim(s, "_") + if s == "" { + s = "tool" + } + return s +} + +func shortHexID(n int) (string, error) { + b := make([]byte, n/2+n%2) + if _, err := rand.Read(b); err != nil { + return "", err + } + return hex.EncodeToString(b)[:n], nil +} + +type toolCallLogger struct { + file *os.File + toolName string + startedAt time.Time + resultRaw json.RawMessage + err error + endOnce sync.Once +} + +func (t *toolCallLogger) WithResult(result any) { + t.resultRaw = marshalResult(result) +} + +func (t *toolCallLogger) WithError(err error) { + t.err = err +} + +func (t *toolCallLogger) End() { + t.endOnce.Do(func() { + defer func() { _ = t.file.Close() }() + + durationMs := time.Since(t.startedAt).Milliseconds() + errStr := "" + if t.err != nil { + errStr = t.err.Error() + } + + end := map[string]any{ + "event": "end", + "duration_ms": durationMs, + "result": nil, + "error": errStr, + } + if t.resultRaw != nil { + end["result"] = json.RawMessage(t.resultRaw) + } + + b, err := json.Marshal(end) + if err != nil { + b = []byte(fmt.Sprintf(`{"event":"end","duration_ms":%d,"result":null,"error":%q}`, durationMs, errStr)) + } + _, _ = t.file.Write(b) + _, _ = t.file.WriteString("\n") + }) +} + +// StartToolCall creates a per-tool-call log file and returns a logger that +// records the call lifecycle. +func StartToolCall(ctx context.Context, base *slog.Logger, toolName string, params any) (ToolCallLogger, error) { + logDirMu.RLock() + dir := logDir + logDirMu.RUnlock() + if dir == "" { + return nil, fmt.Errorf("logging: log directory not initialized; call New first") + } + + toolsDir := filepath.Join(dir, "tools") + if err := os.MkdirAll(toolsDir, 0o700); err != nil { + return nil, fmt.Errorf("logging: create tools log dir: %w", err) + } + + safeName := sanitizeToolName(toolName) + ts := time.Now().Format("20060102_150405.000") + id, err := shortHexID(6) + if err != nil { + return nil, fmt.Errorf("logging: generate tool call id: %w", err) + } + filename := fmt.Sprintf("%s_%s_%s.log", ts, safeName, id) + path := filepath.Join(toolsDir, filename) + + f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600) + if err != nil { + return nil, fmt.Errorf("logging: create tool call log file: %w", err) + } + + startedAt := time.Now() + start := map[string]any{ + "event": "start", + "tool": toolName, + "params": marshalValue(params), + "started_at": startedAt.Format(time.RFC3339Nano), + } + if d, ok := ctx.Deadline(); ok { + start["deadline"] = d.Format(time.RFC3339Nano) + } + + b, err := json.Marshal(start) + if err != nil { + _ = f.Close() + return nil, fmt.Errorf("logging: marshal start record: %w", err) + } + if _, err := f.Write(b); err != nil { + _ = f.Close() + return nil, fmt.Errorf("logging: write start record: %w", err) + } + if _, err := f.WriteString("\n"); err != nil { + _ = f.Close() + return nil, fmt.Errorf("logging: write start record: %w", err) + } + + if base != nil { + base.Info("tool.call.start", "tool", toolName, "file", filename) + } + + return &toolCallLogger{ + file: f, + toolName: toolName, + startedAt: startedAt, + }, nil +} + +func marshalValue(v any) json.RawMessage { + if v == nil { + return []byte("null") + } + b, err := json.MarshalIndent(v, "", " ") + if err != nil { + b, err = json.Marshal(v) + if err != nil { + return mustMarshalString("") + } + } + return b +} + +func marshalResult(v any) json.RawMessage { + if v == nil { + return []byte("null") + } + b, err := json.MarshalIndent(v, "", " ") + if err != nil { + b, err = json.Marshal(v) + if err != nil { + return mustMarshalString("") + } + } + if len(b) > toolCallResultMaxBytes { + trunc := string(b[:toolCallResultMaxBytes]) + "...[truncated]" + return mustMarshalString(trunc) + } + return b +} + +func mustMarshalString(s string) json.RawMessage { + b, err := json.Marshal(s) + if err != nil { + return []byte(`"` + strings.ReplaceAll(s, `"`, `\"`) + `"`) + } + return b +} diff --git a/internal/storage/cluster_repo.go b/internal/storage/cluster_repo.go new file mode 100644 index 0000000..721d58c --- /dev/null +++ b/internal/storage/cluster_repo.go @@ -0,0 +1,252 @@ +package storage + +import ( + "context" + "database/sql" + "encoding/json" + "errors" + "fmt" + "strings" + "time" + + "spark-mcp-go/internal/cluster" +) + +// ErrNotFound is returned when a requested cluster does not exist. +var ErrNotFound = errors.New("storage: cluster not found") + +// ClusterRepo provides CRUD operations for clusters. +type ClusterRepo struct { + db *DB +} + +// Clusters returns a repository bound to this DB handle. +func (d *DB) Clusters() *ClusterRepo { + return &ClusterRepo{db: d} +} + +// Create inserts a new cluster record. +// +// Timestamps are stored as Unix nanoseconds (INTEGER) — sub-second precision +// keeps created_at unique across rapid inserts and makes ORDER BY DESC +// stable without artificial delays. +// +// If a cluster with the same id already exists, the underlying SQLite +// UNIQUE constraint error is wrapped and returned. +func (r *ClusterRepo) Create(ctx context.Context, c *cluster.Cluster) error { + now := time.Now().UnixNano() + c.CreatedAt = time.Unix(0, now) + c.UpdatedAt = c.CreatedAt + + allowlistJSON, err := json.Marshal(c.URLAllowlist) + if err != nil { + return fmt.Errorf("storage: create cluster %s: marshal url_allowlist: %w", c.ID, err) + } + defaultArgsJSON, err := json.Marshal(c.DefaultSubmitArgs) + if err != nil { + return fmt.Errorf("storage: create cluster %s: marshal default_submit_args: %w", c.ID, err) + } + + // TODO(phase-5): encrypt auth_password_enc instead of storing plaintext. + pwdEnc := []byte(c.AuthPassword) + + _, err = r.db.sqlDB.ExecContext(ctx, ` + INSERT INTO clusters ( + id, name, rm_url, shs_url, spark_submit_execute_bin, + is_active, auth_type, auth_username, auth_password_enc, + ssl_verify, ssl_ca_bundle, url_allowlist, default_submit_args, + rate_limit_per_min, created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + c.ID, c.Name, c.RMURL, c.SHSURL, c.SparkSubmitExecuteBin, + boolToInt(c.IsActive), string(c.AuthType), c.AuthUsername, pwdEnc, + boolToInt(c.SSLVerify), c.SSLCABundle, string(allowlistJSON), string(defaultArgsJSON), + c.RateLimitPerMin, now, now, + ) + if err != nil { + if strings.Contains(err.Error(), "UNIQUE constraint failed") { + return fmt.Errorf("storage: create cluster %s: primary key conflict: %w", c.ID, err) + } + return fmt.Errorf("storage: create cluster %s: %w", c.ID, err) + } + return nil +} + +// Get retrieves a cluster by id. If the cluster does not exist, it returns +// ErrNotFound. +func (r *ClusterRepo) Get(ctx context.Context, id string) (*cluster.Cluster, error) { + row := r.db.sqlDB.QueryRowContext(ctx, ` + SELECT id, name, rm_url, shs_url, spark_submit_execute_bin, + is_active, auth_type, auth_username, auth_password_enc, + ssl_verify, ssl_ca_bundle, url_allowlist, default_submit_args, + rate_limit_per_min, created_at, updated_at + FROM clusters + WHERE id = ?`, id) + + c, err := scanCluster(row.Scan) + if errors.Is(err, sql.ErrNoRows) { + return nil, ErrNotFound + } + if err != nil { + return nil, fmt.Errorf("storage: get cluster %s: %w", id, err) + } + return c, nil +} + +// List returns every cluster, including inactive ones, ordered by created_at +// descending. +func (r *ClusterRepo) List(ctx context.Context) ([]*cluster.Cluster, error) { + rows, err := r.db.sqlDB.QueryContext(ctx, ` + SELECT id, name, rm_url, shs_url, spark_submit_execute_bin, + is_active, auth_type, auth_username, auth_password_enc, + ssl_verify, ssl_ca_bundle, url_allowlist, default_submit_args, + rate_limit_per_min, created_at, updated_at + FROM clusters + ORDER BY created_at DESC`) + if err != nil { + return nil, fmt.Errorf("storage: list clusters: %w", err) + } + defer rows.Close() + + var out []*cluster.Cluster + for rows.Next() { + c, err := scanCluster(rows.Scan) + if err != nil { + return nil, fmt.Errorf("storage: list clusters: %w", err) + } + out = append(out, c) + } + if err := rows.Err(); err != nil { + return nil, fmt.Errorf("storage: list clusters: %w", err) + } + return out, nil +} + +// Update modifies an existing cluster. An empty AuthPassword means "do not +// touch the stored password" — the existing auth_password_enc blob is kept +// intact. +func (r *ClusterRepo) Update(ctx context.Context, c *cluster.Cluster) error { + c.UpdatedAt = time.Unix(0, time.Now().UnixNano()) + + allowlistJSON, err := json.Marshal(c.URLAllowlist) + if err != nil { + return fmt.Errorf("storage: update cluster %s: marshal url_allowlist: %w", c.ID, err) + } + defaultArgsJSON, err := json.Marshal(c.DefaultSubmitArgs) + if err != nil { + return fmt.Errorf("storage: update cluster %s: marshal default_submit_args: %w", c.ID, err) + } + + var res sql.Result + if c.AuthPassword == "" { + // Preserve the existing password blob. + res, err = r.db.sqlDB.ExecContext(ctx, ` + UPDATE clusters + SET name = ?, rm_url = ?, shs_url = ?, spark_submit_execute_bin = ?, + is_active = ?, auth_type = ?, auth_username = ?, + ssl_verify = ?, ssl_ca_bundle = ?, url_allowlist = ?, + default_submit_args = ?, rate_limit_per_min = ?, updated_at = ? + WHERE id = ?`, + c.Name, c.RMURL, c.SHSURL, c.SparkSubmitExecuteBin, + boolToInt(c.IsActive), string(c.AuthType), c.AuthUsername, + boolToInt(c.SSLVerify), c.SSLCABundle, string(allowlistJSON), + string(defaultArgsJSON), c.RateLimitPerMin, c.UpdatedAt.UnixNano(), c.ID, + ) + } else { + // TODO(phase-5): encrypt auth_password_enc instead of storing plaintext. + pwdEnc := []byte(c.AuthPassword) + res, err = r.db.sqlDB.ExecContext(ctx, ` + UPDATE clusters + SET name = ?, rm_url = ?, shs_url = ?, spark_submit_execute_bin = ?, + is_active = ?, auth_type = ?, auth_username = ?, auth_password_enc = ?, + ssl_verify = ?, ssl_ca_bundle = ?, url_allowlist = ?, + default_submit_args = ?, rate_limit_per_min = ?, updated_at = ? + WHERE id = ?`, + c.Name, c.RMURL, c.SHSURL, c.SparkSubmitExecuteBin, + boolToInt(c.IsActive), string(c.AuthType), c.AuthUsername, pwdEnc, + boolToInt(c.SSLVerify), c.SSLCABundle, string(allowlistJSON), + string(defaultArgsJSON), c.RateLimitPerMin, c.UpdatedAt.UnixNano(), c.ID, + ) + } + if err != nil { + return fmt.Errorf("storage: update cluster %s: %w", c.ID, err) + } + + n, err := res.RowsAffected() + if err != nil { + return fmt.Errorf("storage: update cluster %s: rows affected: %w", c.ID, err) + } + if n == 0 { + return ErrNotFound + } + return nil +} + +// Delete removes a cluster by id. If the cluster does not exist, it returns +// ErrNotFound. +func (r *ClusterRepo) Delete(ctx context.Context, id string) error { + res, err := r.db.sqlDB.ExecContext(ctx, `DELETE FROM clusters WHERE id = ?`, id) + if err != nil { + return fmt.Errorf("storage: delete cluster %s: %w", id, err) + } + n, err := res.RowsAffected() + if err != nil { + return fmt.Errorf("storage: delete cluster %s: rows affected: %w", id, err) + } + if n == 0 { + return ErrNotFound + } + return nil +} + +// scanFunc abstracts *sql.Row.Scan and *sql.Rows.Scan so a single mapping +// function can serve both QueryRow and Query paths. +type scanFunc func(dest ...any) error + +// scanCluster maps a single SQL row into a cluster.Cluster value. +func scanCluster(scan scanFunc) (*cluster.Cluster, error) { + var c cluster.Cluster + var allowlistJSON, defaultArgsJSON sql.NullString + var pwdEnc []byte + var createdAt, updatedAt int64 + var isActive, sslVerify int + + if err := scan( + &c.ID, &c.Name, &c.RMURL, &c.SHSURL, &c.SparkSubmitExecuteBin, + &isActive, &c.AuthType, &c.AuthUsername, &pwdEnc, + &sslVerify, &c.SSLCABundle, &allowlistJSON, &defaultArgsJSON, + &c.RateLimitPerMin, &createdAt, &updatedAt, + ); err != nil { + return nil, err + } + + c.IsActive = intToBool(isActive) + c.SSLVerify = intToBool(sslVerify) + c.CreatedAt = time.Unix(0, createdAt) + c.UpdatedAt = time.Unix(0, updatedAt) + // TODO(phase-5): decrypt pwdEnc once encryption is introduced. + c.AuthPassword = string(pwdEnc) + + if allowlistJSON.Valid && allowlistJSON.String != "" { + if err := json.Unmarshal([]byte(allowlistJSON.String), &c.URLAllowlist); err != nil { + return nil, fmt.Errorf("storage: scan cluster %s: unmarshal url_allowlist: %w", c.ID, err) + } + } + if defaultArgsJSON.Valid && defaultArgsJSON.String != "" { + if err := json.Unmarshal([]byte(defaultArgsJSON.String), &c.DefaultSubmitArgs); err != nil { + return nil, fmt.Errorf("storage: scan cluster %s: unmarshal default_submit_args: %w", c.ID, err) + } + } + + return &c, nil +} + +func boolToInt(b bool) int { + if b { + return 1 + } + return 0 +} + +func intToBool(i int) bool { + return i != 0 +} diff --git a/internal/storage/cluster_repo_test.go b/internal/storage/cluster_repo_test.go new file mode 100644 index 0000000..f382ba8 --- /dev/null +++ b/internal/storage/cluster_repo_test.go @@ -0,0 +1,306 @@ +package storage + +import ( + "context" + "errors" + "reflect" + "testing" + + "spark-mcp-go/internal/cluster" +) + +func newClusterRepo(t *testing.T) *ClusterRepo { + t.Helper() + db, err := Open(":memory:") + if err != nil { + t.Fatalf("Open in-memory db: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + return db.Clusters() +} + +func testClusterA() *cluster.Cluster { + return &cluster.Cluster{ + ID: "cluster-a", + Name: "Cluster Alpha", + RMURL: "http://rm-a.example.com:8088", + SHSURL: "http://shs-a.example.com:18080", + SparkSubmitExecuteBin: "/usr/bin/spark-submit", + IsActive: true, + AuthType: cluster.AuthBasic, + AuthUsername: "admin-a", + AuthPassword: "secret-1", + SSLVerify: true, + SSLCABundle: "/etc/ssl/ca-bundle-a.pem", + URLAllowlist: []string{"*.a.example.com", "http://hist-a.example.com/*"}, + DefaultSubmitArgs: []string{"--master=yarn", "--deploy-mode=cluster"}, + RateLimitPerMin: 10, + } +} + +func assertClusterEqual(t *testing.T, got, want *cluster.Cluster) { + t.Helper() + + if got.ID != want.ID { + t.Errorf("ID: got %q, want %q", got.ID, want.ID) + } + if got.Name != want.Name { + t.Errorf("Name: got %q, want %q", got.Name, want.Name) + } + if got.RMURL != want.RMURL { + t.Errorf("RMURL: got %q, want %q", got.RMURL, want.RMURL) + } + if got.SHSURL != want.SHSURL { + t.Errorf("SHSURL: got %q, want %q", got.SHSURL, want.SHSURL) + } + if got.SparkSubmitExecuteBin != want.SparkSubmitExecuteBin { + t.Errorf("SparkSubmitExecuteBin: got %q, want %q", got.SparkSubmitExecuteBin, want.SparkSubmitExecuteBin) + } + if got.IsActive != want.IsActive { + t.Errorf("IsActive: got %v, want %v", got.IsActive, want.IsActive) + } + if got.AuthType != want.AuthType { + t.Errorf("AuthType: got %q, want %q", got.AuthType, want.AuthType) + } + if got.AuthUsername != want.AuthUsername { + t.Errorf("AuthUsername: got %q, want %q", got.AuthUsername, want.AuthUsername) + } + if got.AuthPassword != want.AuthPassword { + t.Errorf("AuthPassword: got %q, want %q", got.AuthPassword, want.AuthPassword) + } + if got.SSLVerify != want.SSLVerify { + t.Errorf("SSLVerify: got %v, want %v", got.SSLVerify, want.SSLVerify) + } + if got.SSLCABundle != want.SSLCABundle { + t.Errorf("SSLCABundle: got %q, want %q", got.SSLCABundle, want.SSLCABundle) + } + if !reflect.DeepEqual(got.URLAllowlist, want.URLAllowlist) { + t.Errorf("URLAllowlist: got %v, want %v", got.URLAllowlist, want.URLAllowlist) + } + if !reflect.DeepEqual(got.DefaultSubmitArgs, want.DefaultSubmitArgs) { + t.Errorf("DefaultSubmitArgs: got %v, want %v", got.DefaultSubmitArgs, want.DefaultSubmitArgs) + } + if got.RateLimitPerMin != want.RateLimitPerMin { + t.Errorf("RateLimitPerMin: got %d, want %d", got.RateLimitPerMin, want.RateLimitPerMin) + } + + if !got.CreatedAt.Equal(want.CreatedAt) { + if got.CreatedAt.Unix() != want.CreatedAt.Unix() { + t.Errorf("CreatedAt: got %v, want %v", got.CreatedAt, want.CreatedAt) + } + } + if !got.UpdatedAt.Equal(want.UpdatedAt) { + if got.UpdatedAt.Unix() != want.UpdatedAt.Unix() { + t.Errorf("UpdatedAt: got %v, want %v", got.UpdatedAt, want.UpdatedAt) + } + } +} + +func TestClusterRepo_Create(t *testing.T) { + repo := newClusterRepo(t) + ctx := context.Background() + + want := testClusterA() + if err := repo.Create(ctx, want); err != nil { + t.Fatalf("Create: %v", err) + } + + got, err := repo.Get(ctx, want.ID) + if err != nil { + t.Fatalf("Get after Create: %v", err) + } + assertClusterEqual(t, got, want) +} + +func TestClusterRepo_Get(t *testing.T) { + repo := newClusterRepo(t) + ctx := context.Background() + + want := testClusterA() + if err := repo.Create(ctx, want); err != nil { + t.Fatalf("Create: %v", err) + } + + got, err := repo.Get(ctx, want.ID) + if err != nil { + t.Fatalf("Get: %v", err) + } + assertClusterEqual(t, got, want) + + if _, err := repo.Get(ctx, "nope"); !errors.Is(err, ErrNotFound) { + t.Fatalf("Get missing cluster: got %v, want ErrNotFound", err) + } +} + +func TestClusterRepo_List(t *testing.T) { + repo := newClusterRepo(t) + ctx := context.Background() + + ids := []string{"cluster-a", "cluster-b", "cluster-c"} + for _, id := range ids { + c := testClusterA() + c.ID = id + c.Name = id + if err := repo.Create(ctx, c); err != nil { + t.Fatalf("Create %s: %v", id, err) + } + // Sub-second precision in stored timestamps gives us a stable + // ORDER BY DESC without artificial delays. + } + + got, err := repo.List(ctx) + if err != nil { + t.Fatalf("List: %v", err) + } + if len(got) != len(ids) { + t.Fatalf("List len: got %d, want %d", len(got), len(ids)) + } + + // List must be ordered by created_at DESC: last created first. + wantOrder := []string{"cluster-c", "cluster-b", "cluster-a"} + for i, c := range got { + if c.ID != wantOrder[i] { + t.Errorf("List order[%d]: got %q, want %q", i, c.ID, wantOrder[i]) + } + } +} + +func TestClusterRepo_Update(t *testing.T) { + repo := newClusterRepo(t) + ctx := context.Background() + + c := testClusterA() + if err := repo.Create(ctx, c); err != nil { + t.Fatalf("Create: %v", err) + } + + stored, err := repo.Get(ctx, c.ID) + if err != nil { + t.Fatalf("Get before Update: %v", err) + } + + stored.Name = "Cluster Alpha Updated" + stored.RMURL = "http://rm-a-new.example.com:8088" + stored.IsActive = false + stored.RateLimitPerMin = 20 + // Intentionally leave AuthPassword empty: must preserve existing password. + stored.AuthPassword = "" + + if err := repo.Update(ctx, stored); err != nil { + t.Fatalf("Update: %v", err) + } + + got, err := repo.Get(ctx, c.ID) + if err != nil { + t.Fatalf("Get after Update: %v", err) + } + + if got.Name != stored.Name { + t.Errorf("Name: got %q, want %q", got.Name, stored.Name) + } + if got.RMURL != stored.RMURL { + t.Errorf("RMURL: got %q, want %q", got.RMURL, stored.RMURL) + } + if got.IsActive != false { + t.Errorf("IsActive: got %v, want false", got.IsActive) + } + if got.RateLimitPerMin != 20 { + t.Errorf("RateLimitPerMin: got %d, want 20", got.RateLimitPerMin) + } + if got.AuthPassword != c.AuthPassword { + t.Errorf("AuthPassword should be preserved: got %q, want %q", got.AuthPassword, c.AuthPassword) + } + + // Update a non-existing cluster must return ErrNotFound. + missing := testClusterA() + missing.ID = "missing" + if err := repo.Update(ctx, missing); !errors.Is(err, ErrNotFound) { + t.Fatalf("Update missing cluster: got %v, want ErrNotFound", err) + } +} + +func TestClusterRepo_UpdatePassword(t *testing.T) { + repo := newClusterRepo(t) + ctx := context.Background() + + c := testClusterA() + c.AuthPassword = "old" + if err := repo.Create(ctx, c); err != nil { + t.Fatalf("Create: %v", err) + } + + stored, err := repo.Get(ctx, c.ID) + if err != nil { + t.Fatalf("Get before Update: %v", err) + } + if stored.AuthPassword != "old" { + t.Fatalf("initial password: got %q, want %q", stored.AuthPassword, "old") + } + + stored.AuthPassword = "new" + if err := repo.Update(ctx, stored); err != nil { + t.Fatalf("Update password: %v", err) + } + + got, err := repo.Get(ctx, c.ID) + if err != nil { + t.Fatalf("Get after Update: %v", err) + } + if got.AuthPassword != "new" { + t.Errorf("password: got %q, want %q", got.AuthPassword, "new") + } +} + +func TestClusterRepo_UpdateEmptyPasswordKeepsOldPassword(t *testing.T) { + repo := newClusterRepo(t) + ctx := context.Background() + + c := testClusterA() + c.AuthPassword = "keep-me" + if err := repo.Create(ctx, c); err != nil { + t.Fatalf("Create: %v", err) + } + + stored, err := repo.Get(ctx, c.ID) + if err != nil { + t.Fatalf("Get before Update: %v", err) + } + stored.AuthPassword = "" + if err := repo.Update(ctx, stored); err != nil { + t.Fatalf("Update: %v", err) + } + + got, err := repo.Get(ctx, c.ID) + if err != nil { + t.Fatalf("Get after Update: %v", err) + } + if got.AuthPassword != "keep-me" { + t.Errorf("empty password should keep old value: got %q, want %q", got.AuthPassword, "keep-me") + } +} + +func TestClusterRepo_Delete(t *testing.T) { + repo := newClusterRepo(t) + ctx := context.Background() + + c := testClusterA() + if err := repo.Create(ctx, c); err != nil { + t.Fatalf("Create: %v", err) + } + + if err := repo.Delete(ctx, c.ID); err != nil { + t.Fatalf("Delete existing: %v", err) + } + + if _, err := repo.Get(ctx, c.ID); !errors.Is(err, ErrNotFound) { + t.Fatalf("Get after Delete: got %v, want ErrNotFound", err) + } + + if err := repo.Delete(ctx, c.ID); !errors.Is(err, ErrNotFound) { + t.Fatalf("Delete again: got %v, want ErrNotFound", err) + } + + if err := repo.Delete(ctx, "nope"); !errors.Is(err, ErrNotFound) { + t.Fatalf("Delete missing: got %v, want ErrNotFound", err) + } +} diff --git a/internal/storage/migrate.go b/internal/storage/migrate.go new file mode 100644 index 0000000..aaec3d7 --- /dev/null +++ b/internal/storage/migrate.go @@ -0,0 +1,16 @@ +package storage + +import "fmt" + +// Migrate applies Schema to the underlying database. Because Schema uses +// IF NOT EXISTS for all CREATE statements, a single Exec is sufficient and +// idempotent across restarts. +func (d *DB) Migrate() error { + if d == nil || d.sqlDB == nil { + return fmt.Errorf("storage: migrate: db not open") + } + if _, err := d.sqlDB.Exec(Schema); err != nil { + return fmt.Errorf("storage: migrate: %w", err) + } + return nil +} diff --git a/internal/storage/sqlite.go b/internal/storage/sqlite.go new file mode 100644 index 0000000..fdbdf9e --- /dev/null +++ b/internal/storage/sqlite.go @@ -0,0 +1,85 @@ +package storage + +import ( + "database/sql" + "fmt" + "os" + "path/filepath" + + _ "modernc.org/sqlite" +) + +// Schema defines the SQLite schema for clusters and audit_log. +const Schema = `CREATE TABLE IF NOT EXISTS clusters ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + rm_url TEXT NOT NULL, + shs_url TEXT NOT NULL, + spark_submit_execute_bin TEXT NOT NULL, + is_active INTEGER NOT NULL DEFAULT 1, + auth_type TEXT NOT NULL DEFAULT 'none', + auth_username TEXT, + auth_password_enc BLOB, + ssl_verify INTEGER NOT NULL DEFAULT 1, + ssl_ca_bundle TEXT, + url_allowlist TEXT, + default_submit_args TEXT, + rate_limit_per_min INTEGER NOT NULL DEFAULT 10, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL +); + +CREATE TABLE IF NOT EXISTS audit_log ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + timestamp INTEGER NOT NULL, + actor TEXT NOT NULL, + action TEXT NOT NULL, + cluster_id TEXT, + details TEXT +); +CREATE INDEX IF NOT EXISTS idx_audit_ts ON audit_log(timestamp DESC); +` + +// DB is the storage handle. +type DB struct { + Path string + sqlDB *sql.DB +} + +// Open opens the SQLite database at path, creating the parent directory if +// needed. It enables WAL mode and foreign keys, then applies the schema. +func Open(path string) (*DB, error) { + dir := filepath.Dir(path) + if err := os.MkdirAll(dir, 0o755); err != nil { + return nil, fmt.Errorf("storage: mkdir %s: %w", dir, err) + } + + sqlDB, err := sql.Open("sqlite", path) + if err != nil { + return nil, fmt.Errorf("storage: open sqlite %s: %w", path, err) + } + + if _, err := sqlDB.Exec("PRAGMA journal_mode=WAL"); err != nil { + _ = sqlDB.Close() + return nil, fmt.Errorf("storage: set WAL mode: %w", err) + } + if _, err := sqlDB.Exec("PRAGMA foreign_keys=ON"); err != nil { + _ = sqlDB.Close() + return nil, fmt.Errorf("storage: enable foreign keys: %w", err) + } + + d := &DB{Path: path, sqlDB: sqlDB} + if err := d.Migrate(); err != nil { + _ = sqlDB.Close() + return nil, err + } + return d, nil +} + +// Close releases the underlying database connection. +func (d *DB) Close() error { + if d == nil || d.sqlDB == nil { + return nil + } + return d.sqlDB.Close() +} diff --git a/main.go b/main.go index 56702b8..6d44789 100644 --- a/main.go +++ b/main.go @@ -1,20 +1,88 @@ +// Command spark-mcp-go launches the Spark MCP Server. +// +// Phase 0 scope: wire config + slog + Gin + /healthz. +// Full MCP transport, admin API, and Tool surface land in later phases. package main import ( - "fmt" + "context" + "errors" + "log/slog" + "net/http" + "os" + "os/signal" + "syscall" + "time" + + "github.com/gin-gonic/gin" + + "spark-mcp-go/internal/config" + "spark-mcp-go/internal/logging" ) -// TIP

To run your code, right-click the code and select Run.

Alternatively, click -// the icon in the gutter and select the Run menu item from here.

func main() { - //TIP

Press when your caret is at the underlined text - // to see how GoLand suggests fixing the warning.

Alternatively, if available, click the lightbulb to view possible fixes.

- s := "gopher" - fmt.Println("Hello and welcome, %s!", s) - - for i := 1; i <= 5; i++ { - //TIP

To start your debugging session, right-click your code in the editor and select the Debug option.

We have set one breakpoint - // for you, but you can always add more by pressing .

- fmt.Println("i =", 100/i) + if err := run(); err != nil { + // run() handles its own logging; this is the last-chance report. + slog.Error("fatal", "err", err) + os.Exit(1) } } + +func run() error { + cfg, err := config.Load() + if err != nil { + return err + } + + logger, shutdownLog, err := logging.New(logging.Config{ + Level: cfg.LogLevel, + Dir: cfg.LogDir, + Format: cfg.LogFormat, + LogToStd: true, + }) + if err != nil { + return err + } + defer shutdownLog() + slog.SetDefault(logger) + + if err := os.MkdirAll(cfg.LogDir+"/tools", 0o700); err != nil { + return err + } + + gin.SetMode(gin.ReleaseMode) + r := gin.New() + r.Use(gin.Recovery()) + + r.GET("/healthz", func(c *gin.Context) { + c.JSON(http.StatusOK, gin.H{"ok": true, "version": "0.0.0"}) + }) + + srv := &http.Server{ + Addr: cfg.ListenAddr, + Handler: r, + ReadHeaderTimeout: 10 * time.Second, + } + + errCh := make(chan error, 1) + go func() { + logger.Info("server.start", "addr", cfg.ListenAddr, "data_dir", cfg.DataDir) + if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { + errCh <- err + } + }() + + stop := make(chan os.Signal, 1) + signal.Notify(stop, os.Interrupt, syscall.SIGTERM) + + select { + case err := <-errCh: + return err + case sig := <-stop: + logger.Info("server.shutdown", "signal", sig.String()) + } + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + return srv.Shutdown(ctx) +} diff --git a/progress.md b/progress.md new file mode 100644 index 0000000..da9f33b --- /dev/null +++ b/progress.md @@ -0,0 +1,143 @@ +# Progress Log + +> 会话日志,记录本项目所有重要进展 + +--- + +## 2026-07-10 + +### Session 1: 初始化 + 架构评审 + 计划编写 + +**参与**: User + Claude (Linus persona) + +#### 进展 + +1. **仓库状态确认** (1dfef0f init) + - 单 init 提交,只有 GoLand 模板 main.go + - go.mod 已 pin 所有依赖(全是 // indirect) + - 没有任何业务代码、测试、文档 + +2. **创建 CLAUDE.md** + - 记录项目当前状态 + - 列出标准 Go 命令 + - 标注 4 个待确认决策 + +3. **用户提出初版架构方案** (v0.1) + - 5 个 Tool (spark_submit_cli / yarn_manage_job / fetch_spark_metrics / fetch_cluster_env / analyze_spark_log) + - Gin + MongoDB + MCP + - 3 个"底层元工具" (fetch_url / exec_shell / upload_file) + +4. **Linus 视角评审** (3 个核心问题) + - **存储选型矛盾**: 文档说 SQLite,go.mod pin MongoDB,.gitignore 还有 data/ → 三选一 + - **元工具归属不清**: 内部 Go / 暴露给 LLM / 外部 MCP Server,三种选择对应不同 API + - **Tool 粒度过细**: 5 个 Tool 里 3 个本质是 GET ;analyze_spark_log 混了"抓"和"分析" + - **示例代码 API 不存在**: server.NewStreamableHTTPServer 等在 mcp-go v0.56.0 不存在 + +5. **用户回答 4 个关键决策** (v0.1 → v0.2) + - 存储: **SQLite** (替换 MongoDB) + - Spark 能力: **纯 RM/SHS 读模式** (不提供 spark-submit) + - MCP 传输: **Streamable HTTP** + - Tool 表面: **list_clusters + yarn_job_manage + fetch_url + upload_file** (4 个 Tool,不暴露 exec_shell) + +6. **v0.2 计划编写** + - `task_plan.md` v0.2 — 7 个阶段,30+ 任务 + - `findings.md` — 调研发现、风险分析 + - `progress.md` (本文件) + +7. **范围变更** (v0.2 → v0.3) — 用户中途修正 + - **澄清**: `upload_file` 用途是 **spark-submit 时的 python 脚本上传** + - **冲突**: 与"纯 RM/SHS 不支持提交"决策矛盾 + - **解决**: 用户确认**加 spark_submit** (使用 spark-submit shell 命令直接提交,在 admin 后台配置 binary 路径) + - **`upload_file` 重新定义**: 从"通用 HTTP 上传"改为"本地文件存储" (LLM 提交 content,Server 写到 `./data/`,返回路径) + - **Cluster 配置新增**: 单一 `spark_submit_execute_bin` (v0.3.1 进一步合并 spark_submit_path + spark2_submit_path) + - **Tool 总数**: 4 → **5** (重新引入 `spark_submit`) + +8. **v0.3 计划更新** + - `task_plan.md` 重写,加入 `spark_submit` Tool 详细设计 + - `findings.md` 更新: 新增命令注入风险、v0.3 范围变更章节 + - 风险矩阵更新 + - 决策日志新增 4 条 v0.3 决策 + +9. **v0.3.1 进一步合并** — `spark_submit_path` + `spark2_submit_path` → `spark_submit_execute_bin` 单一字段 + +10. **v0.3.2 强化发现入口** — `list_clusters` 标为强制入口,Cluster 加 `is_active` + +11. **v0.3.3 加回高层 Tool** — `fetch_spark_metrics` + `fetch_cluster_env`(内部调 `fetch_url`) + +12. **v0.3.4 加回规则引擎** — `analyze_spark_log` + 启发式过滤 (data_skew / gc_pressure / bottleneck) + LLM-Ready Prompt。新增 `internal/analyzer/` 包。Tool 总数 5 → **8** + +13. **v0.3.5 大改**(用户给参考实现 spark_executor): + - 拆 `yarn_job_manage` → 4 个独立 RM Tool: `list_applications` / `get_application_status` / `get_application_logs` / `kill_application` + - 加鉴权层 (`none` / `basic` / `kerberos` 桩) + SSL 配置 + - 加 manual redirect following (RM→NM 跨主机 307 必须保留 Authorization) + - `analyze_spark_log` 日志改走 RM `get_application_logs` 降级链 + - Cluster struct 加 8 个鉴权/SSL/allowlist 字段 + - Tool 总数 8 → **11** + - Phase 5 时间 6-7h → 10-12h + +14. **v0.3.6 简化**(用户告知环境用 YARN SimpleAuth): + - 鉴权层去掉 kerberos 桩,改为 `none` / `simple` / `basic` + - `simple` 模式:`user.name` query param(默认 `"yarn"`) + - Hadoop 2.7 兼容性从风险表移除(用户已实测) + - Cluster struct AuthType 注释更新 + +15. **v0.3.7 用户接受所有 6 项建议**: + - Cluster 加 `default_submit_args` + `rate_limit_per_min` + - 删 `applications` 表(纯读不需要) + - `auth_password` PUT 留空 = 保留旧值 + - 加 `audit_log` 表 + 路由 + - 多 `AdminTokens` 支持(env 逗号分隔) + - analyzer 阈值改为 env 可配 + - 无任何未决项 + +16. **v0.3.8 日志基础设施** (用户新要求): + - 用 `log/slog` 替换默认 log + - 终端 text handler + 主文件 JSON handler (multi-handler fan-out) + - Per-Tool 独立日志文件: `data/logs/tools/{ts}_{tool}_{id}.log` + - 文件权限 0600 (含敏感 params) + - T1.5 新增 + T5.13 Tool 集成 + +#### 当前 Tool 表面 (v0.3.8,11 个) + +| # | Tool | 类型 | 备注 | +|---|---|---|---| +| 1 | `list_clusters` | **发现入口 (强制)** | 返回 cluster 全部元信息(含鉴权+allowlist) | +| 2 | `spark_submit` | 业务高层 | 本地 `exec.Command` | +| 3 | `list_applications` | **RM (v0.3.5 新)** | `GET /ws/v1/cluster/apps` | +| 4 | `get_application_status` | **RM (v0.3.5 新)** | `GET /ws/v1/cluster/apps/{id}` | +| 5 | `get_application_logs` | **RM (v0.3.5 新)** | 降级链:aggregated-logs → amContainerLogs → NM | +| 6 | `kill_application` | **RM (v0.3.5 新)** | `PUT /ws/v1/cluster/apps/{id}/state` | +| 7 | `fetch_spark_metrics` | 业务高层 SHS | 内部调 `fetch_url` | +| 8 | `fetch_cluster_env` | 业务高层 RM | 内部调 `fetch_url` | +| 9 | `analyze_spark_log` | 业务高层分析 | 规则引擎 + RM 日志降级 + LLM-Ready Prompt | +| 10 | `fetch_url` | HTTP 原语 | SSRF + 鉴权 + manual redirect | +| 11 | `upload_file` | 本地存储 | 路径白名单 | +| 4 | `fetch_url` | 通用 HTTP 原语 | +| 5 | `upload_file` | 本地文件存储 (v0.3 重新定义) | + +#### 下一步 + +- [ ] **用户审阅 v0.3 计划** +- [ ] 用户确认 / 修改后开始 **Phase 0** 执行 +- [ ] Phase 0 由 Codex 执行 (Go 依赖操作) +- [ ] 每个 Phase 结束: `go build` + `go test` 全绿才能进入下一 Phase + +#### 待用户在执行前澄清的项 (在 task_plan.md "待确认项" 也有) + +- `/mcp` 路由前缀是否带 `/*`? +- ADMIN_TOKEN 和 AGENT_TOKEN 是否合并? +- applications 表是否保留? +- 是否需要 Prometheus metrics? + +--- + +## 会话状态 + +- **当前阶段**: v0.3 计划已写完,等待用户审阅 +- **下一步行动**: 用户批准后 → Phase 0 (依赖调整) +- **阻塞项**: 4 个待确认项(可在 Phase 0-2 边做边问) +- **风险**: mcp-go v0.56.0 API 需在 Phase 6 实施前先查;spark_submit 引入的命令注入面需在 Phase 5.5 严格缓解 +- **关键设计约束** (v0.3 强调): + - `exec.Command(name, args...)` **绝不**走 shell + - `upload_file` **绝不**做远程 HTTP 上传 + - SSRF / 路径白名单 / token 鉴权 三道防线不变 diff --git a/task_plan.md b/task_plan.md new file mode 100644 index 0000000..c0c2028 --- /dev/null +++ b/task_plan.md @@ -0,0 +1,700 @@ +# Spark MCP Server 实现计划 + +> 创建于 2026-07-10 +> 当前状态: 仓库 init 完毕,等待开干 +> 范围: 7 个阶段,预计总工作量 12-15 小时 +> **v0.3 修订**: 加入 `spark_submit` Tool + `upload_file` 改为本地文件存储 + +--- + +## 目标 + +构建一个 **MCP (Model Context Protocol) Server for Apache Spark 运维**,作为统一入口让 LLM Agent 通过: +- YARN RM / SHS REST API **监控**任务 +- 本地 `spark-submit` / `spark2-submit` **提交**任务 +- 通用 HTTP fetch 查询任意 endpoint +- 本地文件存储上传脚本/JAR 给 spark-submit 引用 + +同时提供 Gin HTTP 管理后台配置集群。 + +**非目标** (本次不做): +- 集群部署 / 节点管理 +- Spark UI 渲染 (LLM 自己解析 JSON) +- 多租户 / OAuth / mTLS (留接口) + +--- + +## 已确认的架构决策 + +| 决策点 | 选择 | 备注 | +|---|---|---| +| 存储 | **SQLite** (替换 go.mod 里的 MongoDB) | 单一持久化层,消除 special case | +| Spark 能力 | **读 + 写 (本地提交)** | YARN RM/SHS 读,Server 本地 spark-submit 写 | +| MCP 传输 | **Streamable HTTP** (MCP 2025-03+ 标准) | mark3labs/mcp-go 原生支持 | +| Tool 表面 | **5 个 Tool** | 见下方 Tool 矩阵 | +| HTTP 框架 | **Gin** | Admin API + 挂载 MCP HTTP handler | +| SQLite 驱动 | **modernc.org/sqlite** | 纯 Go,无 CGO | +| spark-submit 路径 | **admin 后台按 cluster 配置** | 每个 cluster 配单一 `spark_submit_execute_bin` (v0.3.1 合并) | +| spark-submit 执行位置 | **Server 本地** (无 SSH) | Server 就是 gateway 机器 | + +--- + +## 架构总览 + +### Tool 矩阵 (v0.3 最终方案) + +| Tool | 类型 | 用途 | 底层 | +|---|---|---|---| +| `list_clusters` | **发现入口 (强制)** | 返回所有 cluster 的完整元信息 (**id, name, rm_url, shs_url, spark_submit_execute_bin, auth, ssl, is_active**)。LLM 调它是任何后续操作的**前提** | SQLite `clusters` 表 | +| `spark_submit` | 业务高层 | 用 cluster 配置的 binary 提交任务 | 本地 `exec.Command(binary, args...)` | +| `list_applications` | **业务高层 (RM)** | 列出 YARN 应用,支持 state/queue/limit 过滤 | RM `GET /ws/v1/cluster/apps` | +| `get_application_status` | **业务高层 (RM)** | 取单个 app 的 state + amContainerLogs 字段 | RM `GET /ws/v1/cluster/apps/{app_id}` | +| `get_application_logs` | **业务高层 (RM)** | 拉聚合 container 日志,404/501 降级到 amContainerLogs + NM | RM `GET /ws/v1/cluster/apps/{app_id}/aggregated-logs` + 3xx 跟随 | +| `kill_application` | **业务高层 (RM)** | 杀掉应用 | RM `PUT /ws/v1/cluster/apps/{app_id}/state` | +| `fetch_spark_metrics` | **业务高层 (SHS)** | 拿 SHS 应用结构化数据 (jobs/stages/executors) | SHS `/api/v1/applications/{app_id}/...` (内部 `fetch_url`) | +| `fetch_cluster_env` | **业务高层 (RM)** | 拿 Yarn 集群资源状态 (nodes/metrics) | RM `/ws/v1/cluster/nodes` (内部 `fetch_url`) | +| `analyze_spark_log` | **业务高层 (分析)** | 启发式检测 + 自动抓相关日志 (走 RM 链路) + LLM-Ready Prompt | 规则引擎 + `get_application_logs` (404/501 降级) | +| `fetch_url` | 通用 HTTP 原语 | LLM 拼任意 URL 抓数据 (SHS 端点都走这条) | 共享 HTTP 客户端 + SSRF + cluster allowlist | +| `upload_file` | 本地文件存储 | LLM 把脚本/JAR 存到 Server `./data/` 供 spark_submit 引用 | 本地文件系统 | + +**Tool 设计取舍**: +- 原 5 个 Tool 方案,现在仍 5 个,但**重组成 5 个真正不同的概念** +- `fetch_url` 是 LLM 唯一的"主动 HTTP 出口" — 业务 Tool 不重新发明轮子 +- `upload_file` **不是 HTTP 上传到远程**,而是**本地文件写入**(LLM 调它存脚本,Server 返回本地路径) +- `spark_submit` 用 `exec.Command(name, args...)` 切片形式传参,LLM 不接触 shell + +### 典型工作流 (LLM 视角) + +``` +1. list_clusters → 看到 cluster "prod" 配置 +2. upload_file → 把 Python 脚本存到 Server 的 ./data/scripts/etl.py +3. spark_submit → 提交任务,args: ["--master", "yarn", "--name", "etl", + "--deploy-mode", "cluster", + "./data/scripts/etl.py"] + → 返回 app_id = "application_12345_0001" +4. yarn_job_manage → 轮询 status,直到 FINISHED +5. fetch_url → 抓 SHS 的 /api/v1/applications/application_12345_0001/jobs + → 拿结构化 metrics +6. fetch_url (SHS logs) → 抓失败 executor 的 stderr +7. (LLM 自己做根因分析) +``` + +### 路由设计 + +``` +:8080 +├── /healthz (公开, 探活) +├── /admin/* (Gin, 需 ADMIN_TOKEN) +│ ├── GET /admin/clusters +│ ├── POST /admin/clusters +│ ├── PUT /admin/clusters/:id +│ └── DELETE /admin/clusters/:id +└── /mcp (Streamable HTTP, 需 AGENT_TOKEN) + └── POST /mcp +``` + +### 项目目录结构 + +``` +spark-mcp-go/ +├── main.go # 入口 +├── go.mod / go.sum +├── internal/ +│ ├── config/ # 环境变量加载 +│ │ └── config.go +│ ├── storage/ # SQLite 存储 +│ │ ├── sqlite.go # DB 连接、迁移 +│ │ └── cluster_repo.go # Cluster CRUD +│ ├── cluster/ # Cluster 业务 +│ │ ├── types.go # Cluster 结构体 +│ │ └── handler.go # /admin/clusters handlers +│ ├── httpclient/ # 共享 HTTP 客户端 (供 fetch_url / yarn_job_manage 用) +│ │ ├── client.go # 带 timeout / size limit +│ │ └── ssrf.go # SSRF 防护 +│ ├── tools/ # MCP Tool 实现 (与协议解耦) +│ │ ├── list_clusters.go +│ │ ├── yarn.go +│ │ ├── fetch_url.go +│ │ ├── upload_file.go # 本地文件存储 +│ │ ├── spark_submit.go +│ │ ├── fetch_spark_metrics.go # 高层 SHS 封装 +│ │ ├── fetch_cluster_env.go # 高层 RM 封装 +│ │ └── analyze_spark_log.go # 规则引擎驱动 +│ ├── analyzer/ # 日志/性能分析 (analyze_spark_log 内部) +│ │ ├── rules.go # 启发式规则定义 (data_skew / gc_pressure / bottleneck) +│ │ └── engine.go # 规则引擎:metrics → findings +│ ├── executor/ # 进程执行封装 +│ │ └── exec.go # 安全的 exec.Command 包装 +│ ├── mcp/ # MCP Server 装配 +│ │ ├── server.go +│ │ └── register.go # 注册 5 个 Tool +│ └── middleware/ +│ └── auth.go # Bearer token 鉴权 +``` + +--- + +## 阶段划分 + +### Phase 0: 清理与依赖调整 (0.5h) + +**目标**: 把 go.mod 调整到位,清空模板代码,建立目录骨架。 + +- [ ] **T0.1** 从 go.mod 移除 `go.mongodb.org/mongo-driver/v2` +- [ ] **T0.2** 添加 `modernc.org/sqlite` +- [ ] **T0.3** `go mod tidy` + `go build ./...` 通过 +- [ ] **T0.4** 删除 `main.go` GoLand 模板 +- [ ] **T0.5** 建立 `internal/{config,storage,cluster,httpclient,tools,executor,mcp,middleware}/` 目录,各放占位文件 +- [ ] **T0.6** 重新写 `main.go` 空壳 +- [ ] **验收**: `go build ./...` 仍能成功,目录结构建立 + +**Executor**: Codex + +--- + +### Phase 1: 配置层 (0.5h) + +**目标**: 从环境变量加载配置。 + +- [ ] **T1.1** `internal/config/config.go` 定义 `Config` (v0.3.7 + v0.3.8 加日志配置): + - 字段: `ListenAddr` (`:8080`), `DataDir` (`./data`), `SQLitePath` (派生), **`AdminTokens []string`** (v0.3.7,逗号分隔 env,必填 ≥1), `AgentToken` (必填), `HTTPClientTimeout` (`30s`), `MaxResponseBytes` (`1MB`), `SparkSubmitTimeout` (`60s`) + - **v0.3.8 日志配置**: + - `LogDir` (`./data/logs`) + - `LogLevel` (`info`,可选 `debug` / `warn` / `error`) + - `LogFormat` (`text` 终端 / `json` 主文件) + - **v0.3.7 analyzer 阈值改为 env**: + - `ANALYZER_DATA_SKEW_RATIO` (`3.0`) + - `ANALYZER_GC_PRESSURE_RATIO` (`0.1`) + - `ANALYZER_BOTTLENECK_SHUFFLE_GB` (`50`) +- [ ] **T1.2** `main.go` 调用 `config.Load()`,失败即退出 +- [ ] **T1.3** 启动时打印配置(隐藏 token) +- [ ] **T1.4** 启动时 `os.MkdirAll(cfg.DataDir, 0755)` 确保 data 目录存在 +- [ ] **T1.5** **Logging 基础设施** (v0.3.8 新,`internal/logging/` 包): + - 使用 `log/slog` (Go 1.21+ stdlib) + - **双输出 multi-handler**: + - 终端: `slog.NewTextHandler(os.Stdout, ...)`(可读,人类阅读) + - 主文件: `slog.NewJSONHandler(file, ...)`(结构化,机器解析) + - 自实现 `MultiHandler` fan-out 到两个 handler + - **主日志文件**: `data/logs/spark-mcp.log` (JSON,所有事件,rotate 由运维负责) + - **Per-Tool 日志文件** (核心需求): `data/logs/tools/{ts}_{tool}_{short_id}.log` + - 每 Tool 调用 **独立**文件 + - 头部: tool 名 + 参数 (JSON) + 开始时间 + - 尾部: 结果 (JSON,截断) + 耗时 + 错误 + - 文件权限 `0600` (owner only,含敏感参数) + - **API**: + ```go + package logging + + func New(cfg LogConfig) (*slog.Logger, func(), error) + func StartToolCall(ctx context.Context, base *slog.Logger, toolName string, params any) (ToolCallLogger, error) + + type ToolCallLogger interface { + WithResult(result any) + WithError(err error) + End() + } + ``` + - 启动时 `os.MkdirAll(LogDir/tools, 0700)` 确保目录存在(权限收紧) +- [ ] **验收**: `go build` 通过,启动时输出配置,data 目录自动创建,**终端看 text 日志 + 主文件 JSON 日志 + 调一个 Tool 后看 per-tool 文件存在** + +**Executor**: Codex + +--- + +### Phase 2: 存储层 (SQLite) (1.5h) + +**目标**: SQLite 存储层,支持 clusters 表的 CRUD。**Cluster 表新增 spark-submit 路径字段**。 + +- [ ] **T2.1** `internal/storage/sqlite.go` + - `Open(path string) (*sql.DB, error)` + - 自动应用 schema migration + - `clusters` 表 (v0.3 + v0.3.2 + v0.3.5 + v0.3.6 + v0.3.7): + ```sql + CREATE TABLE clusters ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + rm_url TEXT NOT NULL, + shs_url TEXT NOT NULL, + spark_submit_execute_bin TEXT NOT NULL, + is_active INTEGER NOT NULL DEFAULT 1, + -- v0.3.6 鉴权 (去掉 kerberos,用户环境用 simple) + auth_type TEXT NOT NULL DEFAULT 'none', -- 'none' | 'simple' | 'basic' + auth_username TEXT, -- simple (user.name) / basic + auth_password_enc BLOB, -- basic,加密存储 (AES-GCM);PUT 留空=保留 + -- v0.3.5 SSL + ssl_verify INTEGER NOT NULL DEFAULT 1, + ssl_ca_bundle TEXT, + -- v0.3.5 URL allowlist (SHS fetch_url 用) + url_allowlist TEXT, -- JSON 数组,glob 模式 + -- v0.3.7 新增 + default_submit_args TEXT, -- JSON 数组,spark_submit 时自动 prepend + rate_limit_per_min INTEGER NOT NULL DEFAULT 10, -- 0=不限,默认 10 次/分钟 + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL + ); + ``` + - `audit_log` 表 (v0.3.7 新增,所有 admin 写操作留痕): + ```sql + CREATE TABLE audit_log ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + timestamp INTEGER NOT NULL, + actor TEXT NOT NULL, -- "admin:"(多 token 支持) + action TEXT NOT NULL, -- "cluster.create" | "cluster.update" | "cluster.delete" + cluster_id TEXT, -- NULL 表示不针对 cluster + details TEXT -- JSON,操作前后关键字段 diff + ); + CREATE INDEX idx_audit_ts ON audit_log(timestamp DESC); + ``` +- [ ] **T2.2** `internal/cluster/types.go`: + ```go + type Cluster struct { + ID string `json:"id"` + Name string `json:"name"` + RMURL string `json:"rm_url"` + SHSURL string `json:"shs_url"` + SparkSubmitExecuteBin string `json:"spark_submit_execute_bin"` + IsActive bool `json:"is_active"` + // v0.3.5 鉴权 (v0.3.6 简化:删 kerberos 字段) + AuthType string `json:"auth_type"` // "none" | "simple" | "basic" + AuthUsername string `json:"auth_username,omitempty"` // simple (user.name) / basic + AuthPassword string `json:"-"` // basic only, 永不出网 + // (Kerberos 字段 v2 才有: AuthPrincipal / AuthKeytabPath) + // v0.3.5 SSL + SSLVerify bool `json:"ssl_verify"` + SSLCABundle string `json:"ssl_ca_bundle,omitempty"` + // v0.3.5 URL allowlist + URLAllowlist []string `json:"url_allowlist,omitempty"` + // v0.3.7 新增 + DefaultSubmitArgs []string `json:"default_submit_args,omitempty"` // spark_submit 自动 prepend + RateLimitPerMin int `json:"rate_limit_per_min"` // 0=不限,默认 10 + CreatedAt int64 `json:"created_at"` + UpdatedAt int64 `json:"updated_at"` + } + ``` + - **LLM 通过 `list_clusters` 拿到完整字段** — RM/SHS endpoint + 鉴权 + allowlist + 默认 args 都是发现内容 + - `AuthPassword` 字段 JSON tag 是 `"-"` 永不出网,DB 里存加密 blob + - **PUT 留空 = 保留旧密码**(避免 admin 端不知道现有密码时清空) + ``` +- [ ] **T2.3** `internal/storage/cluster_repo.go`: 完整 CRUD + `ErrNotFound` 哨兵 +- [ ] **T2.4** 单元测试 `cluster_repo_test.go` 用 `:memory:` DB 测全路径 +- [ ] **验收**: `go test ./internal/storage/...` 全绿 + +**Executor**: Codex + +--- + +### Phase 3: 共享 HTTP 客户端 + SSRF 防护 (1.5h) + +**目标**: 一个共享的 HTTP 客户端,供 `fetch_url` 和 `yarn_job_manage` 用。 + +- [ ] **T3.1** `internal/httpclient/client.go` + - 基于 `net/http` 包装 + - Timeout (默认 30s), MaxResponseBytes (默认 1MB) + - `Do(ctx, req) (*http.Response, error)`,body 已被 truncate +- [ ] **T3.2** `internal/httpclient/ssrf.go` + - `CheckURL(urlStr, allowedHosts) error` + - 私网 IP 黑名单 (IPv4 + IPv6) + - DNS rebinding 防护: 解析时锁定 IP,后续 dial 用 IP + - **allowlist 机制**: cluster URL 的 host 自动加入 allowlist (解决 fetch_url 抓 SHS/RM 的需求) +- [ ] **T3.3** 单元测试: 私网拦截、allowlist 豁免、scheme 校验、超大 body 截断 +- [ ] **T3.4** **手动 redirect following** (v0.3.5 新): + - Go `net/http` 默认跨主机 redirect 会丢 Authorization header + - 自实现 `CheckRedirect` 把原始 Authorization 重新塞到 redirect 后的 request + - 最多 3 次 redirect 后停止 + - **关键场景**: `get_application_logs` 降级路径中,RM→NM 跨主机 307 redirect 必须保留 auth +- [ ] **T3.5** **鉴权层** (v0.3.5 新, v0.3.6 简化): + - `internal/httpclient/auth.go` 定义 `AuthConfig` 接口 + - `none`: 不加任何 header + - `simple`: YARN SimpleAuth(用户环境采用) — 加 `user.name` query param(默认 `"yarn"`,可配) + - `basic`: `req.SetBasicAuth(user, pass)` + - ~~`kerberos`~~: **v1 不实现**(用户环境不需要,集成 gokrb5 是 v2) + - `applyAuth(req *http.Request) error` 统一入口 +- [ ] **T3.6** **SSL 配置** (v0.3.5 新): + - `internal/httpclient/ssl.go` 构造 `*http.Transport` + - `SSLVerify=false` → `InsecureSkipVerify=true` (DANGER,仅测试) + - `SSLCABundle` 非空 → 加载 CA cert 到 `RootCAs` +- [ ] **验收**: `go test ./internal/httpclient/...` 全绿 + +**Executor**: Codex + +--- + +### Phase 4: Cluster 管理 (Admin API) (2h) + +**目标**: `/admin/clusters` 完整 CRUD,**新增 spark-submit 路径字段校验**。 + +- [ ] **T4.1** `internal/middleware/auth.go` (v0.3.7 多 token) + - `RequireAnyToken(tokens []string) gin.HandlerFunc` — 任一 token 匹配即通过,否则 401 + - 记录使用的 token 标识 (hash 前 8 位) 到 `c.Set("admin_token_id", ...)` 供 audit_log 用 +- [ ] **T4.2** `internal/cluster/handler.go` + - `GET/POST/PUT/DELETE /admin/clusters[/:id]` + - 校验: + - `rm_url` / `shs_url` 必须 `http://` 或 `https://` 开头 + - `spark_submit_execute_bin` 必须**绝对路径**(`filepath.IsAbs`) + - 文件**存在**(`os.Stat` 通过) + - 文件**可执行**(`mode&0111 != 0`) + - `id` 非空 + - 校验失败 → 400 + 错误详情 +- [ ] **T4.3** `main.go` 注册 admin route group + audit 拦截器 +- [ ] **T4.4** **audit log 中间件** (v0.3.7): 任何 POST/PUT/DELETE 写一条 `audit_log`,包含 actor + action + cluster_id + details +- [ ] **T4.5** **速率限制中间件** (v0.3.7): per-cluster,基于 `golang.org/x/time/rate`,`spark_submit` Tool 调用前检查 cluster.RateLimitPerMin +- [ ] **T4.6** 集成测试: 完整 CRUD + 鉴权 + 路径校验失败 + audit_log 写入 + 速率限制触发 +- [ ] **验收**: + - 缺 `Authorization` → 401 + - `spark_submit_execute_bin` 是不存在的文件 → 400 + - 正常 cluster 配置 → 200 + - admin 操作后 `audit_log` 表有对应记录 + - `spark_submit` 超过 10/min → 429 + +**Executor**: Codex + +--- + +### Phase 5: MCP Tools 实现 (10-12h, v0.3.5 加 4 个 RM Tool + 鉴权层) + +**目标**: 11 个 Tool 的业务逻辑实现,**与 MCP 协议解耦**。共享鉴权层(由 T3.5 实现)。 + +#### T5.1 `list_clusters` + +- [ ] `internal/tools/list_clusters.go` +- [ ] 签名: `ListClusters(ctx, repo) ([]Cluster, error)` +- [ ] 直接调 `repo.List` + +#### T5.2 `list_applications` (RM, v0.3.5 新) ⭐ + +- [ ] `internal/tools/list_applications.go` +- [ ] 端点: `GET {rm_url}/ws/v1/cluster/apps` +- [ ] Params: + ```go + type ListApplicationsParams struct { + ClusterID string `json:"cluster_id"` + State string `json:"state,omitempty"` // "RUNNING" | "FINISHED" | "FAILED" | "KILLED" | "ACCEPTED" 等 + Queue string `json:"queue,omitempty"` + Limit int `json:"limit,omitempty"` // 默认 100 + } + ``` +- [ ] 流程: 查 cluster → 应用鉴权 → 拼 URL with query string → 调 `doAuthenticatedRequest` +- [ ] 单元测试: 各种 state / queue / limit 组合;cluster inactive → 错误 + +#### T5.3 `get_application_status` (RM, v0.3.5 新) ⭐ + +- [ ] `internal/tools/get_application_status.go` +- [ ] 端点: `GET {rm_url}/ws/v1/cluster/apps/{app_id}` +- [ ] Params: `ClusterID`, `AppID` +- [ ] 返回: 完整 app JSON(含 `state`, `finalStatus`, **`amContainerLogs` URL 字段**) +- [ ] 关键: `amContainerLogs` 字段是 T5.4 降级路径的入口 + +#### T5.4 `get_application_logs` (RM, v0.3.5 新,核心) ⭐ + +- [ ] `internal/tools/get_application_logs.go` +- [ ] **降级链** (用户参考实现关键,3 步): + 1. **首选**: `GET {rm_url}/ws/v1/cluster/apps/{app_id}/aggregated-logs` + 2. **降级 1**: 如果返回 404/501,改调 `get_application_status` 拿 `amContainerLogs` 字段 + 3. **降级 2**: `amContainerLogs` 指向 NM (NodeManager),**手动跟随 3xx Location**(最多 3 次) + 4. **降级 3**: 最终拿到 AM driver 的 stdout +- [ ] Params: + ```go + type GetApplicationLogsParams struct { + ClusterID string `json:"cluster_id"` + AppID string `json:"app_id"` + TailLines int `json:"tail_lines,omitempty"` // 默认 2000 + Filter string `json:"filter,omitempty"` // "ERROR" | "WARN" | "Exception" | "all"(默认) + } + ``` +- [ ] **关键鉴权**: 手动跟随 redirect 时必须保留 Authorization header。Go `net/http` 默认跨主机 redirect 会丢 header,需自实现 `CheckRedirect` 把 `Authorization` 重新塞到 redirect 后的 request 上 + +#### T5.5 `kill_application` (RM, v0.3.5 新) + +- [ ] `internal/tools/kill_application.go` +- [ ] 端点: `PUT {rm_url}/ws/v1/cluster/apps/{app_id}/state` +- [ ] Body: `{"state": "KILLED"}` +- [ ] Params: `ClusterID`, `AppID` +- [ ] 单元测试: 用 httptest mock RM,验证 PUT 请求体和 URL 正确 + +#### T5.6 `fetch_url` (SSRF + 鉴权,关键路径) + +- [ ] `internal/tools/fetch_url.go` +- [ ] Params: `URL` (必填), `Method` (默认 GET), `Headers` (可选), `MaxBytes` (可选) +- [ ] `allowed_hosts` 由调用方(LLM via `list_clusters` 取到的 host)注入 +- [ ] 流程: SSRF 检查 → 调 client → 截断 body → 返回 `{status, headers, body, truncated}` + +#### T5.7 `upload_file` (本地文件存储) + +- [ ] `internal/tools/upload_file.go` +- [ ] Params: + ```go + type UploadFileParams struct { + Path string `json:"path"` // 相对 data dir,如 "scripts/main.py" + Content string `json:"content"` // 文本或 base64 + Encoding string `json:"encoding,omitempty"` // "utf-8" (默认) | "base64" + } + ``` +- [ ] 流程: + 1. 路径白名单校验: `filepath.Clean(params.Path)` 不能 `..` 跳出 `data_dir` + 2. 创建父目录 `os.MkdirAll` + 3. 解码 content (utf-8 直接写,base64 先 decode) + 4. 写文件 (mode 0644) + 5. 返回 `{path: <绝对路径>, size}` +- [ ] 单元测试: 路径跳出 (`../etc/passwd`) 应被拒;二进制 base64 解码正确;大文件 (>1MB) 写成功 + +#### T5.8 `spark_submit` (核心) ⭐ + +- [ ] `internal/executor/exec.go` + - 封装 `exec.CommandContext` + 输出捕获 + - 超时控制、退出码、stdout/stderr 截断(默认 10MB 截断防内存爆) + - 永不调用 `bash -c` / `sh -c` +- [ ] `internal/tools/spark_submit.go` +- [ ] Params (v0.3.1 简化: 移除 Submitter): + ```go + type SparkSubmitParams struct { + ClusterID string `json:"cluster_id"` + Args []string `json:"args"` // binary 参数列表 + TimeoutSeconds int `json:"timeout_seconds,omitempty"` // 默认 60 + } + ``` +- [ ] 流程: + 1. 查 cluster → 校验存在 + IsActive + 2. 取 `cluster.SparkSubmitExecuteBin` 作为 binary (cluster 配置决定,LLM 不能改) + 3. 校验 path 存在 + 可执行 (T4.2 已做,但 runtime 再 check 一次) + 4. **关键**: `exec.CommandContext(ctx, binary, params.Args...)` — **切片传参,绝不走 shell** + 5. 捕获 stdout/stderr + 6. 解析 `app_id` — YARN 提交成功输出 "Submitted application application_xxx_yyy",用 regex `application_\d+_\d+` 提取 + 7. 返回: + ```go + type SparkSubmitResult struct { + Command string `json:"command"` // 完整命令 (for audit,不含二进制路径) + Stdout string `json:"stdout"` // 截断 + Stderr string `json:"stderr"` + ExitCode int `json:"exit_code"` + AppID string `json:"app_id,omitempty"` + } + ``` +- [ ] 单元测试: + - 用 fake binary (Go test helper) 验证 args 切片正确传递 + - 超时场景 + - app_id 解析 regex + - 失败: cluster 不存在、cluster inactive、binary 不存在、binary 不可执行 + +#### T5.9 `fetch_spark_metrics` (高层 SHS 封装) ⭐ + +- [ ] `internal/tools/fetch_spark_metrics.go` +- [ ] **本质**: 内部调用 `fetch_url`,把 SHS URL 拼好 +- [ ] Params: + ```go + type FetchSparkMetricsParams struct { + ClusterID string `json:"cluster_id"` + AppID string `json:"app_id"` + Path string `json:"path,omitempty"` // 默认 "jobs",可选 "stages"/"executors"/"environment" + } + ``` +- [ ] 流程: + 1. 查 cluster → 校验存在 + IsActive,拿 `shs_url` + 2. 拼 URL: `{shs_url}/api/v1/applications/{app_id}/{path}` + 3. **内部**调用 `FetchURL` (T5.3 实现) — 不重新走 SSRF (cluster URL 已经在 allowlist) + 4. 返回 `{url, status, body, truncated}` +- [ ] 单元测试: + - 用 httptest mock SHS,验证 URL 拼接正确 + - cluster 不存在 → 错误 + - cluster inactive → 错误 + - 默认 path = "jobs" + +#### T5.10 `fetch_cluster_env` (高层 RM 封装) ⭐ + +- [ ] `internal/tools/fetch_cluster_env.go` +- [ ] **本质**: 内部调用 `fetch_url`,把 RM URL 拼好 +- [ ] Params: + ```go + type FetchClusterEnvParams struct { + ClusterID string `json:"cluster_id"` + // 可选: "nodes" (默认) | "metrics" | "apps" + } + ``` +- [ ] 流程: + 1. 查 cluster → 校验存在 + IsActive,拿 `rm_url` + 2. 拼 URL: `{rm_url}/ws/v1/cluster/{section}` (默认 `nodes`) + 3. **内部**调用 `FetchURL` (T5.3 实现) + 4. 返回 `{url, status, body, truncated}` +- [ ] 单元测试: + - 用 httptest mock RM,验证 URL 拼接正确 + - 默认 section = "nodes" + - cluster 不存在 / inactive → 错误 + +#### T5.11 `analyze_spark_log` (规则引擎 + RM 日志降级) ⭐ + +- [ ] `internal/analyzer/rules.go` + `engine.go` +- [ ] `internal/tools/analyze_spark_log.go` +- [ ] **指标部分**: 内部调 `fetch_spark_metrics` (T5.9) 拿 stages / executors / jobs +- [ ] **规则部分**: `data_skew` / `gc_pressure` / `bottleneck` (硬编码阈值,接口预留可配) +- [ ] **日志部分** (v0.3.5 修正): **不**走 SHS logs,改调 `get_application_logs` (T5.4) 走 RM 降级链 + - 仅当 findings 非空时拉取,避免无谓请求 +- [ ] **Prompt 部分**: 拼 LLM-Ready 文本 +- [ ] 单元测试: 各种 findings 场景 × 降级链组合 + +#### T5.12 综合单元测试 + +- [ ] 每个 Tool 至少 1 正向 + 1 边界用例 +- [ ] 鉴权层单测 (none / basic / kerberos stub) +- [ ] 降级链单测 (aggregated-logs 404 → amContainerLogs → NM) +- [ ] **per-tool 日志单测** (v0.3.8): mock Tool 调用,验证 per-tool 文件生成 + 内容包含 params/result +- [ ] `go test ./internal/tools/...` 全绿 + +#### T5.13 每个 Tool 集成 `logging.StartToolCall` (v0.3.8) ⭐ + +- [ ] 11 个 Tool 全部接入 +- [ ] Tool 函数入口: + ```go + callLog, _ := logging.StartToolCall(ctx, logger, "list_clusters", params) + defer callLog.End() + ``` +- [ ] 成功路径: `callLog.WithResult(result)` +- [ ] 失败路径: `callLog.WithError(err)` +- [ ] **per-tool 文件**记录原始 params,便于 LLM 行为调试 + +**Executor**: Codex + +--- + +### Phase 6: MCP Server 装配 (2h) + +**目标**: 把 5 个 Tool 注册到 mcp-go Server,挂上 Streamable HTTP handler。 + +- [ ] **T6.1** **查 mark3labs/mcp-go v0.56.0 真实 API** + - `server.NewMCPServer(name, version, opts...)` + - `mcp.NewTool(name, opts...)` + - `mcp.WithString` / `mcp.WithNumber` / `mcp.WithArray` / `mcp.WithObject` + - `server.NewStreamableHTTPServer(s, opts...)` + - **先查源码再写** +- [ ] **T6.2** `internal/mcp/server.go`: `NewServer() *server.MCPServer` +- [ ] **T6.3** `internal/mcp/register.go`: `RegisterTools(s, deps)` + - 注册 5 个 Tool,handler 调 Phase 5 业务函数 + - Tool input schema 严格声明 (LLM 调错就报错) +- [ ] **T6.4** `internal/mcp/streamable.go`: Streamable HTTP handler + - `main.go` 用 `gin.WrapH` 挂到 `/mcp`,加 auth 中间件 +- [ ] **T6.5** 集成测试: `httptest` + JSON-RPC 构造 `tools/call` 请求 +- [ ] **验收**: + - `go build ./...` 通过 + - `curl -X POST http://localhost:8080/mcp -d '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"list_clusters",...}}'` 返回 MCP 响应 + +**Executor**: Codex + +--- + +### Phase 7: 端到端验证 (1h) + +**目标**: 启动 server,跑通完整流程。 + +- [ ] **T7.1** `go build ./...` + `go vet ./...` 无 warning +- [ ] **T7.2** 启动 server,确认 `/healthz` 200 +- [ ] **T7.3** curl 跑 admin CRUD(创建/列出/更新/删除 cluster) +- [ ] **T7.4** 用 curl 发 MCP `tools/call`: + - `list_clusters` → 返回 cluster 列表 + - `upload_file` 上传 `main.py` → 返回路径 + - `spark_submit` (用 fake spark-submit binary,test 模式) → 返回 exit_code + (mock) app_id + - `yarn_job_manage` 用 mock rm_url → 验证 status / kill 路径 + - `fetch_url` 抓公网 URL → 200 + - `fetch_url` 抓 127.0.0.1 → 被 SSRF 拦截 + - `upload_file` path = `"../etc/passwd"` → 路径校验拦截 +- [ ] **T7.5** `go test ./...` 全绿 +- [ ] **验收**: 全部通过 + +**Executor**: CC 验证 + +--- + +## 风险与依赖 + +### 外部依赖 + +- **mark3labs/mcp-go v0.56.0 API 稳定性**: Phase 6.1 需要查真实 API +- **modernc.org/sqlite**: 纯 Go,无外部依赖 + +### 数据结构风险 + +- `applications` 表当前只为后续阶段预留,如果不需要缓存可删除 +- Cluster 配置变更频率极低,SQLite 完全够用 + +### 安全风险 (必须处理) + +| 风险 | 严重度 | 缓解 | +|---|---|---| +| **命令注入 (spark_submit)** | **高** | **`exec.Command(name, args...)` 切片形式,绝不走 shell。binary 由 cluster 配置固定 (v0.3.1),LLM 不能在 Tool 调用中改 binary 路径。LLM 提供的 args 直接作为 binary 的 argv,无字符串拼接** | +| SSRF (fetch_url) | 高 | 私网 IP 黑名单 + cluster URL allowlist + DNS rebinding 锁定 | +| 任意文件读/写 (upload_file) | 中 | 路径白名单 `./data/`,拒绝 `..`、绝对路径、符号链接逃逸 | +| Token 泄露 | 中 | 必须从 env 读,启动日志隐藏 token 字段 | +| 内存爆炸 (大响应/大日志) | 中 | HTTP body 1MB 截断,spark-submit stdout/stderr 10MB 截断,analyze_spark_log 日志 tail 2000 行 | +| 慢请求 DoS | 中 | HTTP client 30s timeout,spark-submit 默认 60s timeout,analyze_spark_log 并行 3 个 SHS fetch (受同一 timeout 约束) | +| spark-submit binary 被替换 | 低 | 启动时 + runtime 都校验 binary 存在 + 可执行 + 绝对路径 | +| 集群配置被改 | 低 | audit log 记录 cluster CRUD 操作 (可选,本阶段不强制) | +| **启发式规则误报** (analyze_spark_log) | 中 | 中 | 规则是 first-pass filter,不是真理。LLM 拿到 findings 后仍需自己判断。Prompt 明确说"请基于以上信息生成根因分析报告" | +| **SHS API 字段变动** (analyze_spark_log) | 中 | 中 | metrics JSON 解析失败时,跳过对应规则不出 finding,不让一个 Tool 崩整个调用 | +| **Hadoop 2.7 兼容性** (v0.3.5 → v0.3.6 已验证) | 低 | 低 | **用户已验证 endpoint 在 2.7 集群上工作**。RM REST API 4 个端点 + SHS 端点都测过 | +| **manual redirect 误用** (v0.3.5) | 中 | 高 | 自实现 `CheckRedirect` 必须严格保留 Authorization,否则 RM→NM 降级会 401/403。代码 review 必查 | + +--- + +## 决策日志 + +| 日期 | 决策 | 理由 | +|---|---|---| +| 2026-07-10 | 存储用 SQLite 而非 MongoDB | 单持久化层,消除 special case | +| 2026-07-10 | 暴露 `fetch_url` (通用 HTTP 原语) | LLM 需要通用 HTTP 能力 | +| 2026-07-10 | MCP 传输用 Streamable HTTP | 现代 MCP 标准 | +| 2026-07-10 | HTTP 客户端独立包,所有 Tool 复用 | SSRF 防护集中 | +| 2026-07-10 | SQLite 驱动选 `modernc.org/sqlite` | 纯 Go,无 CGO | +| 2026-07-10 (v0.2→v0.3) | **加 `spark_submit` Tool** | 用户从"纯读"改为"读+本地提交",业务上需要 | +| 2026-07-10 (v0.2→v0.3) | **`upload_file` 改为本地文件存储** (非 HTTP 上传) | 用途明确为 spark-submit 脚本上传,本地存储更简单安全 | +| 2026-07-10 (v0.2→v0.3) | **spark-submit 在 Server 本地执行** (无 SSH) | 用户明示"使用 spark-submit shell命令直接提交" | +| 2026-07-10 (v0.2→v0.3) | **Cluster 配置加单一 `spark_submit_execute_bin`** (v0.3.1 合并 spark_submit_path + spark2_submit_path) | 用户明示两个 binary 都要可配;后简化合并为单字段,LLM 不必选 binary | +| 2026-07-10 (v0.2→v0.3) | **`spark_submit` 用 `exec.Command` 切片传参** | 绝不走 shell,消除命令注入面 | +| 2026-07-10 (v0.3.1→v0.3.2) | **`list_clusters` 标为强制发现入口,Cluster 加 `is_active` 字段** | LLM 必须能拿到 rm_url/shs_url/spark_submit_execute_bin 才能用其他 Tool。`is_active` 允许禁用 cluster 而不删除 | +| 2026-07-10 (v0.3.2→v0.3.3) | **加回 `fetch_spark_metrics` + `fetch_cluster_env` 作为高层 Tool,内部调用 `fetch_url`** | 业务常用端点封装,LLM 不必拼 SHS/RM URL,但仍保留 `fetch_url` 原语处理长尾查询 | +| 2026-07-10 (v0.3.3→v0.3.4) | **加回 `analyze_spark_log` + 服务端启发式过滤 (data_skew / gc_pressure / bottleneck) + LLM-Ready Prompt** | 用户原 spec 明确要求 [Log Analyzer + Regex Rules + LLM-Ready Prompt] 组件。规则阈值硬编码,接口预留可配 | +| 2026-07-10 (v0.3.4→v0.3.5) | **拆 `yarn_job_manage` 成 4 个 RM Tool:list_applications / get_application_status / get_application_logs / kill_application** | 用户给出参考实现(spark_executor 4 个独立 Python 入口),MCP 端按 1 Tool = 1 端点拆,更符合 MCP 习惯 | +| 2026-07-10 (v0.3.4→v0.3.5) | **加鉴权层 (none / basic / kerberos) + SSL 配置 + manual redirect following** | 集群环境 Spark on YARN + Hadoop 2.7,生产环境需要 SPNEGO。`get_application_logs` RM→NM 降级路径必须保留 Authorization 跨主机 redirect | +| 2026-07-10 (v0.3.4→v0.3.5) | **`analyze_spark_log` 日志部分改走 RM `get_application_logs` 降级链** (不是 SHS) | SHS logs 端点不稳定,RM aggregated-logs + amContainerLogs + NM 是生产链路 | +| 2026-07-10 (v0.3.5→v0.3.6) | **鉴权层简化:`AuthType` 改为 `none / simple / basic`**,Kerberos 推到 v2 | 用户告知集群用 YARN SimpleAuth(`user.name` query param),不需 SPNEGO。v1 简化够用,生产环境部署可上 | +| 2026-07-10 (v0.3.5→v0.3.6) | **Hadoop 2.7 兼容性"已验证"**,从风险表移除 | 用户实测 4 个 RM 端点 + SHS 端点都通,无需 Phase 7 集群验证 | +| 2026-07-10 (v0.3.6→v0.3.7) | **加 per-cluster `default_submit_args` + `rate_limit_per_min`** | 减少 LLM 拼 args 出错,防误调刷爆集群 | +| 2026-07-10 (v0.3.6→v0.3.7) | **删 `applications` 表** | 纯 RM/SHS 读模式不缓存 app 状态 | +| 2026-07-10 (v0.3.6→v0.3.7) | **`auth_password` PUT 留空 = 保留** | admin 端不需要知道现有密码 | +| 2026-07-10 (v0.3.6→v0.3.7) | **加 `audit_log` 表 + 路由** | 留痕 cluster CRUD 操作 | +| 2026-07-10 (v0.3.6→v0.3.7) | **多 `AdminTokens` 支持** (env 逗号分隔) | 多人共用一个 token 不安全,改用多 token 配任意一个通过 | +| 2026-07-10 (v0.3.6→v0.3.7) | **analyzer 阈值改为 env 可配** (`ANALYZER_*`) | 不同集群阈值可能不同,硬编码不灵活 | +| 2026-07-10 (v0.3.7→v0.3.8) | **用 `log/slog` 替换默认 log** (Go 1.21+ stdlib) | 用户要求结构化日志 | +| 2026-07-10 (v0.3.7→v0.3.8) | **双输出:终端 text + 主文件 JSON** | 用户要求终端可读 + 文件持久化 | +| 2026-07-10 (v0.3.7→v0.3.8) | **Per-Tool 独立日志文件** (`data/logs/tools/{ts}_{tool}_{id}.log`) | 用户要求每次工具调用单独记录 tool 名 + 参数 + 结果 + 耗时,便于调试 LLM 行为 | + +--- + +## 待确认项 (执行中可继续澄清) + +(已清空 — v0.3.7 用户接受所有 6 项建议) + +- [ ] **`/mcp` 路由前缀**: `/mcp` 还是 `/mcp/*`? +- [ ] **Token 配置**: `ADMIN_TOKEN` 和 `AGENT_TOKEN` 是否必须分两个?还是合并? +- [ ] **applications 表** (v0.3.7 已删): 纯 RM/SHS 读模式不需要缓存 app 状态 +- [ ] **spark-submit 路径校验强度**: T4.2 已校验存在+可执行,是否还要校验 owner / checksum 防篡改?(本阶段不做) +- [ ] **app_id 解析 regex**: 不同 YARN 版本输出格式可能不同,需保留 regex 可配置 +- [ ] **Metrics / Tracing**: 是否需要集成 Prometheus?当前不包含 + +--- + +## 执行约定 + +- **每个 Phase 结束**: `go build ./...` + `go test ./...` 全绿才能进入下一 Phase +- **代码生成**: 全部走 Codex MCP,使用 `model: "kimi/kimi-k2.7-code"`, `sandbox: "danger-full-access"`, `approval-policy: "on-failure"` +- **API 变更**: 任何 import 路径或公开签名变更,先在 task_plan.md 标注再改 +- **测试覆盖**: 每个 Tool 至少 1 正向 + 1 边界;auth 中间件必须有 401 测试;`exec.Command` 必须切片传参(代码 review 必查) + +--- + +## 进度 + +- [x] Phase 0 - 清理与依赖调整 +- [ ] Phase 1 - 配置层 +- [ ] Phase 2 - 存储层 +- [ ] Phase 3 - 共享 HTTP 客户端 + SSRF 防护 +- [ ] Phase 4 - Cluster 管理 (Admin API) +- [ ] Phase 5 - MCP Tools 实现 (5 个 Tool) +- [ ] Phase 6 - MCP Server 装配 +- [ ] Phase 7 - 端到端验证