tao.chen/spark-mcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity
32 Commits 1 Branch 2 Tags
4af166ea48f7b746c9eb6783ac398ade3cfdeea1
Commit Graph
7 Commits
Author SHA1 Message Date
tao.chenandClaude 157848d87a main: bound startup sweep with context timeout 
Sweep() used to run synchronously at startup without any time bound.
On a large uploads directory this can block the server for 30s or more,
keeping /healthz returning 503 while the reaper works. Add a context
timeout so startup continues if the sweep cannot finish within budget.

The timeout is set to 5 seconds: long enough to clear typical leftover
state, short enough that a stuck sweep will not meaningfully delay
health checks or startup readiness.

Change Store.Sweep to accept a context.Context and check ctx.Err() at
the top of each iteration. If the context is cancelled or times out,
Sweep returns the files deleted so far and ctx.Err().

Add TestStore_Sweep_HonorsContextCancel to ensure a cancelled context
causes Sweep to exit early instead of running to completion.

Deliberately unchanged: there is still no periodic reaper. The server
only sweeps once at startup; this change just bounds that single sweep.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-14 10:18:13 +08:00
tao.chenandClaude f374ebfdb4 uploads,tools,config,main: 将 upload_file 返回路径改为服务器端绝对路径并透传给 spark_submit 
之前 upload_file 把文件写到 DataDir/uploads/<filename>，返回相对路径
uploads/hello.txt。MCP 服务器与 agent 不在同一台机器，agent 无法构造服务
器本地路径，因此 spark_submit 无法定位脚本。

改为由新的 internal/uploads 包 mint 一个 32 字符十六进制 file_id，文件落盘为
DataDir/uploads/<file_id>，元数据写入 <file_id>.meta.json（原始文件名只存在
sidecar 里）。upload_file 返回 {file_id, path, name, size, sha256}，其中 path
是绝对路径。spark_submit 的 description 明确要求 LLM 直接把 upload_file 返回
的 path 放进 args，不要自己构造路径。

为什么只在描述里约束而不在代码里拒绝非 mint 的绝对路径：集群本地已有路径
（如 /opt/spark/examples/pi.py）是合法的 spark-submit 参数，代码不能替
LLM 拒绝。

测试锁定：
- internal/uploads: Save 往返、AbsPath/Validate 非法路径、Sweep 过期/未过期/
  孤立 sidecar
- internal/mcp/tools: upload_file 新响应字段、spark_submit 透传 mint 路径与
  集群本地路径

刻意未做：S3/HDFS 上传、给 spark_submit 新增 file_id 参数、在代码层面拒绝
集群本地绝对路径。

破坏性变更：upload_file 响应从相对 path 改为绝对 path，并新增 file_id/name/
sha256 字段。该工具尚无外部调用者。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-13 19:13:14 +08:00
tao.chenandClaude 7f8960823b config: GIN_MODE 改为环境变量配置 + Load() 自动加载 .env 
main.go 硬编码 gin.ReleaseMode 改成读 cfg.GinMode（默认 release），
方便生产/调试切换 debug/release/test。Load() 开头先调 loadDotenv
(".env"): 缺文件静默跳过，shell env 优先于 .env，跳过注释/空行/
无 '=' 行并 slog.Warn。仅用标准库，不引新依赖。

验证:
  - go build / go vet / go test 全部通过
  - shell env 覆盖 .env (LISTEN_ADDR=:19090 覆盖 .env 的 :18080)
  - 仅 .env 时正确加载 (监听 18080, GIN_MODE=debug 输出 [GIN-debug])
  - 无 .env 且无 shell env 时清晰报错 ADMIN_TOKENS is not set

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-13 13:48:35 +08:00
tao.chenandClaude 705665745e Phase 5 Batch 1: executor + list_clusters + spark_submit + /mcp 路由 
- internal/executor: spark-submit 进程执行, exec.Command 切片形式
  (绝不 shell) + stdout 10MiB tail-cap + app_id regex
  + 5 个单测覆盖 slice-form 注入防御 + 退出码
- internal/mcp/server.go: NewServer + Streamable HTTP Handler
- internal/mcp/tools/{list_clusters, spark_submit, deps, helpers}.go
- mcp-go API 实测修正: WithArray 用 Items() 声明 item 类型;
  StreamableHTTPServer 直接实现 http.Handler, 无 .Handler() 方法
- main.go: 挂 /mcp 路由 (AgentAuth + mcpHandler 手动组合)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-10 16:50:02 +08:00
tao.chenandClaude 2c39091706 Phase 3+4: 共享 HTTP 客户端 + admin API + audit log 
Phase 3 (httpclient):
- shared Client (Timeout + MaxResponseBytes 截断)
- SSRF: 16 段私网 CIDR (含 169.254/::ffff:0:0/96 IPv4-mapped)
  + hostname allowlist (精确/后缀/*. 通配)
  + scheme 校验 (仅 http/https)
- auth 适配器: none / simple (YARN user.name) / basic (SetBasicAuth)
- DoWithRedirect: 跨主机跳转保留 Authorization, max 5 默认
- 19 个测试全绿 (httptest 模拟)

Phase 4 (admin + audit):
- internal/audit: Entry + Repo.Insert/List + MarshalDetails, snake_case JSON
- internal/middleware: AdminAuth/AgentAuth (constant-time 比对)
- internal/admin: 6 端点 (GET/POST/PUT/DELETE /admin/clusters, GET /admin/audit)
  + 写操作触发 audit_log (含 before/after diff)
  + AuthPassword 空=保留旧密码
  + 校验: 必填字段 + auth_type 枚举 + rate_limit>=0
- main.go: 挂载 storage.Open + admin.Mount
- L fix: audit.Entry 加 json tag (smoke test 发现大写 key 不规范)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-10 13:54:24 +08:00
tao.chenandClaude a4e2472716 Phase 0+1+2 续: 项目骨架 + 配置 + 日志 + 存储层 
Phase 0 (项目骨架):
- main.go 改为最小骨架 (config + logging + gin + /healthz + 优雅退出)
- internal/{config,logging,storage,cluster,...}/ 目录占位

Phase 1 (配置 + 日志):
- internal/config: env 解析 + 必填校验 + token 隐藏的 String()
- internal/logging: slog multi-handler 双输出 (终端 text + 主文件 JSON)
  + StartToolCall per-tool 独立文件 (0600), tools 目录 0700

Phase 2 续 (存储层):
- internal/cluster: 16 字段 Cluster struct (AuthPassword json:"-")
- internal/storage: modernc.org/sqlite 接入, WAL 模式, schema 自动迁移
- ClusterRepo CRUD: Create/Get/List/Update/Delete + ErrNotFound
  + AuthPassword 空字符串 = 保留旧密码 (核心约定)
- 7 个单元测试全绿 (:memory: DB)
- created_at/updated_at 改纳秒精度, 消除 List 测试的 sleep 特殊 case

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-10 12:12:53 +08:00
tao.chen 1dfef0f598 init 2026-07-10 10:19:28 +08:00