 tao.chenandClaude
|
e6411b0dc4
|
Phase 5 Batch 4: analyzer 规则引擎 + 3 个高层 Tool
- internal/analyzer: 纯函数规则引擎
- Finding / Severity / Thresholds / Input / StageMetric / ExecutorMetric
- 3 规则: data_skew (max/min ratio), gc_pressure (GC/CPU ratio),
bottleneck (shuffle read+write GB)
- 严重度阶梯: warning / critical (阈值 ×2)
- Analyze 入口数据驱动 for ... range
- 11 个单测 (3 规则 × normal/warning/critical + 零除防御)
- fetch_spark_metrics: SHS 业务封装, format=summary 触发 analyzer
- fetch_cluster_env: RM /cluster/info + /cluster/metrics 聚合
- analyze_spark_log: RM 日志降级链 + LLM-Ready Prompt 拼装
- prompt 结构: cluster 元信息 + log source + log tail +
heuristic findings + suggested LLM analysis
- findings 是 first-pass filter, LLM 做最终判断
- deps.go: +AnalyzerThresholds
- main.go: 注入 cfg 的 3 个阈值
- 端到端实测: analyze_spark_log 完整 prompt 渲染
完成 11 个 Tool 表面 (list_clusters / spark_submit / 4 RM /
fetch_spark_metrics / fetch_cluster_env / analyze_spark_log /
fetch_url / upload_file)
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-07-10 16:53:47 +08:00 |
|
|
tao.chenandClaude
|
d4ad97f595
|
Phase 5 Batch 3: fetch_url + upload_file Tool (底层原语)
- fetch_url: 通用 HTTP 原语
- host allowlist 校验 (cluster.URLAllowlist + RM/SHS host 兜底)
- 复用 httpclient.ApplyAuth + DoWithRedirect
- mcp.WithEnum 约束 method
- 单元测试: 9 子测试 (basic auth 透传 + 跨主机 redirect 保留 auth)
- upload_file: 本地文件存储 (供 spark_submit 引用)
- filename sanitize: [a-zA-Z0-9._-], 拒绝路径逃逸
- filepath.Base 二次校验
- 写到 <DataDir>/uploads/<name>, 权限 0640
- 单元测试: 路径遍历全部拒绝
- deps.go: +DataDir
- main.go: 注入 cfg.DataDir
- 端到端实测: upload_file + spark_submit 引用上传脚本
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-07-10 16:52:10 +08:00 |
|
|
tao.chenandClaude
|
c4ac3cc354
|
Phase 5 Batch 2: 4 RM Tool + rm 客户端 + 日志降级链
- internal/rm/client.go: YARN RM 客户端
- ListApps / GetApp / KillApp / GetLogs
- GetLogs 降级链数据驱动 (amContainerLogs → aggregated-logs → logs)
- 跨主机 redirect 保留 Authorization (URLAllowlist 兜底)
- 4 Tool: list_applications / get_application_status /
get_application_logs / kill_application
- mcp.WithEnum(state) 约束 YARN app 状态
- tool handler 永远 (result, nil), 业务错误用 NewToolResultError
- deps.go: +HTTPClient + MaxResponseBytes
- main.go: 注入 httpclient.Client
- 端到端实测: mock RM 验证 4 Tool + 降级链 + 错误路径
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-07-10 16:50:27 +08:00 |
|