From 307da276bf3efe00418103a056b56f8ab35b61e3 Mon Sep 17 00:00:00 2001
From: Claude <claude@anthropic.com>
Date: Fri, 10 Jul 2026 11:04:19 +0800
Subject: [PATCH] =?UTF-8?q?docs(README):=20add=20RM/SHS=20API=20endpoint?=
 =?UTF-8?q?=20section=20+=20fix=20Spark=20=E9=9B=86=E6=88=90=20row?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

- Drop the now-stale "`yarn logs`" from the 技术栈 table; the service
  no longer shell-outs the yarn CLI, all cluster calls go via httpx
  (yarn_client.py).
- New "RM / SHS API 端点" section between 提交流程 and 配置项:
  - RM: 4 endpoints (/ws/v1/cluster/apps, /{appid},
    /{appid}/aggregated-logs, PUT /{appid}/state) with the
    triggering tool for each.
  - Documents the amContainerLogs 3xx-follow fallback (manual,
    3-hop max) and why (httpx drops Authorization across hosts).
  - SHS: explicitly notes zero built-in calls; history_server_url
    is stored-for-future, today SHS access goes through fetch_url
    + url_allowlist only.
---
 README.md | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/README.md b/README.md
index 867c96a..4750aa4 100644
--- a/README.md
+++ b/README.md
@@ -40,7 +40,7 @@
 | MCP 暴露 | fastapi-mcp |
 | 数据模型 | Pydantic v2 |
 | HTTP 客户端 | httpx、httpx-kerberos |
-| Spark 集成 | `spark-submit`、YARN REST API、`yarn logs` |
+| Spark 集成 | `spark-submit`、YARN REST API (httpx 直连) |
 | 日志 | loguru |
 | 进程管理 | uvicorn、gunicorn |
 | 包管理 | uv |
@@ -221,6 +221,30 @@ get_job_logs
 kill_job
 ```
 
+## RM / SHS API 端点
+
+本服务**不再 shell-out `yarn` CLI**,所有集群交互都通过 `httpx` 直接打 REST(由 `spark_executor/core/yarn_client.py` 统一封装)。鉴权走 `Connection.auth_type`(`none` / `simple` / `basic` / `kerberos` + SPNEGO),SSL 走 `Connection.ssl_verify` / `ssl_ca_bundle`。
+
+### YARN ResourceManager
+
+| 方法 | 端点 | 用途 |
+| --- | --- | --- |
+| `GET` | `/ws/v1/cluster/apps?state=&queue=&limit=` | 列出 YARN 应用(支持 `state` / `queue` / `limit` 过滤),由 `list_applications` 触发 |
+| `GET` | `/ws/v1/cluster/apps/{appid}` | 取单个 app 的 state + amContainerLogs;日志降级路径会复用 |
+| `GET` | `/ws/v1/cluster/apps/{appid}/aggregated-logs` | 拉聚合后的 container 日志,404/501 时自动降级到 amContainerLogs 路径 |
+| `PUT` | `/ws/v1/cluster/apps/{appid}/state`,body `{"state":"KILLED"}` | 杀应用,由 `kill_job` 触发 |
+
+> **日志降级路径**会**手动**跟随 3xx Location 跳到 NodeManager(最多 3 次)抓 AM driver 的 `/stdout`。原因:httpx 默认跨主机重定向会丢掉 `Authorization`,RM→NM 的 307 在很多集群会因此 401/403。driver 日志可以这样取,**executor 日志仍然依赖 `yarn.log-aggregation-enable=true`**,这个不走降级。
+
+### Spark History Server
+
+**当前没有任何内置调用。** `Connection.history_server_url` 字段只是原样存储,留作未来 SHS 专用工具消费(`models.py` 的 docstring 明确写明)。
+
+要走 SHS API(典型路径 `/api/v1/applications/{id}/jobs`、`/api/v1/applications/{id}/stages` 等),**唯一**方式是用 `fetch_url` 工具:
+
+1. 把 SHS 主机 glob 加到对应 `Connection.url_allowlist`(单标签主机如 `ccam*`,或多级域名如 `*.hadoop.internal`)。
+2. 调用 `fetch_url` 时传入完整 SHS URL,鉴权和 SSL 复用同一条 Connection 的配置。
+
 ## 配置项
 
 ### 应用配置