feat: add fetch_url tool for proxying HTTP GET to cluster-internal URLs

Add a new MCP tool that lets the agent fetch URLs on the cluster's
network (YARN tracking UI, Spark History Server, NodeManager web UIs)
when the agent is on a different network and cannot reach those hosts
directly.

The MCP service runs on the YARN RM node, so it can reach every host
the cluster knows about — the agent just needs a way to ask.

Security: SSRF guard via host suffix overlap
  - URL host must share >= 2 labels of suffix with the named
    Connection's yarn_rm_url host (e.g. yarn_rm_url='rm.prod.internal'
    allows 'http://nm01.prod.internal/...')
  - IP literals (10.0.0.1, ::1) rejected
  - Non-http(s) schemes (file://, gopher://, ftp://) rejected
  - Connection with no yarn_rm_url cannot use this tool
- Reuses Connection.auth_for_httpx() and verify_for_httpx() so the
  agent does not need cluster credentials
- Response body capped at 1 MB (truncated=true if larger)
- 30s timeout, follows redirects, loguru INFO audit log on every call

- spark_executor/tools/fetch_url.py: new tool + 2 helpers
  (_host_suffix_overlap, _validate_url_host)
- spark_executor/models.py: FetchUrlResult Pydantic model
- spark_executor/tools/requests.py: FetchUrlRequest with descriptions
- spark_executor/server.py: /fetch_url route, operation_id='fetch_url'
- tests/unit/test_fetch_url.py: 13 unit tests covering all guards,
  truncation, auth/SSL pass-through, redirect follow
- tests/integration/test_mcp_routes.py: assert 21 tool routes
- README.md: 1 row in Spark Executor 工具 table

Tests: 369 passed (up from 356).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Claude
2026-07-09 10:59:25 +08:00
co-authored by Claude Fable 5
parent f8536b63ad
commit 627e70f697
7 changed files with 334 additions and 1 deletions
+27
View File
@@ -21,6 +21,7 @@ from spark_executor.tools.external_jobs import (
get_external_job_status,
get_external_job_result,
)
from spark_executor.tools.fetch_url import fetch_url
from spark_executor.tools.requests import (
ConnectionNameRequest,
EmptyRequest,
@@ -28,6 +29,7 @@ from spark_executor.tools.requests import (
ExternalJobLogsRequest,
ExternalJobStatusRequest,
ExternalJobResultRequest,
FetchUrlRequest,
GetJobLogsRequest,
JobIdRequest,
PendingIdRequest,
@@ -467,3 +469,28 @@ def _read_job_file(req: ReadJobFileRequest):
)
def _update_job_file(req: UpdateJobFileRequest):
return update_job_file(req.script_path, req.content)
# --- HTTP fetch proxy (host allowlist via Connection.yarn_rm_url) ---
@app.post(
"/fetch_url",
operation_id="fetch_url",
summary="Fetch a URL on the cluster's network and return the body",
description=(
"Proxy an HTTP GET to a URL on the cluster's network, returning the "
"response body. Useful when the agent is on a different network from "
"the cluster and cannot reach YARN tracking pages, Spark History "
"Server, or NodeManager web UIs directly.\n\n"
"**Security constraints:** the URL host must share at least 2 labels "
"of suffix with the named Connection's yarn_rm_url host (e.g. if "
"yarn_rm_url is 'rm.prod.internal:8088', you may fetch "
"'http://nm01.prod.internal:8042/...' but NOT 'http://evil.com/...'). "
"IP literals (10.0.0.1, ::1) and non-http(s) schemes (file://, "
"gopher://) are rejected. The Connection's saved auth is reused, so "
"the agent does not need cluster credentials.\n\n"
"**Limits:** response body capped at 1 MB (truncated=true if larger), "
"30s timeout, redirects followed."
),
)
def _fetch_url(req: FetchUrlRequest):
return fetch_url(req.url, req.connection_name)