feat: add fetch_url tool for proxying HTTP GET to cluster-internal URLs
Add a new MCP tool that lets the agent fetch URLs on the cluster's
network (YARN tracking UI, Spark History Server, NodeManager web UIs)
when the agent is on a different network and cannot reach those hosts
directly.
The MCP service runs on the YARN RM node, so it can reach every host
the cluster knows about — the agent just needs a way to ask.
Security: SSRF guard via host suffix overlap
- URL host must share >= 2 labels of suffix with the named
Connection's yarn_rm_url host (e.g. yarn_rm_url='rm.prod.internal'
allows 'http://nm01.prod.internal/...')
- IP literals (10.0.0.1, ::1) rejected
- Non-http(s) schemes (file://, gopher://, ftp://) rejected
- Connection with no yarn_rm_url cannot use this tool
- Reuses Connection.auth_for_httpx() and verify_for_httpx() so the
agent does not need cluster credentials
- Response body capped at 1 MB (truncated=true if larger)
- 30s timeout, follows redirects, loguru INFO audit log on every call
- spark_executor/tools/fetch_url.py: new tool + 2 helpers
(_host_suffix_overlap, _validate_url_host)
- spark_executor/models.py: FetchUrlResult Pydantic model
- spark_executor/tools/requests.py: FetchUrlRequest with descriptions
- spark_executor/server.py: /fetch_url route, operation_id='fetch_url'
- tests/unit/test_fetch_url.py: 13 unit tests covering all guards,
truncation, auth/SSL pass-through, redirect follow
- tests/integration/test_mcp_routes.py: assert 21 tool routes
- README.md: 1 row in Spark Executor 工具 table
Tests: 369 passed (up from 356).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -378,3 +378,27 @@ class ExternalJobResultRequest(BaseModel):
|
||||
...,
|
||||
description="Name of a saved Connection pointing at the YARN cluster.",
|
||||
)
|
||||
|
||||
class FetchUrlRequest(BaseModel):
|
||||
url: str = Field(
|
||||
...,
|
||||
description=(
|
||||
"Absolute http:// or https:// URL to fetch. The host must share "
|
||||
"at least 2 labels of suffix with the named Connection's "
|
||||
"yarn_rm_url host (e.g. if yarn_rm_url is 'rm.prod.internal:8088', "
|
||||
"you may fetch 'http://nm01.prod.internal:8042/...' but NOT "
|
||||
"'http://evil.com/...' or 'http://10.0.0.1/...'). IP literals "
|
||||
"and non-http(s) schemes are rejected. The Connection's saved "
|
||||
"auth is reused for the outbound request — the agent does not "
|
||||
"need cluster credentials."
|
||||
),
|
||||
)
|
||||
connection_name: str = Field(
|
||||
...,
|
||||
description=(
|
||||
"Name of a saved Connection (see list_connections). The "
|
||||
"Connection's yarn_rm_url defines the allowed host domain. "
|
||||
"The Connection's auth_type / auth_user / auth_password / "
|
||||
"ssl_verify / ssl_ca_bundle are reused for the request."
|
||||
),
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user