- fetch_url: revalidate allowlist on every redirect hop (fixes SSRF where
302 to disallowed host / 169.254.169.254 / file:// bypassed the
url_allowlist). Stream response body with iter_bytes and cap at 1MB
so a multi-GB response from an allowlisted host cannot OOM the service.
Reuses the manual-redirect-loop pattern from yarn_client.
- list_applications: stop swallowing 404 (YARN returns 200+empty for
"no match"; 404 means the RM doesn't support the endpoint — surface
the YarnError instead of hiding it as an empty result). Add
Field(ge=1, le=10000) to ListApplicationsRequest.limit so a runaway
limit is rejected at the Pydantic layer with 422.
- save_connection: PATCH semantics for existing records. Re-route to
update_connection when the name already exists so partial updates
(e.g. only master) no longer wipe url_allowlist back to []. Uses an
_UNSET sentinel in the tool function to distinguish "omitted" from
"None" without breaking the existing parameter list.
- README: drop leading space on 5 new connection-tool table rows that
was breaking GitHub Flavored Markdown table continuity.
- Indentation: normalize connections.py and requests.py to 4-space
indent (auth_password/auth_principal/auth_keytab were 3-space).
Co-Authored-By: Claude <noreply@anthropic.com>
Add a new MCP tool that queries YARN's /ws/v1/cluster/apps endpoint
through a named Connection, returning a list of ApplicationSummary
records. Bypasses the local JobStore — useful for enumerating apps
that were not submitted through this service.
API:
list_applications(
connection_name: str, # required, which YARN cluster
state: str | None = None, # YARN state filter: NEW/NEW_SAVING/
# SUBMITTED/ACCEPTED/RUNNING/
# FINISHED/FAILED/KILLED
queue: str | None = None, # YARN queue filter
limit: int = 100, # cap on returned apps (YARN has no
# offset-based pagination; combine
# state/queue filters for big clusters)
) -> list[ApplicationSummary]
Implementation:
- yarn_client.list_applications(config, *, state, queue, limit) -> list[dict]
Returns raw YARN app dicts; raises YarnError on 4xx/5xx; returns
[] on 404 (no apps match). Uses the existing _request helper,
which now accepts a "params" kwarg for query strings (one-line
additive change).
- external_jobs.list_applications(connection_name, state, queue, limit)
-> list[ApplicationSummary]. Looks up the Connection, builds the
YarnClientConfig, calls the yarn_client function, maps each raw
YARN dict to ApplicationSummary (mirroring the manual field-mapping
style of get_job_result). The yarn_client function is imported
as "list_applications_yarn" to avoid name collision.
- ApplicationSummary: 12-field Pydantic model with snake_case names
(application_id, name, user, queue, state, final_status,
application_type, application_tags, started_time, finished_time,
tracking_url, progress). Unused YARN fields (memorySeconds,
vcoreSeconds, preemptedResource*, etc.) are not exposed.
- ListApplicationsRequest: Pydantic body model with Field(description=)
for LLM-facing schema.
- /list_applications route registered with operation_id=
"list_applications", placed next to the other external YARN tools.
Tests:
- 8 new unit tests in test_external_jobs.py (happy path, state/queue/
limit pass-through, default limit, empty list, missing connection,
full field mapping).
- test_mcp_routes.py: assert 23 tool routes.
- README: list_applications row added to the Spark Executor table.
Tests: 390 passed (was 382, +8 net).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add 3 new MCP tools for inspecting YARN applications NOT submitted
through this service: get_external_job_logs, get_external_job_status,
get_external_job_result. Each takes application_id + connection_name
and queries YARN directly, bypassing the local JobStore.
- spark_executor/tools/external_jobs.py: 3 tool functions
- spark_executor/tools/requests.py: 3 new Pydantic body models
(ExternalJobLogsRequest, ExternalJobStatusRequest,
ExternalJobResultRequest)
- spark_executor/server.py: 3 new POST routes with explicit operation_id
- tests/unit/test_external_jobs.py: 7 unit tests
- tests/integration/test_mcp_routes.py: assert 20 tool routes
- README.md: list the 3 new tools
To make the LLM pick the right tool and not guess at field values,
also:
- Add Pydantic field descriptions for 22 fields across 8 request models
(SaveConnectionRequest, UpdatePendingJobRequest, GetJobLogsRequest,
JobIdRequest, PendingIdRequest, ConnectionNameRequest, plus the new
ExternalJob*Request models).
- Update 12 route descriptions with cross-references, prerequisite
context, and 400 behavior notes.
- Refactor _unknown_job_error: an input that looks like a YARN
application_id (starts with 'application_') now returns HTTP 400
(ValueError) with a hint message naming the right external tool;
other not-found cases still return 404 (KeyError). This catches the
common LLM mistake of passing application_id to the internal
get_job_* / kill_job tools.
- 4 new unit tests for the 400 behavior.
Tests: 356 passed (up from 242).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
