- Drop the now-stale "`yarn logs`" from the 技术栈 table; the service
no longer shell-outs the yarn CLI, all cluster calls go via httpx
(yarn_client.py).
- New "RM / SHS API 端点" section between 提交流程 and 配置项:
- RM: 4 endpoints (/ws/v1/cluster/apps, /{appid},
/{appid}/aggregated-logs, PUT /{appid}/state) with the
triggering tool for each.
- Documents the amContainerLogs 3xx-follow fallback (manual,
3-hop max) and why (httpx drops Authorization across hosts).
- SHS: explicitly notes zero built-in calls; history_server_url
is stored-for-future, today SHS access goes through fetch_url
+ url_allowlist only.
We've had a stray tests/.DS_Store showing up as untracked in
`git status` for several commits. The repo already has .gitignore
sections for Python / venv / IDEs / data / test artifacts but
nothing for the macOS file Finder drops into every directory it
touches. Add a minimal macOS section with just .DS_Store (skip
the broader `._*` resource-fork glob since the user asked for the
specific file).
No code or test changes — pure ignore-list update. The stray
.DS_Store on this machine has already been removed from disk, so
this commit touches only .gitignore.
Tests: 405 passed, no test changes (ignore list isn't covered by
tests).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
In 3b698e1 I added the YarnError exception handler and tested it
against get_job_status only. The other four YARN-touching tools
(get_external_job_status, get_external_job_result,
get_external_job_logs, list_applications) were assumed to be
covered by the same handler but never actually exercised against a
YarnError.
Add one integration test per remaining tool. Each one:
- saves a Connection
- mocks the underlying yarn_client symbol the tool uses
(get_application_status / get_application_logs /
list_applications_yarn) to raise YarnError with a distinctive
message
- hits the tool's route via TestClient
- asserts HTTP 502 + the YarnError message in the response detail
The new tests prove the YarnError -> 502 contract holds uniformly:
- test_external_job_status_returns_502_on_yarn_error
(get_application_status raising "not found")
- test_external_job_result_returns_502_on_yarn_error
(get_application_status raising "Connection refused")
- test_external_job_logs_returns_502_on_yarn_error
(get_application_logs raising "HTTP 500 cluster overloaded")
- test_list_applications_returns_502_on_yarn_error
(list_applications_yarn raising "HTTP 503")
Each one uses a different YARN exception message so the test
distinguishes which code path produced the error. Combined with
the existing get_job_status test, the YarnError -> 502 contract
is now verified end-to-end for all five YARN REST-call sites.
Tests: 405 passed (was 401, +4 net).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two follow-ups to the previous error-handling fix (16fe011):
1) Drop the 1 MB cap on read_job_file and update_job_file.
The cap was originally there to keep MCP responses bounded, but it
blocks legitimate use of large PySpark scripts and large update
payloads. With the new model (LLM composes scripts via
write_job_file and edits them via read+update), a fixed 1 MB
cap is more hindrance than protection. MCP response size is
already bounded by the JSON transport and httpx; the tool itself
doesn't need a second limit.
Changes:
- tools/job_file.py: delete MAX_FILE_BYTES constant, drop the two
size checks in read_job_file and update_job_file, update
module docstring.
- tests/unit/test_job_file.py: delete test_update_rejects_
oversized_content and test_update_rejects_1mb_plus_1_byte
(the two tests that asserted the cap), replace with
test_update_accepts_content_larger_than_former_1mb_cap.
- server.py: drop "Caps reads at 1 MB" and "Caps writes at 1 MB"
from the two route descriptions.
2) Add YarnError -> 502 handler.
yarn_client wraps every httpx call: on connect / TLS / timeout /
4xx / 5xx / parse failure it raises YarnError. Previously this
was unhandled, so all six external job tools (get_external_job_*,
list_applications, plus anything else that hits YARN) returned
500 "Internal Server Error" with no detail — the LLM couldn't
tell whether the cluster was down or the request was bad.
Same fix as 16fe011 (which did this for fetch_url directly).
The new handler returns HTTP 502 Bad Gateway with the YarnError
message in the response detail. 502 because the MCP service is
acting as a gateway to YARN — 502 is the standard status for
"upstream didn't respond correctly".
Changes:
- server.py: import YarnError, add @app.exception_handler
returning 502 + the YarnError message.
- tests/integration/test_mcp_routes.py: new test asserts that
when get_application_status raises YarnError, /get_job_status
returns 502 with the YarnError message in the detail.
Note: ValueError (request was bad) is still 400, KeyError (job
not in JobStore) is still 404. The three handlers form a clean
3-way classification of tool-layer errors.
Tests: 401 passed (was 401, +1 YarnError test, -2 cap tests = net -1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
I added "response body capped at 1 MB (truncated boolean)" to the
fetch_url route description in 16fe011, not realising the cap was
already removed in deafb5a (refactor: drop fetch_url body cap and
modernize datetime usage). The body cap is gone from both
fetch_url.py (no _MAX_BODY_BYTES, no [:_MAX_BODY_BYTES] slicing, no
truncated field in the result) and from FetchUrlResult (no
truncated field). The route description now falsely advertised a
limit that no longer exists.
Replace the 1 MB / truncated line with a plain "30s timeout,
redirects followed" line that matches the actual code.
Tests: 401 passed, no test changes (description text is not
asserted).
Note (not part of this change): read_job_file and update_job_file
still have a 1 MB cap in tools/job_file.py:27 (MAX_FILE_BYTES =
1 * 1024 * 1024) and the matching description lines. Tell me if
you want those removed too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Previously, when httpx.get raised an HTTPError (ConnectError for host
unreachable, ReadTimeout for slow servers, RemoteProtocolError, etc.)
the exception bubbled up through the route handler as a bare 500
"Internal Server Error". The LLM got no information about what
actually went wrong — could not tell whether the host was down, the
port was closed, DNS failed, TLS handshake broke, or the request
timed out. The only thing the agent could do was guess.
Wrap the redirect loop in try/except for httpx.HTTPError and
translate to ValueError. The existing exception handler in
server.py turns ValueError into HTTP 400 with the message in the
response detail, so the LLM now sees e.g.:
fetch_url could not reach 'http://nm01.prod.internal:8042/':
ConnectError: Connection refused. Check that the URL is reachable
from the MCP service, the host is in Connection.url_allowlist, and
the connection's auth/SSL settings are correct.
The original exception is chained via `raise ... from exc` so loguru
still records the full traceback with the original type, and the
`__cause__` attribute is set on the ValueError for programmatic
inspection.
Note: upstream HTTP 4xx/5xx responses (server replied, even with an
error status) are NOT translated — the FetchUrlResult carries the
status code and body so the LLM can read what the server actually
said. This is the intentional contrast with the no-response-at-all
case (which now has clear 400 detail).
Tests (3 new in tests/unit/test_fetch_url.py):
- test_fetch_url_raises_400_with_detail_on_connect_error
ConnectError("Connection refused") -> ValueError with
"ConnectError", "Connection refused", the URL, and __cause__
chained.
- test_fetch_url_raises_400_with_detail_on_timeout
ReadTimeout("Timed out reading") -> ValueError with
"ReadTimeout", "Timed out reading", __cause__ chained.
- test_fetch_url_returns_body_for_4xx_5xx_upstream
Upstream 503 with body "Service Unavailable - try again later"
-> FetchUrlResult(status_code=503, body=...). Proves the
intentional contrast.
Route description in server.py updated with a new **Errors** section
explaining the two error paths (no response = 400 with detail, got
a response = body returned).
Tests: 401 passed (was 398, +3 net).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Old description was thin and only mentioned yarn_rm_url / auth_type
as the things to look up. That was written before we added
history_server_url and url_allowlist, and before fetch_url and
list_applications existed.
New description explicitly enumerates the connection fields an
agent is most likely to need (yarn_rm_url, history_server_url,
url_allowlist, auth_*, master/deploy_mode/spark_conf) and names
the tools that consume them (get_external_*, fetch_url,
list_applications, prepare_submit_job) so the LLM knows to call
get_connection first when it needs any of those.
Also adds a security note that the response includes secrets
(auth_password, auth_keytab) — the previous description didn't
warn about this.
Tests: 398 passed, no test changes (description text is not
asserted).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a new optional field to Connection for the Spark History Server
base URL. The field is stored as part of the connection but is not
yet consumed by any tool — for now it's a labeled place to record
where SHS lives on the cluster, and a hook for future SHS-specific
tools. To actually fetch SHS endpoints today, use fetch_url with the
SHS host added to url_allowlist.
The field is Optional[str], default None. Same nullability as
yarn_rm_url. The SHS host and the YARN RM host are usually
different, so this is independent of yarn_rm_url.
Schema:
- models.py: Connection.history_server_url (str | None, default None)
- requests.py:
- SaveConnectionRequest.history_server_url (str | None, default None)
- UpdateConnectionRequest.history_server_url (str | None, default None)
- tools/connections.py: save_connection signature gains
history_server_url with the _UNSET sentinel pattern (same as
yarn_rm_url, url_allowlist, etc.) so the upsert path correctly
distinguishes "not provided" from "explicitly None".
- tools/connections.py: update_connection docstring lists the new
field in the mutable fields set.
Tests (4 new in tests/unit/test_connection_tools.py):
- test_save_connection_with_history_server_url
- test_save_connection_history_server_url_defaults_to_none
- test_update_connection_changes_history_server_url
- test_update_connection_keeps_history_server_url_when_omitted
Tests: 398 passed (was 394, +4 net).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three small but high-leverage text corrections. No code or test
changes — these only affect what the LLM sees in tools/list and
Pydantic schemas.
1) FetchUrlRequest.connection_name description
Old: "The Connection's yarn_rm_url defines the allowed host domain."
New: explicitly says url_allowlist is the host gate, NOT yarn_rm_url.
yarn_rm_url is only used by get_external_* tools. Without this
fix the LLM would try to control fetch scope via yarn_rm_url
(a no-op) instead of url_allowlist.
2) Connection.url_allowlist description
Added: "Set or change via save_connection (pass url_allowlist on
create) or update_connection (PATCH the field on an existing
connection)." Tells the LLM which tools populate the field,
instead of leaving it to guess.
3) /fetch_url route description
Old: "30s timeout, redirects followed."
New: "30s timeout, redirects followed, response body capped at
1 MB (the response includes a truncated boolean when this
kicks in)." LLM previously had no way to discover the 1MB
cap or the truncated indicator; it would just see
short responses and assume that was the full body.
Tests: 394 passed, no test changes (descriptions are not asserted).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- fetch_url: remove the 1MB body cap and the streaming helper. The
manual redirect loop with allowlist re-check (the SSRF fix) is kept
intact; the body is now read in full via resp.text. Drop the now-
meaningless `truncated` field from FetchUrlResult and the tests that
asserted on it. Switched from httpx.stream() back to httpx.get()
for the redirect loop — cleaner without the body cap.
- datetime: replace deprecated datetime.utcnow() with
datetime.now(timezone.utc) in submit.py (3 sites) and
core/job_writer.py (1 site). Update the stale comment in
core/pending_store.py that referenced the old call.
- Clean up an unused `from common.config import settings` import in
submit.py that ruff flagged.
Co-Authored-By: Claude <noreply@anthropic.com>
- fetch_url: revalidate allowlist on every redirect hop (fixes SSRF where
302 to disallowed host / 169.254.169.254 / file:// bypassed the
url_allowlist). Stream response body with iter_bytes and cap at 1MB
so a multi-GB response from an allowlisted host cannot OOM the service.
Reuses the manual-redirect-loop pattern from yarn_client.
- list_applications: stop swallowing 404 (YARN returns 200+empty for
"no match"; 404 means the RM doesn't support the endpoint — surface
the YarnError instead of hiding it as an empty result). Add
Field(ge=1, le=10000) to ListApplicationsRequest.limit so a runaway
limit is rejected at the Pydantic layer with 422.
- save_connection: PATCH semantics for existing records. Re-route to
update_connection when the name already exists so partial updates
(e.g. only master) no longer wipe url_allowlist back to []. Uses an
_UNSET sentinel in the tool function to distinguish "omitted" from
"None" without breaking the existing parameter list.
- README: drop leading space on 5 new connection-tool table rows that
was breaking GitHub Flavored Markdown table continuity.
- Indentation: normalize connections.py and requests.py to 4-space
indent (auth_password/auth_principal/auth_keytab were 3-space).
Co-Authored-By: Claude <noreply@anthropic.com>
Add a new MCP tool that queries YARN's /ws/v1/cluster/apps endpoint
through a named Connection, returning a list of ApplicationSummary
records. Bypasses the local JobStore — useful for enumerating apps
that were not submitted through this service.
API:
list_applications(
connection_name: str, # required, which YARN cluster
state: str | None = None, # YARN state filter: NEW/NEW_SAVING/
# SUBMITTED/ACCEPTED/RUNNING/
# FINISHED/FAILED/KILLED
queue: str | None = None, # YARN queue filter
limit: int = 100, # cap on returned apps (YARN has no
# offset-based pagination; combine
# state/queue filters for big clusters)
) -> list[ApplicationSummary]
Implementation:
- yarn_client.list_applications(config, *, state, queue, limit) -> list[dict]
Returns raw YARN app dicts; raises YarnError on 4xx/5xx; returns
[] on 404 (no apps match). Uses the existing _request helper,
which now accepts a "params" kwarg for query strings (one-line
additive change).
- external_jobs.list_applications(connection_name, state, queue, limit)
-> list[ApplicationSummary]. Looks up the Connection, builds the
YarnClientConfig, calls the yarn_client function, maps each raw
YARN dict to ApplicationSummary (mirroring the manual field-mapping
style of get_job_result). The yarn_client function is imported
as "list_applications_yarn" to avoid name collision.
- ApplicationSummary: 12-field Pydantic model with snake_case names
(application_id, name, user, queue, state, final_status,
application_type, application_tags, started_time, finished_time,
tracking_url, progress). Unused YARN fields (memorySeconds,
vcoreSeconds, preemptedResource*, etc.) are not exposed.
- ListApplicationsRequest: Pydantic body model with Field(description=)
for LLM-facing schema.
- /list_applications route registered with operation_id=
"list_applications", placed next to the other external YARN tools.
Tests:
- 8 new unit tests in test_external_jobs.py (happy path, state/queue/
limit pass-through, default limit, empty list, missing connection,
full field mapping).
- test_mcp_routes.py: assert 23 tool routes.
- README: list_applications row added to the Spark Executor table.
Tests: 390 passed (was 382, +8 net).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two changes:
1) Drop every check except the allowlist lookup.
Old _validate_url_host did: scheme check, host-presence check,
IP-literal check, empty-allowlist check, then glob match.
New _validate_url_host does: parse host, return on glob match,
raise on miss. That's it. The only remaining structural check is
'the URL must have a host' (otherwise the glob has nothing to
test against).
Security implication: scheme (file://, gopher://, ftp://) and
IP literals (10.0.0.1, ::1) are NO LONGER rejected by the
validator. The allowlist is the single source of truth. If the
user writes ['*.*.*.*'], they have opted in to 4-label hosts
including IP literals; if they write ['ccam*'], they get ccam1-
ccam99 and nothing else. The default ['ccam*'] / [] pattern is
tight by construction.
Removed: import ipaddress, the scheme/IP rejection branches, the
'allowlist empty' explicit branch (the empty list naturally
matches nothing).
2) Rename allowed_url_hosts -> url_allowlist.
The previous name was a verbose double-negative ('allowed ... hosts').
The new name is short, modern (allowlist > whitelist), and matches
the pattern of the field (URL hosts allowed). Renamed in:
- Connection (models.py)
- SaveConnectionRequest, UpdateConnectionRequest, FetchUrlRequest
(requests.py)
- _validate_url_host, _host_matches_any_glob parameters
(fetch_url.py)
- save_connection / update_connection call sites and error
messages (connections.py, fetch_url.py)
- Route descriptions (server.py)
- All test files
- README
No backward-compat alias: the field was added in 0291f36 and
hasn't shipped, so no production migration. Local dev data
(data/connections.json, gitignored) with allowed_url_hosts set
will be silently dropped by Pydantic v2 (default for extra
fields is ignore) — those connections lose their allowlist and
fetch_url will reject everything until re-saved.
Test cleanup:
- Removed 9 obsolete tests (scheme/IP/suffix rejection)
- Renamed allowed_url_hosts -> url_allowlist in 11 surviving tests
- Added 4 new tests documenting the 'allowlist is the only gate'
model: IP literal accepted, HTTPS accepted, no-host rejected,
empty allowlist rejected, error message mentions url_allowlist
-1 obsolete test, net -4 from 386 -> 382 tests passing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Part 1 — fetch_url validation simplification
Now that Connection.allowed_url_hosts exists, it's the ONLY check.
The old 2-label suffix-overlap rule against yarn_rm_url is gone.
- Connection.allowed_url_hosts: list[str] = Field(default_factory=list)
(was list[str] | None = None). Default empty list means the
connection has no fetch access; the user must explicitly opt in
via save_connection or update_connection.
- _validate_url_host drops the yarn_rm_url parameter and the
_host_suffix_overlap helper. Now: scheme/host/IP checks, then
empty-reject, then glob match. Reject everything else.
- fetch_url's error message now points the user at
Connection.allowed_url_hosts as the fix.
- The label-by-label glob check from the previous commit is kept
(SSRF guard: 'ccam*' matches 'ccam50' but NOT 'ccam50.evil.com').
Part 2 — update_connection tool
PATCH-style update for an existing Connection record. Only the
fields the caller provides are changed. Same pattern as
update_pending_job: req.model_dump(exclude_none=True), with
'name' popped before passing to the store.
To CLEAR a field (e.g. drop auth_password), use delete_connection
+ save_connection. This is YAGNI; the alternative (model_fields_set
to distinguish 'omitted' from 'null') adds surface for bugs.
- ConnectionStore.update(name, **fields): get, model_copy(update=...),
save. Lock + atomic write. Re-validates the patched record.
- update_connection tool function: passes fields through to the
store, logs which fields were changed.
- UpdateConnectionRequest Pydantic model: 12 mutable fields + name.
- /update_connection route with operation_id='update_connection'.
- 9 new unit tests in test_connection_tools.py (PATCH, dict/list
replace-not-merge, unknown name 404, validation of patched record,
disk persistence).
- test_fetch_url.py: existing 21 tests updated; 3 new tests for
the empty/None/missing allowed_url_hosts cases.
- test_mcp_routes.py: assert 22 tool routes.
- README: update_connection row added; fetch_url row updated.
Tests: 386 passed (up from 377, +9 net).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The default 'URL host shares >= 2 labels of suffix with yarn_rm_url
host' rule is too strict for clusters whose hostnames are single-label
(e.g. 'ccam1' through 'ccam99'). The user's cluster is reachable at
http://ccam1:8088, and they want fetch_url to work for any ccamN
host — but ccam1 and ccam50 share 0 suffix labels, so the existing
rule rejects everything.
Add an opt-in allowlist field on Connection:
allowed_url_hosts: list[str] | None
Each entry is an fnmatch glob pattern. The URL host is allowed if it
matches ANY pattern, regardless of the suffix rule. Save a Connection
with ['ccam*'] to allow ccam1, ccam2, ..., ccam99.
SSRF safety: the implementation does NOT use vanilla fnmatch on the
flat host string (because '*' in fnmatch crosses '.', so 'ccam*' would
match 'ccam50.evil.com' — a security hole). Instead, both the pattern
and the host are split on '.' and matched LABEL-BY-LABEL with the
label counts required to match exactly. So 'ccam*' matches 'ccam50'
but NOT 'ccam50.evil.com' (different label counts).
Test coverage:
- 8 new tests in tests/unit/test_fetch_url.py (glob allow, glob deny,
dot-boundary SSRF test, multiple globs, empty list, None fallback,
error message hint)
- All 21 fetch_url tests pass; 377 total.
- spark_executor/models.py: Connection.allowed_url_hosts (with description)
- spark_executor/tools/requests.py: SaveConnectionRequest.allowed_url_hosts
- spark_executor/tools/fetch_url.py: new _host_matches_any_glob helper;
_validate_url_host now takes allowed_hosts and checks globs BEFORE
the suffix rule
- tests/unit/test_fetch_url.py: 8 new tests
- README.md: fetch_url row mentions the allowlist
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a new MCP tool that lets the agent fetch URLs on the cluster's
network (YARN tracking UI, Spark History Server, NodeManager web UIs)
when the agent is on a different network and cannot reach those hosts
directly.
The MCP service runs on the YARN RM node, so it can reach every host
the cluster knows about — the agent just needs a way to ask.
Security: SSRF guard via host suffix overlap
- URL host must share >= 2 labels of suffix with the named
Connection's yarn_rm_url host (e.g. yarn_rm_url='rm.prod.internal'
allows 'http://nm01.prod.internal/...')
- IP literals (10.0.0.1, ::1) rejected
- Non-http(s) schemes (file://, gopher://, ftp://) rejected
- Connection with no yarn_rm_url cannot use this tool
- Reuses Connection.auth_for_httpx() and verify_for_httpx() so the
agent does not need cluster credentials
- Response body capped at 1 MB (truncated=true if larger)
- 30s timeout, follows redirects, loguru INFO audit log on every call
- spark_executor/tools/fetch_url.py: new tool + 2 helpers
(_host_suffix_overlap, _validate_url_host)
- spark_executor/models.py: FetchUrlResult Pydantic model
- spark_executor/tools/requests.py: FetchUrlRequest with descriptions
- spark_executor/server.py: /fetch_url route, operation_id='fetch_url'
- tests/unit/test_fetch_url.py: 13 unit tests covering all guards,
truncation, auth/SSL pass-through, redirect follow
- tests/integration/test_mcp_routes.py: assert 21 tool routes
- README.md: 1 row in Spark Executor 工具 table
Tests: 369 passed (up from 356).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add 3 new MCP tools for inspecting YARN applications NOT submitted
through this service: get_external_job_logs, get_external_job_status,
get_external_job_result. Each takes application_id + connection_name
and queries YARN directly, bypassing the local JobStore.
- spark_executor/tools/external_jobs.py: 3 tool functions
- spark_executor/tools/requests.py: 3 new Pydantic body models
(ExternalJobLogsRequest, ExternalJobStatusRequest,
ExternalJobResultRequest)
- spark_executor/server.py: 3 new POST routes with explicit operation_id
- tests/unit/test_external_jobs.py: 7 unit tests
- tests/integration/test_mcp_routes.py: assert 20 tool routes
- README.md: list the 3 new tools
To make the LLM pick the right tool and not guess at field values,
also:
- Add Pydantic field descriptions for 22 fields across 8 request models
(SaveConnectionRequest, UpdatePendingJobRequest, GetJobLogsRequest,
JobIdRequest, PendingIdRequest, ConnectionNameRequest, plus the new
ExternalJob*Request models).
- Update 12 route descriptions with cross-references, prerequisite
context, and 400 behavior notes.
- Refactor _unknown_job_error: an input that looks like a YARN
application_id (starts with 'application_') now returns HTTP 400
(ValueError) with a hint message naming the right external tool;
other not-found cases still return 404 (KeyError). This catches the
common LLM mistake of passing application_id to the internal
get_job_* / kill_job tools.
- 4 new unit tests for the 400 behavior.
Tests: 356 passed (up from 242).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Dockerfile: COPY files_mcp package, mkdir -p /app/data/files at build
time, set ENV FILES_MCP_ROOT=/app/data/files as the default sandbox
root (overridable at runtime).
- docker-compose.yml: surface FILES_MCP_ROOT in the env block with the
same default, so operators can override it via .env or -e.
FILES_MCP_ROOT must exist at process start (path_guard._init_root
checks is_dir()), so the directory is created during build rather than
deferred to entrypoint — fail-fast at image build beats a runtime
RuntimeError on every container start.
Default lives under /app/data so it rides the existing data volume
(./data:/app/data) and persists across container restarts alongside
connections.json / pending_jobs.json / logs/.
Verified locally: docker compose config --quiet passes; FILES_MCP_ROOT
resolves to /app/data/files in the rendered config. Full image build
not run (docker daemon unavailable in this environment); please run
`docker compose up -d --build` to confirm in your environment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds tests/integration/test_main_lifespan.py with 5 cases that exercise
the end-to-end main.app lifespan:
- default (no MCP_SERVICES) mounts both services in BUILTIN order
- MCP_SERVICES=files mounts only files
- MCP_SERVICES='' mounts zero
- MCP_SERVICES=bogus raises RuntimeError at startup
- /health on the root app is unaffected by MCP filtering
Tests use TestClient and mock main.init_mcp_server to record mount
calls, since fastapi-mcp's HTTP endpoints 404 for bare GETs regardless
of mount state (the MCP transport requires an initialize handshake
before any tool call). Mocking avoids that coupling.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- main.py: drops the hard-coded McpService class and MCP_SERVICES list.
Lifespan now calls discover_and_filter() which does import + env
filtering. Unset MCP_SERVICES reproduces today's behavior exactly.
- spark_executor/service.py: new file exporting SERVICE = McpService(
name='spark_executor', app=spark_executor.app, mount_path='/spark-executor-mcp').
- files_mcp/service.py: new file exporting SERVICE = McpService(
name='files', app=files_mcp.app, mount_path='/files-mcp').
The McpService dataclass itself moved to common/mcp_service.py in the
prior commit. 327 existing tests pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds common/mcp_service.py with the McpService dataclass (moved from
main.py), BUILTIN_SERVICES list, _filter_by_env, and discover_and_filter.
MCP_SERVICES env var acts as a whitelist (unset=all, empty=zero).
Adds tests/unit/test_mcp_service.py with 12 cases covering: load order,
broken-import handling (warn vs raise by env), missing/wrong-typed
SERVICE, duplicate name and duplicate mount_path detection, env filter
(unset/empty/subset/order), and unknown-name error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Self-review caught a contradiction: the spec's prose said
'MCP_SERVICES="" (empty) -> zero services mount' but the code example
in the spec used 'if not raw: return services' which conflates unset
and empty (both fall through to 'all mount').
Fix: _filter_by_env now uses 'MCP_SERVICES' not in os.environ to
distinguish unset (backward compat: all mount) from empty (explicit
choice: zero mount). Test cases already specified the correct behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds docs/superpowers/specs/2026-06-30-pluggable-mcp-design.md describing
a refactor of main.py: McpService dataclass moves to common/mcp_service.py
alongside BUILTIN_SERVICES list and discover_and_filter() helper. Each
service package gets a service.py exporting a SERVICE McpService instance.
Deployments opt in/out via MCP_SERVICES env whitelist (unset = all).
Branch: feat/files-mcp (continues from file-MCP service work; same branch,
not pushed; can be split into separate PRs later)
Backward compat: 0 existing tools / tests / dependencies changed;
unsetting MCP_SERVICES reproduces today's behavior exactly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- main.py: import files_app, append McpService entry to MCP_SERVICES.
Lifespan already iterates the list, so no edit to the lifespan function.
- tests/conftest.py: autouse fixture rebinds files_mcp.core.path_guard._ROOT
to a per-test tmp_path so the sandbox is fresh for every test.
- files_mcp/__init__.py: drop the try/except around the server import now
that server.py exists, restoring fail-fast on a missing/corrupt server
module in production. The _init_root() try/except stays, since pytest
collection can import the package before FILES_MCP_ROOT is set.
Verified via uv run pytest: 327 tests pass, no regressions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds files_mcp/server.py (FastAPI title=Files MCP, 9 POST routes, and
3 exception handlers: KeyError->404, ValueError->400, FileExistsError->409).
The third handler is the only one not present in spark_executor.server;
handlers are registered on this app only, so spark_executor is unaffected.
Adds tests/integration/test_files_mcp_routes.py with 12 end-to-end HTTP
tests covering CRUD happy paths, error codes (404/400/409), and
introspection (list_dir, stat, search, move, copy).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds files_mcp/tools/requests.py with 9 Pydantic body models (one per
tool, since fastapi-mcp passes args as JSON body) and
files_mcp/tools/files.py with 9 thin wrappers that call path_guard
then fs_ops and return model_dump() form. Adds tests/unit/test_files_tool.py
covering happy path, sandbox escape, and KeyError/FileExistsError
propagation for each wrapper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds files_mcp/models.py (FileEntry, OperationResult) and
files_mcp/core/fs_ops.py with 9 pure file operations on already-resolved
Path objects: create/read/update/delete/list_dir/stat/search/move/copy.
All writes go through tempfile.mkstemp + os.replace for atomicity;
read enforces utf-8 + 1MB cap; create/update enforce strict presence
semantics (create: absent or overwrite; update: present). 39 unit tests
in tests/unit/test_fs_ops.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the files_mcp/ sibling package and core/path_guard.py which
resolves arbitrary input paths and rejects any that escape the
FILES_MCP_ROOT sandbox (handles .., ~, symlinks, NUL, empty). Adds
tests/unit/test_path_guard.py covering accept, escape modes,
symlink escape, and _init_root failure paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds docs/superpowers/plans/2026-06-30-files-mcp.md — 5-task
implementation plan covering scaffold+path_guard, models+fs_ops,
business wrappers+request models, server+routes+integration tests,
and main.py+conftest wiring. 77 new tests across 3 unit files and
1 integration file; verified acceptance includes the spark_executor
test suite (242 existing) still passing unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds docs/superpowers/specs/2026-06-30-files-mcp-design.md describing a
sibling FastAPI sub-app (mount: /files-mcp) that exposes 9 file operations
(create/read/update/delete/list_dir/stat/search/move/copy) over fastapi-mcp,
sandboxed to FILES_MCP_ROOT with utf-8 text only and atomic writes.
Branch: feat/files-mcp
Mount path: /files-mcp
New package: files_mcp/
Backward compat: 0 existing tools / tests / dependencies changed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Audit of the previous 'salvage application_id from stderr' change
(e4fb97c) revealed the fix was incomplete:
- The pending record now had pending.application_id set on failure.
- But get_job_logs / get_job_status / kill_job all look up the
application_id through JobStore.get_either(application_id), which
scans the on-disk jobs.json for a Job record matching that
application_id. No Job record was being created on the failure
path, so the recovered application_id was unreachable through
the public tool surface — get_job_logs would 404 with 'No Job
found for id=...application_xxx'.
Fix: when confirm_submit_job's failure branch successfully extracts
an application_id from stderr, it now also creates a Job record (same
shape as the success path: 12-char hex job_id, application_id,
script_path, queue, submit_time, connection, yarn_rm_url). The Job's
only job in this case is to act as the lookup index from the
LLM-facing tools into the underlying YARN app.
The job_id is generated even when application_id recovery fails (so
we always set pending.job_id), but the Job record is only put when
application_id is non-None. That keeps the 'spark-submit died locally
before reaching YARN' case clean: no Job record, get_job_logs gets
the standard 'no logs available' error from yarn_client, not a
404 from JobStore.
Tests:
- test_confirm_recovers_application_id_from_failed_spark_submit now
asserts both that pending.application_id is set AND that a Job
record exists and is findable by both job_id AND application_id.
- test_confirm_failed_spark_submit_without_application_id_keeps_it_none
asserts the negative case: no application_id, no Job record.
Fixture fix: _fresh() now rebinds submit.job_store to use tmp_path.
Previously the test wrote to ./data/jobs.json (the real data dir),
which left stale records between test runs and broke any test that
scanned by application_id. This is a latent bug exposed by the new
assertions; fixing it here also makes every other confirm test
hermetic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Spark-submit can fail AFTER YARN has already accepted the application
(lost RM connection, PySpark script exited non-zero on the cluster
side, auth expired mid-submit, ...). In those cases the stderr still
contains 'Submitted application <id>', but the previous
confirm_submit_job implementation threw away that signal and recorded
the pending as FAILED with no application_id. The user had no way to
fetch the YARN logs of the failed job.
Fix:
- run_spark_submit now attaches the failed CompletedProcess to the
SparkSubmitError as .result, so callers can still inspect stderr.
- confirm_submit_job's failure branch tries to parse
(application_id, tracking_url) from exc.result.stderr. If found,
it writes them onto the pending record before re-raising. The
status is still FAILED and the error message is still set — the
user sees the failure normally, but the pending now also has a
log target they can pass to get_application_logs.
- If the stderr has no application_id (e.g. spark-submit failed
locally before reaching YARN), the parse attempt returns None and
the pending is FAILED with no log target — exactly the previous
behavior in that case.
Tests:
- test_confirm_marks_failed_on_spark_submit_error: existing test
now also asserts application_id is None (no result attached).
- test_confirm_recovers_application_id_from_failed_spark_submit:
failed run with 'Submitted application X' in stderr → pending is
FAILED + application_id is set.
- test_confirm_failed_spark_submit_without_application_id_keeps_it_none:
failed run with no application_id in stderr → pending is FAILED
+ application_id is None.
Full suite 248 passed (+2 from 246).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
write_job_file now validates that the incoming code string is encodable
as UTF-8 BEFORE doing any file I/O. A lone surrogate (U+D800..U+DFFF) —
typically the result of a broken decode somewhere upstream in the
agent's pipeline — cannot be represented in UTF-8, and would otherwise
either silently produce mojibake on disk or crash with an opaque
UnicodeEncodeError from open().
Behavior:
- try: code.encode('utf-8')
- on UnicodeEncodeError: log a warning and raise ValueError with a
message that points the agent at 're-generate the code as plain
Unicode'. ValueError -> HTTP 400 via the existing server.py
exception handler, so the agent gets a clean error and no file is
written.
- normal CJK / emoji / accented Latin still work — the encode check
is essentially free.
Tests:
- test_write_accepts_valid_utf8_including_cjk_and_emoji: positive
case for the common LLM output patterns.
- test_write_rejects_lone_surrogate: negative case, also asserts
that tmp_path is empty (no half-written .py left behind).
Full suite 246 passed (+2 from 244).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
confirm_submit_job no longer retries on SparkSubmitError. A failed
single attempt is recorded as-is:
- pending.status = 'FAILED' with the error message in pending.error
- the function re-raises the SparkSubmitError so the caller (and the
agent) sees the failure immediately
- the agent can call get_pending_job to see the persisted FAILED
state, including the captured error
The 'retry' code path was hiding real failures behind transient
recovery: a spark-submit that died because the script is broken looks
the same as one that died because YARN was momentarily unreachable.
Without retries, the agent gets a clean FAILED state and a clear
exception to act on, instead of a delayed, ambiguous result.
Manual recovery is still possible: re-calling confirm_submit_job on a
FAILED pending resets it to PENDING and runs a single fresh attempt
(see test_confirm_resets_failed_then_succeeds). The reset path is
useful for 'I fixed the script, try again' flows; it just is no longer
the default on every failure.
Removed:
- the for-attempt loop and time.sleep in submit.py
- import time in submit.py and test_submit_tool.py
- confirm_max_retries and confirm_retry_delay_seconds from
common/config.py (env vars, snapshot, reload)
- 2 retry tests (test_confirm_retries_spark_submit_failure_then_succeeds,
test_confirm_exhausts_retries_and_sets_failed)
- test_confirm_marks_failed_on_spark_submit_error updated to assert
the new single-attempt + raise contract
- test_confirm_resets_failed_and_retries renamed to
test_confirm_resets_failed_then_succeeds (no more retry loop to
exercise)
Full suite 244 passed (3 fewer than before, matching the removed
tests).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
YARN's 307 Location header is always a fully-qualified absolute URL.
The previous implementation used urllib.parse.urljoin to handle the
hypothetical case of a relative Location, but that's a defensive
codepath for a scenario that does not occur in practice. Drop it:
- Remove 'from urllib.parse import urljoin'.
- _request_following_redirects now assigns the Location value
straight to the next request URL.
- Update the docstring to reflect the new contract.
- Remove the test that exercised the relative-Location path.
Full suite 246 passed (1 fewer than the previous commit, matching the
test that was removed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
httpx follows redirects by default, but it does NOT forward the
Authorization header across host boundaries. In YARN deployments where
the ResourceManager 307-redirects the NodeManager log fetch to a
different host (load balancer, Knox, NM selection), the follow-up
request lands unauthenticated and returns 401/403.
Replace the _request call in _fetch_logs_via_am_container with
_request_following_redirects, which:
- Walks the 3xx Location chain (up to 3 hops).
- Re-applies auth + verify on every hop.
- Resolves relative Location URLs against the current request URL.
- Raises YarnError on 3xx-without-Location (misconfigured server) and
on hop count overflow (redirect loop protection).
301/302/303/307/308 are all treated as 'follow the Location' per
RFC 7231 — the method/body handling for the rest is up to httpx when
we re-issue the request.
Tests: 4 new (cross-host 307, no-Location 307, two-hop chain, relative
Location). Full suite 247 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The previous commit (32ee167) added a regex-based HTML directory-listing
parser as a fallback when the NodeManager wraps /stdout in HTML (Knox,
wrapped vendor YARN builds). On review, the parser is too brittle to
trust in production:
- Apache-style listings vary across Hadoop distributions; some add
extra rows, icons, or anchors that change the regex hit set.
- Some clusters return the listing but 4xx each individual log file
(auth layer in the way), so the fallback silently returns empty.
- The behavior the agent observes for the same application_id becomes
environment-dependent in ways that are hard to reason about.
Revert to the simple, predictable path: GET /apps/{appid}, read
amContainerLogs, GET {amContainerLogs}/stdout, return its text. If the
cluster wraps /stdout in HTML, the agent gets a clear
'log-aggregation-enable' error and can ask the user to enable log
aggregation or use a different fetch mechanism — much easier to debug
than a half-parsed directory listing.
Removes:
- _looks_like_html, _parse_log_listing_html helpers
- The bare-URL fallback path and its try/except wrapper
- 4 unit tests that exercised the parser
Full suite back to 243 passed (matches fd0036c).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Some YARN deployments (Knox-proxied, wrapped vendor builds) return an
HTML directory listing page for /node/containerlogs/.../stdout instead of
the raw log bytes. The previous fix assumed a clean text/plain response
and failed every log fetch on those clusters.
Two new private helpers in yarn_client.py:
- _looks_like_html(text): cheap prolog sniff (<!doctype html, <html,
<?xml) on the first ~200 chars.
- _parse_log_listing_html(html): extract plain log-file names via a
href="..." regex; drop ../, absolute paths, sort toggles (?C=N;O=D),
and absolute URLs.
_fetch_logs_via_am_container now does:
1. GET /apps/{appid} (unchanged).
2. Read app.amContainerLogs.
3. Fast path: GET {amContainerLogs}/stdout. If 2xx and NOT HTML, return.
4. Fallback: GET {amContainerLogs} (bare URL), parse the HTML listing,
fetch each log file individually, and concatenate with === filename ===
headers in document order.
5. Raise _logs_unavailable_error if both paths yield nothing.
Tests: 3 new (test_logs_falls_back_to_directory_listing_when_stdout_returns_html,
test_logs_html_filter_strips_sort_links_and_parent_dir,
test_logs_html_filter_handles_prelaunch_files,
test_logs_raises_when_directory_listing_empty) — full suite 247 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The previous _fetch_logs_via_nodemanager walked
/apps/{appid}/appattempts -> /apps/{appid}/appattempts/{id}/containers,
but YARN ResourceManager does not expose the second hop. Container
info lives on NodeManager and is not reachable from RM REST, so the
fallback returned _logs_unavailable_error on every cluster without
/aggregated-logs (Hadoop 2.x, log aggregation disabled).
Replace the walk with: GET /apps/{appid} -> read app.amContainerLogs
-> GET {amContainerLogs}/stdout. The URL is provided directly by the
app response, so no container enumeration is needed.
Returns AM (driver) container logs only. Executor container logs
still require yarn.log-aggregation-enable=true (primary /aggregated-logs
path); the existing _logs_unavailable_error message already says that,
so the contract is preserved.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three changes the user wanted in the operation skill, all aimed at
making the LLM agent pause and ask the human instead of silently
choosing defaults or auto-confirming:
1. At task submission, ASK for connection (which one to use) and
app_name. The service has no default for app_name and rejects
implicit queue/memory/cores/num_executors defaults, so the
agent has to surface these to the user anyway — better to do
it deliberately than to call a tool, get 400, and re-ask.
2. Before prepare_submit_job, ASK for the submit parameters
(queue, executor_memory, executor_cores, num_executors,
extra_args). The agent tells the user the server-side defaults
it WOULD use so they can approve or override, instead of
picking on their behalf.
3. Before confirm_submit_job, present a one-screen summary of
exactly what will run (connection, app_name, script, queue,
resources, tracking URL) and wait for the user's explicit
affirmative ('yes', 'confirm', 'go', 'y', even an emoji).
A non-answer or 'wait' / 'let me think' is NOT consent. The
agent NEVER calls confirm_submit_job without that go-ahead.
Where the changes live in the skill:
* §0 Mindset — added a 4th 'must' about not auto-confirming,
cross-referencing the new §0.1.
* §0.1 Mandatory user confirmations — new sub-section, spells out
the three checkpoints in detail (what to ask, what counts as
consent, what doesn't).
* §2 The canonical happy path — the numbered list now interleaves
CHECKPOINT 1/2/3 lines with the tool calls, so the pauses are
visually unmistakable.
* §8 Common pitfalls — two new rows: 'auto-confirming without
explicit yes' and 'picking defaults on the user's behalf'.
* §9 End-to-end example — the word-count walkthrough now shows
the full ask flow with sample agent / user dialogue at each
checkpoint, ending in 'only NOW may you call confirm_submit_job'.
The example also adds a 'kill_job: ASK THE USER FIRST' note, since
killing a running YARN app is the same class of irreversible action
as confirming a new one.
File: 449 → 557 lines (+108).
No code changes; tests not affected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The MCP tool was named `generate_job_file` from Stage 2 but it does
NOT generate PySpark code — the calling LLM writes the code in its own
context, and this tool only persists it to a file under
SPARK_EXECUTOR_JOBS_DIR so `spark-submit` can see it. The misleading
`generate_` prefix sent agents (and humans) looking for a code
generator that doesn't exist.
This commit folds three related polish changes into one (split later
with rebase -i if you want them as separate history):
1. The rename itself:
- `tools/generate.py` → `tools/write_job.py`
- `generate_job_file` → `write_job_file`
- `GenerateJobFileRequest` → `WriteJobFileRequest`
- `/generate_job_file` route → `/write_job_file`
- `operation_id="generate_job_file"` → `operation_id="write_job_file"`
The internal helper `core.job_writer.write_job_file` (which just
writes bytes to disk with no SQL guard) is imported with an
`_write_to_disk` alias to avoid the name collision with the
MCP-exposed function in the same module.
The description for the tool now explicitly states 'this tool
does NOT generate PySpark code. The calling LLM is expected to
have already written the code; this tool only persists it.'
2. Skill for LLM agents operating the service
(`docs/superpowers/skills/spark-executor-mcp-operate/SKILL.md`,
449 lines). Covers the 16 tools, the two-step prepare/confirm
flow, the dual-ID contract (job_id vs application_id), the
PendingSubmission state machine, the Connection profile, the
job-file workflow, the error reference, common pitfalls, and a
full end-to-end word-count example.
3. Default `executor_memory` lowered 4G → 2G
(`_DEFAULTS_TO_CONFIRM` in `server.py`). Mirrors the matching
change in `test_mcp_routes.py` and the 5 unit tests that
reference the default. Aligns with the lighter workloads the
service is sized for in its current container profile.
Also tracked in git for the first time:
- `docs/superpowers/plans/2026-06-24-spark-executor-mcp.md`
(the original Stage 1/2/3 design plan, updated to use the new
tool name throughout).
Test rename:
- `tests/unit/test_generate_tool.py` → `test_write_job_tool.py`
- the new test file picks up an extra assertion that the SQL guard
rejects a `DROP TABLE` statement at write time.
243 tests pass (was 242; +1 new SQL-guard assertion). Zero regressions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude
tao.chen