Audit of the previous 'salvage application_id from stderr' change
(e4fb97c) revealed the fix was incomplete:
- The pending record now had pending.application_id set on failure.
- But get_job_logs / get_job_status / kill_job all look up the
application_id through JobStore.get_either(application_id), which
scans the on-disk jobs.json for a Job record matching that
application_id. No Job record was being created on the failure
path, so the recovered application_id was unreachable through
the public tool surface — get_job_logs would 404 with 'No Job
found for id=...application_xxx'.
Fix: when confirm_submit_job's failure branch successfully extracts
an application_id from stderr, it now also creates a Job record (same
shape as the success path: 12-char hex job_id, application_id,
script_path, queue, submit_time, connection, yarn_rm_url). The Job's
only job in this case is to act as the lookup index from the
LLM-facing tools into the underlying YARN app.
The job_id is generated even when application_id recovery fails (so
we always set pending.job_id), but the Job record is only put when
application_id is non-None. That keeps the 'spark-submit died locally
before reaching YARN' case clean: no Job record, get_job_logs gets
the standard 'no logs available' error from yarn_client, not a
404 from JobStore.
Tests:
- test_confirm_recovers_application_id_from_failed_spark_submit now
asserts both that pending.application_id is set AND that a Job
record exists and is findable by both job_id AND application_id.
- test_confirm_failed_spark_submit_without_application_id_keeps_it_none
asserts the negative case: no application_id, no Job record.
Fixture fix: _fresh() now rebinds submit.job_store to use tmp_path.
Previously the test wrote to ./data/jobs.json (the real data dir),
which left stale records between test runs and broke any test that
scanned by application_id. This is a latent bug exposed by the new
assertions; fixing it here also makes every other confirm test
hermetic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Spark-submit can fail AFTER YARN has already accepted the application
(lost RM connection, PySpark script exited non-zero on the cluster
side, auth expired mid-submit, ...). In those cases the stderr still
contains 'Submitted application <id>', but the previous
confirm_submit_job implementation threw away that signal and recorded
the pending as FAILED with no application_id. The user had no way to
fetch the YARN logs of the failed job.
Fix:
- run_spark_submit now attaches the failed CompletedProcess to the
SparkSubmitError as .result, so callers can still inspect stderr.
- confirm_submit_job's failure branch tries to parse
(application_id, tracking_url) from exc.result.stderr. If found,
it writes them onto the pending record before re-raising. The
status is still FAILED and the error message is still set — the
user sees the failure normally, but the pending now also has a
log target they can pass to get_application_logs.
- If the stderr has no application_id (e.g. spark-submit failed
locally before reaching YARN), the parse attempt returns None and
the pending is FAILED with no log target — exactly the previous
behavior in that case.
Tests:
- test_confirm_marks_failed_on_spark_submit_error: existing test
now also asserts application_id is None (no result attached).
- test_confirm_recovers_application_id_from_failed_spark_submit:
failed run with 'Submitted application X' in stderr → pending is
FAILED + application_id is set.
- test_confirm_failed_spark_submit_without_application_id_keeps_it_none:
failed run with no application_id in stderr → pending is FAILED
+ application_id is None.
Full suite 248 passed (+2 from 246).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
confirm_submit_job no longer retries on SparkSubmitError. A failed
single attempt is recorded as-is:
- pending.status = 'FAILED' with the error message in pending.error
- the function re-raises the SparkSubmitError so the caller (and the
agent) sees the failure immediately
- the agent can call get_pending_job to see the persisted FAILED
state, including the captured error
The 'retry' code path was hiding real failures behind transient
recovery: a spark-submit that died because the script is broken looks
the same as one that died because YARN was momentarily unreachable.
Without retries, the agent gets a clean FAILED state and a clear
exception to act on, instead of a delayed, ambiguous result.
Manual recovery is still possible: re-calling confirm_submit_job on a
FAILED pending resets it to PENDING and runs a single fresh attempt
(see test_confirm_resets_failed_then_succeeds). The reset path is
useful for 'I fixed the script, try again' flows; it just is no longer
the default on every failure.
Removed:
- the for-attempt loop and time.sleep in submit.py
- import time in submit.py and test_submit_tool.py
- confirm_max_retries and confirm_retry_delay_seconds from
common/config.py (env vars, snapshot, reload)
- 2 retry tests (test_confirm_retries_spark_submit_failure_then_succeeds,
test_confirm_exhausts_retries_and_sets_failed)
- test_confirm_marks_failed_on_spark_submit_error updated to assert
the new single-attempt + raise contract
- test_confirm_resets_failed_and_retries renamed to
test_confirm_resets_failed_then_succeeds (no more retry loop to
exercise)
Full suite 244 passed (3 fewer than before, matching the removed
tests).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The MCP tool was named `generate_job_file` from Stage 2 but it does
NOT generate PySpark code — the calling LLM writes the code in its own
context, and this tool only persists it to a file under
SPARK_EXECUTOR_JOBS_DIR so `spark-submit` can see it. The misleading
`generate_` prefix sent agents (and humans) looking for a code
generator that doesn't exist.
This commit folds three related polish changes into one (split later
with rebase -i if you want them as separate history):
1. The rename itself:
- `tools/generate.py` → `tools/write_job.py`
- `generate_job_file` → `write_job_file`
- `GenerateJobFileRequest` → `WriteJobFileRequest`
- `/generate_job_file` route → `/write_job_file`
- `operation_id="generate_job_file"` → `operation_id="write_job_file"`
The internal helper `core.job_writer.write_job_file` (which just
writes bytes to disk with no SQL guard) is imported with an
`_write_to_disk` alias to avoid the name collision with the
MCP-exposed function in the same module.
The description for the tool now explicitly states 'this tool
does NOT generate PySpark code. The calling LLM is expected to
have already written the code; this tool only persists it.'
2. Skill for LLM agents operating the service
(`docs/superpowers/skills/spark-executor-mcp-operate/SKILL.md`,
449 lines). Covers the 16 tools, the two-step prepare/confirm
flow, the dual-ID contract (job_id vs application_id), the
PendingSubmission state machine, the Connection profile, the
job-file workflow, the error reference, common pitfalls, and a
full end-to-end word-count example.
3. Default `executor_memory` lowered 4G → 2G
(`_DEFAULTS_TO_CONFIRM` in `server.py`). Mirrors the matching
change in `test_mcp_routes.py` and the 5 unit tests that
reference the default. Aligns with the lighter workloads the
service is sized for in its current container profile.
Also tracked in git for the first time:
- `docs/superpowers/plans/2026-06-24-spark-executor-mcp.md`
(the original Stage 1/2/3 design plan, updated to use the new
tool name throughout).
Test rename:
- `tests/unit/test_generate_tool.py` → `test_write_job_tool.py`
- the new test file picks up an extra assertion that the SQL guard
rejects a `DROP TABLE` statement at write time.
243 tests pass (was 242; +1 new SQL-guard assertion). Zero regressions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add an update_pending_job MCP tool that lets callers modify parameters
of a PENDING submission before confirm_submit_job:
- queue, executor_memory, executor_cores, num_executors
- app_name
- extra_args
- script_path (re-validates file existence and SQL guard)
Only PENDING submissions can be updated; SUBMITTED/CANCELLED/FAILED are
rejected with HTTP 400.
Harden confirm_submit_job for unreliable test environments and transient
spark-submit failures:
- Add confirm_max_retries and confirm_retry_delay_seconds settings
(env-configurable).
- SUBMITTED pending returns cached result without re-running spark-submit.
- FAILED pending is reset to PENDING and retried.
- CANCELLED pending is still rejected.
- On success, persist pending as SUBMITTED before creating the in-memory Job
so a job_store failure cannot leave the record as PENDING while the YARN
app is already running.
- Persist tracking_url on PendingSubmission for idempotent returns.
Tests cover retry-then-success, max-retries-exceeded, idempotency,
FAILED reset, and pending-saved-before-job-store.
Restore defaults for queue/executor_memory/executor_cores/num_executors
in prepare_submit_job, but require the caller to explicitly confirm
them. If any defaulted field is omitted, the route returns HTTP 400
listing the defaults and asks the caller to resubmit with explicit
values.
app_name remains required (no meaningful default). extra_args remains
optional.
Tests cover rejection of unconfirmed defaults and acceptance of
explicitly confirmed defaults.
Remove defaults from queue, executor_memory, executor_cores,
num_executors, and app_name in prepare_submit_job. All must now be
explicitly confirmed by the caller.
Also add extra_args to the PendingSubmission snapshot so users can
confirm non-conf spark-submit flags (e.g. --jars, --py-files) at
prepare time; confirm_submit_job passes them through to
build_spark_submit_command.
This is an intentional breaking change to the MCP tool contract:
callers can no longer rely on implicit defaults.
- Add optional app_name to PendingSubmission and prepare_submit_job,
passed through PrepareSubmitJobRequest for user-provided tracking.
- Rewrite PendingStore to shard records by UTC date under
data/pending_jobs/YYYY-MM-DD.json instead of a single monolithic
pending_jobs.json.
- Legacy single-file pending_jobs.json remains readable; first save()
migrates it and renames the old file to avoid double reads.
- Tests cover date sharding, multi-date list, legacy read, legacy
migration, and app_name round-trip.
Two problems with the prior prepare_submit_job flow:
1. The agent could pass a script_path that only existed in its own
context (LLM-generated code not yet on disk) or a path on the host
filesystem that's invisible inside the container. The new check
surfaces this as a 400 with a clear remediation hint instead of
letting spark-submit fail later with an opaque FileNotFoundError -> 500.
2. The container-isolation issue: any path the agent gives is interpreted
inside the container. The two ways a file can legitimately exist there
are (a) generate_job_file(code=...) just wrote it to
SPARK_EXECUTOR_JOBS_DIR (the default ./data/jobs/ is the only
gitignored dir that survives restarts), or (b) a host dir was
mounted via -v. The error message spells both out so an agent can
self-correct.
Implementation:
- submit.py: new _check_script_path() that raises ValueError (-> 400)
when the path is missing, empty, or a directory. Called in both
prepare_submit_job and confirm_submit_job (defense in depth).
- prepare_submit_job checks the connection FIRST (KeyError -> 404)
before the script (ValueError -> 400), so an agent with both problems
sees the more fundamental 'unknown connection' error first.
- server.py / requests.py: route description and Pydantic field
description spell out the generate_job_file pattern so an LLM
reading the tool schema learns the right next step.
8 new tests; 8 existing tests adjusted to create real files (they used
synthetic /tmp/*.py paths that don't exist).
yarn_client.py no longer invokes the 'yarn' binary via subprocess; it uses
httpx against /ws/v1/cluster/apps/* endpoints. This means the runtime image
no longer needs the Hadoop client installation — the only YARN-side
dependency left in the container is the config dir consumed by
spark-submit itself.
New model field:
- Job.yarn_rm_url: str | None
- PendingSubmission.yarn_rm_url: str | None (snapshotted at prepare)
prepare_submit_job snapshots Connection.yarn_rm_url into the pending
record (consistent with the existing master/deploy_mode/spark_conf
snapshot pattern); confirm_submit_job copies it onto the Job so
status/logs/kill can use it without re-looking-up the connection.
Resolution order for the RM URL at runtime:
1. Job.yarn_rm_url (preferred — survives connection edits/deletes)
2. Connection.yarn_rm_url fallback (if a future tool is added that
doesn't go through a Job)
3. YARN_RESOURCE_MANAGER_URL env var
Errors:
- YarnConfigError (HTTP 4xx semantics) when URL is missing/malformed
- YarnError for HTTP 4xx/5xx from the RM, network failures, missing
state field, or unparseable log responses
10 new tests in test_yarn_client.py cover the REST surface:
success, 404, 5xx, missing state field, env-var fallback, malformed
URL, log 404 with log-aggregation hint, kill PUT body shape, and
httpx connection-error wrapping.
