The previous _fetch_logs_via_nodemanager walked
/apps/{appid}/appattempts -> /apps/{appid}/appattempts/{id}/containers,
but YARN ResourceManager does not expose the second hop. Container
info lives on NodeManager and is not reachable from RM REST, so the
fallback returned _logs_unavailable_error on every cluster without
/aggregated-logs (Hadoop 2.x, log aggregation disabled).
Replace the walk with: GET /apps/{appid} -> read app.amContainerLogs
-> GET {amContainerLogs}/stdout. The URL is provided directly by the
app response, so no container enumeration is needed.
Returns AM (driver) container logs only. Executor container logs
still require yarn.log-aggregation-enable=true (primary /aggregated-logs
path); the existing _logs_unavailable_error message already says that,
so the contract is preserved.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three changes the user wanted in the operation skill, all aimed at
making the LLM agent pause and ask the human instead of silently
choosing defaults or auto-confirming:
1. At task submission, ASK for connection (which one to use) and
app_name. The service has no default for app_name and rejects
implicit queue/memory/cores/num_executors defaults, so the
agent has to surface these to the user anyway — better to do
it deliberately than to call a tool, get 400, and re-ask.
2. Before prepare_submit_job, ASK for the submit parameters
(queue, executor_memory, executor_cores, num_executors,
extra_args). The agent tells the user the server-side defaults
it WOULD use so they can approve or override, instead of
picking on their behalf.
3. Before confirm_submit_job, present a one-screen summary of
exactly what will run (connection, app_name, script, queue,
resources, tracking URL) and wait for the user's explicit
affirmative ('yes', 'confirm', 'go', 'y', even an emoji).
A non-answer or 'wait' / 'let me think' is NOT consent. The
agent NEVER calls confirm_submit_job without that go-ahead.
Where the changes live in the skill:
* §0 Mindset — added a 4th 'must' about not auto-confirming,
cross-referencing the new §0.1.
* §0.1 Mandatory user confirmations — new sub-section, spells out
the three checkpoints in detail (what to ask, what counts as
consent, what doesn't).
* §2 The canonical happy path — the numbered list now interleaves
CHECKPOINT 1/2/3 lines with the tool calls, so the pauses are
visually unmistakable.
* §8 Common pitfalls — two new rows: 'auto-confirming without
explicit yes' and 'picking defaults on the user's behalf'.
* §9 End-to-end example — the word-count walkthrough now shows
the full ask flow with sample agent / user dialogue at each
checkpoint, ending in 'only NOW may you call confirm_submit_job'.
The example also adds a 'kill_job: ASK THE USER FIRST' note, since
killing a running YARN app is the same class of irreversible action
as confirming a new one.
File: 449 → 557 lines (+108).
No code changes; tests not affected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The MCP tool was named `generate_job_file` from Stage 2 but it does
NOT generate PySpark code — the calling LLM writes the code in its own
context, and this tool only persists it to a file under
SPARK_EXECUTOR_JOBS_DIR so `spark-submit` can see it. The misleading
`generate_` prefix sent agents (and humans) looking for a code
generator that doesn't exist.
This commit folds three related polish changes into one (split later
with rebase -i if you want them as separate history):
1. The rename itself:
- `tools/generate.py` → `tools/write_job.py`
- `generate_job_file` → `write_job_file`
- `GenerateJobFileRequest` → `WriteJobFileRequest`
- `/generate_job_file` route → `/write_job_file`
- `operation_id="generate_job_file"` → `operation_id="write_job_file"`
The internal helper `core.job_writer.write_job_file` (which just
writes bytes to disk with no SQL guard) is imported with an
`_write_to_disk` alias to avoid the name collision with the
MCP-exposed function in the same module.
The description for the tool now explicitly states 'this tool
does NOT generate PySpark code. The calling LLM is expected to
have already written the code; this tool only persists it.'
2. Skill for LLM agents operating the service
(`docs/superpowers/skills/spark-executor-mcp-operate/SKILL.md`,
449 lines). Covers the 16 tools, the two-step prepare/confirm
flow, the dual-ID contract (job_id vs application_id), the
PendingSubmission state machine, the Connection profile, the
job-file workflow, the error reference, common pitfalls, and a
full end-to-end word-count example.
3. Default `executor_memory` lowered 4G → 2G
(`_DEFAULTS_TO_CONFIRM` in `server.py`). Mirrors the matching
change in `test_mcp_routes.py` and the 5 unit tests that
reference the default. Aligns with the lighter workloads the
service is sized for in its current container profile.
Also tracked in git for the first time:
- `docs/superpowers/plans/2026-06-24-spark-executor-mcp.md`
(the original Stage 1/2/3 design plan, updated to use the new
tool name throughout).
Test rename:
- `tests/unit/test_generate_tool.py` → `test_write_job_tool.py`
- the new test file picks up an extra assertion that the SQL guard
rejects a `DROP TABLE` statement at write time.
243 tests pass (was 242; +1 new SQL-guard assertion). Zero regressions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two user-reported bugs, same root cause: the in-memory JobStore + the
'job_id must be the 12-char hex' tool contract.
Bug 1: 'Unknown job_id' reported frequently
JobStore was a process-local dict (spark_executor/core/job_store.py).
Under gunicorn workers > 1, a job created by confirm_submit_job
landing on worker A was invisible to worker B, so a follow-up
get_job_status / get_job_result / get_job_logs / kill_job landing on
a different worker returned 'Unknown job_id'. Same multi-worker
problem that bit the MCP session layer; only the affected data was
different.
Bug 2: 'get_job_logs frequently confuses job_id and application_id'
confirm_submit_job returns BOTH identifiers in SubmitResult, but
get_job_logs (and friends) only accepted the local 12-char job_id
and never said so in their description. When the agent passed the
YARN application_id, the error message itself was misleading:
'Unknown job_id: application_17400000001_0001' — the agent had
passed an id, just the wrong kind.
This change fixes both at the root:
* JobStore is now JSON-backed at data/jobs.json (atomic tempfile +
os.replace), with cross-process safety via fcntl.flock on a sibling
.lock file. Stage 3's SQLite migration is still planned; the file
format is intentionally simple so it is a straight
'for j in read_all(): db.insert(j)'.
* New JobStore.get_either(uid) looks up by job_id first, then
application_id. All four job-lifecycle tools (get_job_status,
get_job_result, get_job_logs, kill_job) call get_either instead
of get(job_id), so the agent can pass either identifier and get
the same answer.
* The 'neither matched' KeyError now spells out both id forms and
what they look like, so the agent isn't left guessing.
* server.py tool descriptions for the four job tools explicitly
state 'job_id accepts BOTH identifiers' so this is visible to the
LLM at tool-selection time, not only at error time.
Tests:
* test_job_store.py: tmp_path isolation, persistence across
instances, human-readable JSON, corrupt-file resilience,
get_either (by job_id, by application_id, collision preference,
unknown), put idempotency.
* test_{logs,status,kill,result}_tool.py: per-test tmp_path fixture,
'accepts application_id' regression for each tool, and an
explicit assertion that the unknown-id error message mentions
BOTH id forms. test_result_raises_keyerror_for_unknown_job's
match pattern updated for the new message.
242 tests pass (was 226; +16 new). Zero regressions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fastapi-mcp uses each route's OpenAPI operationId as the MCP tool name
exposed to LLM clients (fastapi_mcp/server.py:619-620). When the
operation_id is omitted, FastAPI auto-generates a name like
`prepare_submit_job_prepare_submit_job_post` (function name + path +
method), which is verbose and ugly for the agent.
This change adds an explicit `operation_id=` to all 17 MCP tool
routes in spark_executor/server.py, named after the handler function
(stripping the leading underscore). Concretely, the LLM client now sees
tool names like `prepare_submit_job`, `confirm_submit_job`,
`get_job_status`, etc. — same as the route paths.
`/health` (the only non-MCP GET route) is left untouched; its
auto-generated name (`health_check_health_get`) is never exposed.
Also adds a regression test
`test_every_mcp_route_has_explicit_clean_operation_id` in
tests/integration/test_mcp_routes.py that:
* fetches the live OpenAPI spec,
* asserts every MCP tool route has an operationId,
* rejects FastAPI's auto-generated format (e.g. operationId ending
in `_post` / `_get`),
* asserts no two routes share an operationId (fastapi-mcp keys
its tool registry by name, so duplicates would shadow silently).
All 226 tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MCP sessions live in a per-process in-memory dict on whatever worker
handled the initialize request (StreamableHTTPSessionManager._server_instances).
gunicorn round-robins HTTP requests across workers, so the same
mcp-session-id frequently lands on a different worker and returns
'Session not found' / 'Invalid or expired session ID'. fastapi-mcp
hardcodes stateless=False, and gunicorn has no built-in sticky-session
support, so there is no in-process workaround.
This change keeps the existing workers=2 default (the project does not
want to give up the modest parallelism it gives) but adds:
* A comment block above workers= explaining the constraint, the
workers=1 fix, and the nginx sticky-session escape hatch.
* An on_starting(server) hook that logs a WARNING whenever
workers > 1, naming the symptom and the fix so the misconfig is
loud at deploy time rather than silent at request time.
* Three regression tests in tests/unit/test_gunicorn_config.py
asserting the hook exists, is silent for workers=1, and warns
(with the expected text) for workers>1.
* A 'MCP session affinity' note in CLAUDE.md key conventions so
the constraint is recorded for future maintainers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docker-entrypoint.sh now:
- Reads SPARK_EXECUTOR_SPARK_SUBMIT_BIN with default spark-submit and
validates that the configured binary resolves in PATH.
- If the configured binary is an absolute path, prepends its directory
to PATH so subprocess.run can find it.
- Still validates JAVA_HOME/bin/java and SPARK_HOME/bin/spark-submit.
- Writes /etc/profile.d/spark_executor_env.sh with the resolved
JAVA_HOME, SPARK_HOME, SPARK_EXECUTOR_SPARK_SUBMIT_BIN, and PATH so
interactive `docker exec` shells see the same environment as the
main gunicorn process.
This fixes two reported issues:
1. SPARK_EXECUTOR_SPARK_SUBMIT_BIN was ignored because the binary was
not guaranteed to be on PATH.
2. `docker exec` shells did not see JAVA_HOME/SPARK_HOME in PATH.
Add an update_pending_job MCP tool that lets callers modify parameters
of a PENDING submission before confirm_submit_job:
- queue, executor_memory, executor_cores, num_executors
- app_name
- extra_args
- script_path (re-validates file existence and SQL guard)
Only PENDING submissions can be updated; SUBMITTED/CANCELLED/FAILED are
rejected with HTTP 400.
Delete compression="gz" from both debug and info file sinks so logs
are kept as plain text files. Simplifies log inspection and avoids the
small CPU overhead of nightly gzip compression.
Older pending records (created before app_name was required) may not
have the field. If such a record exists in the date-sharded pending
store, any operation that loads the store fails Pydantic validation and
blocks prepare/list/get/cancel.
Make app_name optional with default None on the persistence model while
keeping it required in PrepareSubmitJobRequest, so new submissions still
must provide it but old records can be loaded and cancelled.
Harden confirm_submit_job for unreliable test environments and transient
spark-submit failures:
- Add confirm_max_retries and confirm_retry_delay_seconds settings
(env-configurable).
- SUBMITTED pending returns cached result without re-running spark-submit.
- FAILED pending is reset to PENDING and retried.
- CANCELLED pending is still rejected.
- On success, persist pending as SUBMITTED before creating the in-memory Job
so a job_store failure cannot leave the record as PENDING while the YARN
app is already running.
- Persist tracking_url on PendingSubmission for idempotent returns.
Tests cover retry-then-success, max-retries-exceeded, idempotency,
FAILED reset, and pending-saved-before-job-store.
Restore defaults for queue/executor_memory/executor_cores/num_executors
in prepare_submit_job, but require the caller to explicitly confirm
them. If any defaulted field is omitted, the route returns HTTP 400
listing the defaults and asks the caller to resubmit with explicit
values.
app_name remains required (no meaningful default). extra_args remains
optional.
Tests cover rejection of unconfirmed defaults and acceptance of
explicitly confirmed defaults.
Remove defaults from queue, executor_memory, executor_cores,
num_executors, and app_name in prepare_submit_job. All must now be
explicitly confirmed by the caller.
Also add extra_args to the PendingSubmission snapshot so users can
confirm non-conf spark-submit flags (e.g. --jars, --py-files) at
prepare time; confirm_submit_job passes them through to
build_spark_submit_command.
This is an intentional breaking change to the MCP tool contract:
callers can no longer rely on implicit defaults.
- Add optional app_name to PendingSubmission and prepare_submit_job,
passed through PrepareSubmitJobRequest for user-provided tracking.
- Rewrite PendingStore to shard records by UTC date under
data/pending_jobs/YYYY-MM-DD.json instead of a single monolithic
pending_jobs.json.
- Legacy single-file pending_jobs.json remains readable; first save()
migrates it and renames the old file to avoid double reads.
- Tests cover date sharding, multi-date list, legacy read, legacy
migration, and app_name round-trip.
YARN ResourceManager REST can return either JSON or XML depending on
the Accept / Content-Type headers. We always parse responses with
resp.json(), so an ambiguous or XML response would break the client.
Add Accept: application/json to every _request call so the response
format is pinned unambiguously. This is compatible with both Hadoop
2.x/CDH 5 and Hadoop 3.x RM endpoints.
Tests: assert that httpx.request receives headers={"Accept": "application/json"}.
Add per-Connection ssl_verify / ssl_ca_bundle plus global defaults so
CDH 5 / on-prem clusters with self-signed certs or custom CA bundles can
be queried without patching code.
- Connection gets ssl_verify (bool|None) and ssl_ca_bundle (str|None)
- Settings gets ssl_verify_default and ssl_ca_bundle_default
- New YarnClientConfig dataclass carries the resolved verify= value
- _request passes verify= through to httpx.request
- All public yarn_client functions now take YarnClientConfig instead of
a bare yarn_rm_url string; tool call sites resolve the Connection
- SaveConnectionRequest exposes the two new fields
Tests cover per-connection CA bundle, per-connection verify=False,
global default fallback, and connection-not-found error.
The function had hardcoded --master / --deploy-mode / --queue /
--executor-memory / --executor-cores / --num-executors plus a
`spark_conf` dict for --conf, with no way to pass other flags like
--jars, --py-files, --files, --driver-memory, --name, etc.
Add `extra_args: dict[str, str] | None = None` that emits `--{key}
{value}` pairs, placed after the --conf loop and before the script
path. Default None preserves the existing cmd shape exactly.
Structured (dict) instead of raw list[str] so LLM agents writing
tool-call payloads can't accidentally pack flag+value into a single
string. Key order in the dict is preserved.
Tests: 4 cases — None default, single pair, multiple pairs in order,
no collision with spark_conf. uv run pytest -> 175 passed.
`get_job_status` returns YARN state + the raw response blob, so the
terminal fields (finalStatus, diagnostics, trackingUrl, startedTime,
finishedTime) are buried inside `raw` and not surfaced in a structured
form. Add a new tool that parses them.
`get_job_result(job_id)` reuses `yarn_client.get_application_status` and
extracts:
- finalStatus (SUCCEEDED / FAILED / KILLED / UNDEFINED)
- diagnostics (YARN final message)
- tracking_url (Spark Web UI)
- started_time / finished_time (epoch ms)
All five fields are optional: running jobs have no `finishedTime`, and
older YARN versions (CDH 5 / H2) may omit some fields. Missing fields
stay None — never raise.
Coexistence with `get_job_status` is intentional: the latter is for
polling the running YARN state, the former is the terminal view.
Tests: 4 cases — happy path, running job (no finishedTime), bare-minimum
raw (all optionals None), unknown job_id raises KeyError. uv run pytest
-> 171 passed.
`/ws/v1/cluster/apps/{id}/aggregated-logs` is a Hadoop 3+ endpoint; on
CDH 5 / Hadoop 2.6 it returns 404 and the old code surfaced a hard error
to the user. Add capability detection in `get_application_logs`: on 404 or
501 from the aggregated-logs endpoint, walk `appattempts` -> `containers`
-> NodeManager `/node/containerlogs/{id}/{user}/` and concatenate the
results.
5xx on the aggregated-logs endpoint still raises immediately (no fallback
attempted for genuine server errors). The H3 fast path is unchanged.
Constraints honored:
- No `subprocess` / `yarn` CLI. The whole point of yarn_client.py is to
avoid that dependency, so the fallback stays on `httpx` + REST.
- No new pyproject.toml / uv.lock dependencies.
- Public signature of `get_application_logs` unchanged; `tools/logs.py`
needs no changes.
- No version detection — pure capability detection (try, react to status).
Tests: replaced the old "raise on 404" test with 5 new cases (H3 fast
path, NM fallback, NM path empty, multiple containers, 5xx immediate
raise). `uv run pytest` -> 167 passed.
The Dockerfile installs openjdk-17-jre-headless and sets
JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64, but the entrypoint
default was still /usr/lib/jvm/java-11-openjdk-amd64. The
validation caught the mismatch at container start:
[entrypoint] FATAL: JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
but /usr/lib/jvm/java-11-openjdk-amd64/bin/java is missing or
not executable
That's exactly the failure mode the validation is designed to catch
- the deployment config got out of sync. Aligning all four files
(Dockerfile, entrypoint, .env.example, docker-compose.yml) to
java-17-openjdk-amd64.
No app code touched. 165/165 still pass.
Hardcoding 'spark-submit' as cmd[0] in build_spark_submit_command
breaks for hosts where:
- both Spark 1.x and 2.x/3.x are installed and 'spark-submit' resolves
to the wrong one (use 'spark2-submit' or 'spark3-submit' explicitly)
- the user wants to launch via the PySpark entrypoint ('pyspark')
- a custom wrapper script sits on PATH (e.g. a credentials-injecting
'spark-submit-wrapper')
New env var SPARK_EXECUTOR_SPARK_SUBMIT_BIN. Default is 'spark-submit'
(preserves the current behavior for everyone). Override in .env /
docker-compose.yml to change.
common/config.py:
- new Settings.spark_submit_bin field
- env-var resolution in from_env() with default 'spark-submit'
- included in reload() so tests work
spark_executor/core/spark_submit.py:
- cmd[0] reads settings.spark_submit_bin (was hardcoded 'spark-submit')
.env.example: new section with the override and example values.
docker-compose.yml: forwards the var with the standard 'spark-submit'
default.
Tests: 2 new (settings.spark_submit_bin='spark2-submit', ='pyspark')
plus existing tests updated to use the settings-restore fixture so
mutations don't leak between tests.
165/163 still pass.
Three fixes requested:
1. Spark 4.1.2 -> Spark 3.1.2 (Hadoop 2.7 prebuilt). Compatible with the
JDK 11 build below and a more conservative choice for production.
2. JDK 17 -> JDK 11. openjdk-11-jre-headless package; JAVA_HOME points
at /usr/lib/jvm/java-11-openjdk-amd64.
3. /usr/bin/env 'bach' no such file: defensive shebang fix. The
downloaded spark-3.1.2-bin-hadoop2.7.tgz happens to have a clean
shebang, but older 4.x distributions (and any future typo in a
release) would break the same way we just saw. The Dockerfile now
runs:
find /opt/spark/bin -type f -exec sed -i '1s|^.*$|#!/usr/bin/env bash|' {} +
which rewrites the first line of every bin/* script to a known-good
shebang. Idempotent, defensive, costs nothing.
4. docker-entrypoint.sh: simplified and made validation explicit.
Old version used an awk/sed pipeline to strip /usr/lib/jvm/ and
/opt/spark/bin from the existing PATH before prepending the new
values. That had a subtle bug: if the new JAVA_HOME was itself
under /usr/lib/jvm/ (e.g. /usr/lib/jvm/java-11-openjdk-amd64), the
strip would remove the new path too. New version just prepends the
resolved paths and leaves the old PATH alone. The new paths win
because they come first.
5. docker-entrypoint.sh: now validates the resolved paths BEFORE
exporting them. If JAVA_HOME/bin/java or SPARK_HOME/bin/spark-submit
are missing, the container fails fast with a clear hint instead of
letting a job submission die with an opaque 'no such file'. Also
logs the effective 'java' and 'spark-submit' paths (and java
version) to stderr at every start, so docker logs make the
resolution visible.
6. .env.example + docker-compose.yml: default JAVA_HOME updated to
/usr/lib/jvm/java-11-openjdk-amd64. Spark client 3.1.2 (hadoop2.7)
noted in the comment as the working combo.
163/146 still pass (no code changes to the app; Dockerfile + entrypoint
+ docs only). The new entrypoint was smoke-tested locally: validation
fires as expected (the local dev box has no JDK 11, which is exactly
the kind of misconfig the validation now catches at container start).
Closes the review-and-edit loop for LLM-generated PySpark code:
generate_job_file(code=...) -> {script_path}
read_job_file(script_path=...) -> {content, path, size}
update_job_file(path, content) -> {path, bytes_written}
prepare_submit_job(path) -> {pending_id, ...}
Or, in a single edit cycle:
1. generate (LLM writes initial draft)
2. read (LLM or human inspects)
3. update (overwrite with edited version)
4. prepare (submit for two-step confirmation)
Safety:
- read_job_file has no path restriction (read-only; useful for
inspecting any file the agent can see: scripts, logs/, hadoop-conf/)
- update_job_file is sandboxed to settings.jobs_dir (the same dir
generate_job_file writes to). Rejects paths outside that tree,
including ../-traversal attempts. This protects host-mounted
configs (/etc/passwd, hadoop-conf/*) from being overwritten by
the agent.
- 1 MB cap on both reads and writes so MCP responses stay bounded.
Pydantic body models (ReadJobFileRequest, UpdateJobFileRequest) follow
the Stage 1 pattern so tools/call roundtrips long code strings without
the FastAPI query-length 422.
Tests (13 new):
- 9 unit tests: read success/missing/empty/dir, update success/outside/
relative-escape/missing/oversize/1mb+1, full edit cycle round-trip
- 4 integration tests: read via MCP, missing file 400, write+readback
via MCP, outside-jobs_dir rejection via MCP
163/146 still pass. Live verified end-to-end: generate -> read v1
-> update -> read v2; update /etc/passwd correctly 400'd with
'script_path must be under ... data/jobs/'.
Production log noise: every container start was emitting 14 INFO lines
listing every tool name and full description, which is dev-time
debugging info that doesn't belong in INFO.
Now:
- INFO: one line per service with the count
'MCP service \042spark_executor\042: 14 tool(s) registered'
- DEBUG: per-tool name + description, prefixed with the service name
so multi-service deploys are grep-friendly
' spark_executor._prepare_submit_job_prepare_submit_job_post - ...'
Operators can flip SPARK_EXECUTOR_LOG_LEVEL=INFO for production (no
per-tool spam) or DEBUG for dev (full inventory).
146/146 still pass. Live verified: at INFO level only the count line
shows; at DEBUG level both the count and per-tool lines show.
The agent can now write PySpark that runs DROP/DELETE/UPDATE/etc. on
production tables. Add a static guard that rejects anything other than
SELECT and INSERT at two enforcement points:
1. generate_job_file: validates BEFORE writing to disk. Agent gets
immediate feedback ('rewrite to use only SELECT/INSERT') rather
than learning at submit time.
2. prepare_submit_job: re-validates the script content (reads the
file) as a defense-in-depth check. Catches host-mounted files,
manually-edited files, anything that bypassed generate_job_file.
How it works:
- common/sql_guard.py extracts Python string literals whose first
keyword is a SQL verb (catches spark.sql('...'), f-strings, and any
raw SQL literal)
- sqlparse splits each literal into statements; we check the first
keyword against the policy (SELECT/INSERT/WITH allowed; DROP,
DELETE, UPDATE, TRUNCATE, ALTER, CREATE, REPLACE, MERGE, GRANT,
REVOKE, SET, SHOW, KILL, EXEC, etc. forbidden)
- WITH recurses into the CTE body to catch WITH x AS (DROP ...) ...
- The MCP layer maps ValueError -> HTTP 400 (existing handler)
Test coverage:
- 28 unit tests in test_sql_guard.py cover: extraction (single/double/
f-string, English false positives, multi-literal), statement
classification (SELECT, INSERT, DROP, DELETE, UPDATE, TRUNCATE,
ALTER, CREATE, multi-statement, CTE bodies, comments)
- 2 integration tests verify MCP layer returns 400 with the policy
explanation at both generate_job_file and prepare_submit_job
Limitations (documented in sql_guard.py docstring):
- f-strings where the SQL is built at runtime (e.g. f'SELECT * FROM
{user_input}') look like SELECTs at static-analysis time. The
guard catches the static literal; the runtime substitution is the
caller's responsibility.
- pyspark.sql.functions.expr('...') accepts SQL inline; not currently
caught. (Future work.)
146/146 still pass. Live verified: DROP TABLE -> MCP 400 with policy
explanation; SELECT -> MCP 200 + file written to ./data/jobs/.
The lifespan was hardcoded to mount just one MCP service
(spark_executor_app at /spark-executor-mcp). To add a second MCP
service, you'd have to edit the lifespan function body — error-prone
and not obvious where to add a new entry.
Refactor:
- New McpService dataclass (name, app, mount_path)
- MCP_SERVICES list at module top — adding a new service is one append
- Lifespan iterates MCP_SERVICES; the per-service mount + tool-list
logic is now a helper (_log_mcp_tools)
- Tool-list log now prefixes the service name:
'MCP tools registered (spark_executor, 14):'
instead of just '(14):', so multi-service startups are unambiguous
116/116 still pass. Live verified: gunicorn 2-worker startup logs
each service with its tool list and the source app version.
Future services just need a McpService entry — no lifespan edit. The
MCP_SERVICES list has a docstring example showing the one-liner.
Three problems with the prior Dockerfile:
1. JAVA_HOME and SPARK_HOME were hardcoded at build time (ENV ...),
so docker-compose / .env overrides had no effect.
2. PATH was constructed at build time using those hardcoded values,
so even if you could override the env vars, PATH would still
reference the old literal paths (e.g. /usr/lib/jvm/java-17-openjdk-amd64/bin
hardcoded into PATH at image build).
3. There was no .env.example entry for either, so operators had no
template to follow.
Fix:
- docker-entrypoint.sh: re-derives PATH from the (possibly overridden)
JAVA_HOME and SPARK_HOME at every container start. Strips any stale
JDK/Spark bin dirs from PATH first so a restart with a new override
actually changes which / resolve.
- Dockerfile: COPY + ENTRYPOINT [entrypoint.sh], CMD [gunicorn main:app].
The existing ENV JAVA_HOME and ENV SPARK_HOME stay as the defaults
so the image works out of the box; users override via .env.
- .env.example: new 'Java + Spark install paths' section, default
values match the Dockerfile, comments explain the entrypoint
re-derivation.
- docker-compose.yml: forwards JAVA_HOME and SPARK_HOME with
container-side defaults matching the Dockerfile.
Verified:
- 116/116 tests still pass
- Direct entrypoint run with JAVA_HOME=/fake/jdk shows PATH rebuilt as
/app/.venv/bin:/fake/jdk/bin:/fake/spark/bin:... (override took effect)
Note: uses awk instead of 'paste -sd:' for portability across macOS
(BSD paste doesn't support -s) and Linux (GNU does).
common/logging.py used hardcoded 'data/logs/{debug,info}/' paths,
which forced logs into the same dir as connections.json. Operators
couldn't put logs on a dedicated volume or a different disk for
retention policy reasons.
Now the base log dir comes from settings.log_dir (env var
SPARK_EXECUTOR_LOG_DIR, default <data_dir>/logs). Subdirs debug/ and
info/ are auto-created. Same behavior in dev (./data/logs/), but in
prod you can do:
SPARK_EXECUTOR_LOG_DIR=/var/log/spark-executor # dedicated volume
SPARK_EXECUTOR_LOG_DIR=/mnt/slow-storage/logs # cold storage for old logs
Other changes:
- .env.example: new SPARK_EXECUTOR_LOG_DIR entry, split into its own
'Loguru file sinks' section
- docker-compose.yml: forwards SPARK_EXECUTOR_LOG_DIR with the
container-side default /app/data/logs (the existing ./data:/app/data
volume mount already covers this)
- common/config.py: Settings gets a new log_dir field, defaults
derived from data_dir; reload() also resets it
116/116 still pass. Live smoke verified: with
SPARK_EXECUTOR_LOG_DIR=/tmp/spark-logs, both
/tmp/spark-logs/info/2026-06-25.log
/tmp/spark-logs/debug/2026-06-25.log
are created on the first request.
.env.example now lists every env var read by the app, with a comment
explaining the meaning and the default. The previously-missing ones
(from the common/config.py refactor) are added:
- SPARK_EXECUTOR_DATA_DIR
- SPARK_EXECUTOR_JOBS_DIR
- SPARK_EXECUTOR_LOG_LEVEL
docker-compose.yml forwards all of them with sensible container-side
defaults (SPARK_EXECUTOR_DATA_DIR=/app/data, which is the volume
mount point from the host).
Split the env block in compose into two commented sections
('common/config.py knobs' vs 'gunicorn.conf.py knobs') so it's clear
which file each one is consumed by.
The GUNICORN_LOGLEVEL entry is also added (was missing). All other
GUNICORN_* knobs were already documented.
116/116 still pass.
Before: SPARK_EXECUTOR_DATA_DIR, SPARK_EXECUTOR_JOBS_DIR, and
YARN_RESOURCE_MANAGER_URL were each read directly via os.environ.get()
inside the module that used them. Log level was hardcoded DEBUG in
common/logging.py. There was no single file showing what the full set
of env vars the app reads is.
After: common/config.py defines a single Settings dataclass that
reads all env vars at import time and exposes them as fields on a
module-level singleton. App code uses "from common.config import
settings; settings.data_dir" etc. New SPARK_EXECUTOR_LOG_LEVEL env
var controls stderr + info file verbosity (debug file always gets full
DEBUG).
Improvements:
- One file lists every env var the app reads (was: grep the codebase)
- Tests can monkeypatch fields on the settings singleton directly
instead of monkeypatching the env + reloading
- Adding a new env var means adding one field in config.py, not
editing 3+ call sites
- settings.reload() method for tests that prefer env-var style
Out of scope (kept where they are):
- GUNICORN_* env vars live in gunicorn.conf.py (gunicorn concept)
- PYTHONUNBUFFERED in Dockerfile (Python runtime flag)
- SPARK_SUBMIT_OPTS not in config (JVM flag, not Python)
Test changes:
- test_job_writer.py: settings.jobs_dir instead of monkeypatching
SPARK_EXECUTOR_JOBS_DIR
- test_yarn_client.py: settings.yarn_resource_manager_url instead of
monkeypatching YARN_RESOURCE_MANAGER_URL
- test_generate_tool.py: same as job_writer
- Each test file gets an autouse fixture that snapshots+restores
settings so one test mutation does not leak into the next
116/116 still pass. Live verified: SPARK_EXECUTOR_LOG_LEVEL=INFO
suppresses DEBUG loguru output as expected.
Two problems with the prior prepare_submit_job flow:
1. The agent could pass a script_path that only existed in its own
context (LLM-generated code not yet on disk) or a path on the host
filesystem that's invisible inside the container. The new check
surfaces this as a 400 with a clear remediation hint instead of
letting spark-submit fail later with an opaque FileNotFoundError -> 500.
2. The container-isolation issue: any path the agent gives is interpreted
inside the container. The two ways a file can legitimately exist there
are (a) generate_job_file(code=...) just wrote it to
SPARK_EXECUTOR_JOBS_DIR (the default ./data/jobs/ is the only
gitignored dir that survives restarts), or (b) a host dir was
mounted via -v. The error message spells both out so an agent can
self-correct.
Implementation:
- submit.py: new _check_script_path() that raises ValueError (-> 400)
when the path is missing, empty, or a directory. Called in both
prepare_submit_job and confirm_submit_job (defense in depth).
- prepare_submit_job checks the connection FIRST (KeyError -> 404)
before the script (ValueError -> 400), so an agent with both problems
sees the more fundamental 'unknown connection' error first.
- server.py / requests.py: route description and Pydantic field
description spell out the generate_job_file pattern so an LLM
reading the tool schema learns the right next step.
8 new tests; 8 existing tests adjusted to create real files (they used
synthetic /tmp/*.py paths that don't exist).
Exposes a single new MCP tool: generate_job_file(code) -> {script_path}.
Wiring follows the Stage 1 conventions (Pydantic body model for the
route, loguru DEBUG/INFO logging, summary+description for the startup
log, MCP arg-pattern via the body model so tools/call roundtrips long
code strings without 422-ing on FastAPI query length limits).
Flow:
1. LLM calls generate_job_file(code=...) -> {script_path}
2. LLM calls prepare_submit_job(connection=...,
script_path=...) -> {pending_id}
3. User reviews the file + pending record
4. LLM calls confirm_submit_job(pending_id=...) -> spark-submit runs
The output directory is SPARK_EXECUTOR_JOBS_DIR (default ./data/jobs/,
gitignored, persists across container restarts via the existing
./data volume mount in docker-compose.yml).
3 new tests:
- Unit: env-var override, default fallback in tmp cwd
- Integration: end-to-end body call with a > FastAPI-query-limit code
string (the canary test that would have caught the Stage 1
query-params-422 bug)
Live MCP smoke verified: tools/list shows 14 tools (13 from Stage 1 +
the new one), tools/call generate_job_file returns the absolute path
under ./data/jobs/.
spark_executor/core/job_writer.py provides write_job_file(code, jobs_dir=None)
that resolves the output directory in this order:
1. explicit jobs_dir argument (for tests + programmatic override)
2. SPARK_EXECUTOR_JOBS_DIR environment variable (operator override)
3. DEFAULT_JOBS_DIR = './data/jobs' (gitignored, persists via volume mount)
Filenames are job_<UTC-stamp>_<random-hex>.py — same-second writes still
get distinct names via the random suffix, so concurrent agents won't
clobber each other.
8 tests cover: arg>env>default precedence, dir auto-creation, env-var
override, default fallback in a tmp cwd, and uniqueness under concurrent
calls.
YARN is the dominant use case, and the value of master for YARN is the
literal string 'yarn' (not a URL) — there is no useful variation. Making
it a default saves every YARN user from typing the same value every time:
save_connection(name='prod') # master defaults to 'yarn'
save_connection(name='dev', master='spark://10.0.0.5:7077') # override
Added a field_validator on master that catches common typos
('yarn-cluster', 'http://...', empty string) at save time with a clear
error message instead of letting them reach spark-submit.
Backward compatible: existing Connection records without an explicit
master field default to 'yarn' on load (Pydantic applies defaults on
deserialization), so already-saved data still works.
3 new tests cover the default, the validator, and that other cluster
managers (spark://, k8s://, local[N]) still pass.
docker-compose.yml:
- Builds the Dockerfile, exposes :8000
- Mounts ./data (persistent) and ./hadoop-conf (ro) for cluster configs
- Forwards YARN_RESOURCE_MANAGER_URL via env var (fallback for REST client)
- Healthcheck via /openapi.json (cheap liveness probe; MCP initialize
handshake would be more accurate but requires session-id plumbing)
- restart: unless-stopped for production
- Resource limit section commented out (server itself is lightweight)
.env.example:
- Documents the env vars compose expects
- .env is gitignored
main.py lifespan now enumerates every MCP tool with its description at
startup so operators can see at a glance what the server exposes.
The auto-generated '### Responses:' suffix that fastapi-mcp appends to
each tool description is truncated before logging to keep the startup
log scannable.
Docstrings added to all 12 route handlers in server.py so the MCP tool
descriptions are meaningful (without them, fastapi-mcp would emit empty
descriptions). Each route now carries a summary + description used by
the auto-generated OpenAPI schema.
Spark 4.1.2 requires Java 17 at runtime (it boots a JVM via the
bin/spark-submit shell script). The previous Dockerfile only installed
curl, ca-certificates, and tar — leaving spark-submit unable to find
the 'java' binary.
A JRE is sufficient (not the full JDK) because spark-submit does not
compile Java at runtime — it loads pre-built JARs from $SPARK_HOME/jars/.
JRE-only keeps the image ~160 MB smaller than JDK-headless.
Sets JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 and prepends it to
PATH so spark-submit (which checks JAVA_HOME) finds the right JVM.
Combined with the prior yarn-REST rewrite, the runtime image now needs
exactly three things in addition to python:3.12-slim:
- uv (from ghcr.io/astral-sh/uv)
- openjdk-17-jre-headless
- Spark 4.1.2 from Tsinghua mirror
yarn_client.py no longer invokes the 'yarn' binary via subprocess; it uses
httpx against /ws/v1/cluster/apps/* endpoints. This means the runtime image
no longer needs the Hadoop client installation — the only YARN-side
dependency left in the container is the config dir consumed by
spark-submit itself.
New model field:
- Job.yarn_rm_url: str | None
- PendingSubmission.yarn_rm_url: str | None (snapshotted at prepare)
prepare_submit_job snapshots Connection.yarn_rm_url into the pending
record (consistent with the existing master/deploy_mode/spark_conf
snapshot pattern); confirm_submit_job copies it onto the Job so
status/logs/kill can use it without re-looking-up the connection.
Resolution order for the RM URL at runtime:
1. Job.yarn_rm_url (preferred — survives connection edits/deletes)
2. Connection.yarn_rm_url fallback (if a future tool is added that
doesn't go through a Job)
3. YARN_RESOURCE_MANAGER_URL env var
Errors:
- YarnConfigError (HTTP 4xx semantics) when URL is missing/malformed
- YarnError for HTTP 4xx/5xx from the RM, network failures, missing
state field, or unparseable log responses
10 new tests in test_yarn_client.py cover the REST surface:
success, 404, 5xx, missing state field, env-var fallback, malformed
URL, log 404 with log-aggregation hint, kill PUT body shape, and
httpx connection-error wrapping.
- common/logging.py: improve format to timestamp|LEVEL|module:func:line - message
- core/ layer: DEBUG log every subprocess invocation (cmd, rc, byte counts),
JSON load/dump events, parsed application_id. ERROR log on failures.
- tools/ layer: DEBUG log every public tool entry with key parameters,
INFO log on business outcomes (saved/submitted/killed/...).
- New tests/unit/test_logging.py: capture loguru output via in-memory sink
and assert DEBUG + INFO messages are emitted for representative flows.
Tool functions raise KeyError for unknown ids and ValueError for invalid
state transitions. Without handlers, FastAPI returned a bare 500
"Internal Server Error" to MCP clients — useless for an agent that needs
to know whether the id was wrong vs the cluster was down.
KeyError -> 404 with the message ("Unknown job_id: foo").
ValueError -> 400 with the message (e.g. "pending_id p_xyz is in status
'CANCELLED', not PENDING").
The plan's route signatures used query parameters, which worked for direct
HTTP callers and unit tests, but fastapi-mcp's HTTP transport passes
tools/call arguments as a JSON body. dict-typed parameters like
spark_conf arrived as a string and the route returned 422.
Refactor each route to take a single Pydantic body model (saved in
spark_executor/tools/requests.py). Underlying tool functions unchanged.
Integration tests in tests/integration/test_mcp_routes.py now exercise
the full body-based roundtrip (save_connection with spark_conf, prepare
→ list → get, list_connections with empty body).
