ClaudeandClaude Fable 5
523e6a9c76
The MCP tool was named `generate_job_file` from Stage 2 but it does
NOT generate PySpark code — the calling LLM writes the code in its own
context, and this tool only persists it to a file under
SPARK_EXECUTOR_JOBS_DIR so `spark-submit` can see it. The misleading
`generate_` prefix sent agents (and humans) looking for a code
generator that doesn't exist.
This commit folds three related polish changes into one (split later
with rebase -i if you want them as separate history):
1. The rename itself:
- `tools/generate.py` → `tools/write_job.py`
- `generate_job_file` → `write_job_file`
- `GenerateJobFileRequest` → `WriteJobFileRequest`
- `/generate_job_file` route → `/write_job_file`
- `operation_id="generate_job_file"` → `operation_id="write_job_file"`
The internal helper `core.job_writer.write_job_file` (which just
writes bytes to disk with no SQL guard) is imported with an
`_write_to_disk` alias to avoid the name collision with the
MCP-exposed function in the same module.
The description for the tool now explicitly states 'this tool
does NOT generate PySpark code. The calling LLM is expected to
have already written the code; this tool only persists it.'
2. Skill for LLM agents operating the service
(`docs/superpowers/skills/spark-executor-mcp-operate/SKILL.md`,
449 lines). Covers the 16 tools, the two-step prepare/confirm
flow, the dual-ID contract (job_id vs application_id), the
PendingSubmission state machine, the Connection profile, the
job-file workflow, the error reference, common pitfalls, and a
full end-to-end word-count example.
3. Default `executor_memory` lowered 4G → 2G
(`_DEFAULTS_TO_CONFIRM` in `server.py`). Mirrors the matching
change in `test_mcp_routes.py` and the 5 unit tests that
reference the default. Aligns with the lighter workloads the
service is sized for in its current container profile.
Also tracked in git for the first time:
- `docs/superpowers/plans/2026-06-24-spark-executor-mcp.md`
(the original Stage 1/2/3 design plan, updated to use the new
tool name throughout).
Test rename:
- `tests/unit/test_generate_tool.py` → `test_write_job_tool.py`
- the new test file picks up an extra assertion that the SQL guard
rejects a `DROP TABLE` statement at write time.
243 tests pass (was 242; +1 new SQL-guard assertion). Zero regressions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
74 lines
2.5 KiB
Python
74 lines
2.5 KiB
Python
# coding=utf-8
|
|
import io
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from spark_executor.core import connection_store, pending_store
|
|
from spark_executor.core.connection_store import ConnectionStore
|
|
from spark_executor.core.pending_store import PendingStore
|
|
from spark_executor.models import Connection
|
|
from spark_executor.tools import connections, submit
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _fresh_stores(tmp_path: Path, monkeypatch):
|
|
monkeypatch.setattr(connection_store, "DEFAULT_DATA_DIR", str(tmp_path))
|
|
monkeypatch.setattr(connection_store, "store", ConnectionStore())
|
|
monkeypatch.setattr(pending_store, "DEFAULT_DATA_DIR", str(tmp_path))
|
|
monkeypatch.setattr(pending_store, "store", PendingStore())
|
|
connections.store = connection_store.store
|
|
submit.conn_store = connection_store.store
|
|
submit.pending_store = pending_store.store
|
|
submit.job_store = submit.job_store.__class__() # fresh in-memory job store
|
|
|
|
|
|
@pytest.fixture
|
|
def log_capture():
|
|
"""Attach an in-memory sink to loguru so tests can assert on emitted lines."""
|
|
from common.logging import logger
|
|
buf = io.StringIO()
|
|
handler_id = logger.add(buf, level="DEBUG", format="{level}|{message}")
|
|
yield buf
|
|
logger.remove(handler_id)
|
|
|
|
|
|
def test_save_connection_emits_info_log(log_capture):
|
|
connections.save_connection(name="prod", master="yarn")
|
|
text = log_capture.getvalue()
|
|
assert "INFO" in text
|
|
assert "save_connection enter" in text
|
|
assert "DEBUG" in text
|
|
assert "connection saved" in text
|
|
|
|
|
|
def test_prepare_submit_job_emits_debug_and_info(log_capture, tmp_path):
|
|
connections.save_connection(name="prod", master="yarn", deploy_mode="cluster")
|
|
script = tmp_path / "demo.py"
|
|
script.write_text("print('hi')\n")
|
|
log_capture.truncate(0); log_capture.seek(0)
|
|
submit.prepare_submit_job(
|
|
connection="prod",
|
|
script_path=str(script),
|
|
queue="research",
|
|
executor_memory="2G",
|
|
executor_cores=2,
|
|
num_executors=2,
|
|
app_name="test-app",
|
|
)
|
|
text = log_capture.getvalue()
|
|
assert "DEBUG|prepare_submit_job enter" in text
|
|
assert "INFO|prepare_submit_job ok" in text
|
|
assert f"script_path={script}" in text
|
|
assert "queue=research" in text
|
|
|
|
|
|
def test_get_unknown_pending_job_emits_debug(log_capture):
|
|
log_capture.truncate(0); log_capture.seek(0)
|
|
import pytest as _pytest
|
|
with _pytest.raises(KeyError):
|
|
submit.get_pending_job("p_doesnotexist")
|
|
text = log_capture.getvalue()
|
|
assert "DEBUG|get_pending_job enter" in text
|
|
assert "p_doesnotexist" in text
|