ClaudeandClaude Fable 5
523e6a9c76
The MCP tool was named `generate_job_file` from Stage 2 but it does
NOT generate PySpark code — the calling LLM writes the code in its own
context, and this tool only persists it to a file under
SPARK_EXECUTOR_JOBS_DIR so `spark-submit` can see it. The misleading
`generate_` prefix sent agents (and humans) looking for a code
generator that doesn't exist.
This commit folds three related polish changes into one (split later
with rebase -i if you want them as separate history):
1. The rename itself:
- `tools/generate.py` → `tools/write_job.py`
- `generate_job_file` → `write_job_file`
- `GenerateJobFileRequest` → `WriteJobFileRequest`
- `/generate_job_file` route → `/write_job_file`
- `operation_id="generate_job_file"` → `operation_id="write_job_file"`
The internal helper `core.job_writer.write_job_file` (which just
writes bytes to disk with no SQL guard) is imported with an
`_write_to_disk` alias to avoid the name collision with the
MCP-exposed function in the same module.
The description for the tool now explicitly states 'this tool
does NOT generate PySpark code. The calling LLM is expected to
have already written the code; this tool only persists it.'
2. Skill for LLM agents operating the service
(`docs/superpowers/skills/spark-executor-mcp-operate/SKILL.md`,
449 lines). Covers the 16 tools, the two-step prepare/confirm
flow, the dual-ID contract (job_id vs application_id), the
PendingSubmission state machine, the Connection profile, the
job-file workflow, the error reference, common pitfalls, and a
full end-to-end word-count example.
3. Default `executor_memory` lowered 4G → 2G
(`_DEFAULTS_TO_CONFIRM` in `server.py`). Mirrors the matching
change in `test_mcp_routes.py` and the 5 unit tests that
reference the default. Aligns with the lighter workloads the
service is sized for in its current container profile.
Also tracked in git for the first time:
- `docs/superpowers/plans/2026-06-24-spark-executor-mcp.md`
(the original Stage 1/2/3 design plan, updated to use the new
tool name throughout).
Test rename:
- `tests/unit/test_generate_tool.py` → `test_write_job_tool.py`
- the new test file picks up an extra assertion that the SQL guard
rejects a `DROP TABLE` statement at write time.
243 tests pass (was 242; +1 new SQL-guard assertion). Zero regressions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
56 lines
2.0 KiB
Python
56 lines
2.0 KiB
Python
# coding=utf-8
|
|
import os
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from common import config
|
|
from spark_executor.tools import write_job
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _restore_settings():
|
|
snapshot = config.Settings(
|
|
data_dir=config.settings.data_dir,
|
|
jobs_dir=config.settings.jobs_dir,
|
|
yarn_resource_manager_url=config.settings.yarn_resource_manager_url,
|
|
log_level=config.settings.log_level,
|
|
)
|
|
yield
|
|
config.settings.data_dir = snapshot.data_dir
|
|
config.settings.jobs_dir = snapshot.jobs_dir
|
|
config.settings.yarn_resource_manager_url = snapshot.yarn_resource_manager_url
|
|
config.settings.log_level = snapshot.log_level
|
|
|
|
|
|
def test_write_writes_code_and_returns_path(monkeypatch, tmp_path: Path):
|
|
config.settings.jobs_dir = str(tmp_path / "data" / "jobs")
|
|
out = write_job.write_job_file(
|
|
"from pyspark.sql import SparkSession\n"
|
|
"spark = SparkSession.builder.getOrCreate()\n"
|
|
)
|
|
assert "script_path" in out
|
|
p = out["script_path"]
|
|
assert os.path.isabs(p)
|
|
assert p.startswith(str(tmp_path / "data" / "jobs"))
|
|
assert p.endswith(".py")
|
|
with open(p) as f:
|
|
assert "SparkSession.builder.getOrCreate()" in f.read()
|
|
|
|
|
|
def test_write_uses_settings_jobs_dir(tmp_path: Path):
|
|
config.settings.jobs_dir = str(tmp_path / "custom")
|
|
out = write_job.write_job_file("x = 1\n")
|
|
assert out["script_path"].startswith(str(tmp_path / "custom"))
|
|
assert os.path.isfile(out["script_path"])
|
|
|
|
|
|
def test_write_rejects_sql_violation(tmp_path: Path):
|
|
"""The SQL guard is the whole reason this isn't a bare file-write
|
|
tool: the LLM might have produced a script with a forbidden
|
|
statement and we want to fail at write time, not at prepare time."""
|
|
from spark_executor.tools.write_job import SqlGuardViolation
|
|
config.settings.jobs_dir = str(tmp_path / "data" / "jobs")
|
|
with pytest.raises(SqlGuardViolation):
|
|
write_job.write_job_file("spark.sql('DROP TABLE users')\n")
|