diff --git a/docs/superpowers/skills/spark-executor-mcp-operate/SKILL.md b/docs/superpowers/skills/spark-executor-mcp-operate/SKILL.md index af9a87f..b909e14 100644 --- a/docs/superpowers/skills/spark-executor-mcp-operate/SKILL.md +++ b/docs/superpowers/skills/spark-executor-mcp-operate/SKILL.md @@ -42,6 +42,70 @@ Three things you MUST internalize before calling any tool: `write_job_file(code=...)` to drop it on the container's filesystem where `spark-submit` can see it. +4. **Three decisions MUST be approved by the human user before you act on + them — never pick defaults silently, never auto-confirm.** See §0.1 + below. This is non-negotiable: confirm_submit_job starts a real YARN + job that costs cluster time and may write to production data paths, + and the user is the one who has to live with the result. Get their + explicit "yes, confirm" before pulling the trigger. + +### 0.1. Mandatory user confirmations (the three checkpoints) + +For every Spark job the user asks you to run, you must stop and ask the +human user to approve these three decisions in order. Do not pick +defaults on their behalf. Do not assume "they'd obviously want prod / +default queue / 2G / 2 cores / 2 executors" — those are YOUR defaults, +not theirs. + +**Checkpoint 1 — when the user first asks to run a job:** +ASK the user for: +- Which **connection** to submit to. Show the existing connections + (from `list_connections`) as options; if the right one doesn't exist, + ask for the new connection's full profile (name, master, deploy_mode, + yarn_rm_url, spark_conf, auth). See §5 for the field reference. +- The **app_name** (human-readable label for the YARN app). The + service has no default for this; without it `prepare_submit_job` + returns 400. + +Do not skip past this even if the user is in a hurry or the request +seems obvious. "Run a daily job on prod" still needs you to confirm +which "daily job", which "prod", and what app_name to label it with. + +**Checkpoint 2 — before `prepare_submit_job`:** +ASK the user for the values of these submit parameters, even if they +have server-side defaults: +- `app_name` (if you didn't get it at Checkpoint 1) +- `queue` +- `executor_memory` +- `executor_cores` +- `num_executors` +- any `extra_args` (jars, py-files, …) needed for this script + +Tell the user the server-side defaults you WOULD use if they don't care +(say "I'll use default queue, 2G memory, 2 cores, 2 executors unless +you say otherwise") so they can either approve or override. The +service itself rejects implicit defaults (returns 400 listing the +assumed values) — this is by design, and you should respect it by +being the one to surface the assumed values, not by silently letting +the service 400 and then re-asking. + +**Checkpoint 3 — before `confirm_submit_job` (the irreversible one):** +Present a one-screen summary of exactly what will happen, and wait for +the user to say "confirm" / "yes" / "do it" (or any unambiguous +affirmative). The summary must include: +- The connection name and its master / deploy_mode / yarn_rm_url +- The app_name and the script path +- queue, executor_memory, executor_cores, num_executors +- The YARN tracking URL that will be returned (you don't know it yet, + but you can say "you'll get a tracking URL once it starts") + +The user's exact words don't need to be "confirm" — any clear +affirmative works ("go", "ship it", "y", "🚀", "do it"). A non-answer +or a "wait" / "hold on" / "let me think" is NOT consent; wait. + +After they say yes, you may call `confirm_submit_job`. **Never call +`confirm_submit_job` without that explicit go-ahead.** + --- ## 1. Tool index @@ -73,15 +137,24 @@ Three things you MUST internalize before calling any tool: ## 2. The canonical happy path: submit a PySpark job ``` -1. save_connection # one-time, per cluster -2. write_job_file # write the LLM's PySpark to disk -3. read_job_file # (optional) sanity-check what was written -4. prepare_submit_job # snapshot + persist; no YARN call yet -5. confirm_submit_job # NOW spark-submit runs -6. get_job_status / logs # poll while running -7. get_job_result # terminal view when done +-- CHECKPOINT 1: ask user for connection + app_name -- +1. list_connections / save_connection # pick or create the right profile +2. write_job_file # write the LLM's PySpark to disk +3. read_job_file # (optional) sanity-check what was written +-- CHECKPOINT 2: ask user for queue / memory / cores / num_executors -- +4. prepare_submit_job # snapshot + persist; no YARN call yet +-- CHECKPOINT 3: present summary, wait for explicit "yes, confirm" -- +5. confirm_submit_job # NOW spark-submit runs (irreversible) +6. get_job_status / logs # poll while running +7. get_job_result # terminal view when done ``` +The three `CHECKPOINT` lines above are mandatory pauses. Do NOT +collapse them or skip past them. See §0.1 for the full rules; the +short version is: ask the user before each tool call that affects +cluster state, and never call `confirm_submit_job` without their +explicit go-ahead. + ### 2.1 Save a connection (one-time per cluster) ```python @@ -354,30 +427,33 @@ raised. | `cancel_pending_job` on a running YARN app | Returns 400 — submissions are terminal once submitted | Use `kill_job(job_id=...)` instead, which talks to YARN REST | | Re-pointing a Connection and expecting in-flight jobs to follow | They don't — the Job snapshots `yarn_rm_url` at confirm time | Re-issue with a new `prepare_submit_job` if you actually need a different RM | | Calling lifecycle tools with the wrong ID type | Used to fail; **fixed in the 2026-06-29 update** — both IDs are accepted | Just pass whichever you have; the service tries `job_id` first | +| Auto-confirming without the user's explicit "yes" | Triggers an irreversible YARN job the user may not have wanted — wastes cluster time, may write to wrong data paths | Per §0.1, **always** present a one-screen summary and wait for the user to say "confirm" / "yes" / equivalent before calling `confirm_submit_job`. Silence, "hmm", or "let me think" are NOT consent. | +| Picking connection / queue / executor defaults on the user's behalf | The user knows their cluster's cost / SLA / data layout better than you do; your defaults may not match their intent | Per §0.1, ASK at Checkpoint 1 (connection + app_name) and Checkpoint 2 (queue / memory / cores / num_executors). Surface the defaults you'd use so they can approve or override, don't just pick. | | "Session not found" at the MCP layer | Multi-worker gunicorn + in-process session state | Out of scope for this skill; the service warns on startup if `GUNICORN_WORKERS>1`; set `GUNICORN_WORKERS=1` or front with a sticky LB | --- -## 9. End-to-end example (LLM agent's perspective) +## 9. End-to-end example (with the three checkpoints) User: "Run a word count on `s3a://my-bucket/books/*.txt` and write the result back to `s3a://my-bucket/wc/`." ``` -# 1. Cluster profile (assumes this is a fresh setup) -save_connection( - name="prod", - master="yarn", - deploy_mode="cluster", - yarn_rm_url="http://rm.prod.example.com:8088", - spark_conf={ - "spark.sql.shuffle.partitions": "200", - "spark.hadoop.fs.s3a.access.key": "...", - "spark.hadoop.fs.s3a.secret.key": "...", - }, -) +# ============================================================ +# CHECKPOINT 1 — ask the user for connection + app_name FIRST +# ============================================================ +# +# Agent: "I see you have these saved connections: +# - prod (yarn RM http://rm.prod.example.com:8088, cluster, kerberos) +# - dev (yarn RM http://rm.dev.example.com:8088, client, none) +# Which one should I submit to? And what app_name should I label +# the YARN app with?" +# User: "prod, call it wordcount-books" +# +# (If neither connection is right, ask for the full profile and call +# save_connection before continuing — see §2.1 for the fields.) -# 2. Compose the PySpark code IN YOUR CONTEXT +# 1. Compose the PySpark code IN YOUR CONTEXT pyspark_code = ''' from pyspark.sql import SparkSession spark = SparkSession.builder.appName("wordcount").getOrCreate() @@ -390,32 +466,60 @@ df = words.toDF(["word", "count"]) df.write.mode("overwrite").parquet("s3a://my-bucket/wc/") ''' -# 3. Persist to disk +# 2. Persist to disk script_path = write_job_file(code=pyspark_code)["script_path"] -# 4. Sanity-check (optional but recommended) +# 3. Sanity-check (optional but recommended) contents = read_job_file(script_path=script_path)["content"] # ... review in context ... -# 5. Prepare (snapshot — no YARN call yet) +# ============================================================ +# CHECKPOINT 2 — ask the user for submit parameters +# ============================================================ +# +# Agent: "I'll use these unless you say otherwise: +# queue=default, executor_memory=2G, executor_cores=2, +# num_executors=2. Any overrides (different queue, more memory, +# jars/py-files to add)?" +# User: "use research queue, 4G memory, 2 executors" +# +# (If the user says "defaults are fine", that's still an explicit +# approval — you may then call prepare_submit_job with those values.) + +# 4. Prepare (snapshot — no YARN call yet) pending = prepare_submit_job( connection="prod", script_path=script_path, app_name="wordcount-books", - queue="default", - executor_memory="2G", + queue="research", + executor_memory="4G", executor_cores=2, num_executors=2, ) pending_id = pending["pending_id"] -# 6. Confirm (NOW spark-submit runs) +# ============================================================ +# CHECKPOINT 3 — present summary, wait for explicit "confirm" +# ============================================================ +# +# Agent: "About to run on YARN: +# • connection: prod (yarn, cluster, http://rm.prod.example.com:8088) +# • app_name: wordcount-books +# • script: +# • queue: research +# • resources: 2G memory × 2 cores × 2 executors +# This will invoke spark-submit and start a real YARN application. +# Confirm? (y / n)" +# User: "yes, go" +# ─── only NOW may you call confirm_submit_job ─── + +# 5. Confirm (NOW spark-submit runs) result = confirm_submit_job(pending_id=pending_id) job_id = result["job_id"] # STORE application_id = result["application_id"] # STORE tracking_url = result["tracking_url"] # STORE -# 7. Poll +# 6. Poll import time while True: st = get_job_status(job_id=job_id).state @@ -423,12 +527,16 @@ while True: break time.sleep(30) -# 8. Final result +# 7. Final result res = get_job_result(job_id=job_id) print(res.final_status, res.diagnostics) # If something went wrong and you want to see why: # logs = get_job_logs(job_id=job_id, tail_chars=20000) + +# (If you need to kill the job: ASK THE USER FIRST, then call +# kill_job(job_id=job_id). Don't kill running jobs without an explicit +# "kill it" — the user may have wanted to let it finish.) ``` ---