fix(job_store): persist Jobs to disk and accept either ID in job tools

Two user-reported bugs, same root cause: the in-memory JobStore + the
'job_id must be the 12-char hex' tool contract.

Bug 1: 'Unknown job_id' reported frequently
  JobStore was a process-local dict (spark_executor/core/job_store.py).
  Under gunicorn workers > 1, a job created by confirm_submit_job
  landing on worker A was invisible to worker B, so a follow-up
  get_job_status / get_job_result / get_job_logs / kill_job landing on
  a different worker returned 'Unknown job_id'. Same multi-worker
  problem that bit the MCP session layer; only the affected data was
  different.

Bug 2: 'get_job_logs frequently confuses job_id and application_id'
  confirm_submit_job returns BOTH identifiers in SubmitResult, but
  get_job_logs (and friends) only accepted the local 12-char job_id
  and never said so in their description. When the agent passed the
  YARN application_id, the error message itself was misleading:
  'Unknown job_id: application_17400000001_0001' — the agent had
  passed an id, just the wrong kind.

This change fixes both at the root:

  * JobStore is now JSON-backed at data/jobs.json (atomic tempfile +
    os.replace), with cross-process safety via fcntl.flock on a sibling
    .lock file. Stage 3's SQLite migration is still planned; the file
    format is intentionally simple so it is a straight
    'for j in read_all(): db.insert(j)'.

  * New JobStore.get_either(uid) looks up by job_id first, then
    application_id. All four job-lifecycle tools (get_job_status,
    get_job_result, get_job_logs, kill_job) call get_either instead
    of get(job_id), so the agent can pass either identifier and get
    the same answer.

  * The 'neither matched' KeyError now spells out both id forms and
    what they look like, so the agent isn't left guessing.

  * server.py tool descriptions for the four job tools explicitly
    state 'job_id accepts BOTH identifiers' so this is visible to the
    LLM at tool-selection time, not only at error time.

Tests:
  * test_job_store.py: tmp_path isolation, persistence across
    instances, human-readable JSON, corrupt-file resilience,
    get_either (by job_id, by application_id, collision preference,
    unknown), put idempotency.
  * test_{logs,status,kill,result}_tool.py: per-test tmp_path fixture,
    'accepts application_id' regression for each tool, and an
    explicit assertion that the unknown-id error message mentions
    BOTH id forms. test_result_raises_keyerror_for_unknown_job's
    match pattern updated for the new message.

242 tests pass (was 226; +16 new). Zero regressions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

spark_executor/server.py

@@ -181,7 +181,12 @@ def _cancel_pending_job(req: PendingIdRequest):
     description=(
         "Return the YARN application state (RUNNING / SUCCEEDED / FAILED / "
         "KILLED / ACCEPTED / NEW / NEW_SAVING / SUBMITTED / etc.) plus the "
-        "raw YARN REST response body."
+        "raw YARN REST response body.\n\n"
+        "**job_id accepts BOTH identifiers** returned by "
+        "confirm_submit_job: the local job_id (12-char hex, e.g. "
+        "'a1b2c3d4e5f6') and the YARN application_id (e.g. "
+        "'application_17400000001_0001'). The lookup is by job_id first, "
+        "then by application_id."
     ),
 )
 def _get_job_status(req: JobIdRequest):
@@ -196,7 +201,12 @@ def _get_job_status(req: JobIdRequest):
         "Return a terminal-oriented view of a Spark job: final_status, "
         "diagnostics, tracking_url, started_time, and finished_time. "
         "This is distinct from get_job_status, which is for polling the "
-        "running YARN state and returns the raw YARN response."
+        "running YARN state and returns the raw YARN response.\n\n"
+        "**job_id accepts BOTH identifiers** returned by "
+        "confirm_submit_job: the local job_id (12-char hex, e.g. "
+        "'a1b2c3d4e5f6') and the YARN application_id (e.g. "
+        "'application_17400000001_0001'). The lookup is by job_id first, "
+        "then by application_id."
     ),
 )
 def _get_job_result(req: JobIdRequest):
@@ -210,7 +220,12 @@ def _get_job_result(req: JobIdRequest):
     description=(
         "Pull aggregated logs from the YARN ResourceManager. Returns the last "
         "tail_chars characters (default 5000). Requires yarn.log-aggregation-enable "
-        "to be true on the target cluster."
+        "to be true on the target cluster.\n\n"
+        "**job_id accepts BOTH identifiers** returned by "
+        "confirm_submit_job: the local job_id (12-char hex, e.g. "
+        "'a1b2c3d4e5f6') and the YARN application_id (e.g. "
+        "'application_17400000001_0001'). The lookup is by job_id first, "
+        "then by application_id."
     ),
 )
 def _get_job_logs(req: GetJobLogsRequest):
@@ -221,7 +236,12 @@ def _get_job_logs(req: GetJobLogsRequest):
     "/kill_job",
     operation_id="kill_job",
     summary="Kill a running job",
-    description="PUT state=KILLED to YARN REST API for the job's application_id.",
+    description=(
+        "PUT state=KILLED to YARN REST API for the job's application_id. "
+        "**job_id accepts BOTH identifiers** returned by "
+        "confirm_submit_job: the local job_id (12-char hex) and the YARN "
+        "application_id. The lookup is by job_id first, then by application_id."
+    ),
 )
 def _kill_job(req: JobIdRequest):
     return kill_job(req.job_id)