fix: drop file tools 1 MB cap + add YarnError -> 502 handler

Two follow-ups to the previous error-handling fix (16fe011):

1) Drop the 1 MB cap on read_job_file and update_job_file.

   The cap was originally there to keep MCP responses bounded, but it
   blocks legitimate use of large PySpark scripts and large update
   payloads. With the new model (LLM composes scripts via
   write_job_file and edits them via read+update), a fixed 1 MB
   cap is more hindrance than protection. MCP response size is
   already bounded by the JSON transport and httpx; the tool itself
   doesn't need a second limit.

   Changes:
   - tools/job_file.py: delete MAX_FILE_BYTES constant, drop the two
     size checks in read_job_file and update_job_file, update
     module docstring.
   - tests/unit/test_job_file.py: delete test_update_rejects_
     oversized_content and test_update_rejects_1mb_plus_1_byte
     (the two tests that asserted the cap), replace with
     test_update_accepts_content_larger_than_former_1mb_cap.
   - server.py: drop "Caps reads at 1 MB" and "Caps writes at 1 MB"
     from the two route descriptions.

2) Add YarnError -> 502 handler.

   yarn_client wraps every httpx call: on connect / TLS / timeout /
   4xx / 5xx / parse failure it raises YarnError. Previously this
   was unhandled, so all six external job tools (get_external_job_*,
   list_applications, plus anything else that hits YARN) returned
   500 "Internal Server Error" with no detail — the LLM couldn't
   tell whether the cluster was down or the request was bad.

   Same fix as 16fe011 (which did this for fetch_url directly).
   The new handler returns HTTP 502 Bad Gateway with the YarnError
   message in the response detail. 502 because the MCP service is
   acting as a gateway to YARN — 502 is the standard status for
   "upstream didn't respond correctly".

   Changes:
   - server.py: import YarnError, add @app.exception_handler
     returning 502 + the YarnError message.
   - tests/integration/test_mcp_routes.py: new test asserts that
     when get_application_status raises YarnError, /get_job_status
     returns 502 with the YarnError message in the detail.

   Note: ValueError (request was bad) is still 400, KeyError (job
   not in JobStore) is still 404. The three handlers form a clean
   3-way classification of tool-layer errors.

Tests: 401 passed (was 401, +1 YarnError test, -2 cap tests = net -1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

spark_executor/tools/job_file.py

@@ -16,7 +16,10 @@ Safety:
   - update_job_file:  must be under SPARK_EXECUTOR_JOBS_DIR
     (settings.jobs_dir) so the agent cannot overwrite host-mounted
     configs or arbitrary files on the container FS.
-  - 1 MB cap on both read and write payloads to keep MCP responses bounded.
+  - No size cap on read or write payloads — the caller controls the
+    file size. MCP response size is bounded by httpx and the JSON
+    transport; for very large files the LLM should split via
+    write_job_file and read in chunks via read_job_file + update.
 """
 import os
 from pathlib import Path
@@ -24,9 +27,6 @@ from pathlib import Path
 from common.config import settings
 from common.logging import logger
 
-MAX_FILE_BYTES = 1 * 1024 * 1024  # 1 MB
-
-
 class ScriptFileError(ValueError):
     """Raised when read/update fails. -> HTTP 400 via the FastAPI ValueError
     handler in server.py.
@@ -68,16 +68,10 @@ def _check_writable(script_path: str) -> None:
 def read_job_file(script_path: str) -> dict[str, object]:
     """Return the text content of an existing script file.
 
-    Caps the read at 1 MB to keep MCP responses bounded; raises
-    ScriptFileError (-> 400) if the file is missing or too large.
+    No size cap. Raises ScriptFileError (-> 400) if the file is missing.
     """
     _check_readable(script_path)
     size = os.path.getsize(script_path)
-    if size > MAX_FILE_BYTES:
-        raise ScriptFileError(
-            f"Script is too large to read back ({size} bytes > {MAX_FILE_BYTES} "
-            f"byte cap). Edit it via a host volume mount instead."
-        )
     logger.debug(f"read_job_file enter script_path={script_path} size={size}")
     with open(script_path, encoding="utf-8") as f:
         content = f.read()
@@ -88,17 +82,12 @@ def read_job_file(script_path: str) -> dict[str, object]:
 def update_job_file(script_path: str, content: str) -> dict[str, object]:
     """Overwrite an existing script file with new content.
 
-    Restricted to paths under settings.jobs_dir. Caps writes at 1 MB.
-    Raises ScriptFileError (-> 400) if the path is missing, outside
-    the allowed dir, or the content is too large.
+    Restricted to paths under settings.jobs_dir. No size cap.
+    Raises ScriptFileError (-> 400) if the path is missing or outside
+    the allowed dir.
     """
     _check_writable(script_path)
     encoded_size = len(content.encode("utf-8"))
-    if encoded_size > MAX_FILE_BYTES:
-        raise ScriptFileError(
-            f"content is too large ({encoded_size} bytes > {MAX_FILE_BYTES} "
-            f"byte cap). Split the script into multiple files."
-        )
     logger.debug(
         f"update_job_file enter script_path={script_path} "
         f"new_bytes={encoded_size}"