fix: drop file tools 1 MB cap + add YarnError -> 502 handler

Two follow-ups to the previous error-handling fix (16fe011):

1) Drop the 1 MB cap on read_job_file and update_job_file.

   The cap was originally there to keep MCP responses bounded, but it
   blocks legitimate use of large PySpark scripts and large update
   payloads. With the new model (LLM composes scripts via
   write_job_file and edits them via read+update), a fixed 1 MB
   cap is more hindrance than protection. MCP response size is
   already bounded by the JSON transport and httpx; the tool itself
   doesn't need a second limit.

   Changes:
   - tools/job_file.py: delete MAX_FILE_BYTES constant, drop the two
     size checks in read_job_file and update_job_file, update
     module docstring.
   - tests/unit/test_job_file.py: delete test_update_rejects_
     oversized_content and test_update_rejects_1mb_plus_1_byte
     (the two tests that asserted the cap), replace with
     test_update_accepts_content_larger_than_former_1mb_cap.
   - server.py: drop "Caps reads at 1 MB" and "Caps writes at 1 MB"
     from the two route descriptions.

2) Add YarnError -> 502 handler.

   yarn_client wraps every httpx call: on connect / TLS / timeout /
   4xx / 5xx / parse failure it raises YarnError. Previously this
   was unhandled, so all six external job tools (get_external_job_*,
   list_applications, plus anything else that hits YARN) returned
   500 "Internal Server Error" with no detail — the LLM couldn't
   tell whether the cluster was down or the request was bad.

   Same fix as 16fe011 (which did this for fetch_url directly).
   The new handler returns HTTP 502 Bad Gateway with the YarnError
   message in the response detail. 502 because the MCP service is
   acting as a gateway to YARN — 502 is the standard status for
   "upstream didn't respond correctly".

   Changes:
   - server.py: import YarnError, add @app.exception_handler
     returning 502 + the YarnError message.
   - tests/integration/test_mcp_routes.py: new test asserts that
     when get_application_status raises YarnError, /get_job_status
     returns 502 with the YarnError message in the detail.

   Note: ValueError (request was bad) is still 400, KeyError (job
   not in JobStore) is still 404. The three handlers form a clean
   3-way classification of tool-layer errors.

Tests: 401 passed (was 401, +1 YarnError test, -2 cap tests = net -1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

tests/integration/test_mcp_routes.py

@@ -8,6 +8,7 @@ from fastapi.testclient import TestClient
 from spark_executor.core import connection_store, pending_store
 from spark_executor.core.connection_store import ConnectionStore
 from spark_executor.core.pending_store import PendingStore
+from spark_executor.models import Connection
 from spark_executor.server import app
 from spark_executor.tools import connections, submit
 
@@ -544,3 +545,37 @@ def test_update_pending_job_route_rejects_submitted(tmp_path, monkeypatch):
     )
     assert r.status_code == 400
     assert "only PENDING submissions can be updated" in r.json()["detail"]
+
+
+# --- YarnError -> 502 handler ---
+
+
+def test_yarn_error_returns_502_with_detail(tmp_path, monkeypatch):
+    """When a YARN REST call raises YarnError (host unreachable, YARN
+    returned 4xx/5xx, parse failure), the route must surface HTTP 502
+    with the YarnError message in the detail — not 500 'Internal
+    Server Error' with no info. Mirrors the same fix for fetch_url.
+    """
+    from unittest.mock import patch
+    from spark_executor.core.yarn_client import YarnError
+
+    connection_store.store.save(
+        Connection(
+            name="prod",
+            master="yarn",
+            yarn_rm_url="http://ccam1:8088",
+        )
+    )
+    c = TestClient(app)
+    with patch(
+        "spark_executor.tools.status.get_application_status",
+        side_effect=YarnError("YARN connection failed: ConnectError: Connection refused"),
+    ):
+        r = c.post(
+            "/get_job_status",
+            json={"job_id": "a1b2c3d4e5f6"},
+        )
+    assert r.status_code == 502
+    detail = r.json()["detail"]
+    assert "YARN connection failed" in detail
+    assert "Connection refused" in detail

tests/unit/test_job_file.py

@@ -118,25 +118,17 @@ def test_update_rejects_missing_file(tmp_path: Path):
         update_job_file(str(tmp_path / "nope.py"), "x\n")
 
 
-def test_update_rejects_oversized_content(tmp_path: Path, monkeypatch):
-    """Cap at 1 MB so the MCP response stays bounded."""
+def test_update_accepts_content_larger_than_former_1mb_cap(tmp_path: Path):
+    """The 1 MB cap was removed (commit a5e6d1f and following). Content
+    larger than the old cap must now be accepted; size is bounded by
+    the MCP transport, not the tool."""
     config.settings.jobs_dir = str(tmp_path)
     p = tmp_path / "big.py"
     p.write_text("# small\n")
-    # Synthesize 2 MB of content (don't actually write 2 MB to disk)
     big = "x" * (2 * 1024 * 1024)
-    with pytest.raises(ScriptFileError, match="too large"):
-        update_job_file(str(p), big)
-
-
-def test_update_rejects_1mb_plus_1_byte(tmp_path: Path):
-    """Exactly at the boundary: 1 MB + 1 byte must be rejected."""
-    config.settings.jobs_dir = str(tmp_path)
-    p = tmp_path / "x.py"
-    p.write_text("# t\n")
-    just_over = "x" * (1024 * 1024 + 1)
-    with pytest.raises(ScriptFileError, match="too large"):
-        update_job_file(str(p), just_over)
+    out = update_job_file(str(p), big)
+    assert out["bytes_written"] == len(big)
+    assert p.read_text() == big
 
 
 # --- round-trip: write -> read -> update -> read ---