fix: address review findings on fetch-url-tool

- fetch_url: revalidate allowlist on every redirect hop (fixes SSRF where
  302 to disallowed host / 169.254.169.254 / file:// bypassed the
  url_allowlist). Stream response body with iter_bytes and cap at 1MB
  so a multi-GB response from an allowlisted host cannot OOM the service.
  Reuses the manual-redirect-loop pattern from yarn_client.

- list_applications: stop swallowing 404 (YARN returns 200+empty for
  "no match"; 404 means the RM doesn't support the endpoint — surface
  the YarnError instead of hiding it as an empty result). Add
  Field(ge=1, le=10000) to ListApplicationsRequest.limit so a runaway
  limit is rejected at the Pydantic layer with 422.

- save_connection: PATCH semantics for existing records. Re-route to
  update_connection when the name already exists so partial updates
  (e.g. only master) no longer wipe url_allowlist back to []. Uses an
  _UNSET sentinel in the tool function to distinguish "omitted" from
  "None" without breaking the existing parameter list.

- README: drop leading space on 5 new connection-tool table rows that
  was breaking GitHub Flavored Markdown table continuity.

- Indentation: normalize connections.py and requests.py to 4-space
  indent (auth_password/auth_principal/auth_keytab were 3-space).

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Claude
2026-07-09 14:23:16 +08:00
co-authored by Claude
parent 7fbad97a87
commit 7d4e512cb0
9 changed files with 331 additions and 93 deletions
+13
View File
@@ -5,6 +5,7 @@ from unittest.mock import patch
import pytest
from spark_executor.core import connection_store
from spark_executor.core.yarn_client import YarnError
from spark_executor.models import ApplicationSummary, Connection
from spark_executor.tools import connections, external_jobs
from spark_executor.tools.requests import ListApplicationsRequest
@@ -290,3 +291,15 @@ def test_list_applications_maps_yarn_json_to_summary(fresh_stores):
assert summary.finished_time == 0
assert summary.tracking_url == "http://rm:8088/proxy/application_42"
assert summary.progress == 12.5
def test_list_applications_raises_on_404(fresh_stores):
external_jobs.conn_store.save(
Connection(name="prod", master="yarn", yarn_rm_url="http://rm:8088")
)
with patch(
"spark_executor.tools.external_jobs.list_applications_yarn",
side_effect=YarnError("YARN list applications failed: 404"),
):
with pytest.raises(YarnError, match="YARN list applications failed"):
external_jobs.list_applications("prod")