feat(fetch_url): per-Connection allowed_url_hosts glob allowlist

The default 'URL host shares >= 2 labels of suffix with yarn_rm_url
host' rule is too strict for clusters whose hostnames are single-label
(e.g. 'ccam1' through 'ccam99'). The user's cluster is reachable at
http://ccam1:8088, and they want fetch_url to work for any ccamN
host — but ccam1 and ccam50 share 0 suffix labels, so the existing
rule rejects everything.

Add an opt-in allowlist field on Connection:

  allowed_url_hosts: list[str] | None

Each entry is an fnmatch glob pattern. The URL host is allowed if it
matches ANY pattern, regardless of the suffix rule. Save a Connection
with ['ccam*'] to allow ccam1, ccam2, ..., ccam99.

SSRF safety: the implementation does NOT use vanilla fnmatch on the
flat host string (because '*' in fnmatch crosses '.', so 'ccam*' would
match 'ccam50.evil.com' — a security hole). Instead, both the pattern
and the host are split on '.' and matched LABEL-BY-LABEL with the
label counts required to match exactly. So 'ccam*' matches 'ccam50'
but NOT 'ccam50.evil.com' (different label counts).

Test coverage:
- 8 new tests in tests/unit/test_fetch_url.py (glob allow, glob deny,
  dot-boundary SSRF test, multiple globs, empty list, None fallback,
  error message hint)
- All 21 fetch_url tests pass; 377 total.

- spark_executor/models.py: Connection.allowed_url_hosts (with description)
- spark_executor/tools/requests.py: SaveConnectionRequest.allowed_url_hosts
- spark_executor/tools/fetch_url.py: new _host_matches_any_glob helper;
  _validate_url_host now takes allowed_hosts and checks globs BEFORE
  the suffix rule
- tests/unit/test_fetch_url.py: 8 new tests
- README.md: fetch_url row mentions the allowlist

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Claude
2026-07-09 11:22:55 +08:00
co-authored by Claude Fable 5
parent 627e70f697
commit 0291f36b01
5 changed files with 209 additions and 23 deletions
+15
View File
@@ -69,6 +69,21 @@ class Connection(BaseModel):
auth_principal: str | None = None
auth_keytab: str | None = None
allowed_url_hosts: list[str] | None = Field(
default=None,
description=(
"Optional list of fnmatch glob patterns for hosts the fetch_url tool "
"may access, in addition to the default 'shares >= 2 labels of suffix "
"with yarn_rm_url host' rule. Useful for clusters whose hostnames do "
"NOT share a 2+ label suffix — e.g. single-label hosts like 'ccam1'-"
"'ccam99' (configure ['ccam*']) or HDFS namenode on a different "
"subdomain ('*.hadoop.internal'). Patterns are matched against the "
"URL host only (no port, no path). fnmatch rules apply: '*' does NOT "
"match '.', so 'ccam*' matches 'ccam50' but not 'ccam50.evil.com'. "
"Default None means the suffix rule alone applies."
),
)
@field_validator("master")
@classmethod
def _check_master(cls, v: str) -> str: