feat: Connection.master defaults to 'yarn'

YARN is the dominant use case, and the value of master for YARN is the
literal string 'yarn' (not a URL) — there is no useful variation. Making
it a default saves every YARN user from typing the same value every time:

    save_connection(name='prod')           # master defaults to 'yarn'
    save_connection(name='dev', master='spark://10.0.0.5:7077')  # override

Added a field_validator on master that catches common typos
('yarn-cluster', 'http://...', empty string) at save time with a clear
error message instead of letting them reach spark-submit.

Backward compatible: existing Connection records without an explicit
master field default to 'yarn' on load (Pydantic applies defaults on
deserialization), so already-saved data still works.

3 new tests cover the default, the validator, and that other cluster
managers (spark://, k8s://, local[N]) still pass.
This commit is contained in:
Claude
2026-06-25 10:05:27 +08:00
parent 504447048d
commit 516541a4df
2 changed files with 43 additions and 2 deletions
+18 -2
View File
@@ -5,7 +5,7 @@
"""
from datetime import datetime
from pydantic import BaseModel, Field
from pydantic import BaseModel, Field, field_validator
class Job(BaseModel):
@@ -32,11 +32,27 @@ class SubmitResult(BaseModel):
class Connection(BaseModel):
name: str
master: str
# Defaults to "yarn" because that's the literal string spark-submit wants
# for --master when targeting YARN. Override for Standalone (spark://...),
# Kubernetes (k8s://...), or local mode.
master: str = "yarn"
deploy_mode: str = "cluster"
yarn_rm_url: str | None = None
spark_conf: dict[str, str] = Field(default_factory=dict)
@field_validator("master")
@classmethod
def _check_master(cls, v: str) -> str:
"""Catch common typos like 'yarn-cluster' or 'http://...'. """
if v == "yarn":
return v
if v.startswith(("spark://", "k8s://", "mesos://", "local")):
return v
raise ValueError(
f"master must be 'yarn', 'spark://...', 'k8s://...', 'mesos://...', "
f"or 'local[/N]'; got {v!r}"
)
class PendingSubmission(BaseModel):
pending_id: str