refactor: replace yarn CLI shell-out with YARN REST API

yarn_client.py no longer invokes the 'yarn' binary via subprocess; it uses
httpx against /ws/v1/cluster/apps/* endpoints. This means the runtime image
no longer needs the Hadoop client installation — the only YARN-side
dependency left in the container is the config dir consumed by
spark-submit itself.

New model field:
  - Job.yarn_rm_url: str | None
  - PendingSubmission.yarn_rm_url: str | None (snapshotted at prepare)

prepare_submit_job snapshots Connection.yarn_rm_url into the pending
record (consistent with the existing master/deploy_mode/spark_conf
snapshot pattern); confirm_submit_job copies it onto the Job so
status/logs/kill can use it without re-looking-up the connection.

Resolution order for the RM URL at runtime:
  1. Job.yarn_rm_url (preferred — survives connection edits/deletes)
  2. Connection.yarn_rm_url fallback (if a future tool is added that
     doesn't go through a Job)
  3. YARN_RESOURCE_MANAGER_URL env var

Errors:
  - YarnConfigError (HTTP 4xx semantics) when URL is missing/malformed
  - YarnError for HTTP 4xx/5xx from the RM, network failures, missing
    state field, or unparseable log responses

10 new tests in test_yarn_client.py cover the REST surface:
success, 404, 5xx, missing state field, env-var fallback, malformed
URL, log 404 with log-aggregation hint, kill PUT body shape, and
httpx connection-error wrapping.
This commit is contained in:
Claude
2026-06-24 17:33:36 +08:00
parent 0d7b1d1ec3
commit b3564737f7
12 changed files with 252 additions and 91 deletions
+2
View File
@@ -15,6 +15,7 @@ class Job(BaseModel):
queue: str
submit_time: datetime
connection: str
yarn_rm_url: str | None = None
class JobStatus(BaseModel):
@@ -42,6 +43,7 @@ class PendingSubmission(BaseModel):
connection: str
master: str
deploy_mode: str
yarn_rm_url: str | None = None
script_path: str
queue: str
executor_memory: str