feat(submit): configurable confirm retries and idempotent confirm

Harden confirm_submit_job for unreliable test environments and transient
spark-submit failures:

- Add confirm_max_retries and confirm_retry_delay_seconds settings
  (env-configurable).
- SUBMITTED pending returns cached result without re-running spark-submit.
- FAILED pending is reset to PENDING and retried.
- CANCELLED pending is still rejected.
- On success, persist pending as SUBMITTED before creating the in-memory Job
  so a job_store failure cannot leave the record as PENDING while the YARN
  app is already running.
- Persist tracking_url on PendingSubmission for idempotent returns.

Tests cover retry-then-success, max-retries-exceeded, idempotency,
FAILED reset, and pending-saved-before-job-store.

This commit is contained in:

Claude

2026-06-26 15:51:51 +08:00

parent 9e7b425f09

commit a613c7bdb8

5 changed files with 240 additions and 29 deletions

									
										tests/unit/test_pending_store.py
									
		+1
		
												View File
												
				@@ -1,4 +1,5 @@

				# coding=utf-8

				import json

				from datetime import datetime

				from pathlib import Path