refactor(submit): drop spark-submit retry loop; confirm raises on failure

confirm_submit_job no longer retries on SparkSubmitError. A failed
single attempt is recorded as-is:

  - pending.status = 'FAILED' with the error message in pending.error
  - the function re-raises the SparkSubmitError so the caller (and the
    agent) sees the failure immediately
  - the agent can call get_pending_job to see the persisted FAILED
    state, including the captured error

The 'retry' code path was hiding real failures behind transient
recovery: a spark-submit that died because the script is broken looks
the same as one that died because YARN was momentarily unreachable.
Without retries, the agent gets a clean FAILED state and a clear
exception to act on, instead of a delayed, ambiguous result.

Manual recovery is still possible: re-calling confirm_submit_job on a
FAILED pending resets it to PENDING and runs a single fresh attempt
(see test_confirm_resets_failed_then_succeeds). The reset path is
useful for 'I fixed the script, try again' flows; it just is no longer
the default on every failure.

Removed:
  - the for-attempt loop and time.sleep in submit.py
  - import time in submit.py and test_submit_tool.py
  - confirm_max_retries and confirm_retry_delay_seconds from
    common/config.py (env vars, snapshot, reload)
  - 2 retry tests (test_confirm_retries_spark_submit_failure_then_succeeds,
    test_confirm_exhausts_retries_and_sets_failed)
  - test_confirm_marks_failed_on_spark_submit_error updated to assert
    the new single-attempt + raise contract
  - test_confirm_resets_failed_and_retries renamed to
    test_confirm_resets_failed_then_succeeds (no more retry loop to
    exercise)

Full suite 244 passed (3 fewer than before, matching the removed
tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

common/config.py

@@ -76,13 +76,6 @@ class Settings:
     # The binary is resolved against $PATH (set by docker-entrypoint.sh).
     spark_submit_bin: str = "spark-submit"
 
-    # --- Confirm retry policy ---
-    # confirm_submit_job retries spark-submit on SparkSubmitError to tolerate
-    # transient client/network issues in test environments or unstable clusters.
-    # Max retries (not counting the first attempt) and delay between attempts.
-    confirm_max_retries: int = 3
-    confirm_retry_delay_seconds: float = 5.0
-
     # --- SSL/TLS defaults for YARN REST calls ---
     ssl_verify_default: bool = True
     ssl_ca_bundle_default: str | None = None
@@ -113,12 +106,6 @@ class Settings:
             ssl_ca_bundle_default=(
                 os.environ.get("SPARK_EXECUTOR_SSL_CA_BUNDLE_DEFAULT") or None
             ),
-            confirm_max_retries=int(
-                os.environ.get("SPARK_EXECUTOR_CONFIRM_MAX_RETRIES", "3")
-            ),
-            confirm_retry_delay_seconds=float(
-                os.environ.get("SPARK_EXECUTOR_CONFIRM_RETRY_DELAY_SECONDS", "5.0")
-            ),
         )
 
     def reload(self) -> "Settings":
@@ -136,8 +123,6 @@ class Settings:
         self.spark_submit_bin = fresh.spark_submit_bin
         self.ssl_verify_default = fresh.ssl_verify_default
         self.ssl_ca_bundle_default = fresh.ssl_ca_bundle_default
-        self.confirm_max_retries = fresh.confirm_max_retries
-        self.confirm_retry_delay_seconds = fresh.confirm_retry_delay_seconds
         return self