feat: env-configurable spark-submit binary name
Hardcoding 'spark-submit' as cmd[0] in build_spark_submit_command
breaks for hosts where:
- both Spark 1.x and 2.x/3.x are installed and 'spark-submit' resolves
to the wrong one (use 'spark2-submit' or 'spark3-submit' explicitly)
- the user wants to launch via the PySpark entrypoint ('pyspark')
- a custom wrapper script sits on PATH (e.g. a credentials-injecting
'spark-submit-wrapper')
New env var SPARK_EXECUTOR_SPARK_SUBMIT_BIN. Default is 'spark-submit'
(preserves the current behavior for everyone). Override in .env /
docker-compose.yml to change.
common/config.py:
- new Settings.spark_submit_bin field
- env-var resolution in from_env() with default 'spark-submit'
- included in reload() so tests work
spark_executor/core/spark_submit.py:
- cmd[0] reads settings.spark_submit_bin (was hardcoded 'spark-submit')
.env.example: new section with the override and example values.
docker-compose.yml: forwards the var with the standard 'spark-submit'
default.
Tests: 2 new (settings.spark_submit_bin='spark2-submit', ='pyspark')
plus existing tests updated to use the settings-restore fixture so
mutations don't leak between tests.
165/163 still pass.
This commit is contained in:
@@ -65,6 +65,17 @@ class Settings:
|
||||
# file sink always captures DEBUG (full audit trail).
|
||||
log_level: str = "DEBUG"
|
||||
|
||||
# --- Spark CLI binary name ---
|
||||
# The program name used in cmd[0] when invoking the Spark client.
|
||||
# Defaults to 'spark-submit' (the standard Spark 2.x / 3.x / 4.x CLI).
|
||||
# Override via SPARK_EXECUTOR_SPARK_SUBMIT_BIN for:
|
||||
# - 'spark2-submit' on a system where both Spark 1.x and 2.x are
|
||||
# installed and spark-submit points to the wrong one
|
||||
# - a custom wrapper script (e.g. '/usr/local/bin/spark-submit-wrapper')
|
||||
# - 'pyspark' if you want to launch via the PySpark entrypoint
|
||||
# The binary is resolved against $PATH (set by docker-entrypoint.sh).
|
||||
spark_submit_bin: str = "spark-submit"
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> "Settings":
|
||||
data_dir = os.environ.get("SPARK_EXECUTOR_DATA_DIR", "./data")
|
||||
@@ -81,6 +92,9 @@ class Settings:
|
||||
# `or None` collapses empty string to None for the URL fallback
|
||||
yarn_resource_manager_url=os.environ.get("YARN_RESOURCE_MANAGER_URL") or None,
|
||||
log_level=os.environ.get("SPARK_EXECUTOR_LOG_LEVEL", "DEBUG"),
|
||||
spark_submit_bin=os.environ.get(
|
||||
"SPARK_EXECUTOR_SPARK_SUBMIT_BIN", "spark-submit"
|
||||
),
|
||||
)
|
||||
|
||||
def reload(self) -> "Settings":
|
||||
@@ -95,6 +109,7 @@ class Settings:
|
||||
self.log_dir = fresh.log_dir
|
||||
self.yarn_resource_manager_url = fresh.yarn_resource_manager_url
|
||||
self.log_level = fresh.log_level
|
||||
self.spark_submit_bin = fresh.spark_submit_bin
|
||||
return self
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user