Claude
cb909f7fea
Three fixes requested:
1. Spark 4.1.2 -> Spark 3.1.2 (Hadoop 2.7 prebuilt). Compatible with the
JDK 11 build below and a more conservative choice for production.
2. JDK 17 -> JDK 11. openjdk-11-jre-headless package; JAVA_HOME points
at /usr/lib/jvm/java-11-openjdk-amd64.
3. /usr/bin/env 'bach' no such file: defensive shebang fix. The
downloaded spark-3.1.2-bin-hadoop2.7.tgz happens to have a clean
shebang, but older 4.x distributions (and any future typo in a
release) would break the same way we just saw. The Dockerfile now
runs:
find /opt/spark/bin -type f -exec sed -i '1s|^.*$|#!/usr/bin/env bash|' {} +
which rewrites the first line of every bin/* script to a known-good
shebang. Idempotent, defensive, costs nothing.
4. docker-entrypoint.sh: simplified and made validation explicit.
Old version used an awk/sed pipeline to strip /usr/lib/jvm/ and
/opt/spark/bin from the existing PATH before prepending the new
values. That had a subtle bug: if the new JAVA_HOME was itself
under /usr/lib/jvm/ (e.g. /usr/lib/jvm/java-11-openjdk-amd64), the
strip would remove the new path too. New version just prepends the
resolved paths and leaves the old PATH alone. The new paths win
because they come first.
5. docker-entrypoint.sh: now validates the resolved paths BEFORE
exporting them. If JAVA_HOME/bin/java or SPARK_HOME/bin/spark-submit
are missing, the container fails fast with a clear hint instead of
letting a job submission die with an opaque 'no such file'. Also
logs the effective 'java' and 'spark-submit' paths (and java
version) to stderr at every start, so docker logs make the
resolution visible.
6. .env.example + docker-compose.yml: default JAVA_HOME updated to
/usr/lib/jvm/java-11-openjdk-amd64. Spark client 3.1.2 (hadoop2.7)
noted in the comment as the working combo.
163/146 still pass (no code changes to the app; Dockerfile + entrypoint
+ docs only). The new entrypoint was smoke-tested locally: validation
fires as expected (the local dev box has no JDK 11, which is exactly
the kind of misconfig the validation now catches at container start).
79 lines
3.2 KiB
Bash
79 lines
3.2 KiB
Bash
# Spark Executor MCP — environment template
|
|
# Copy to .env and edit. .env is gitignored.
|
|
#
|
|
# All env vars in this file are read by common/config.py (the single source
|
|
# of truth for application config). The only exception is the GUNICORN_*
|
|
# block at the bottom — those are read by gunicorn.conf.py.
|
|
|
|
# --- Data persistence ---
|
|
# Base directory for connections.json, pending_jobs.json, loguru logs/,
|
|
# and (by default) jobs/. Mount this from the host in production so
|
|
# state survives container restarts. The default ./data/ is fine in dev.
|
|
#
|
|
# SPARK_EXECUTOR_DATA_DIR=./data
|
|
# SPARK_EXECUTOR_DATA_DIR=/var/lib/spark-executor/data
|
|
|
|
# --- Job files (LLM-generated PySpark) ---
|
|
# Where generate_job_file writes PySpark source. Defaults to
|
|
# <SPARK_EXECUTOR_DATA_DIR>/jobs. Override to point at a larger disk
|
|
# (e.g. /var/spark-jobs) when the data volume is small.
|
|
#
|
|
# SPARK_EXECUTOR_JOBS_DIR=./data/jobs
|
|
# SPARK_EXECUTOR_JOBS_DIR=/var/spark-jobs
|
|
|
|
# --- YARN REST client ---
|
|
# Fallback URL when a Job's yarn_rm_url (snapshotted from its Connection
|
|
# at prepare_submit_job time) is unset. Set this OR per-Connection via
|
|
# save_connection.
|
|
#
|
|
# Examples:
|
|
# YARN_RESOURCE_MANAGER_URL=http://yarn-rm.prod.internal:8088
|
|
# YARN_RESOURCE_MANAGER_URL=https://yarn-rm.staging.example.com:8088
|
|
YARN_RESOURCE_MANAGER_URL=
|
|
|
|
# --- Loguru file sinks ---
|
|
# Base directory for loguru output. Subdirs debug/ and info/ are created
|
|
# automatically; rotated daily, gzipped, kept 30 days. Defaults to
|
|
# <SPARK_EXECUTOR_DATA_DIR>/logs. Override to point at a dedicated log
|
|
# volume (e.g. /var/log/spark-executor) or a network mount.
|
|
#
|
|
# SPARK_EXECUTOR_LOG_DIR=./data/logs
|
|
# SPARK_EXECUTOR_LOG_DIR=/var/log/spark-executor
|
|
|
|
# --- Loguru verbosity ---
|
|
# For stderr + the info-level file sink. The debug-level file sink
|
|
# always captures full DEBUG (audit trail regardless of level).
|
|
# DEBUG - default; full verbosity
|
|
# INFO - quieter; recommended for production
|
|
#
|
|
# SPARK_EXECUTOR_LOG_LEVEL=DEBUG
|
|
# SPARK_EXECUTOR_LOG_LEVEL=INFO
|
|
|
|
# --- Optional: JVM flags forwarded to spark-submit ---
|
|
# Useful for proxies, custom truststores, or driver memory caps.
|
|
# SPARK_SUBMIT_OPTS=-Dhttps.proxyHost=proxy.corp -Dhttps.proxyPort=3128
|
|
|
|
# --- Java + Spark install paths ---
|
|
# Where the openjdk-11-jre-headless JDK lives, and where the Spark
|
|
# distribution was extracted during the image build. Defaults match the
|
|
# Dockerfile's ENTRYPOINT script. Override if you mount a different
|
|
# Java (e.g. /usr/lib/jvm/java-17-openjdk-arm64 on some ARM hosts) or
|
|
# a pre-installed Spark from a host volume (e.g. /opt/spark-3.1.2-bin-hadoop2.7).
|
|
#
|
|
# The container's docker-entrypoint.sh re-derives PATH from these values
|
|
# at every start, so overriding them here actually changes which `java`
|
|
# and `spark-submit` binaries the gunicorn process picks up.
|
|
#
|
|
# JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
|
|
# SPARK_HOME=/opt/spark
|
|
|
|
# --- Gunicorn process model (see gunicorn.conf.py; NOT read by common/config.py) ---
|
|
# Defaults shown. These are read by gunicorn directly, not by the app.
|
|
# GUNICORN_WORKERS=2
|
|
# GUNICORN_THREADS=1
|
|
# GUNICORN_TIMEOUT=120 # generous; yarn logs can be slow
|
|
# GUNICORN_GRACEFUL_TIMEOUT=30
|
|
# GUNICORN_KEEPALIVE=5
|
|
# GUNICORN_BIND=0.0.0.0:8000
|
|
# GUNICORN_LOGLEVEL=info
|