Three problems with the prior Dockerfile:
1. JAVA_HOME and SPARK_HOME were hardcoded at build time (ENV ...),
so docker-compose / .env overrides had no effect.
2. PATH was constructed at build time using those hardcoded values,
so even if you could override the env vars, PATH would still
reference the old literal paths (e.g. /usr/lib/jvm/java-17-openjdk-amd64/bin
hardcoded into PATH at image build).
3. There was no .env.example entry for either, so operators had no
template to follow.
Fix:
- docker-entrypoint.sh: re-derives PATH from the (possibly overridden)
JAVA_HOME and SPARK_HOME at every container start. Strips any stale
JDK/Spark bin dirs from PATH first so a restart with a new override
actually changes which / resolve.
- Dockerfile: COPY + ENTRYPOINT [entrypoint.sh], CMD [gunicorn main:app].
The existing ENV JAVA_HOME and ENV SPARK_HOME stay as the defaults
so the image works out of the box; users override via .env.
- .env.example: new 'Java + Spark install paths' section, default
values match the Dockerfile, comments explain the entrypoint
re-derivation.
- docker-compose.yml: forwards JAVA_HOME and SPARK_HOME with
container-side defaults matching the Dockerfile.
Verified:
- 116/116 tests still pass
- Direct entrypoint run with JAVA_HOME=/fake/jdk shows PATH rebuilt as
/app/.venv/bin:/fake/jdk/bin:/fake/spark/bin:... (override took effect)
Note: uses awk instead of 'paste -sd:' for portability across macOS
(BSD paste doesn't support -s) and Linux (GNU does).
Spark 4.1.2 requires Java 17 at runtime (it boots a JVM via the
bin/spark-submit shell script). The previous Dockerfile only installed
curl, ca-certificates, and tar — leaving spark-submit unable to find
the 'java' binary.
A JRE is sufficient (not the full JDK) because spark-submit does not
compile Java at runtime — it loads pre-built JARs from $SPARK_HOME/jars/.
JRE-only keeps the image ~160 MB smaller than JDK-headless.
Sets JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 and prepends it to
PATH so spark-submit (which checks JAVA_HOME) finds the right JVM.
Combined with the prior yarn-REST rewrite, the runtime image now needs
exactly three things in addition to python:3.12-slim:
- uv (from ghcr.io/astral-sh/uv)
- openjdk-17-jre-headless
- Spark 4.1.2 from Tsinghua mirror
Claude