Three problems with the prior Dockerfile:
1. JAVA_HOME and SPARK_HOME were hardcoded at build time (ENV ...),
so docker-compose / .env overrides had no effect.
2. PATH was constructed at build time using those hardcoded values,
so even if you could override the env vars, PATH would still
reference the old literal paths (e.g. /usr/lib/jvm/java-17-openjdk-amd64/bin
hardcoded into PATH at image build).
3. There was no .env.example entry for either, so operators had no
template to follow.
Fix:
- docker-entrypoint.sh: re-derives PATH from the (possibly overridden)
JAVA_HOME and SPARK_HOME at every container start. Strips any stale
JDK/Spark bin dirs from PATH first so a restart with a new override
actually changes which / resolve.
- Dockerfile: COPY + ENTRYPOINT [entrypoint.sh], CMD [gunicorn main:app].
The existing ENV JAVA_HOME and ENV SPARK_HOME stay as the defaults
so the image works out of the box; users override via .env.
- .env.example: new 'Java + Spark install paths' section, default
values match the Dockerfile, comments explain the entrypoint
re-derivation.
- docker-compose.yml: forwards JAVA_HOME and SPARK_HOME with
container-side defaults matching the Dockerfile.
Verified:
- 116/116 tests still pass
- Direct entrypoint run with JAVA_HOME=/fake/jdk shows PATH rebuilt as
/app/.venv/bin:/fake/jdk/bin:/fake/spark/bin:... (override took effect)
Note: uses awk instead of 'paste -sd:' for portability across macOS
(BSD paste doesn't support -s) and Linux (GNU does).
common/logging.py used hardcoded 'data/logs/{debug,info}/' paths,
which forced logs into the same dir as connections.json. Operators
couldn't put logs on a dedicated volume or a different disk for
retention policy reasons.
Now the base log dir comes from settings.log_dir (env var
SPARK_EXECUTOR_LOG_DIR, default <data_dir>/logs). Subdirs debug/ and
info/ are auto-created. Same behavior in dev (./data/logs/), but in
prod you can do:
SPARK_EXECUTOR_LOG_DIR=/var/log/spark-executor # dedicated volume
SPARK_EXECUTOR_LOG_DIR=/mnt/slow-storage/logs # cold storage for old logs
Other changes:
- .env.example: new SPARK_EXECUTOR_LOG_DIR entry, split into its own
'Loguru file sinks' section
- docker-compose.yml: forwards SPARK_EXECUTOR_LOG_DIR with the
container-side default /app/data/logs (the existing ./data:/app/data
volume mount already covers this)
- common/config.py: Settings gets a new log_dir field, defaults
derived from data_dir; reload() also resets it
116/116 still pass. Live smoke verified: with
SPARK_EXECUTOR_LOG_DIR=/tmp/spark-logs, both
/tmp/spark-logs/info/2026-06-25.log
/tmp/spark-logs/debug/2026-06-25.log
are created on the first request.
.env.example now lists every env var read by the app, with a comment
explaining the meaning and the default. The previously-missing ones
(from the common/config.py refactor) are added:
- SPARK_EXECUTOR_DATA_DIR
- SPARK_EXECUTOR_JOBS_DIR
- SPARK_EXECUTOR_LOG_LEVEL
docker-compose.yml forwards all of them with sensible container-side
defaults (SPARK_EXECUTOR_DATA_DIR=/app/data, which is the volume
mount point from the host).
Split the env block in compose into two commented sections
('common/config.py knobs' vs 'gunicorn.conf.py knobs') so it's clear
which file each one is consumed by.
The GUNICORN_LOGLEVEL entry is also added (was missing). All other
GUNICORN_* knobs were already documented.
116/116 still pass.
docker-compose.yml:
- Builds the Dockerfile, exposes :8000
- Mounts ./data (persistent) and ./hadoop-conf (ro) for cluster configs
- Forwards YARN_RESOURCE_MANAGER_URL via env var (fallback for REST client)
- Healthcheck via /openapi.json (cheap liveness probe; MCP initialize
handshake would be more accurate but requires session-id plumbing)
- restart: unless-stopped for production
- Resource limit section commented out (server itself is lightweight)
.env.example:
- Documents the env vars compose expects
- .env is gitignored
Claude