- Dockerfile: COPY files_mcp package, mkdir -p /app/data/files at build
time, set ENV FILES_MCP_ROOT=/app/data/files as the default sandbox
root (overridable at runtime).
- docker-compose.yml: surface FILES_MCP_ROOT in the env block with the
same default, so operators can override it via .env or -e.
FILES_MCP_ROOT must exist at process start (path_guard._init_root
checks is_dir()), so the directory is created during build rather than
deferred to entrypoint — fail-fast at image build beats a runtime
RuntimeError on every container start.
Default lives under /app/data so it rides the existing data volume
(./data:/app/data) and persists across container restarts alongside
connections.json / pending_jobs.json / logs/.
Verified locally: docker compose config --quiet passes; FILES_MCP_ROOT
resolves to /app/data/files in the rendered config. Full image build
not run (docker daemon unavailable in this environment); please run
`docker compose up -d --build` to confirm in your environment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three fixes requested:
1. Spark 4.1.2 -> Spark 3.1.2 (Hadoop 2.7 prebuilt). Compatible with the
JDK 11 build below and a more conservative choice for production.
2. JDK 17 -> JDK 11. openjdk-11-jre-headless package; JAVA_HOME points
at /usr/lib/jvm/java-11-openjdk-amd64.
3. /usr/bin/env 'bach' no such file: defensive shebang fix. The
downloaded spark-3.1.2-bin-hadoop2.7.tgz happens to have a clean
shebang, but older 4.x distributions (and any future typo in a
release) would break the same way we just saw. The Dockerfile now
runs:
find /opt/spark/bin -type f -exec sed -i '1s|^.*$|#!/usr/bin/env bash|' {} +
which rewrites the first line of every bin/* script to a known-good
shebang. Idempotent, defensive, costs nothing.
4. docker-entrypoint.sh: simplified and made validation explicit.
Old version used an awk/sed pipeline to strip /usr/lib/jvm/ and
/opt/spark/bin from the existing PATH before prepending the new
values. That had a subtle bug: if the new JAVA_HOME was itself
under /usr/lib/jvm/ (e.g. /usr/lib/jvm/java-11-openjdk-amd64), the
strip would remove the new path too. New version just prepends the
resolved paths and leaves the old PATH alone. The new paths win
because they come first.
5. docker-entrypoint.sh: now validates the resolved paths BEFORE
exporting them. If JAVA_HOME/bin/java or SPARK_HOME/bin/spark-submit
are missing, the container fails fast with a clear hint instead of
letting a job submission die with an opaque 'no such file'. Also
logs the effective 'java' and 'spark-submit' paths (and java
version) to stderr at every start, so docker logs make the
resolution visible.
6. .env.example + docker-compose.yml: default JAVA_HOME updated to
/usr/lib/jvm/java-11-openjdk-amd64. Spark client 3.1.2 (hadoop2.7)
noted in the comment as the working combo.
163/146 still pass (no code changes to the app; Dockerfile + entrypoint
+ docs only). The new entrypoint was smoke-tested locally: validation
fires as expected (the local dev box has no JDK 11, which is exactly
the kind of misconfig the validation now catches at container start).
Three problems with the prior Dockerfile:
1. JAVA_HOME and SPARK_HOME were hardcoded at build time (ENV ...),
so docker-compose / .env overrides had no effect.
2. PATH was constructed at build time using those hardcoded values,
so even if you could override the env vars, PATH would still
reference the old literal paths (e.g. /usr/lib/jvm/java-17-openjdk-amd64/bin
hardcoded into PATH at image build).
3. There was no .env.example entry for either, so operators had no
template to follow.
Fix:
- docker-entrypoint.sh: re-derives PATH from the (possibly overridden)
JAVA_HOME and SPARK_HOME at every container start. Strips any stale
JDK/Spark bin dirs from PATH first so a restart with a new override
actually changes which / resolve.
- Dockerfile: COPY + ENTRYPOINT [entrypoint.sh], CMD [gunicorn main:app].
The existing ENV JAVA_HOME and ENV SPARK_HOME stay as the defaults
so the image works out of the box; users override via .env.
- .env.example: new 'Java + Spark install paths' section, default
values match the Dockerfile, comments explain the entrypoint
re-derivation.
- docker-compose.yml: forwards JAVA_HOME and SPARK_HOME with
container-side defaults matching the Dockerfile.
Verified:
- 116/116 tests still pass
- Direct entrypoint run with JAVA_HOME=/fake/jdk shows PATH rebuilt as
/app/.venv/bin:/fake/jdk/bin:/fake/spark/bin:... (override took effect)
Note: uses awk instead of 'paste -sd:' for portability across macOS
(BSD paste doesn't support -s) and Linux (GNU does).
Spark 4.1.2 requires Java 17 at runtime (it boots a JVM via the
bin/spark-submit shell script). The previous Dockerfile only installed
curl, ca-certificates, and tar — leaving spark-submit unable to find
the 'java' binary.
A JRE is sufficient (not the full JDK) because spark-submit does not
compile Java at runtime — it loads pre-built JARs from $SPARK_HOME/jars/.
JRE-only keeps the image ~160 MB smaller than JDK-headless.
Sets JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 and prepends it to
PATH so spark-submit (which checks JAVA_HOME) finds the right JVM.
Combined with the prior yarn-REST rewrite, the runtime image now needs
exactly three things in addition to python:3.12-slim:
- uv (from ghcr.io/astral-sh/uv)
- openjdk-17-jre-headless
- Spark 4.1.2 from Tsinghua mirror
