Claude
cb909f7fea
Three fixes requested:
1. Spark 4.1.2 -> Spark 3.1.2 (Hadoop 2.7 prebuilt). Compatible with the
JDK 11 build below and a more conservative choice for production.
2. JDK 17 -> JDK 11. openjdk-11-jre-headless package; JAVA_HOME points
at /usr/lib/jvm/java-11-openjdk-amd64.
3. /usr/bin/env 'bach' no such file: defensive shebang fix. The
downloaded spark-3.1.2-bin-hadoop2.7.tgz happens to have a clean
shebang, but older 4.x distributions (and any future typo in a
release) would break the same way we just saw. The Dockerfile now
runs:
find /opt/spark/bin -type f -exec sed -i '1s|^.*$|#!/usr/bin/env bash|' {} +
which rewrites the first line of every bin/* script to a known-good
shebang. Idempotent, defensive, costs nothing.
4. docker-entrypoint.sh: simplified and made validation explicit.
Old version used an awk/sed pipeline to strip /usr/lib/jvm/ and
/opt/spark/bin from the existing PATH before prepending the new
values. That had a subtle bug: if the new JAVA_HOME was itself
under /usr/lib/jvm/ (e.g. /usr/lib/jvm/java-11-openjdk-amd64), the
strip would remove the new path too. New version just prepends the
resolved paths and leaves the old PATH alone. The new paths win
because they come first.
5. docker-entrypoint.sh: now validates the resolved paths BEFORE
exporting them. If JAVA_HOME/bin/java or SPARK_HOME/bin/spark-submit
are missing, the container fails fast with a clear hint instead of
letting a job submission die with an opaque 'no such file'. Also
logs the effective 'java' and 'spark-submit' paths (and java
version) to stderr at every start, so docker logs make the
resolution visible.
6. .env.example + docker-compose.yml: default JAVA_HOME updated to
/usr/lib/jvm/java-11-openjdk-amd64. Spark client 3.1.2 (hadoop2.7)
noted in the comment as the working combo.
163/146 still pass (no code changes to the app; Dockerfile + entrypoint
+ docs only). The new entrypoint was smoke-tested locally: validation
fires as expected (the local dev box has no JDK 11, which is exactly
the kind of misconfig the validation now catches at container start).
92 lines
3.6 KiB
YAML
92 lines
3.6 KiB
YAML
# spark-executor-mcp — production runtime
|
|
#
|
|
# Bring up with:
|
|
# docker compose up -d --build
|
|
#
|
|
# Prerequisites (one-time):
|
|
# 1. Put your YARN/Hadoop client configs in ./hadoop-conf/ (must contain
|
|
# core-site.xml + yarn-site.xml + hdfs-site.xml matching the target
|
|
# cluster). The Dockerfile's RUN mkdir -p already creates the dir, but
|
|
# it will be empty until you populate it.
|
|
# 2. Set YARN_RESOURCE_MANAGER_URL in .env (or export it in your shell).
|
|
# This is the fallback when a Connection was saved without yarn_rm_url.
|
|
#
|
|
# After it starts, the MCP endpoint is at:
|
|
# http://localhost:8000/spark-executor-mcp (initialize -> tools/list -> tools/call)
|
|
|
|
services:
|
|
mcp-server:
|
|
build:
|
|
context: .
|
|
dockerfile: Dockerfile
|
|
image: mcp-server:latest
|
|
container_name: mcp-tools
|
|
|
|
restart: unless-stopped
|
|
|
|
ports:
|
|
- "8000:8000"
|
|
|
|
environment:
|
|
# Stream Python output to stdout/stderr line-by-line (live loguru output).
|
|
PYTHONUNBUFFERED: "1"
|
|
|
|
# --- common/config.py knobs (see .env.example for full docs) ---
|
|
# Base dir for connections.json, pending_jobs.json, loguru logs/, jobs/.
|
|
# Defaults to ./data inside the container (mounted from host via volumes below).
|
|
SPARK_EXECUTOR_DATA_DIR: ${SPARK_EXECUTOR_DATA_DIR:-/app/data}
|
|
|
|
# Where generate_job_file writes LLM-generated PySpark code.
|
|
# Defaults to <SPARK_EXECUTOR_DATA_DIR>/jobs.
|
|
SPARK_EXECUTOR_JOBS_DIR: ${SPARK_EXECUTOR_JOBS_DIR:-/app/data/jobs}
|
|
|
|
# Base dir for loguru file output (debug/ + info/ subdirs are
|
|
# auto-created). Defaults to <SPARK_EXECUTOR_DATA_DIR>/logs.
|
|
SPARK_EXECUTOR_LOG_DIR: ${SPARK_EXECUTOR_LOG_DIR:-/app/data/logs}
|
|
|
|
# Fallback YARN RM URL used by the REST client when a Connection's
|
|
# yarn_rm_url is not set or a Job lacks a snapshot. Leave empty if
|
|
# you always set yarn_rm_url per Connection via save_connection.
|
|
YARN_RESOURCE_MANAGER_URL: ${YARN_RESOURCE_MANAGER_URL:-}
|
|
|
|
# Loguru verbosity for stderr + info file. DEBUG | INFO.
|
|
SPARK_EXECUTOR_LOG_LEVEL: ${SPARK_EXECUTOR_LOG_LEVEL:-DEBUG}
|
|
|
|
# --- Runtime paths (consumed by docker-entrypoint.sh, not common/config.py) ---
|
|
# Override to point at a different JDK install or pre-mounted Spark
|
|
# distribution. The entrypoint re-derives PATH from these at every
|
|
# container start, so overrides actually take effect.
|
|
# JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
|
|
# SPARK_HOME=/opt/spark
|
|
JAVA_HOME: ${JAVA_HOME:-/usr/lib/jvm/java-11-openjdk-amd64}
|
|
SPARK_HOME: ${SPARK_HOME:-/opt/spark}
|
|
|
|
# --- gunicorn.conf.py knobs (NOT read by common/config.py) ---
|
|
# 2 workers is a good default for a small MCP service; raise for
|
|
# high-concurrency deploys.
|
|
GUNICORN_WORKERS: ${GUNICORN_WORKERS:-2}
|
|
GUNICORN_TIMEOUT: ${GUNICORN_TIMEOUT:-120}
|
|
GUNICORN_BIND: ${GUNICORN_BIND:-0.0.0.0:8000}
|
|
|
|
# Optional: pass JVM options to spark-submit (e.g. for proxies, memory).
|
|
# SPARK_SUBMIT_OPTS: "-Dhttps.proxyHost=..."
|
|
|
|
volumes:
|
|
# Persist Connections, PendingSubmissions, and loguru logs across
|
|
# container restarts. Gitignored.
|
|
- ./data:/app/data
|
|
|
|
# Real Hadoop/YARN client configs read by spark-submit at submit time.
|
|
# Read-only so the running container cannot mutate cluster config.
|
|
- ./hadoop-conf:/etc/hadoop/conf:ro
|
|
|
|
|
|
# No resource limits — spark-submit talks to YARN, which does the actual
|
|
# heavy lifting. The MCP server itself is lightweight (FastAPI + httpx).
|
|
# Uncomment to cap if needed:
|
|
# deploy:
|
|
# resources:
|
|
# limits:
|
|
# cpus: "1.0"
|
|
# memory: 1G
|