Claude
f8e367d43c
Before: SPARK_EXECUTOR_DATA_DIR, SPARK_EXECUTOR_JOBS_DIR, and
YARN_RESOURCE_MANAGER_URL were each read directly via os.environ.get()
inside the module that used them. Log level was hardcoded DEBUG in
common/logging.py. There was no single file showing what the full set
of env vars the app reads is.
After: common/config.py defines a single Settings dataclass that
reads all env vars at import time and exposes them as fields on a
module-level singleton. App code uses "from common.config import
settings; settings.data_dir" etc. New SPARK_EXECUTOR_LOG_LEVEL env
var controls stderr + info file verbosity (debug file always gets full
DEBUG).
Improvements:
- One file lists every env var the app reads (was: grep the codebase)
- Tests can monkeypatch fields on the settings singleton directly
instead of monkeypatching the env + reloading
- Adding a new env var means adding one field in config.py, not
editing 3+ call sites
- settings.reload() method for tests that prefer env-var style
Out of scope (kept where they are):
- GUNICORN_* env vars live in gunicorn.conf.py (gunicorn concept)
- PYTHONUNBUFFERED in Dockerfile (Python runtime flag)
- SPARK_SUBMIT_OPTS not in config (JVM flag, not Python)
Test changes:
- test_job_writer.py: settings.jobs_dir instead of monkeypatching
SPARK_EXECUTOR_JOBS_DIR
- test_yarn_client.py: settings.yarn_resource_manager_url instead of
monkeypatching YARN_RESOURCE_MANAGER_URL
- test_generate_tool.py: same as job_writer
- Each test file gets an autouse fixture that snapshots+restores
settings so one test mutation does not leak into the next
116/116 still pass. Live verified: SPARK_EXECUTOR_LOG_LEVEL=INFO
suppresses DEBUG loguru output as expected.
55 lines
1.8 KiB
Python
55 lines
1.8 KiB
Python
# coding=utf-8
|
|
"""
|
|
@Time :2026/6/24
|
|
@Author :tao.chen
|
|
|
|
Writes LLM-generated PySpark code to disk so the agent can then call
|
|
prepare_submit_job(script_path=...) on the returned path.
|
|
|
|
Resolution order for the output directory:
|
|
1. explicit jobs_dir argument (used in tests)
|
|
2. SPARK_EXECUTOR_JOBS_DIR environment variable (operator override)
|
|
3. DEFAULT_JOBS_DIR constant (./data/jobs/ — gitignored, persists across
|
|
container restarts when ./data is mounted)
|
|
"""
|
|
import os
|
|
import secrets
|
|
from datetime import datetime
|
|
|
|
from common.config import settings
|
|
from common.logging import logger
|
|
|
|
ENV_JOBS_DIR = "SPARK_EXECUTOR_JOBS_DIR"
|
|
|
|
|
|
def resolve_jobs_dir(jobs_dir: str | None = None) -> str:
|
|
"""Pick the effective jobs directory in priority order: arg > settings.jobs_dir.
|
|
|
|
`settings.jobs_dir` itself is computed in `common/config.py` as:
|
|
SPARK_EXECUTOR_JOBS_DIR env var (if set) > <data_dir>/jobs
|
|
"""
|
|
if jobs_dir is not None:
|
|
return jobs_dir
|
|
return settings.jobs_dir
|
|
|
|
|
|
def write_job_file(code: str, jobs_dir: str | None = None) -> str:
|
|
"""Write `code` to <jobs_dir>/job_<timestamp>_<rand>.py; return abs path.
|
|
|
|
Creates the directory if missing. Filename is unique per call (timestamp
|
|
down to the second + 3 random bytes) so concurrent agents cannot collide.
|
|
"""
|
|
effective_dir = resolve_jobs_dir(jobs_dir)
|
|
os.makedirs(effective_dir, exist_ok=True)
|
|
stamp = datetime.utcnow().strftime("%Y%m%d%H%M%S")
|
|
name = f"job_{stamp}_{secrets.token_hex(3)}.py"
|
|
path = os.path.join(effective_dir, name)
|
|
abs_path = os.path.abspath(path)
|
|
logger.debug(
|
|
f"write_job_file enter jobs_dir={effective_dir} code_bytes={len(code)}"
|
|
)
|
|
with open(abs_path, "w", encoding="utf-8") as f:
|
|
f.write(code)
|
|
logger.info(f"write_job_file ok script_path={abs_path} bytes={len(code)}")
|
|
return abs_path
|