Claude
f8e367d43c
Before: SPARK_EXECUTOR_DATA_DIR, SPARK_EXECUTOR_JOBS_DIR, and
YARN_RESOURCE_MANAGER_URL were each read directly via os.environ.get()
inside the module that used them. Log level was hardcoded DEBUG in
common/logging.py. There was no single file showing what the full set
of env vars the app reads is.
After: common/config.py defines a single Settings dataclass that
reads all env vars at import time and exposes them as fields on a
module-level singleton. App code uses "from common.config import
settings; settings.data_dir" etc. New SPARK_EXECUTOR_LOG_LEVEL env
var controls stderr + info file verbosity (debug file always gets full
DEBUG).
Improvements:
- One file lists every env var the app reads (was: grep the codebase)
- Tests can monkeypatch fields on the settings singleton directly
instead of monkeypatching the env + reloading
- Adding a new env var means adding one field in config.py, not
editing 3+ call sites
- settings.reload() method for tests that prefer env-var style
Out of scope (kept where they are):
- GUNICORN_* env vars live in gunicorn.conf.py (gunicorn concept)
- PYTHONUNBUFFERED in Dockerfile (Python runtime flag)
- SPARK_SUBMIT_OPTS not in config (JVM flag, not Python)
Test changes:
- test_job_writer.py: settings.jobs_dir instead of monkeypatching
SPARK_EXECUTOR_JOBS_DIR
- test_yarn_client.py: settings.yarn_resource_manager_url instead of
monkeypatching YARN_RESOURCE_MANAGER_URL
- test_generate_tool.py: same as job_writer
- Each test file gets an autouse fixture that snapshots+restores
settings so one test mutation does not leak into the next
116/116 still pass. Live verified: SPARK_EXECUTOR_LOG_LEVEL=INFO
suppresses DEBUG loguru output as expected.
58 lines
1.5 KiB
Python
58 lines
1.5 KiB
Python
# coding=utf-8
|
|
"""
|
|
@Time :2026/6/24
|
|
@Author :tao.chen
|
|
|
|
Process-wide loguru configuration. Import `logger` from here in every
|
|
module instead of instantiating new loggers.
|
|
|
|
Log level is controlled via `SPARK_EXECUTOR_LOG_LEVEL` (see common/config.py):
|
|
DEBUG - default; full verbosity
|
|
INFO - quieter; recommended for production
|
|
|
|
Levels used in this project:
|
|
DEBUG - entry/exit of public tools, subprocess commands, file I/O paths
|
|
INFO - business events (job submitted, status changed, connection saved)
|
|
WARNING - recoverable problems (transient YARN issues, retry-able)
|
|
ERROR - raised exceptions (caller will see the traceback)
|
|
"""
|
|
import sys
|
|
from pathlib import Path
|
|
from loguru import logger
|
|
|
|
from common.config import settings
|
|
|
|
Path("data/logs/debug").mkdir(parents=True, exist_ok=True)
|
|
Path("data/logs/info").mkdir(parents=True, exist_ok=True)
|
|
|
|
logger.remove()
|
|
logger.add(
|
|
sys.stderr,
|
|
level=settings.log_level,
|
|
format=(
|
|
"<green>{time:HH:mm:ss.SSS}</green> | "
|
|
"<level>{level: <7}</level> | "
|
|
"<cyan>{name}</cyan>:<cyan>{function}</cyan>:<cyan>{line}</cyan> - "
|
|
"<level>{message}</level>"
|
|
),
|
|
enqueue=True,
|
|
colorize=True,
|
|
)
|
|
|
|
logger.add(
|
|
"data/logs/debug/{time:YYYY-MM-DD}.log",
|
|
level="DEBUG", # always full at the file level for audit
|
|
enqueue=True,
|
|
retention="30 days",
|
|
compression="gz",
|
|
colorize=True,
|
|
)
|
|
|
|
logger.add(
|
|
"data/logs/info/{time:YYYY-MM-DD}.log",
|
|
level=settings.log_level,
|
|
enqueue=True,
|
|
retention="30 days",
|
|
compression="gz",
|
|
encoding="utf-8",
|
|
) |