refactor: env-configurable loguru file paths

common/logging.py used hardcoded 'data/logs/{debug,info}/' paths,
which forced logs into the same dir as connections.json. Operators
couldn't put logs on a dedicated volume or a different disk for
retention policy reasons.

Now the base log dir comes from settings.log_dir (env var
SPARK_EXECUTOR_LOG_DIR, default <data_dir>/logs). Subdirs debug/ and
info/ are auto-created. Same behavior in dev (./data/logs/), but in
prod you can do:

  SPARK_EXECUTOR_LOG_DIR=/var/log/spark-executor   # dedicated volume
  SPARK_EXECUTOR_LOG_DIR=/mnt/slow-storage/logs   # cold storage for old logs

Other changes:
  - .env.example: new SPARK_EXECUTOR_LOG_DIR entry, split into its own
    'Loguru file sinks' section
  - docker-compose.yml: forwards SPARK_EXECUTOR_LOG_DIR with the
    container-side default /app/data/logs (the existing ./data:/app/data
    volume mount already covers this)
  - common/config.py: Settings gets a new log_dir field, defaults
    derived from data_dir; reload() also resets it

116/116 still pass. Live smoke verified: with
SPARK_EXECUTOR_LOG_DIR=/tmp/spark-logs, both
  /tmp/spark-logs/info/2026-06-25.log
  /tmp/spark-logs/debug/2026-06-25.log
are created on the first request.
This commit is contained in:
Claude
2026-06-25 10:57:50 +08:00
parent 52624aacff
commit 13e39b39f3
4 changed files with 36 additions and 7 deletions
+4
View File
@@ -40,6 +40,10 @@ services:
# Defaults to <SPARK_EXECUTOR_DATA_DIR>/jobs.
SPARK_EXECUTOR_JOBS_DIR: ${SPARK_EXECUTOR_JOBS_DIR:-/app/data/jobs}
# Base dir for loguru file output (debug/ + info/ subdirs are
# auto-created). Defaults to <SPARK_EXECUTOR_DATA_DIR>/logs.
SPARK_EXECUTOR_LOG_DIR: ${SPARK_EXECUTOR_LOG_DIR:-/app/data/logs}
# Fallback YARN RM URL used by the REST client when a Connection's
# yarn_rm_url is not set or a Job lacks a snapshot. Leave empty if
# you always set yarn_rm_url per Connection via save_connection.