refactor: env-configurable loguru file paths
common/logging.py used hardcoded 'data/logs/{debug,info}/' paths,
which forced logs into the same dir as connections.json. Operators
couldn't put logs on a dedicated volume or a different disk for
retention policy reasons.
Now the base log dir comes from settings.log_dir (env var
SPARK_EXECUTOR_LOG_DIR, default <data_dir>/logs). Subdirs debug/ and
info/ are auto-created. Same behavior in dev (./data/logs/), but in
prod you can do:
SPARK_EXECUTOR_LOG_DIR=/var/log/spark-executor # dedicated volume
SPARK_EXECUTOR_LOG_DIR=/mnt/slow-storage/logs # cold storage for old logs
Other changes:
- .env.example: new SPARK_EXECUTOR_LOG_DIR entry, split into its own
'Loguru file sinks' section
- docker-compose.yml: forwards SPARK_EXECUTOR_LOG_DIR with the
container-side default /app/data/logs (the existing ./data:/app/data
volume mount already covers this)
- common/config.py: Settings gets a new log_dir field, defaults
derived from data_dir; reload() also resets it
116/116 still pass. Live smoke verified: with
SPARK_EXECUTOR_LOG_DIR=/tmp/spark-logs, both
/tmp/spark-logs/info/2026-06-25.log
/tmp/spark-logs/debug/2026-06-25.log
are created on the first request.
This commit is contained in:
@@ -40,6 +40,10 @@ services:
|
||||
# Defaults to <SPARK_EXECUTOR_DATA_DIR>/jobs.
|
||||
SPARK_EXECUTOR_JOBS_DIR: ${SPARK_EXECUTOR_JOBS_DIR:-/app/data/jobs}
|
||||
|
||||
# Base dir for loguru file output (debug/ + info/ subdirs are
|
||||
# auto-created). Defaults to <SPARK_EXECUTOR_DATA_DIR>/logs.
|
||||
SPARK_EXECUTOR_LOG_DIR: ${SPARK_EXECUTOR_LOG_DIR:-/app/data/logs}
|
||||
|
||||
# Fallback YARN RM URL used by the REST client when a Connection's
|
||||
# yarn_rm_url is not set or a Job lacks a snapshot. Leave empty if
|
||||
# you always set yarn_rm_url per Connection via save_connection.
|
||||
|
||||
Reference in New Issue
Block a user