Airflow: Loading logging configuration from .yaml or .ini file

[d-lee]

It would be great to implement loading of logging configuration from an external config file. It would allow for a much more flexibility and control over logging output from various modules without a need to modify any code.

This way we can dynamically switch between different log formatters (e.g. standard and debug. with debug being more verbose.. say outputting function names, code line numbers, etc which we would not normally need in a standard log output); or reduce/elevate logging level for specific modules (e.g. have airflow log at INFO level while restrict boto and pycrypto loggers to WARNING); or define several different log handlers (e.g. have a separate error handler that only captures ERROR and CRITICAL events and stores them in a separate file)

[mistercrunch]

Is there a standard for doing this? We want to stay as close to standard as possible.

[d-lee]

I am not sure there is a standard but I believe that the commonly accepted practice is to use logging.config (https://docs.python.org/2/library/logging.config.html). You can use either dictConfig, if you load from .yaml, .json; or fileConfig if you use .ini style config files. fileConfig is slowly phased out, some things you can do with dictConfig are not implemented (and not going to be implemented) in fileConfig.

Ideally this should be combined with hierarchical approach to logging. I.e. each module defines it's own logger via logger = logging.getLogger(__name__) and then do all logging operations using this module level logger instead of a root logger (logger.info('Foo bar')).

This guy has a fairly comprehensive example of best logging practices in Python - http://victorlin.me/posts/2012/08/26/good-logging-practice-in-python

Privately we use modified settings.py with alttered configure_logging() that loads config from logging.yaml.

I can file a PR if anyone else is interested

[sergiohgz]

Hi @d-lee and @mistercrunch,

We need a module for changing the standard logging configuration to follow our applications standards because we index them and use for metrics and looking for problems. If need help with this, I can do it.

Thank you so much

[Attsun1031]

As @d-lee said, and as far as I know, loading log configuration from ini file or yaml file is ideal and standard way.
I prefer ini style by following reasons.

• You can write log config in airflow.cfg
• To read yaml, you need 3rd party library

I worry about run command log because log filename is dynamically decided at runtime, so we cannot set at static conifguration file. （https://github.com/airbnb/airflow/blob/master/airflow/bin/cli.py#L141）