Hydra: [Feature Request] Automatically create directory hierarchy for custom loggers with file handlers

[meder-yembo]

🚀 Feature Request

Create the whole path to a log file specified in a custom logging YAML file.

Motivation

I create a file hierarchy when I run my application code, and my logs are written to a directory that do not exist when the initial run command is called. Instead it's written to <some_new_directory>/logs/<name>.log. Within this <some_new_directory> I only want to create a subdirectory on-demand, as in only if a file is to be written to it. Basically, I want to avoid empty folders. One such sub-folder is the logs folder. I only want this folder created if a log is to be written. So, I have customized my a logger as follows:

# @package _group_
version: 1
formatters:
  yembo:
    format: '[%(asctime)s] [%(name)s:%(pathname)s:%(lineno)d] [%(levelname)s]: %(message)s'
    dateformat: '%Y-%m-%dT%H:%M:%S'
filters:
  module_filter: 
    name: ${env:NAME}
handlers:
  console:
    level: ${env:LEVEL}
    class: logging.StreamHandler
    formatter: yembo
    stream: ext://sys.stdout
    filters: [module_filter]
  file:
    level: ${env:LEVEL}
    class: logging.FileHandler
    formatter: yembo
    filename: ${env:ROOT}/logs/${env:NAME}.log # <<<<<<<<< This is the key part
    filters: [module_filter]
root:
  level: ${env:LEVEL}
  handlers: [console, file]
  filters: [module_filter]

disable_existing_loggers: false

However, I must manually create the ${env:ROOT}/logs directory before running this or I get a FileNotFoundError when trying to write to the log file.

Pitch

I believe that this info is passed directly to configure a Python logger, so it's a Python logger issue in reality. However, as Hydra aims to provided customizable logging workflows easily, it might be useful to parse this dict for file handlers and confirm the the directory path exists, creating it if it doesn't.

Describe alternatives you've considered

The most obvious alternative is to make the intermediate directory structure manually (i.e. through some other application code).

Are you willing to open a pull request? (See CONTRIBUTING)

I'm not sure I'd have the expertise to do this well.

Additional context

[omry]

Hi, thanks for the feature request.
You are correct, the logging config is passed as is (after interpolations are resolved) to Python logging.dictConfig function.

I feel like parsing the logging config is not a healthy area for Hydra to take responsibility for.
Different logging handlers may have different "sensible" logic for creation of the path hierarchy. This really should be a feature request for the Python file handler.

[meder-yembo]

Got it. That's fair.