Hydra: [Feature Request] Set all loggers log level to DEBUG except hydra

[Khoa-NT]

🚀 Feature Request

I want to set the logging to DEBUG level for all my logger except hydra.

Motivation

So I added a Config file: hydra/job_logging/custom.yaml following this Customizing logging

hydra:
  # python logging configuration for tasks
  job_logging:
    root:
      level: DEBUG

Is your feature request related to a problem? Please describe.

It returns error:

Traceback (most recent call last):
  File "train.py", line 22, in <module>
    main()
  File "/opt/conda/lib/python3.7/site-packages/hydra/main.py", line 24, in decorated_main
    strict=strict,
  File "/opt/conda/lib/python3.7/site-packages/hydra/_internal/utils.py", line 174, in run_hydra
    overrides=args.overrides,
  File "/opt/conda/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 86, in run
    job_subdir_key=None,
  File "/opt/conda/lib/python3.7/site-packages/hydra/plugins/common/utils.py", line 101, in run_job
    configure_log(config.hydra.job_logging, config.hydra.verbose)
  File "/opt/conda/lib/python3.7/site-packages/hydra/plugins/common/utils.py", line 24, in configure_log
    logging.config.dictConfig(conf)
  File "/opt/conda/lib/python3.7/logging/config.py", line 800, in dictConfig
    dictConfigClass(config).configure()
  File "/opt/conda/lib/python3.7/logging/config.py", line 496, in configure
    raise ValueError("dictionary doesn't specify a version")
ValueError: dictionary doesn't specify a version

Pitch

Describe the solution you'd like

I know we can set the hydra.verbose=[logger_1, logger_2, logger_3]
But it's time-consuming if I have a lot of loggers.

[omry]

That example seems wrong, I will take a look.

This is the full job logging config, see the version?

You have two choices:

1. have a full job logging specification (working example)
2. override the existing job logging specification (which is what you are actually trying to do).

The problem with 2, is that that you are actually replacing the hydra/job_logging with your own custom logging config, so really you are doing 1.

Try this:
Move the logging config override name outside of hydra/job_logging, for example into a log_override group:

defaults:
  - log_override: debug

log_override/debug.yaml:

hydra:
  job_logging:
    root:
      level: DEBUG

As a side note, you can also just override the log level from the command line (without all of this mess):

$ python foo.py hydra.job_logging.root.level=DEBUG

[omry]

I know we can set the hydra.verbose=[logger_1, logger_2, logger_3]
But it's time-consuming if I have a lot of loggers.

Another thing:
If your loggers are hierarchical:

• module.logger_1
• module.logger_2
• module.logger_3

You should be able to use

hydra.verbose=module

[Khoa-NT]

That example seems wrong, I will take a look.

This is the full job logging config, see the version?

You have two choices:

1. have a full job logging specification (working example)
2. override the existing job logging specification (which is what you are actually trying to do).

The problem with 2, is that that you are actually replacing the hydra/job_logging with your own custom logging config, so really you are doing 1.

From the Overview and following tutorials, I thought that the customize hydra config will override the existent original hydra configs instead of replacing them.

Thank you for clarifying my misunderstanding.

Try this:
Move the logging config override name outside of hydra/job_logging, for example into a log_override group:

defaults:
  - log_override: debug

log_override/debug.yaml:

hydra:
  job_logging:
    root:
      level: DEBUG

As a side note, you can also just override the log level from the command line (without all of this mess):

$ python foo.py hydra.job_logging.root.level=DEBUG

I followed your guides but also, all the DEBUG loggers from python lib are also printed.
Ex:

[2020-04-14 09:36:00,026][matplotlib][DEBUG] - $HOME=/root
[2020-04-14 09:36:00,027][matplotlib][DEBUG] - CONFIGDIR=/root/.config/matplotlib
[2020-04-14 09:36:00,027][matplotlib][DEBUG] - matplotlib data path: /opt/conda/lib/python3.7/site-packages/matplotlib/mpl-data
[2020-04-14 09:36:00,031][matplotlib][DEBUG] - loaded rc file /opt/conda/lib/python3.7/site-packages/matplotlib/mpl-data/matplotlibrc
[2020-04-14 09:36:00,033][matplotlib][DEBUG] - matplotlib version 3.1.3
[2020-04-14 09:36:00,033][matplotlib][DEBUG] - interactive is False
[2020-04-14 09:36:00,033][matplotlib][DEBUG] - platform is linux
[2020-04-14 09:36:00,034][matplotlib][DEBUG] - loaded modules: [

I know we can set the hydra.verbose=[logger_1, logger_2, logger_3]
But it's time-consuming if I have a lot of loggers.

Another thing:
If your loggers are hierarchical:

• module.logger_1
• module.logger_2
• module.logger_3

You should be able to use

hydra.verbose=module

I have many loggers in different python files but I think this is the best solution at this moment.
I will try this.
Because It prevents the logger from printing the odd DEBUG from the third party.

[omry]

Closing this.
If there is more interest in this feature I can consider adding some kind of support.