Loguru: How to configure multiple logger like standard logging.config.dictConfig

[Gideon-koutian]

hi,I read through the readme and api reference but didnot find out how to get multiple logger instances.
here is my configure of logging loggers:

in my project,i rely on getLogger("xxx") to process the log instead of getLogger() to distribute the log. And I don't want some of the logs to be passed to multiple loggers, so I added propagate: no to each logger.
in loguru's api reference，I only found the filter Parameters in logger.add to handle multiple loggers.
but mine multiple loggers are stored in the same log, if i used filter=“xxx.log”,log record will send to multiple loggers,so how to deal with my scene

[Delgan]

Hey @Gideon-koutian!

I think you could use .bind() to replicate your current logging configuration.

For example, instead of logger = logging.getLogger("xxx"), you could do logger = logger.bind(name="xxx").

As a result, each message logged with this bound logger will contain the name value in the extra record dict that you can use to filter logs adequately.

# Convenient function to avoid repetition, but you could simply go with a lambda
def make_filter(name):
    def filter(record):
        return record["extra"].get("name") == name
    return filter

logger.add("params.log", level="INFO", filter=make_filter("params"))
logger.add("debug.log", level="DEBUG", filter=make_filter("debug"))
logger.add("api.log", level="INFO", filter=make_filter("api"))
logger.add("scheduler.log", level="INFO", filter=make_filter("scheduler"))

api_logger = logger.bind(name="api")
api_logger.info("This message is only propagated to the 'API' handler")

Depending on how you intend to handle incoming log messages, maybe can you use just one custom sink and filter them in your function directly:

def sink(message):
    record = message.record
    name = record["extra"].get("name")

    if name == "params":
        do_something(message)
    elif name == "debug":
        do_something_else(message)
    ...

logger.add(sink)

I admit that this looks less convenient than built-in logging, though. 😕

Do you think using Loguru like this would suit your needs?

[Gideon-koutian]

first I changed my configuration to this

logger.add(
    handlers.RotatingFileHandler(filename="log/jetty_stdout.log", mode='a', backupCount=9, maxBytes=eval('1024*1024'),
                                 encoding="utf-8", delay=True), level=logging.INFO, format=_format, enqueue=True,
    backtrace=True, filter=lambda record: True if "LOG" in record["extra"] else False)
logger.add(
    handlers.RotatingFileHandler(filename="log/scheduler_stdout.log", mode='a', backupCount=9,
                                 maxBytes=eval('1024*1024'),
                                 encoding="utf-8", delay=True), level=logging.INFO, format=_format, enqueue=True,
    backtrace=True, filter=lambda record: True if "SCHEDULER" in record["extra"] else False)
logger.add(
    handlers.TimedRotatingFileHandler(filename="log/error.log", when="midnight", interval=1, backupCount=6, utc=True,
                                      delay=False, encoding="utf-8"), level=logging.ERROR, format=_format, enqueue=True,
    backtrace=True, filter=lambda record: True if "LOG" in record["extra"] else False)
logger.add(
    handlers.TimedRotatingFileHandler(filename="log/api_jetty_stdout.log", when="midnight", interval=1, backupCount=6,
                                      utc=True, delay=False, encoding="utf-8"), level=logging.INFO, format=_format,
    enqueue=True, backtrace=True, filter=lambda record: True if "API" in record["extra"] else False)

````

Then, I replaced my original logging configuration code with this configuration.
But I found loguru missed supports the delay parameter in logging.Hander, fortunately, no effect
```python
logger.configure(**{
    "handlers": [
        {"sink": sys.stdout, "format": _format, "backtrace": True, "catch": True,
         "filter": lambda record: False if "SCHEDULER" in record["extra"] else True},
        {"sink": LOGGING_JERTY, "level": logging.INFO, "format": _format, "backtrace": True,
         "catch": True, "rotation": "10 MB", "retention": 9, "compression": "gz",
         "filter": lambda record: False if "SCHEDULER" in record["extra"] else True},
        {"sink": LOGGING_ERROR, "level": logging.ERROR, "format": _format, "backtrace": True,
         "catch": True, "rotation": "w0", "retention": 6, "compression": "gz",
         "filter": lambda record: False if "SCHEDULER" in record["extra"] else True},
        {"sink": LOGGING_SCHEDULER, "level": logging.INFO, "format": _format, "backtrace": True,
         "catch": True, "rotation": "10 MB", "retention": 1, "compression": "gz",
         "filter": lambda record: True if "SCHEDULER" in record["extra"] else False}
    ]
})

After that, I will try to change this code to a configuration file.😁
and I hope to have more interaction with the Hander of the standard library.

[Delgan]

I hope this is not too much burden compared to the standard logging library. :grimacing:

But I found loguru missed supports the delay parameter in logging.Hander, fortunately, no effect

Do you mean there is a problem with delay parameter of Loguru's sink? The code logger.add("file.log", delay=True) should work if the sink ("file.log" here) is a str or a Path. It has no effect if the sink is of another type.

I hope to have more interaction with the Hander of the standard library.

What would you suggest to Loguru?

[Delgan]

@Gideon-koutian Are you satisfied with your final configuration? Anything to add to this issue? 🙂

[Delgan]

Feel free to re-open the issue if you have others questions. 👍

[D3f0]

I'd like to allow to define/override logging settings in a separate file (a TOML file), that provides a set of dictionaries. I see that some sink settings expect to receive sys.stderr, and reverences that cannot be "serialized". Any suggestions?

[Delgan]

Hi @D3f0. 🙂

The problem you describe is one of the reasons I did not implement any method to load a configuration from a file.

Honestly, I don't what would be the best solution. The standard logging library partly solves it by specifying a special syntax: _Access to external objects_.
So, you could state that 'ext://sys.stderr' stands for sys.stderr, and then tranform the TOML dict in your Python script. Depending on the dynamicity you are looking for, you could simply use pattern matching like {"ext:://sys.stderr": sys.stderr} or implement a proper resolver as done by the standard library: cpython/logging/config.py.

But when it comes to parametrizing handlers with functions, it's even more complicated. If you wish to support this too, I guess you need to define a set of pre-wrote functions in your Python script that you can parametrize in the TOML file by using their identifier.

Basically, the simplest solution would look like this I think:

resolver = {
    "ext://stderr": sys.stderr,
    "ext://stdout": sys.stdout,
    "ext://database": lambda msg: db.update(msg.record),
}

for handler in toml_config["handlers"]:
     for key, value in handler.items():
         handler[key] = resolver.get(value, value)

loguru.config(toml_config)

I suppose you probably already have thought of this solution. I don't have any better idea for now, but I'm interested to know the solution you will choose. Alternatively, you could also define the logging configuration dict inside some kind of config.py file.