Loguru: how do i Use loguru in multi modules?

[popeaaa]

if i have 2 python script using loguru works together, but in different folders. how to set loguru so i can get one log file ?

[Delgan]

Hi @popeaaa.

You can log to the same file from different modules (i.e. Python files composing your application), just call .add() only once, the imported logger is the same object so share the same handlers.

Do you mean you have two applications run independently and executed by two different processes?

Well, if you .add("file.log") from the two apps, it risks to produce corrupted logs obviously. So you need some kind of communication for synchronization between the two processes. Maybe using the built-in SocketHandler with one server (writing logs to file) and one client (sending logs to server)?

[popeaaa]

Hi @popeaaa.

You can log to the same file from different modules (i.e. Python files composing your application), just call .add() only once, the imported logger is the same object so share the same handlers.

Do you mean you have two applications run independently and executed by two different processes?

Well, if you .add("file.log") from the two apps, it risks to produce corrupted logs obviously. So you need some kind of communication for synchronization between the two processes. Maybe using the built-in SocketHandler with one server (writing logs to file) and one client (sending logs to server)?

thank you ,it is 2 processes

[Delgan]

So I think you can use the snippet from the Logging Cookbook: Sending and receiving logging events across a network.

That would look like this:

# server.py
import time
import socketserver
import threading
import pickle
import struct
from loguru import logger

class LogRecordStreamHandler(socketserver.StreamRequestHandler):

    def handle(self):
        while True:
            chunk = self.connection.recv(4)
            if len(chunk) < 4:
                break
            slen = struct.unpack('>L', chunk)[0]
            chunk = self.connection.recv(slen)
            while len(chunk) < slen:
                chunk = chunk + self.connection.recv(slen - len(chunk))
            record = pickle.loads(chunk)
            logger.opt(raw=True).log(record['levelno'], record['msg'] + '\n')

address = ('localhost', 9999)
server = socketserver.TCPServer(address, LogRecordStreamHandler)

thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()

while 1:
    time.sleep(1)
    logger.info("Message from server")

# client.py
import time
from loguru import logger
from logging.handlers import SocketHandler

socket_handler = SocketHandler('localhost', 9999)

logger.remove()
logger.add(socket_handler)

while 1:
    time.sleep(1)
    logger.info("Message from client")

I tested it and it seems to work, but this is not as elegant as I would like. Ideally, you would just log the unpickled record, so that formatting depends on your server (and not client) handlers. Keep in mind that pickle is unsafe, so you maybe better should serialize the logs using JSON.

[popeaaa]

So I think you can use the snippet from the Logging Cookbook: Sending and receiving logging events across a network.

That would look like this:

# server.py
import time
import socketserver
import threading
import pickle
import struct
from loguru import logger

class LogRecordStreamHandler(socketserver.StreamRequestHandler):

    def handle(self):
        while True:
            chunk = self.connection.recv(4)
            if len(chunk) < 4:
                break
            slen = struct.unpack('>L', chunk)[0]
            chunk = self.connection.recv(slen)
            while len(chunk) < slen:
                chunk = chunk + self.connection.recv(slen - len(chunk))
            record = pickle.loads(chunk)
            logger.opt(raw=True).log(record['levelno'], record['msg'] + '\n')

address = ('localhost', 9999)
server = socketserver.TCPServer(address, LogRecordStreamHandler)

thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()

while 1:
    time.sleep(1)
    logger.info("Message from server")

# client.py
import time
from loguru import logger
from logging.handlers import SocketHandler

socket_handler = SocketHandler('localhost', 9999)

logger.remove()
logger.add(socket_handler)

while 1:
    time.sleep(1)
    logger.info("Message from client")

I tested it and it seems to work, but this is not as elegant as I would like. Ideally, you would just log the unpickled record, so that formatting depends on your server (and not client) handlers. Keep in mind that pickle is unsafe, so you maybe better should serialize the logs using JSON.

thanks a lot! happy Chinese new year!

[Mango-kid]

Im not sure the above example will work from multiple clients. I think you would have to remove after each message sent.

[Delgan]

@Mango-kid What do you mean? What should be removed?

[Mango-kid]

If you want to have multiple log clients connected you will have to

logger.remove()
logger.debug("some log message")
logger.add(socket_handler)

for every logging event. I dont think the server in the above cookbook code will handle more that one connection because each logger leaves the connection open.

[Delgan]

Oh, yes you're correct, thanks for pointing this out! The above snippet was a "translation" of the existing logging cookbook, but it needs to be adapted to accept multiple clients at the same time, indeed.