Lightning: Write logs to disk automatically

[cdecker]

Besides keeping logs in memory for live inspection, and printing them to stdout, we should also have a log file to which log-entries get appended. This should make debugging a lot easier since users tend to not tee logs into files until something bad happens, and telling them that they should save logs in case it happens again is just not helpful.

[gabridome]

[niceToHave] It would be nice to put in place some kind of log rotation too maybe.

[jb55]

@gabridome

[niceToHave] It would be nice to put in place some kind of log rotation too maybe.

that's probably best left to tools like logrotate

[gabridome]

that's probably best left to tools like logrotate

Absolutely but as @cdecker says:

This should make debugging a lot easier since users tend to not tee logs into files until something bad happens, and telling them that they should save logs in case it happens again is just not helpful.

If our aim is to manage log information for debugging purposes, maybe it is better to manage them in a proper way ourselves also helping him to find the information we need. Logrotate is not a user oriented tool. Just pondering.

I have only today zipped my lightning.log file (700M) into a 25M one. I have sweared to myself I will configure logrotate but I don't think the average user will do it.

[cdecker]

Well, my idea is to basically give the usual hooks for logrotate et al., i.e., register a SIGUSR1 handler (used for IO logging on subdaemons but master it's not yet) that reopens the log-file. This allows logrotate to move the file (not invalidating the fd that lightningd uses to append logs lines), then issue SIGUSR1 to lightningd, which causes it to detach from the moved one and create a new one, and then logrotate can do whatever it pleases with the rotated log.

[rustyrussell]

SIGHUP is standard for log rotation. But I agree...

[cdecker]

1847 only partially addressed this issue. Its main goal is to always log to file, independently of whether we also log to stdout or not. The problem is that by default we just log to stdout and users are losing the logs in the scrollback buffers of their terminals making it really hard to trace issues.