Archivebox: Question: Logging

[MartinThoma]

I'm currently writing an article about logging in Python. As a part of that, I look at other projects and how logging is done there. I've seen logging_util.py and was wondering why this project does not use the built-in logging module.

Could somebody share some thoughts around logging? Why is it done like that in this project? Do you consider this a best practice?

[pirate]

It's not a best practice, I just tend to write my own logging utils because I like very fine-grained control over how colored log output appears and I find the logging builtin library to be somewhat clunky.

[MartinThoma]

Thank you for answering!

If it's not a best practice, what would you consider a best practice for logging?

Have you given 3rd party logging libraries a try (structlog, loguru, pysnooper, or something else)? If so, what do you think of them?

[MartinThoma]

I find the logging builtin library to be somewhat clunky

Could you explain that a bit more? What exactly is it that you don't like?

I think I have heard about loguru at Python bytes and they mentioned that it deals pretty well with colored output (e.g. over various consoles / terminals / within PyCharm)

[pirate]

Those libraries would all be great for a long-running background daemon that writes to a logfile, e.g. archivebox server, but the output is far too verbose/hard-to-tune for a one-shot cli command like archivebox add, archivebox version, etc.

Because I sometimes want timestamps, I sometimes want color, I sometimes want structured output sections, but not always, I it's easier for me to just hand code it than use a logger that imposes those things all the time or never.

Take for example this output from archivebox add:

There are multiple tiers of indentation that are context-dependent, multiple colors for different types of notifications that are tuned for UX, not for strict internal consistency with any particular "log level" e.g. DEBUG/INFO/etc. (which a library would enforce). I also have multiple colors on a single line to highlight important parts of the line, something not any of the tools let you do easily.

I also dislike having to import/declare a logger at the top of every file, something that neither matches with my mental model of logging, nor is enforced by any linter, so it's easy to forget and cause subtle logging errors in certain environments.

At the end of the day, why use a dependency for something that I can code in half a day by hand, and matches my needs exactly. In general in my Python projects, I try to only use dependencies for big internal features that cannot easily be written by hand. I dislike the NPM-style of dependency management that encourages hundreds of 10-line dependencies for small conveniences.

Not to mention, it's much more straightforward when debugging to read something like this, than to have to read the docs on a 3rd party dependency and understand how it works:

print('{green}# ArchiveBox Imports{reset}'.format(**ANSI))
print('{green}from archivebox.core.models import Snapshot, User{reset}'.format(**ANSI))
print('{green}from archivebox import *\n    {}{reset}'.format("\n    ".join(list_subcommands().keys()), **ANSI))
print()
print('[i] Welcome to the ArchiveBox Shell!')
print('    https://github.com/pirate/ArchiveBox/wiki/Usage#Shell-Usage')
print()
print('    {lightred}Hint:{reset} Example use:'.format(**ANSI))
print('        print(Snapshot.objects.filter(is_archived=True).count())')
print('        Snapshot.objects.get(url="https://example.com").as_json()')
print('        add("https://example.com/some/new/url")')

---

In a new project I would consider using either normal print statements, or if absolutely necessary, the logging python built-in library as a best practice. Unless you have a massive project or a project thats core UX centers around its logfiles, I would not add additional dependencies to manage logging. (archivebox is <6k lines of python)