Bigchaindb: [logging] decide on log levels and document their usage

[sbellem]

Python's logging module provides the following levels of logging:

| Level | Numeric value |
| ------------------ | -------------------- |
| CRITICAL | 50 |
| ERROR | 40 |
| WARNING | 30 |
| INFO | 20 |
| DEBUG | 10 |
| NOTSET | 0 |

We already wish to use DEBUG, INFO, WARNING and ERROR. This ticket is mostly concerned with:

• [ ] clarifying whether we wish to also use CRITICAL, and
• [ ] subsequently producing a bit of documentation that helps guiding developers to know which level of logging to use based on specific cases.

As an example, an error stating that the genesis block is missing could be viewed as CRITICAL whereas an error caused by an invalid signature could simply be viewed as an ERROR.

Furthermore:

• [ ] When do wish to add DEBUG level logs?
• [ ] When do we put INFO level logs?
• [ ] Can there be too much INFO logs?
• [ ] What are the consequences of having too much INFO logs?
• [ ] Can there be to much DEBUG logs?
• [ ] What are the consequences of having too much DEBUG logs?
• [ ] What is the difference between a WARNING log and an ERROR log.
• [ ] Logger.exception() should be used instead of Logger.error() when handling exceptions. (Thanks @sohkai https://github.com/bigchaindb/bigchaindb/issues/1203#issuecomment-281095255)

And so forth. @sohkai and @krish7919 and others: feel free to add to the above concerns.

Some deliverables:

• [ ] Docs for devs (in the standard readthedocs format).
  
  
  
  • [ ] Refer to When to use logging as a starting point.
  
  
• [ ] Slide deck for devs.

[sohkai]

Something small to remember: Logger.exception() should be used instead of Logger.error() when handling exceptions.

It should also be noted that we can filter events to create notifications via external web services or email (e.g. send an email for all CRITICAL errors).

[sbellem]

Events also have an importance which the developer _ascribes_ to the event; the importance can also be called the level or severity.

[trentmc]

My three cents, in italics.

• clarifying whether we wish to also use CRITICAL: _I'm fine either way. Looks like it might be easier to actually include._
• subsequently producing a bit of documentation that helps guiding developers to know which level of logging to use based on specific cases: _Sure. Eg INFO for end users, DEBUG for devs. or end users trying to reproduce an error. Be sure to take advantage of 3rd party docs too. Avoid overkill on our side._
• When do wish to add DEBUG level logs? : _Following the guidelines above. Also, we will coalesce over time based on feedback amongst each other._
• When do we put INFO level logs? _See above._
• Can there be too much INFO logs? _Yes. And there can be too little, which is annoying. If it's too much, push it into DEBUG logs._
• What are the consequences of having too much INFO logs? _Annoyances to people running in production because too much to sift through. Monstrous log files that take non-negligible disk space. Code maintenance is a bit painful because so many log lines._
• Can there be to [sic] much DEBUG logs? _Yes, but you really have to try. Bigger issue is usually not enough DEBUG logs._
• What are the consequences of having too much DEBUG logs? _Huge log file, code maintenance is a bit painful because so many log lines._
• What is the difference between a WARNING log and an ERROR log. _WARNING isn't when an error occurs, but something that coulda been dangerous happened._
• Logger.exception() should be used instead of Logger.error() when handling exceptions. (Thanks @sohkai #1203 (comment))

[r-marques]

At this point BigchainDB does not handle CRITICAL faults.
Regarding the other debug levels at least at this point it should be up to the developers and reviewers discretion.

I say we close this. @sbellem ?