Flux: Leveled logging

[prometherion]

Describe the feature

Actually, filtering logs per level is quite tricky: users that want to get only error or warning messages have to parse output via _grep_ or any other viable solution, even worse if they want to get messages above a selected threshold (e.g.: just _warn_ and _err_).

Since flux is already using the dependency go-kit also for logging, would be nice to implement this feature decorating the logger thanks to the level package of the same dependency.

level allows several logging levels according to the code base and this feature wouldn't be breaking: further, no needs of test refactoring or a new test suite.

What would the new user story look like?

1. providing the new --verbosity flag with a default value for getting actual logging expectations in order to don't break backward compatibility or users expecting of getting a full and detailed log, as it is right now.
2. allowed values for --verbosity flag would be shadowed to level ones since they're reasonable and easy to understand at first glance, so:
   
   
   
   • allow-debug to get all the logs (actual situation)
   
   
   • allow-info to get _info_, _warn_ and _error_ messages
   
   
   • allow-warn to get _warn_ and _error_ messages
   
   
   • allow-error to get only _error_ messages
   
   
3. refactoring the codebase

Expected behavior

Get the ability to get log messages just for a given level without the need for various _grep_ or _jq_ piping in case of JSON format output.

[alexanderbuhler]

+1 on this one. We've just integrated with datadog and flux messages are being treated as errors by default. Some flux logs contain info= or err= so we can map them to a log level, others are missing any clue about level/verbosity, like this one:

ts=2020-02-11T12:00:49.890827678Z caller=warming.go:206 component=warmer updated=datadog/agent successful=0 attempted=2