Opa: How to suppress health check logs?

[kennethmyers]

Whenever the /health endpoint is hit, logs such as the following are produced (when running with the default log_level: INFO):

{"client_addr":"[::1]:63959","level":"info","msg":"Received request.","req_id":2,"req_method":"GET","req_path":"/health","time":"2020-07-10T12:42:31-05:00"}
{"client_addr":"[::1]:63959","level":"info","msg":"Sent response.","req_id":2,"req_method":"GET","req_path":"/health","resp_bytes":2,"resp_duration":0.318927,"resp_status":200,"time":"2020-07-10T12:42:31-05:00"}

While it is desirable to log HTTP requests in general, logging each and every health check requests floods the logs with useless data, especially when running OPA in a system with frequent health checks such as Kubernetes.

Is there a way to suppress these health check logs, but still keep all other HTTP request logs?

If there is not a way currently, would the maintainers be open to a PR to implement such a feature?

Thanks

[tsandall]

I tend to agree but I'm surprised that health check access logs outpace access logs from policy queries. How frequently do you have the health check invoked? Perhaps another question is what are you relying on the OPA server logs for? Do you have the console decision logger enabled (e.g., --set decision_logs.console=true)?

One thing we've wanted to do for a little while is get away from the low-level HTTP access logs. Instead, by default, OPA would log INFO & ERROR logs about async operations like bundle downloads and status reports as well as decision logs. The HTTP access logs would be relegated to DEBUG where they rightly belong IMO. Would this satisfy your requirements?

EDIT: @patrick-east WDYT?

Related: #2173

[kennethmyers]

The health check runs every 10 seconds, which is the default frequency for Kubernetes "liveness" probes.

We do have the decision logger enabled and that is the actual information we care about retrieving from the OPA server logs. So in our case, yes, relegating HTTP access logs to DEBUG would satisfy our requirements (I also agree it makes more sense for HTTP access logs to fall under DEBUG).

Thank you for the response and for your work on this great project!

[Lirt]

Totally agree with @kennethmyers. Health check logs are extremely annoying during troubleshooting even on empty cluster. I usually look there for decision logs or check the input format for Kubernetes operations.

@tsandall Your idea with splitting these messages by log levels sounds great to me.