Docs: refactor log files documentation

[taroface]

We currently have multiple articles covering various log files that could be unified in their approach:

• https://www.cockroachlabs.com/docs/v20.1/debug-and-error-logs.html
• https://www.cockroachlabs.com/docs/v20.1/query-behavior-troubleshooting.html#sql-logging
• https://www.cockroachlabs.com/docs/v20.1/sql-audit-logging.html

We should:

• Consolidate log files into a single reference document
  
  
  
  • Differentiate primary and secondary log files
  
  
• Refactor guides to reference log files where relevant
• Be sure to include guidance on obtaining logs from DB Console

[taroface]

@piyush-singh to summarize the use cases for each of the log types here. Thanks!

[piyush-singh]

The main user journeys I see for our logging are the following:

• Proactive and reactive security - Our users will want to use tools like SQL Audit logging and connection/auth logging to feed intrusion detection/prevention systems and SIEMs. Further, in the event of a breach, customers will want to use this information to track which data may have been compromised.
• Performance troubleshooting - Tools like the slow query log, traces in logs, and other sql performance logging tools are helpful to our users when trying to identify problems queries and root cause their slowness.
• Observability - CockroachDB's logs are useful for other general observability reasons, for example, errors during setup or configuration of a cluster will be logged to the main crdb log files. This would include things like node-node connection errors, or other errors that are not surfaced via other mechanisms such as the Admin UI.

cc @knz in case I missed any crucial use cases for our logs.

[piyush-singh]

cc @thtruo as we were chatting about this today

[taroface]

This should include a table of "logging groups" as outlined in this PR: https://github.com/cockroachdb/cockroach/pull/50599

[knz]

One more use case: configuration assistance.

During initial cluster deployment, or whenever the deployment parameters "outside" of crdb are changed, the operator may use the log files to support the configuration process -- e.g to confirm that everything is working as expected.

Generally I would divide "observability" between

• troubleshootability, which is really transparency in the face of exceptional events / errors
• health indicators, which is really a stream of log events that report that "everything is OK"

From an engineering process perspective, the development of good logging is different for these two use cases. I suspect the docs could be different as well.

[jseldess]

I think this has been resolved with the 21.1 logging docs work. Please reopen if that's not right.