Crossplane: Errors and failures should be logged

[jbw976]

Is this a bug report or feature request?

• Feature Request

What should the feature do: Some of the Crossplane controllers do not log anything at all when there are errors or failures. We do a good job of capturing errors in status conditions on CRDs, but there is sometimes zero logging to help troubleshoot an issue. We should take a pass through the controllers and log when any errors or failures happen.

What is use case behind this feature: It greatly helps troubleshooting and helping out our users when they can simply share the crossplane pod logs that will show all failures that occurred. Currently, we have to help them find the correct object to look up status conditions on and often this is a chain of objects, e.g. bucket claim -> concrete bucket -> provider object. This requires a lot of back and forth with the user and it's not intuitive for them to figure out on their own.

See the following Slack conversation for a great example of this:
https://crossplane.slack.com/archives/CEF5N8X08/p1551900157039900

I understand avoiding log bloat, but especially in the early phases of a project we need more consolidated logging of errors to streamline troubleshooting with our users and to allow them to self-service debug as well. We can always dial back logging later on as the project gets more stable.

[jbw976]

@displague ran into this as well while walking through the gitlab and/or wordpress examples. Take for example a MySQLInstance (mysql claim) that the user creates. A key breakdown in the user's ability to troubleshoot why their mysql isn't coming up is because we generate a concrete mysql instance that they have no knowledge of, e.g., cloudsqlinstances.database.gcp.crossplane.io.

Just looking at their MySQLInstance instance won't give them enough information to know that the real underlying error is probably going to actually be on the CloudSQLInstance object instead. Furthermore, the concrete object is currently in a different namespace than the claim, which makes it even harder to figure out.

A huge step to making Crossplane easier to troubleshoot/debug would be to write failures for all objects to the log. This gives one single place for users to look when something goes wrong, instead of having to deduce the right object type in the right namespace to get more information from the conditions or events on that object. That's way too much to ask users to figure out right now and short term help with logging would go a long way.

We can do it at DEBUG level.

[negz]

I think it's really important to remember that most users of Crossplane won't necessarily have access to the pod logs to debug what is happening. I'm all for making it easier to figure out what Crossplane is doing, but let's not forget that we have several tools in our toolbox here, including:

• Info level logging (core audience: cluster operators)
• Debug logging (core audience: Crossplane developers)
• The Synced conditioned status (core audience: Crossplane users)
• Kubernetes events (core audience: Crossplane users)
• Timeseries (e.g. Prometheus) data (core audience: cluster operators)

Just looking at their MySQLInstance instance won't give them enough information to know that the real underlying error is probably going to actually be on the CloudSQLInstance object instead. Furthermore, the concrete object is currently in a different namespace than the claim, which makes it even harder to figure out.

This is an important case to consider, but I would argue the solution is to ensure people who create a MySQLInstance can easily discover and view the CloudSQLInstance that satisfies that claim. Not only to view the status of the underlying managed resource, but to determine what shape of managed resource was provisioned to satsify your claim.

[turkenh]

I agree having the required information in events and status of resources should be enough for users. We should better focus on providing some kind of tooling which pieces things together.
A trace command in cli looks could help here.

[negz]

Relates to https://github.com/crossplaneio/crossplane/pull/858