Linkerd2: Surface controller logs in CLI or dashboard

Created on 18 Dec 2018 · 10Comments · Source: linkerd/linkerd2

Feature Request

What problem are you trying to solve?

When things go wrong in the controller, the controller will usually log an error. However, these errors are not surfaced to users who are interacting with Linkerd primarily through the CLI and web UI. Furthermore, even if you are looking for the logs, it can be difficult to know which pod and which container's logs to look at.

How should the problem be solved?

Consider adding a "logs" page to the Linkerd web dashboard which displays aggregated logs from all the controller containers. Ideally with filter controls to select which container (default: all) and what log level (default: error).

Any alternatives you've considered?

A handcrafted kubectl command to get logs from the desired containers and an artisanal grep command to select out the appropriate log levels.

areusability areweb

Source

adleong

All 10 comments

I'd love to see this in the CLI (basically stern on the backend) before the GUI.

grampelberg on 18 Dec 2018

👍1

I would love to get this started. Should this be its own command? I was thinking maybe this could be added as an extension to the get command but that might feel a little weird. We could just add a new logs command similar to kubectl.

dadjeibaah on 19 Dec 2018

linkerd logs sounds awesome. There's actually a list of things that I'd love to see there (based off the most recent debugging issues I've had).

In priority order:

Control plane container logs

linkerd logs

# This feels as close to the kubectl UX as possible, but I'm not sure the resource category helps much on the control plane side
linkerd logs deploy/controller -c tap

By default, all the components (ca, controller, grafana, prometheus, web).
Colorized based on log level (and maybe in the message itself? timestamp vs message).
Filters to reduce by component name (controller/tap for example).

This should work at the deployment level and not the pod level. Folks can use kubectl logs for the specific container logs themselves.

Proxy logs

linkerd -n emojivoto logs deploy web

This seems like a good idea, but thinking about it a little bit more ... I'm not sure what value we'd provide over stern here. It feels like we could add some helpful metadata, just not sure what that'd be.

Modify log level

Changing the proxy log levels feels magical to me right now. I have to go ask each time to understand what needs to be turned on or off. It would be nice to have a command that turns it on for you and provides the list of what you can toggle.

I'm not sure what the command should be here, and it should work with inject.

grampelberg on 19 Dec 2018

This all looks good! I really like the logs command you have above. I feel like kubectl logs can get a little wordy to type out when trying to debug stuff in control plane components. I think we can have a shorter command if we don't use the deploy/ prefix as you mentioned. It makes things even more concise.

The colorized output is a good idea, but I think it would be helpful to have it distinguish the different loglines generated from a pod/container rather than have it distinguish the log level similar to how docker logs work.

dadjeibaah on 19 Dec 2018

I'm more interested in making the lines scannable than anything else. I suspect that would require something different than either pod/container or log level. I wonder what good examples there are of this. @hawkw I know you've been playing around with log outputs.

grampelberg on 19 Dec 2018

👍1

@grampelberg I've got a lot in the pipe as far as the proxy logs go --- one thing I think you'll be happy about is that it'll be easier to _change_ the proxy log level for different components at runtime. I imagine proxies could have an admin route for setting the log level, so we could provide a nice UI for changing it.