Linkerd2: Debug container

[grampelberg]

What problem are you trying to solve?

The current proxy and control plane containers are stripped down to reduce size. While that's awesome, it makes debugging (especially network level stuff) a little bit of a pain.

How should the problem be solved?

I would like to have a new flag for inject and install that adds a new sidecar container which contains a whole bunch of debugging tooling and the correct permissions to run. The flow could be:

$ linkerd install --debug-mode
$ linkerd inject --debug-mode

The container should have raised privileges and mount some host volumes. What other paths on the host would be valuable? Maybe

apiVersion: v1
kind: Pod
metadata: 
  name: linkerd-debug
spec: 
  volumes:
    - name: host-config
      hostPath:
        path: /etc/default
    - name: proc
      hostPath:
        path: /proc
  containers: 
    - name: linkerd-debug
      image: gcr.io/linkerd-io/debug:stable-2.2.0
      securityContext:
        privileged: true
      volumeMounts:
        - mountPath: /etc/default
          name: host-config
        - mountPath: /host/proc
          name: proc

I would like the base container to have the following packages pre-installed:

• tcpdump
• netstat
• strace
• bpftrace
• lsof
• telnet
• curl
• nghttp
• tar

Open questions

• It would be nice to make sure https://github.com/eldadru/ksniff works for the tcpdump use cases with this container in the pod.
• It seems like it'd be cool to have a linkerd debug deploy/foobar that automates some of this. Not sure there's any real benefit over the top of kubectl exec though.

[grampelberg]

@hawkw what tools and mounts am I missing?

[hawkw]

• I use tshark a lot and generally prefer it over tcpdump (especially because it's HTTP2-aware)
• maybe lsof so you don't have to grep /proc/fd? though I think netstat is probably sufficient

[hawkw]

This is so obvious that I nearly forgot, but: it should have curl and nghttp (for making HTTP/2 requests). Maybe telnet as well?

[olix0r]

I use tshark a lot and generally prefer it over tcpdump (especially because it's HTTP2-aware)

That's purely UI, though. tshark can read pcap saved by tcpdump, etc.

[hawkw]

@olix0r yeah, I suppose one can always just kubectl cp the pcap back onto their dev machine, I just never remember that's an option. Good point.

(in that case we should definitely make sure that the debug container has tar installed, as kubectl cp requires it...)

[grampelberg]

It should be possible to use https://github.com/eldadru/ksniff for all the wireshark goodness.

[grampelberg]

There should be symbols in place to be able to use gdb for both the proxy and control plane components.

[dadjeibaah]

While reading all referenced issues and the description, I've got a couple of questions:

• What's the reasoning behind basing this on Alpine vs. our own Linkerd base image? It seems to me that only the controller uses a scratch image. Issue #2167 mentions that this is for security reasons. Should we be concerned since we would be adding a debug container possibly in the control plane as well?
• I see a couple of articles on the Internet, arguing that ss is favored over netstat, wondering what your thoughts are about this.

[siggy]

Re: Alpine vs. Linkerd's base image (gcr.io/linkerd-io/base:2017-10-30.01). I'd be fine using the Linkerd base image, and then adding more debug-specific tooling on top of it. To be clear, I'm not suggesting we modify the Linkerd base image itself. I think the security concerns arise from running a non-scratch image in production full-time. Deploying a one-off debug container for a limited period of time carries less risk.

Re: ss vs. netstat, why not both? As this is a debug image not intended for production, I'd err on the side of overloading it with utilities.

[grampelberg]

• What's the reasoning behind basing this on Alpine

No good reason, I just like it better. I'm good using the debian base. It definitely needs to be updated, but that's in progress.

• Should we be concerned since we would be adding a debug container possibly in the control plane as well?

Debug container should not be anywhere by default, and injected alongside the proxy container. It is more valuable for workloads that aren't the control plane.

[dadjeibaah]

Thanks for clarifying, I was able to get a debug container working. Do we want this container to run as a sidecar or as a pod running alongside a meshed pod? I ask because I would imagine that if we wanted to use something like ksniff, we would need to have the container running inside the meshed pod in order to capture packets coming in and out of the meshed pod.

[grampelberg]

It'll need to be in the same network namespace, so inside a meshed pod.

[ihcsim]

Not sure if kubectl-debug will be useful here. A bit similar to kubectl exec, but (I think) it injects a debug container (via a daemonset agent) into a pod using the kubectl debug plugin. If anything, for inspiration maybe.

[dadjeibaah]

@ihcsim, kubectl-debug is super cool. Interesting approach having a debug server auto inject debug containers on demand.

[grampelberg]

Woah, that's cool. I didn't realize you could go directly to the kubelet and modify pods at run time. Not sure it'll work for us though ...

[olix0r]

There's another question as to whether this pod should run as the proxy user or not. Remember that the proxy user is not iptabled through the proxy and communicates directly with the external world... so you'll be unable to debug certain things, depending on which decision is made.

[olix0r]

What does install --debug-mode do exactly? Does it configure _all_ injected pods to be configured with debug mode?

[olix0r]

I'm -1 on alpine -- we're using a debian variant everywhere else, why have to pull another set of layers only to end up with set of nonstandardish tools? I also don't see any value in coupling to this to linkerd's proxy image -- base image would be fine. Let's not have to build/distribute a new image every time the proxy changes.

I think it's fine for this container to have an arbitrary set of tools but i think it should be useful out of the box without having to exec into it or do anything. To that end, I suggest that the debug container have an ENTRYPOINT [ "tshark", "-i", "any" ] so that we get a simple text dump summarizing all packets. if we don't do this, it's virtually impossible to capture to traffic during pod initialization...

The one caveat is that this might make it more difficult to run a full tcpdump in the container while tshark is running -- we'd have to test that. Also, it would be really nice to be able to configure arguments to tshark, so should --debug-mode take an optional argument that can configure, and perhaps disable, tshark?

[grampelberg]

What does install --debug-mode do exactly? Does it configure all injected pods to be configured with debug mode?

I'd imagine that it gets added to the control plane and that's it. Honestly the linkerd inject behavior is more likely what folks want, we'll just need to make it obvious that you can re-inject control plane components.

I'm -1 on alpine -- we're using a debian variant everywhere else, why have to pull another set of layers only to end up with set of nonstandardish tools?

I've found the package availability on alpine to be much better than debian. That said, it is good enough. Updated to remove the reference and leave it as an implementation detail =)

To that end, I suggest that the debug container have an ENTRYPOINT [ "tshark", "-i", "any" ]

I violently hate images that use ENTRYPOINT. That said, having a default other than a sleep sounds like a good idea and I'm +1 on having it be the traffic flowing through. Feels kinda like a verbose +++ option for logging.

so should --debug-mode take an optional argument that can configure, and perhaps disable, tshark?

I'd prefer to keep the configuration surface area as minimal as possible. Feels like someone could edit the manifest themselves to change options or just exec and run a second copy (need to verify that works).

[dadjeibaah]

@olix0r, +1 on @grampelberg comment on how the flag would work adding the flag --debug-mode would only install the debug container for that command. i.e. debug mode on install would only install it for the control plane.

I'm -1 on alpine -- we're using a debian variant everywhere else, why have to pull another set of layers > only to end up with set of nonstandardish tools?

I agree that finding certain packages for debian is a challenge. When taking a stab at this issue earlier, I wasn't able to find packages like bpftrace for debian, and they had to be built from source. However, It sounds like the MVP for this story is just getting tcpdumps. I am fine with sticking with the base image but we will have to revisit what other tools we want in this container.

[olix0r]

@dadjeibaah

@olix0r, +1 on @grampelberg comment on how the flag would work adding the flag --debug-mode would only install the debug container for that command. i.e. debug mode on install would only install it for the control plane.

But that's inconsistent with all of the other flags that are shared between the install and inject flow... Any inject-flag that's passed at install-time configures how inject works for all pods. I don't think this flag should behave differently from others, so I think the question is whether this makes sense as a default ever... If not, this should probably be an inject-only flag, as I think Thomas suggested.

@grampelberg

I'd prefer to keep the configuration surface area as minimal as possible. Feels like someone could edit the manifest themselves to change options or just exec and run a second copy (need to verify that works).

I guess my main qualm is calling it "debug-mode" -- i'd expect it to be something like "debug-mode=disabled" by default and have a possible debug-mode=tshark, whatever.

If this is really just boolean, then we should make it an enable/disable flag like the others. But I think it would be smart to take an argument for now so we have the opportunity to expand later.

[dadjeibaah]

@grampelberg, what are your thoughts on extending linkerd logs to retrieve the tshark output from the debug container. I was thinking something like:

linkerd logs --data-plane deploy/voting -n emojivoto. Maybe we don't need to restrict it to just the tshark output maybe every cotainer within that deployment?

[grampelberg]

@dadjeibaah I like including the debug container. I think we should stay away from all containers as that really feels like us wrapping stern.