Linkerd2: Tap is not working with edge-20.10.2

Created on 15 Oct 2020 · 12Comments · Source: linkerd/linkerd2

Bug Report

What is the issue?

Tap is not working.

How can it be reproduced?

Install edge-20.10.2 with emojivoto app, and try to tap.

Logs, error output, etc

 linkerd-tap-c8b54b98f-nztjc tap time="2020-10-15T13:04:33Z" level=info msg="Tapping 4 pods for target: type:\"namespace\" name:\"emojivoto\""
 linkerd-tap-c8b54b98f-nztjc tap time="2020-10-15T13:04:33Z" level=info msg="Establishing tap on 100.84.149.138:4190"
 linkerd-tap-c8b54b98f-nztjc tap time="2020-10-15T13:04:33Z" level=info msg="Establishing tap on 100.84.141.196:4190"
 linkerd-tap-c8b54b98f-nztjc tap time="2020-10-15T13:04:33Z" level=info msg="Establishing tap on 100.84.157.248:4190"
 linkerd-tap-c8b54b98f-nztjc tap time="2020-10-15T13:04:33Z" level=info msg="Establishing tap on 100.84.137.241:4190"
 linkerd-tap-c8b54b98f-nztjc tap time="2020-10-15T13:04:33Z" level=error msg="[100.84.149.138] encountered an error: rpc error: code = Unknown desc = OK: HTTP status code 200; transport: missing content-type field"
 linkerd-tap-c8b54b98f-nztjc tap time="2020-10-15T13:04:33Z" level=error msg="[100.84.141.196] encountered an error: rpc error: code = Unknown desc = OK: HTTP status code 200; transport: missing content-type field"
 linkerd-tap-c8b54b98f-nztjc tap time="2020-10-15T13:04:33Z" level=error msg="[100.84.157.248] encountered an error: rpc error: code = Unknown desc = OK: HTTP status code 200; transport: missing content-type field"
 linkerd-tap-c8b54b98f-nztjc tap time="2020-10-15T13:04:33Z" level=error msg="[100.84.137.241] encountered an error: rpc error: code = Unknown desc = OK: HTTP status code 200; transport: missing content-type field"

`linkerd check` output

All Green

Environment

Kubernetes Version: 1.16
Cluster Environment: (GKE, AKS, kops, ...) EKS
Host OS: Linux
Linkerd version: edge-20.10.2

Possible solution

Revert to the latest stable (2.8.1)

Additional context

Source

shaikatz

Most helpful comment

I can confirm that this solved the tapping issue :-)
Thanks for your help guys.

shaikatz on 20 Oct 2020

🎉2 🚀1

All 12 comments

Thanks for the report @shaikatz.

I have not been able to reproduce this so far. I've installed Linkerd and emojivoto on a 1.16 and 1.18 cluster. I'm using edge-20.10.2. I'm running commands like linkerd tap -n emojivoto deploy/voting and linkerd tap ns/emojivoto. Both work correctly, as well as additional ones.

Could you provide the commands you have tried running and any additional details you can think of? Thanks!

kleimkuhler on 15 Oct 2020

Running linkerd tap -n emojivoto deploy/voting produce empty output for me.
This is more verbose output I could've grab from the tap pod:

linkerd-tap-55db6c9589-2cf8x tap time="2020-10-15T14:17:42Z" level=info msg="Tapping 1 pods for target: namespace:\"emojivoto\" type:\"deployment\" name:\"voting\""
linkerd-tap-55db6c9589-2cf8x linkerd-proxy [    86.310131683s]  WARN ThreadId(01) logical{dst=100.84.155.170:4190}:concrete: linkerd2_proxy_discover::buffer: Discovery stream ended!
linkerd-tap-55db6c9589-2cf8x linkerd-tap-55db6c9589-2cf8x linkerd-proxy [    86.310796249s]  WARN ThreadId(01) outbound:accept{peer.addr=100.84.138.208:55342 target.addr=100.84.155.170:4190}:http{v=h2}: linkerd2_app_core::errors: Failed to proxy request: request required the identity 'voting.emojivoto.serviceaccount.identity.linkerd.cluster.local' but no identity found
linkerd-tap-55db6c9589-2cf8x tap linkerd-proxy [   106.693163259s]  WARN ThreadId(01) outbound:accept{peer.addr=100.84.138.208:55680 target.addr=100.84.155.170:4190}:http{v=h2}: linkerd2_app_core::errors: Failed to proxy request: request required the identity 'voting.emojivoto.serviceaccount.identity.linkerd.cluster.local' but no identity found
time="2020-10-15T14:17:42Z" level=info msg="Establishing tap on 100.84.155.170:4190"
linkerd-tap-55db6c9589-2cf8x linkerd-tap-55db6c9589-2cf8x linkerd-proxy [   111.166116810s]  WARN ThreadId(01) outbound:accept{peer.addr=100.84.138.208:55788 target.addr=100.84.155.170:4190}:http{v=h2}: linkerd2_app_core::errors: Failed to proxy request: request required the identity 'voting.emojivoto.serviceaccount.identity.linkerd.cluster.local' but no identity found
linkerd-tap-55db6c9589-2cf8x tap time="2020-10-15T14:17:42Z" level=error msg="[100.84.155.170] encountered an error: rpc error: code = Unknown desc = OK: HTTP status code 200; transport: missing content-type field"

voting pod logs:

time="2020-10-15T14:16:11Z" level=info msg="running version edge-20.10.2"
[     0.9087298s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.9105626s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.9111747s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.9115677s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.9123045s]  INFO ThreadId(01) linkerd2_proxy: Local identity is voting.emojivoto.serviceaccount.identity.linkerd.cluster.local
[     0.9132806s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.9136951s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.9574257s]  INFO ThreadId(06) outbound: linkerd2_app: listen.addr=127.0.0.1:4140
[     0.9648778s]  INFO ThreadId(06) inbound: linkerd2_app: listen.addr=0.0.0.0:4143
[     0.501898190s]  INFO ThreadId(10) daemon:identity: linkerd2_app: Certified identity: voting.emojivoto.serviceaccount.identity.linkerd.cluster.local

Forgot to mention also that I'm using cilium (not as a CNI but only for the network policies layer), can it somehow be related?

shaikatz on 15 Oct 2020

@kleimkuhler @shaikatz I just installed a kind 1.16.15 with Cillium as per this resource and I installed the latest Linkerd edge and it works correctly.

⛵ kind-kind in linkerd2 on  alex/addon-config-b-gone [$?] via 🐹 v1.14.9
❯ k version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-09T11:26:42Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.15", GitCommit:"2adc8d7091e89b6e3ca8d048140618ec89b39369", GitTreeState:"clean", BuildDate:"2020-09-14T07:56:51Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

⛵ kind-kind in linkerd2 on  alex/addon-config-b-gone [$?] via 🐹 v1.14.9
❯ k version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-09T11:26:42Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.15", GitCommit:"2adc8d7091e89b6e3ca8d048140618ec89b39369", GitTreeState:"clean", BuildDate:"2020-09-14T07:56:51Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

⛵ kind-kind in linkerd2 on  alex/addon-config-b-gone [$?] via 🐹 v1.14.9
❯ linkerd version
Client version: edge-20.10.2
Server version: edge-20.10.2

⛵ kind-kind in linkerd2 on  alex/addon-config-b-gone [$?] via 🐹 v1.14.9
❯ linkerd tap ns/linkerd
req id=1:0 proxy=out src=<nil>:0 dst=10.244.1.31:4191 tls=true :method=GET :authority=10.244.1.31:4191 :path=/metrics
rsp id=1:0 proxy=out src=<nil>:0 dst=10.244.1.31:4191 tls=true :status=200 latency=6305µs
end id=1:0 proxy=out src=<nil>:0 dst=10.244.1.31:4191 tls=true duration=137µs response-length=3181B
req id=1:1 proxy=in  src=10.244.2.249:46744 dst=10.244.2.213:9090 tls=no_tls_from_remote :method=GET :authority=10.244.2.213:9090 :path=/-/ready
rsp id=1:1 proxy=in  src=10.244.2.249:46744 dst=10.244.2.213:9090 tls=no_tls_from_remote :status=200 latency=816µs
end id=1:1 proxy=in  src=10.244.2.249:46744 dst=10.244.2.213:9090 tls=no_tls_from_remote duration=76µs response-length=21B
req id=1:2 proxy=out src=<nil>:0 dst=10.244.3.211:9996 tls=true :method=GET :authority=10.244.3.211:9996 :path=/metrics
req id=1:0 proxy=in  src=10.244.2.213:56260 dst=10.244.3.211:9996 tls=true :method=GET :authority=10.244.3.211:9996 :path=/metrics
rsp id=1:0 proxy=in  src=10.244.2.213:56260 dst=10.244.3.211:9996 tls=true :status=200 latency=6092µs
rsp id=1:2 proxy=out src=<nil>:0 dst=10.244.3.211:9996 tls=true :status=200 latency=8004µs
end id=1:0 proxy=in  src=10.244.2.213:56260 dst=10.244.3.211:9996 tls=true duration=1599µs response-length=3031B
end id=1:2 proxy=out src=<nil>:0 dst=10.244.3.211:9996 tls=true duration=1291µs response-length=3031B
req id=1:0 proxy=in  src=10.244.1.161:37540 dst=10.244.1.31:9995 tls=no_tls_from_remote :method=GET :authority=10.244.1.31:9995 :path=/ping
rsp id=1:0 proxy=in  src=10.244.1.161:37540 dst=10.244.1.31:9995 tls=no_tls_from_remote :status=200 latency=726µs
end id=1:0 proxy=in  src=10.244.1.161:37540 dst=10.244.1.31:9995 tls=no_tls_from_remote duration=84µs response-length=5B
req id=1:3 proxy=out src=<nil>:0 dst=10.244.1.137:4191 tls=true :method=GET :authority=10.244.1.137:4191 :path=/metrics
rsp id=1:3 proxy=out src=<nil>:0 dst=10.244.1.137:4191 tls=true :status=200 latency=4830µs
end id=1:3 proxy=out src=<nil>:0 dst=10.244.1.137:4191 tls=true duration=123µs response-length=3162B
req id=1:0 proxy=in  src=10.244.1.161:36936 dst=10.244.1.76:9994 tls=no_tls_from_remote :method=GET :authority=10.244.1.76:9994 :path=/ping
rsp id=1:0 proxy=in  src=10.244.1.161:36936 dst=10.244.1.76:9994 tls=no_tls_from_remote :status=200 latency=272µs
end id=1:0 proxy=in  src=10.244.1.161:36936 dst=10.244.1.76:9994 tls=no_tls_from_remote duration=26µs response-length=5B
req id=1:4 proxy=out src=<nil>:0 dst=10.244.3.93:4191 tls=true :method=GET :authority=10.244.3.93:4191 :path=/metrics
rsp id=1:4 proxy=out src=<nil>:0 dst=10.244.3.93:4191 tls=true :status=200 latency=1127µs
end id=1:4 proxy=out src=<nil>:0 dst=10.244.3.93:4191 tls=true duration=33µs response-length=2731B
^C

⛵ kind-kind in linkerd2 on  alex/addon-config-b-gone [$?] via 🐹 v1.14.9 took 2s
❯ linkerd tap ns/emojivoto
req id=0:0 proxy=out src=<nil>:0 dst=10.244.3.69:8080 tls=true :method=GET :authority=web-svc.emojivoto:80 :path=/api/list
req id=0:0 proxy=in  src=10.244.1.4:58616 dst=10.244.3.69:8080 tls=true :method=GET :authority=web-svc.emojivoto:80 :path=/api/list
req id=0:1 proxy=out src=<nil>:0 dst=10.244.3.163:8080 tls=true :method=POST :authority=emoji-svc.emojivoto:8080 :path=/emojivoto.v1.EmojiService/ListAll
req id=0:0 proxy=in  src=10.244.3.69:34194 dst=10.244.3.163:8080 tls=true :method=POST :authority=emoji-svc.emojivoto:8080 :path=/emojivoto.v1.EmojiService/ListAll
rsp id=0:0 proxy=in  src=10.244.3.69:34194 dst=10.244.3.163:8080 tls=true :status=200 latency=1301µs
end id=0:0 proxy=in  src=10.244.3.69:34194 dst=10.244.3.163:8080 tls=true grpc-status=OK duration=130µs response-length=2140B
rsp id=0:1 proxy=out src=<nil>:0 dst=10.244.3.163:8080 tls=true :status=200 latency=2837µs
rsp id=0:0 proxy=out src=<nil>:0 dst=10.244.3.69:8080 tls=true :status=200 latency=8451µs
end id=0:1 proxy=out src=<nil>:0 dst=10.244.3.163:8080 tls=true grpc-status=OK duration=405µs response-length=2140B
rsp id=0:0 proxy=in  src=10.244.1.4:58616 dst=10.244.3.69:8080 tls=true :status=200 latency=6425µs
end id=0:0 proxy=in  src=10.244.1.4:58616 dst=10.244.3.69:8080 tls=true duration=1652µs response-length=4513B
end id=0:0 proxy=out src=<nil>:0 dst=10.244.3.69:8080 tls=true duration=1368µs response-length=4513B
req id=0:1 proxy=out src=<nil>:0 dst=10.244.3.69:8080 tls=true :method=GET :authority=web-svc.emojivoto:80 :path=/api/vote
req id=0:2 proxy=in  src=10.244.1.4:58616 dst=10.244.3.69:8080 tls=true :method=GET :authority=web-svc.emojivoto:80 :path=/api/vote
req id=0:3 proxy=out src=<nil>:0 dst=10.244.3.163:8080 tls=true :method=POST :authority=emoji-svc.emojivoto:8080 :path=/emojivoto.v1.EmojiService/FindByShortcode
req id=0:1 proxy=in  src=10.244.3.69:34194 dst=10.244.3.163:8080 tls=true :method=POST :authority=emoji-svc.emojivoto:8080 :path=/emojivoto.v1.EmojiService/FindByShortcode
rsp id=0:1 proxy=in  src=10.244.3.69:34194 dst=10.244.3.163:8080 tls=true :status=200 latency=1354µs
end id=0:1 proxy=in  src=10.244.3.69:34194 dst=10.244.3.163:8080 tls=true grpc-status=OK duration=176µs response-length=23B
rsp id=0:3 proxy=out src=<nil>:0 dst=10.244.3.163:8080 tls=true :status=200 latency=2890µs
end id=0:3 proxy=out src=<nil>:0 dst=10.244.3.163:8080 tls=true grpc-status=OK duration=135µs response-length=23B
req id=0:4 proxy=out src=<nil>:0 dst=10.244.3.93:8080 tls=true :method=POST :authority=voting-svc.emojivoto:8080 :path=/emojivoto.v1.VotingService/VoteTrophy
req id=0:0 proxy=in  src=10.244.3.69:42152 dst=10.244.3.93:8080 tls=true :method=POST :authority=voting-svc.emojivoto:8080 :path=/emojivoto.v1.VotingService/VoteTrophy
rsp id=0:0 proxy=in  src=10.244.3.69:42152 dst=10.244.3.93:8080 tls=true :status=200 latency=978µs
end id=0:0 proxy=in  src=10.244.3.69:42152 dst=10.244.3.93:8080 tls=true grpc-status=OK duration=108µs response-length=5B
rsp id=0:4 proxy=out src=<nil>:0 dst=10.244.3.93:8080 tls=true :status=200 latency=2203µs
end id=0:4 proxy=out src=<nil>:0 dst=10.244.3.93:8080 tls=true grpc-status=OK duration=90µs response-length=5B
rsp id=0:2 proxy=in  src=10.244.1.4:58616 dst=10.244.3.69:8080 tls=true :status=200 latency=11350µs
end id=0:2 proxy=in  src=10.244.1.4:58616 dst=10.244.3.69:8080 tls=true duration=36µs response-length=0B
rsp id=0:1 proxy=out src=<nil>:0 dst=10.244.3.69:8080 tls=true :status=200 latency=13038µs
end id=0:1 proxy=out src=<nil>:0 dst=10.244.3.69:8080 tls=true duration=35µs response-length=0B
req id=0:2 proxy=out src=<nil>:0 dst=10.244.3.69:8080 tls=true :method=GET :authority=web-svc.emojivoto:80 :path=/api/list
req id=0:5 proxy=in  src=10.244.1.4:58616 dst=10.244.3.69:8080 tls=true :method=GET :authority=web-svc.emojivoto:80 :path=/api/list
req id=0:6 proxy=out src=<nil>:0 dst=10.244.3.163:8080 tls=true :method=POST :authority=emoji-svc.emojivoto:8080 :path=/emojivoto.v1.EmojiService/ListAll
req id=0:2 proxy=in  src=10.244.3.69:34194 dst=10.244.3.163:8080 tls=true :method=POST :authority=emoji-svc.emojivoto:8080 :path=/emojivoto.v1.EmojiService/ListAll
rsp id=0:2 proxy=in  src=10.244.3.69:34194 dst=10.244.3.163:8080 tls=true :status=200 latency=1296µs
end id=0:2 proxy=in  src=10.244.3.69:34194 dst=10.244.3.163:8080 tls=true grpc-status=OK duration=114µs response-length=2140B
rsp id=0:6 proxy=out src=<nil>:0 dst=10.244.3.163:8080 tls=true :status=200 latency=2649µs
end id=0:6 proxy=out src=<nil>:0 dst=10.244.3.163:8080 tls=true grpc-status=OK duration=147µs response-length=2140B
rsp id=0:5 proxy=in  src=10.244.1.4:58616 dst=10.244.3.69:8080 tls=true :status=200 latency=6111µs
rsp id=0:2 proxy=out src=<nil>:0 dst=10.244.3.69:8080 tls=true :status=200 latency=8041µs
end id=0:5 proxy=in  src=10.244.1.4:58616 dst=10.244.3.69:8080 tls=true duration=1670µs response-length=4513B
end id=0:2 proxy=out src=<nil>:0 dst=10.244.3.69:8080 tls=true duration=1489µs response-length=4513B
req id=0:3 proxy=out src=<nil>:0 dst=10.244.3.69:8080 tls=true :method=GET :authority=web-svc.emojivoto:80 :path=/api/vote
req id=0:7 proxy=in  src=10.244.1.4:58616 dst=10.244.3.69:8080 tls=true :method=GET :authority=web-svc.emojivoto:80 :path=/api/vote
req id=0:8 proxy=out src=<nil>:0 dst=10.244.3.163:8080 tls=true :method=POST :authority=emoji-svc.emojivoto:8080 :path=/emojivoto.v1.EmojiService/FindByShortcode
req id=0:3 proxy=in  src=10.244.3.69:34194 dst=10.244.3.163:8080 tls=true :method=POST :authority=emoji-svc.emojivoto:8080 :path=/emojivoto.v1.EmojiService/FindByShortcode
rsp id=0:3 proxy=in  src=10.244.3.69:34194 dst=10.244.3.163:8080 tls=true :status=200 latency=1415µs
end id=0:3 proxy=in  src=10.244.3.69:34194 dst=10.244.3.163:8080 tls=true grpc-status=OK duration=138µs response-length=45B
rsp id=0:8 proxy=out src=<nil>:0 dst=10.244.3.163:8080 tls=true :status=200 latency=2928µs
end id=0:8 proxy=out src=<nil>:0 dst=10.244.3.163:8080 tls=true grpc-status=OK duration=145µs response-length=45B
req id=0:9 proxy=out src=<nil>:0 dst=10.244.3.93:8080 tls=true :method=POST :authority=voting-svc.emojivoto:8080 :path=/emojivoto.v1.VotingService/VoteStuckOutTongueWinkingEye
req id=0:1 proxy=in  src=10.244.3.69:42152 dst=10.244.3.93:8080 tls=true :method=POST :authority=voting-svc.emojivoto:8080 :path=/emojivoto.v1.VotingService/VoteStuckOutTongueWinkingEye
rsp id=0:1 proxy=in  src=10.244.3.69:42152 dst=10.244.3.93:8080 tls=true :status=200 latency=1276µs
end id=0:1 proxy=in  src=10.244.3.69:42152 dst=10.244.3.93:8080 tls=true grpc-status=OK duration=142µs response-length=5B
rsp id=0:9 proxy=out src=<nil>:0 dst=10.244.3.93:8080 tls=true :status=200 latency=2945µs
end id=0:9 proxy=out src=<nil>:0 dst=10.244.3.93:8080 tls=true grpc-status=OK duration=137µs response-length=5B
rsp id=0:7 proxy=in  src=10.244.1.4:58616 dst=10.244.3.69:8080 tls=true :status=200 latency=10099µs
end id=0:7 proxy=in  src=10.244.1.4:58616 dst=10.244.3.69:8080 tls=true duration=43µs response-length=0B
rsp id=0:3 proxy=out src=<nil>:0 dst=10.244.3.69:8080 tls=true :status=200 latency=11813µs
end id=0:3 proxy=out src=<nil>:0 dst=10.244.3.69:8080 tls=true duration=40µs response-length=0B
^C

This might be related to EKS then? :thinking:

Pothulapati on 15 Oct 2020

@shaikatz You mentioned that you are using Cillium only as networkPolicy :thinking: and wondering if this would help somehow.

My main worry is that the stable worked but the edge does not which is confusing. :thinking:

Pothulapati on 15 Oct 2020

The article mentions that the check command was failing, but it passes for me so I don't think it's related to my issue.

My first assumption was that I'm misconfiguring something since I'm new to linkerd, but after reverting back to stable and seeing that everything is working, I'm pretty sure something is broken with the latest release and my current setup.

I'll be happy to provide any further info that can help you guys understand the root cause of that issue.

shaikatz on 15 Oct 2020

@shaikatz Can you tweak Linkerd install to do verbose logs using inkerd install --controller-log-level debug --proxy-log-level debug and post the tap's and its proxy's logs? That would be helpful!

Pothulapati on 15 Oct 2020

@Pothulapati the debug logs provide hundreds of lines, which of them I should forward to you and in which format?
I've started by adding a gist for the tap logs, grepped for Failed to proxy request + 20 lines before and after. (https://gist.github.com/shaikatz/a6ba3846d905f1d7f5d29849ebfabecc)

Also the proxy logs, a lot of them might not be relevant because there is also some default emojivoto traffic, but I hope you could find the interesting lines.
https://gist.github.com/shaikatz/e81368f73816215bb1d194dc80cbf908

shaikatz on 15 Oct 2020

@shaikatz I created a 1.16 EKS cluster with Cillium following this doc and was not able to replicate the problem.

Can you provide more information on how Cillium is installed in your environment and any specific configuration on your cluster?

Pothulapati on 16 Oct 2020

The guide you attached instruct how to replace aws-cni with cilium, which is not something I'm doing, I'm using chainingMode.
My setup is done by installing cilium using chart version 1.8.4 with those values:

cilium:
  global:
    cni:
      chainingMode: aws-cni
    masquerade: false
    tunnel: disabled
    nodeinit:
      enabled: true
  config:
    upgradeCompatibility: '1.7'

I've also configured external SNAT:
kubectl set env daemonset -n kube-system aws-node AWS_VPC_K8S_CNI_EXTERNALSNAT=true (docs)

shaikatz on 18 Oct 2020

I notice in your tap controller's logs:

linkerd-tap-7755c4549f-4k895 linkerd-proxy [   143.740471252s] DEBUG ThreadId(01) logical{dst=100.84.147.236:4190}:concrete: linkerd2_app::dst::default_resolve: Synthesizing endpoint error=rejected discovery addr=100.84.147.236:4190

This indicates that the pod network (including the IP 100.84.147.236) is not in the configured set of discoverable networks.
The default set of networks includes only private IPs https://github.com/linkerd/linkerd2/blob/4f16a234aadd45ce70fa19fafbe5c8b5714ba9a5/charts/linkerd2/values.yaml#L77

It appears that the network 100.64.0.0/10 is also private and could be added to this list. We should consider this for stable-2.9.

In the meantime, you would have to use Helm to install Linkerd and update this configuration in the values.yml, as I do not believe this configuration can be controlled via the CLI.

olix0r on 19 Oct 2020

Ahh, That seems like the issue.

In the meantime, you would have to use Helm to install Linkerd and update this configuration in the values.yml, as I do not believe this configuration can be controlled via the CLI.

I think this is possible by passing the following configuration file into the --config flag through CLI:

global:
  proxy:
    destinationGetNetworks: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,100.64.0.0/10"

@shaikatz Can you try this and post your findings? :+1:

Pothulapati on 19 Oct 2020

I can confirm that this solved the tapping issue :-)
Thanks for your help guys.

shaikatz on 20 Oct 2020

🎉2 🚀1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Wire up stats and dashboards for Jobs

klingerf · 3Comments

change css variables for color to be more descriptive

franziskagoltz · 3Comments

Add Support for PodDisruptionBudgets in Helm Chart

tustvold · 4Comments

Disable custom cluster domain changes during control plane upgrade

ihcsim · 4Comments

Publish Helm chart to Helm Hub

ihcsim · 4Comments

Linkerd2: Tap is not working with edge-20.10.2

Bug Report

What is the issue?

How can it be reproduced?

Logs, error output, etc

linkerd check output

Environment

Possible solution

Additional context

Most helpful comment

All 12 comments

Related issues

`linkerd check` output