Linkerd2: Can't Reach StatefulSet Pods Via Stable Network ID

Created on 12 Feb 2019 · 13Comments · Source: linkerd/linkerd2

Bug Report

What is the issue?

A meshed client isn't able to reach a meshed statefulset pod via the pod's stable network ID. This works if the client isn't meshed.

How can it be reproduced?

Deploy a meshed statefulset nginx

cat <<EOF | linkerd inject - | kubectl apply -f -
kind: Service 
apiVersion: v1
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  clusterIP: None
  selector:
    app: nginx
  ports:
  - name: http
    port: 80
    targetPort: http

---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 3
  serviceName: nginx
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          name: http
EOF

Use a curl pod to reach nginx:

$ kubectl run curl --image=appropriate/curl --restart=Never --command -- sleep 3600

# this works
$ kubectl exec curl -- curl --silent nginx-0.nginx.default.svc.cluster.local
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

Now meshed the curl pod and notice that it can't reach the nginx pod:

$ kubectl run curl --image=appropriate/curl --restart=Never --dry-run -o yaml --command -- sleep 3600 | linkerd inject - | kubectl apply -f -

# stuck
$ kubectl  exec curl -c curl -- curl --silent nginx-0.nginx.default.svc.cluster.local

# this works
$ kubectl exec curl -c curl -- curl --silent nginx.default
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
...

Logs, error output, etc

Proxy logs from the curl client:

WARN admin={bg=resolver} linkerd2_proxy::control::destination::background::destination_set Destination.Get stream errored for NameAddr { name: "nginx-0.nginx.default.svc.cluster.local", port: 80 }: Grpc(Status { code: Unknown, message: "resolver [&{k8sDNSZoneLabels:[] controllerNamespace:linkerd endpointsWatcher:0xc4205e9f20 profileWatcher:0xc4203f2ae0}] found error resolving host [nginx-0.nginx.default.svc.cluster.local] port [80]: not a service: nginx-0.nginx.default.svc.cluster.local" })
WARN admin={bg=resolver} linkerd2_proxy::control::destination::background::destination_set Destination.Get stream errored for NameAddr { name: "nginx-0.nginx.default.svc.cluster.local", port: 80 }: Grpc(Status { code: Unknown, message: "resolver [&{k8sDNSZoneLabels:[] controllerNamespace:linkerd endpointsWatcher:0xc4205e9f20 profileWatcher:0xc4203f2ae0}] found error resolving host [nginx-0.nginx.default.svc.cluster.local] port [80]: not a service: nginx-0.nginx.default.svc.cluster.local" })

`linkerd check` output

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

linkerd-existence
-----------------
√ control plane namespace exists
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-api
-----------
√ control plane pods are ready
√ can query the control plane API
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus

linkerd-service-profile
-----------------------
√ no invalid service profiles

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match

Environment

Kubernetes Version: v1.13.2
Cluster Environment: Minikube
Host OS: Ubuntu 16.04
Linkerd version: edge-19.2.2

Possible solution

Additional context

arecontroller areproxy bug prioritP0

Source

ihcsim

Most helpful comment

I think that now we can solve this completely in the destination service by (1) modifying the Get endpoint to serve responses for statefulset names -- this will enable identity, etc, for statefulset traffic -- and (2) modifying GetProfiles to drop the instance ID from the service name before resolving the profile.

olix0r on 15 Jul 2019

🎉4

All 13 comments

Same here!

Environment

Kubernetes Version: v1.13.2
Cluster Environment: bare-metal + kube-router CNI
Host OS: ContainerLinux 1967.4.0
Linkerd version: edge-19.2.2

christianhuening on 12 Feb 2019

@hawkw @adleong any ideas on the scope for this and what pieces of code it'll touch?

grampelberg on 7 Mar 2019

@grampelberg I'd have to double-check to be sure, but I think this will only require a change to the control plane's Destination service. I don't think a proxy change is necessary.

hawkw on 7 Mar 2019

If the work isn't massive, this is a great getting started issue as it introduces folks to how discovery is working. I'd love to see if anyone's interested in tackling it (assuming the changes aren't massive).

grampelberg on 7 Mar 2019

👍1

this also doesn't seem to work for any headless services that expose pods via a fully numeric prefix. not the same error, but the failure is also in the discovery service and affects the same kind of workload/use case.

https://github.com/linkerd/linkerd2/blob/65a228fe6c4a4daac6190889834692790d0d926d/controller/api/destination/k8s_resolver.go#L213

is an invalid assumption that generates thousands of retries per second to the controller api from the proxies attempting to resolve valid dns names.

lancehitchcock on 7 Apr 2019

👍1

@lancehitchcock is it possible to share a k8s manifest that exhibits this behavior? i'd love to capture this in an integration test.

olix0r on 7 Apr 2019

any service with .spec.clusterIP of None exposes pods in the DNS SRV API this way.

Prometheus specifically uses this for scraping if you use dns sd.

using the original example from above and adding a second service with a different name (like nginx-headless) and the clusterIP set to None would be sufficient to give you the SRV DNS entries under _http._tcp.nginx-headless.default.svc.cluster.local

lancehitchcock on 7 Apr 2019

you can verify this with dig SRV _http._tcp.nginx-headless.default.svc.cluster.local

the manifest would look like this:

kind: Service 
apiVersion: v1
metadata:
  name: nginx-headless
  labels:
    app: nginx
spec:
  clusterIP: None
  selector:
    app: nginx
  ports:
  - name: http
    port: 80
    targetPort: http

edit: it is possible that for statefulset specifically, you may have to specify this service as the .spec.serviceName but this is not necessary for deployment types

lancehitchcock on 7 Apr 2019

;; QUESTION SECTION:
;_http._tcp.<service name>.<namespace>.svc.cluster.local.        IN SRV

;; ANSWER SECTION:
_http._tcp.<service name>.<namespace>.svc.cluster.local. 30 IN SRV 10 50 80 3763656465393934.<service name>.<namespace>.svc.cluster.local.
_http._tcp.<service name>.<namespace>.svc.cluster.local. 30 IN SRV 10 50 80 3236616662636137.<service name>.<namespace>.svc.cluster.local.

;; ADDITIONAL SECTION:
3763656465393934.<service name>.<namespace>.svc.cluster.local. 30 IN A 10.4.1.23
3236616662636137.<service name>.<namespace>.svc.cluster.local. 30 IN A 10.4.0.7

the result would look something like this, and calls to those endpoints through a sidecar proxy will fail

lancehitchcock on 7 Apr 2019

Verified / confirmed the same behavior that @lancehitchcock saw on 6 April:

Environment:
Kubernetes Version: v1.13.6-gke.13
Cluster Environment: Google Kubernetes Engine
Host OS: Google Container Optimized OS 1.12.7-gke.7
Linkerd version: stable-2.3.2

memory on 24 Jun 2019

Behold, an all-in-one demo of the issue:

https://gist.github.com/memory/21cf2a81637253889f8fe3b11e4e3bc8

With linkerd injection disabled, the gethello pod should be able to successfully query the three pods of the hello statefulset:

checking http://hello-0.hello.default.svc.cluster.local/
200
checking http://hello-1.hello.default.svc.cluster.local/
200
checking http://hello-2.hello.default.svc.cluster.local/
200

With injection _enabled_ (change lines 35 and 88), all requests will return 503 errors:

checking http://hello-0.hello.default.svc.cluster.local/
503
checking http://hello-1.hello.default.svc.cluster.local/
503
checking http://hello-2.hello.default.svc.cluster.local/
503

memory on 11 Jul 2019

👍2

olix0r on 15 Jul 2019

🎉4

Hi @memory, I ran into a strange issue when testing your demo. Your hello pods declare their container port as 8080 but actually just listen on port 80. This causes Linkerd to attempt to connect to port 8080 and get a connection refused error. For some reason, if Linkerd is not used, Kubernetes will forward the connection to port 80 even though the target port is defined as 8080, which is very surprising to me.

Anyway, I've put together https://github.com/linkerd/linkerd2/pull/3113 to add stateful set support. If I modify your demo to declare 80 as the hello container port, this PR fixes the issue.

Please take a look and let me know what you think!

adleong on 20 Jul 2019

👍1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

"linkerd inject" does not work on cronjobs?

geekmush · 4Comments

Disable custom cluster domain changes during control plane upgrade

ihcsim · 4Comments

Document service discovery and load balancing

briansmith · 4Comments

Add Support for PodDisruptionBudgets in Helm Chart

tustvold · 4Comments

Developer Documentation

manimaul · 3Comments

Linkerd2: Can't Reach StatefulSet Pods Via Stable Network ID

Bug Report

What is the issue?

How can it be reproduced?

Logs, error output, etc

linkerd check output

Environment

Possible solution

Additional context

Most helpful comment

All 13 comments

Environment

Related issues

`linkerd check` output