Coredns: Readiness probe failed 8181: connect: connection refused

[msschl]

Same behavior as in #3219

coredns version: 1.6.2

$ kubectl get pods -n kube-system -o wide | grep coredns
coredns-5644d7b6d9-4sbks                 0/1     Running   0          20h   10.244.1.2    lxkubenode01     <none>           <none>
coredns-5644d7b6d9-57dwc                 0/1     Running   0          20h   10.244.2.5    lxkubenode02     <none>           <none>

Events:

Events:
  Type     Reason     Age                   From                   Message
  ----     ------     ----                  ----                   -------
  Warning  Unhealthy  40s (x7381 over 20h)  kubelet, lxkubenode02  Readiness probe failed: Get http://10.244.2.5:8181/ready: dial tcp 10.244.2.5:8181: connect: connection refused

Telnet:

$ telnet 10.244.2.5 8080
Trying 10.244.2.5...
Connected to 10.244.2.5.
Escape character is '^]'.
$ telnet 10.244.2.5 8181
Trying 10.244.2.5...
telnet: Unable to connect to remote host: Connection refused

CoreDNS ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.my-org.com in-addr.arpa ip6.arpa {
           pods verified
           upstream
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward .  172.18.2.21
        cache 30
        loop
        reload
        loadbalance
    }

CoreDNS logs:

$ kubectl logs coredns-5644d7b6d9-57dwc -n kube-system
.:53
2019-10-26T17:06:05.744Z [INFO] plugin/reload: Running configuration MD5 = 1b7f8bcd3236f0c1573a0cf40262700c
2019-10-26T17:06:05.745Z [INFO] CoreDNS-1.6.2
2019-10-26T17:06:05.745Z [INFO] linux/amd64, go1.12.8, 795a3eb
CoreDNS-1.6.2
linux/amd64, go1.12.8, 795a3eb

 readinessProbe:
  failureThreshold: 3
  httpGet:
    path: /ready
    port: 8181
    scheme: HTTP
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 1

[msschl]

Updated to CoreDNS v1.6.4. Same issue

Logs:

$ kubectl logs coredns-6c5cb5bb5f-vgcvh -n kube-system -f
.:53
[INFO] plugin/reload: Running configuration MD5 = 38112becb4cbc8801a33dd4726e839c3
   ______                ____  _   _______
  / ____/___  ________  / __ \/ | / / ___/      ~ CoreDNS-1.6.4
 / /   / __ \/ ___/ _ \/ / / /  |/ /\__ \       ~ linux/amd64, go1.13.1, b139ba3-dirty
/ /___/ /_/ / /  /  __/ /_/ / /|  /___/ /
\____/\____/_/   \___/_____/_/ |_//____/
[INFO] Reloading
[INFO] plugin/reload: Running configuration MD5 = a1c45fe3a8384a4525bf5f9c1ff473a1
[INFO] Reloading complete

[rajansandeep]

You need to add the ready plugin for coredns to report readiness.

[msschl]

Thanks. It works...

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.as-gmbh.de in-addr.arpa ip6.arpa {
           pods verified
           upstream
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        ready :8181
        forward .  172.18.2.21
        cache 30
        loop
        reload
        loadbalance
    }

[miekg]

/label: kubernetes

[ykfq]

Update coredns image from v1.5.0 to current version v1.6.9, then the error got fixed.

[miekg]

/close
/lock

On Wed, 3 Jun 2020, 15:00 ykfq, [e-mail] wrote:

Update coredns image from v1.5.0 to current version v1.6.9, then the error
got fixed.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/coredns/coredns/issues/3403#issuecomment-638180682,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACWIWZTB7WMAW6XL7GBVETRUZCPHANCNFSM4JFRYBJQ
.