Okd: [vSphere] API server and other pods on one of the masters in CrashLoopBackoff

[jomeier]

Hi,

beta 3 (but also seen on beta 1 and beta 2):

After installation everything looks fine over a few days. From time to time the overview page in the web ui shows update status "failing" where it was before "up to date".

I had a look on the cluster operators and I see this:

dns: Updating, 4.4.0-0.okd-2020-04-16-074048-beta3, Not all DNS DaemonSets available.
ClusterOperatorCOetcd:  Unknown, 4.4.0-0.okd-2020-04-16-074048-beta3, -

In the namespace openshift-dns I see the dns-default pod on master1 in crashloop with this logs:

E0424 10:33:58.591863       1 reflector.go:125] github.com/coredns/coredns/plugin/kubernetes/controller.go:302: Failed to list *v1.Service: Get https://172.30.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.30.0.1:443: connect: no route to host
E0424 10:33:58.591845       1 reflector.go:125] github.com/coredns/coredns/plugin/kubernetes/controller.go:304: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.30.0.1:443: connect: no route to host
E0424 10:33:58.591863       1 reflector.go:125] github.com/coredns/coredns/plugin/kubernetes/controller.go:302: Failed to list *v1.Service: Get https://172.30.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.30.0.1:443: connect: no route to host
E0424 10:33:58.591856       1 reflector.go:125] github.com/coredns/coredns/plugin/kubernetes/controller.go:309: Failed to list *v1.Namespace: Get https://172.30.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 172.30.0.1:443: connect: no route to host
E0424 10:33:58.591856       1 reflector.go:125] github.com/coredns/coredns/plugin/kubernetes/controller.go:309: Failed to list *v1.Namespace: Get https://172.30.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 172.30.0.1:443: connect: no route to host
E0424 10:33:58.591856       1 reflector.go:125] github.com/coredns/coredns/plugin/kubernetes/controller.go:309: Failed to list *v1.Namespace: Get https://172.30.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 172.30.0.1:443: connect: no route to host
E0424 10:33:58.591856       1 reflector.go:125] github.com/coredns/coredns/plugin/kubernetes/controller.go:309: Failed to list *v1.Namespace: Get https://172.30.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 172.30.0.1:443: connect: no route to host
[INFO] SIGTERM: Shutting down servers then terminating

The error always manifests in some pods being in crashloopbackoff:

meier_jo@MU746291:~$ ./oc get pods --all-namespaces -o wide | grep -i crashloop
openshift-apiserver-operator                            openshift-apiserver-operator-79bf686d47-s4q8c                     0/1       CrashLoopBackOff   573        7d18h     10.254.0.9     chmuokd4master1     <none>           <none>
openshift-apiserver                                     apiserver-5db4b56b6f-w7q7c                                        0/1       CrashLoopBackOff   624        7d1h      10.254.0.54    chmuokd4master1     <none>           <none>
openshift-dns                                           dns-default-qf599                                                 1/2       CrashLoopBackOff   741        10d       10.254.0.2     chmuokd4master1     <none>           <none>
openshift-etcd-operator                                 etcd-operator-f6c4bbb5c-46vv2                                     0/1       CrashLoopBackOff   576        7d18h     10.254.0.10    chmuokd4master1     <none>           <none>
openshift-kube-apiserver-operator                       kube-apiserver-operator-58765974dc-r49qm                          0/1       CrashLoopBackOff   573        7d18h     10.254.0.25    chmuokd4master1     <none>           <none>
openshift-kube-controller-manager-operator              kube-controller-manager-operator-5d8d5f7b97-zjj4b                 0/1       CrashLoopBackOff   576        7d18h     10.254.0.13    chmuokd4master1     <none>           <none>
openshift-kube-storage-version-migrator-operator        kube-storage-version-migrator-operator-f7986b6b6-mf7gj            0/1       CrashLoopBackOff   575        7d18h     10.254.0.20    chmuokd4master1     <none>           <none>
openshift-machine-api                                   cluster-autoscaler-operator-766758d4d4-f95wp                      1/2       CrashLoopBackOff   624        7d18h     10.254.0.27    chmuokd4master1     <none>           <none>
openshift-service-catalog-apiserver-operator            openshift-service-catalog-apiserver-operator-649c5c78d8-89snc     0/1       CrashLoopBackOff   572        7d18h     10.254.0.34    chmuokd4master1     <none>           <none>

Logs of openshift-apiserver on master1:

[core@chmuokd4master1 ~]$ sudo crictl logs aa2b34d17d6b4
Copying system trust bundle
F0424 10:25:16.735857       1 cmd.go:72] unable to load configmap based request-header-client-ca-file: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 172.30.0.1:443: connect: no route to host

Because this happened a few times either we have a problem in our cluster setup or it is a problem with OKD.

Any help is appreciated and welcome.

Greetings,

Josef

[jomeier]

The api server seems to complain about that the api server is not reachable (172.30.0.1:443).

[vrutkovs]

Could you collect the logs from kube-apiserver / etcd containers on the node?

[jomeier]

@vrutkovs : just to be clear: You mean with must-gather or in /var/log/pods/… ?

[vrutkovs]

must-gather won't work with api server unreachable.

You'd need to:
a) ssh on master nodes
b) get container id: crictl ps -a | grep for apiservers, etcd, dns and sdn
c) get logs: crictl logs <containerid>

[jomeier]

The must-gather tarball:
https://drive.google.com/file/d/1Xseel7TvOXL82pamGT6RkOhTR-SqXalf/view?usp=sharing

[jomeier]

In the api server logs I saw something about "api rate limiting" or similar.

We are using ArgoCD for GitOp on our OKD 4 cluster. And it seems as if ArgoCD begins hammering against the API server if it finds resources which can't be applied to k8s. This seems to be a bug:

https://argoproj.slack.com/archives/CASHNF6MS/p1587737753356400

I also can see in Argo's logs that it constantly reconciles, tries to do something. On a different Argo instance that runs on OKD 3.11 where no problematic resources are in the Git repo, it always pauses for a few minutes before it starts the reconcile loop.

But the developers are not sure if it really Argo is hammering or if the noobaa operator is doing that.

What I'm wondering about is, why it can crash the api server/dns/etcd pods on one of the OKD 4.4 masters.

I disabled ArgoCD on OKD 4, restarted the crashlooping pods on OKD 4 and everything works as expected again.

I'm pretty sure if I start ArgoCD again and it starts running amok again, that it will not take long again and OKD 4 API server pods will crash again.

This shouldn't happen in my opinion. OKD should be tougher than everything else :-)

[jomeier]

@vrutkovs recommended to increase the api burst limit. It's described here how I can do that:

https://docs.openshift.com/container-platform/4.3/scalability_and_performance/recommended-host-practices.html

This could fix the problem but as described above I would expect that the API server would not crash. And still I'm not 100% confident that API hammering by ArgoCD was the root cause but everything looks like it was.

[jomeier]

And here is a discussion about ArgoCD maybe creating this problem here:

https://argoproj.slack.com/archives/CASHNF6MS/p1587737753356400

[jomeier]

Update:
ArgoCD seems not to be the problem, but the Noobaa operator that was created with ArgoCD. The noobaa operator seems to continuously update it's Custom Resources and ArgoCD is only reacting to it. It creates additional API calls but the root cause of the "strange" behaviour of Argo seems to be in the Noobaa Operator.

Thats a beautiful example of being on the wrong track for hours. Currently I'm not even sure if the number of API calls per second is the root cause of my initial problem or if it is something completely different.

Everything described by me before does not explain the problem with the crashlooping api server/dns/etcd pods :-/

But hopefully the OKD experts can see something in the must-gather logs. Maybe it's only a configuration problem in our setup. Sorry for spamming around.

[vrutkovs]

You should report that to Noobaa operator upstream - there's no reason they

kube-apiserver gets stuck:

I0417 07:53:57.315280       1 servicehostname.go:40] syncing servicenetwork hostnames: [172.30.0.1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local openshift openshift.default openshift.default.svc openshift.default.svc.cluster.local]
            E0417 07:56:45.265016       1 leaderelection.go:367] Failed to update lock: etcdserver: request timed out
            E0417 07:58:17.930382       1 leaderelection.go:367] Failed to update lock: etcdserver: request timed out
            E0417 07:59:26.910329       1 leaderelection.go:331] error retrieving resource lock openshift-kube-apiserver/cert-regeneration-controller-lock: Get https://localhost:6443/api/v1/namespaces/openshift-kube-apiserver/configmaps/cert-regeneration-controller-lock?timeout=35s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)

that happened because etcd leader has become too slow:

etcdserver: read-only range request "key:\"/kubernetes.io/clusterroles/system:registry\" " with result "range_response_count:0 size:7" took too long (17.250412247s) to execute
....

then it got killed by healthcheck, new leader elected and all the requests have been reset.

It appears etcd storage is not fast enough for too many requests from API server

[jomeier]

Hi Vadim. Thanks for looking into that!

Could this be the reason that on our master1 API server listening on 172.30.0.1 : 443 can't be reached anymore? Because that was reported by all pods that were in the crashloopbackoff.

Which container is responsible to listen on IP address 172.30.0.1 port 443 ? kube-apiserver or openshift-apiserver?

Shouldn't the requests of the pods on master1 go to the remaining masters automatically? That's what I understand multi masters are determined for (HA). But every pod on master1, that has something to do with the api server, reported "no route to host".

Is there anything I can do to make API server and etcd more robust with config parameters? Is that the right place to raise the API burst limit ?

[jomeier]

I found most of my (kubeapi noob) answers here:

https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/

Its iptables that routes all traffic from 172.30.0.1:443 to the kube-apiserver container(s?).

[vrutkovs]

Lets close this - this is effectively an issue in Noobaa operator which is triggered by ArgoCD usage. I don't think there's anything we can do about that on OKD side