Minikube: verifying ingress addon pods : timed out waiting for the condition: timed out waiting for the condition

[staticdev]

Steps to reproduce the issue:

1. minikube start --vm-driver=docker --memory 3072
2. minikube addons enable ingress

Full output of failed command: tried 3 times.
😄 minikube v1.12.1 on Ubuntu 20.04
✨ Using the docker driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating docker container (CPUs=2, Memory=3072MB) ...
🐳 Preparing Kubernetes v1.18.3 on Docker 19.03.2 ...
🔎 Verifying Kubernetes components...
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube"
🔎 Verifying ingress addon...

💣 enable failed: run callbacks: running callbacks: [verifying ingress addon pods : timed out waiting for the condition: timed out waiting for the condition]

😿 minikube is exiting due to an error. If the above message is not useful, open an issue:
👉 https://github.com/kubernetes/minikube/issues/new/choose

Full output of minikube start command used, if not already included:
-

Optional: Full output of minikube logs command:

[tstromberg]

Can you please share the output of:

• kubectl descr po -A

The only thing new in v.12.1 is an updated version of ingress: https://github.com/kubernetes/minikube/pull/8705

I was not able to replicate this issue on Docker/Linux or Hyperkit/macOS. My suspicion is that perhaps the download failed? Regardless, we can do a better job of showing an actionable error message.

[staticdev]

I saw on #8705 that nginx-ingress-controller or kube-webhook-certgen, I could try downgrading separately each of these components to try to isolate the error. But I am not sure how is the best way for me to do this test.

kubectl descr po -A

```NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-66bff467f8-pkgjv 1/1 Running 0 4m34s
kube-system etcd-minikube 1/1 Running 0 4m31s
kube-system ingress-nginx-admission-create-f4kgh 0/1 Completed 0 4m17s
kube-system ingress-nginx-admission-patch-v7wth 0/1 Completed 0 4m17s
kube-system ingress-nginx-controller-69ccf5d9d8-8pmdm 1/1 Running 0 4m17s
kube-system kube-apiserver-minikube 1/1 Running 0 4m31s
kube-system kube-controller-manager-minikube 1/1 Running 0 4m31s
kube-system kube-proxy-7m7tl 1/1 Running 0 4m34s
kube-system kube-scheduler-minikube 1/1 Running 0 4m31s
kube-system metrics-server-5f956b6d5f-9zfvx 1/1 Running 0 4m22s
kube-system storage-provisioner 1/1 Running 0 4m35s

[shyamkhakhar9]

I also face the same issue while trying to addons and I have ingress nginx yaml where when I apply kubectl it gives me following error.. how to resolve that

Error from server (InternalError): error when creating "ingress-service.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.kube-system.svc:443/extensions/v1beta1/ingresses?timeout=30s: dial tcp 10.97.204.103:443: connect: connection refused

[staticdev]

@tstromberg update: after some more tests, I see that most of the times it gives the error, but sometimes not. As you said, the max waiting time may be on the edge of the average time it takes. I would like to isolate which version update is causing this increased time so that we can upstream this issue to the component with degraded performance.

How can I do that? Do I need to recompile minikube from source or there is an easier way to do it?

[chitranshi21]

Hi,
I am getting the same error.
enable failed: run callbacks: running callbacks: [verifying ingress addon pods : timed out waiting for the condition: timed out waiting for the condition]
minikube version - V1.12.1
Any updates ?

[anotherandi]

I tried down-dating to 1.12.0 but had the same issue on windows 10.

[chukmunnlee]

I manage to get enable the ingress add on by enabling cni and install Cilium. Have not tried with other providers

I'm on 1.12.3

[jaurestchoffo]

Still no solution for this problem ??? I've been struggling for a while for my project and I have to support soon
thank you in advance for any solution
that is what i have

🔎 Verifying ingress addon...

💣 enable failed: run callbacks: running callbacks: [verifying ingress addon pods : timed out waiting for the condition: timed out waiting for the condition]

😿 minikube is exiting due to an error. If the above message is not useful, open an issue:
👉 https://github.com/kubernetes/minikube/issues/new/choose

[vvlookman]

Because the image is not pulled, try workaround:

$ kubectl get pod -A

List all the system pod names

$ kubectl describe pod/ingress-nginx-controller-xxxxxx --namespace=kube-system

Find out the image name which has not been pulled successfully, in my case is quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0

$ minikube ssh
$ docker pull quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0

Pull the image in minikube vm manully

[djvasquezvm]

Same problem. I followed your steps but my problem was:
Warning FailedMount 12m (x7 over 12m) kubelet, minikube MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found

So I realized it was SELinux. If you have SELinux enforcing, try using permissive mode.

Worked for me!

Because the image is not pulled, try workaround:

$ kubectl get pod -A

List all the system pod names

$ kubectl describe pod/ingress-nginx-controller-xxxxxx --namespace=kube-system

Find out the image name which has not been pulled successfully, in my case is quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0

$ minikube ssh
$ docker pull quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0

Pull the image in minikube vm manully

[ELTEGANI]

I faced the same I found the problem from my poor internet connection

[839928622]

I faced the same I found the problem from my poor internet connection
did u get it fix later on ?

[ELTEGANI]

I faced the same I found the problem from my poor internet connection
did u get it fix later on ?

Yeb fixed it

[renich]

In my case, I live in México and I have a 100 Mbps connection. Everything installed correctly but not the nginx ingress:

0s          Normal    BackOff             pod/ingress-nginx-controller-789d9c4dc-7wpkv    Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20"
0s          Warning   Failed              pod/ingress-nginx-controller-789d9c4dc-7wpkv    Error: ImagePullBackOff
0s          Normal    BackOff             pod/ingress-nginx-controller-789d9c4dc-7wpkv    Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20"
0s          Warning   Failed              pod/ingress-nginx-controller-789d9c4dc-7wpkv    Error: ImagePullBackOff
0s          Normal    Pulling             pod/ingress-nginx-controller-789d9c4dc-7wpkv    Pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20"
0s          Warning   Failed              pod/ingress-nginx-controller-789d9c4dc-7wpkv    Error: ErrImagePull
0s          Warning   Failed              pod/ingress-nginx-controller-789d9c4dc-7wpkv    Failed to pull image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20": rpc error: code = Unknown desc = Invalid image name "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20", unknown transport "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller"
0s          Normal    BackOff             pod/ingress-nginx-controller-789d9c4dc-7wpkv    Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20"
0s          Warning   Failed              pod/ingress-nginx-controller-789d9c4dc-7wpkv    Error: ImagePullBackOff
0s          Warning   Failed              pod/ingress-nginx-controller-789d9c4dc-7wpkv    Error: ImagePullBackOff

[renich]

It seems that the URL being used isn't correct. If you go here: https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/US/ingress-nginx/controller@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20 it works, but the URL logged is: us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20

If you put them side by side, ìt's easy to notice the difference:

us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20
us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20

[renich]

So, for me, edititng the deployment with the correct image URL seems to have worked.

To do this, just: kubectl -n kube-system edit deployment ingress-nginx-controller and look for "image:" and change it to:

image: us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20

[pimenvibritania]

So, for me, edititng the deployment with the correct image URL seems to have worked.

To do this, just: kubectl -n kube-system edit deployment ingress-nginx-controller and look for "image:" and change it to:

image: us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20

WOOWWW Thank You!! its work!

[WillNilges]

I'm still having trouble. I edited the image url, but I'm still not able to pull the image

[wnilges@thingsboard-minikube ~]$ kubectl get pods -A
NAMESPACE     NAME                                        READY   STATUS              RESTARTS   AGE
kube-system   coredns-f9fd979d6-lm5j8                     1/1     Running             0          26m
kube-system   etcd-minikube                               1/1     Running             0          26m
kube-system   ingress-nginx-admission-create-gxwkc        0/1     ImagePullBackOff    0          9m19s
kube-system   ingress-nginx-admission-patch-mz5bx         0/1     ImagePullBackOff    0          9m8s
kube-system   ingress-nginx-controller-7977f85d85-xbnwr   0/1     ContainerCreating   0          9m45s
kube-system   kube-apiserver-minikube                     1/1     Running             0          26m
kube-system   kube-controller-manager-minikube            1/1     Running             0          26m
kube-system   kube-proxy-2p9wr                            1/1     Running             0          26m
kube-system   kube-scheduler-minikube                     1/1     Running             0          26m
kube-system   storage-provisioner                         1/1     Running             1          26m

[WillNilges]

For reference, here's my full minikube start attempt:

[wnilges@thingsboard-minikube root]$  minikube start --addons=ingress --alsologtostderr -v=5

(It's really big)

[WillNilges]

Weather update: Works on my local Ubuntu 20 machine, not so much on my Fedora Server 32 VM. _Highly_ wack.

[Hongbo-Miao]

Thanks @renich ! Met same issue. After kubectl -n kube-system edit deployment ingress-nginx-controller, change

us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20

to

us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20

(You can try verify by opening first URL in browser, which gets 404 error, the second one which is also version 0.34.1 which can be opened)

Basically just need remove :v0.34.1.

After editing, sine it is Vim, just type :wq or :x and hit Enter.

And then do minikube addons enable ingress again.

[danielbeeke]

In my case I had something running on port 80, that can also happen.