Ingress-nginx: Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: context deadline exceeded

[luppie]

Hi!

I've installed kubernetes cluster on three servers: l000d01kms001 (K8s master) and l000d01ksl001 & l000d01ksl001 - all basen Centos 7.5

Installed software:

docker-ce-cli-19.03.6-3.el7.x86_64
docker-ce-19.03.6-3.el7.x86_64
kubelet-1.17.3-0.x86_64
kubernetes-cni-0.7.5-0.x86_64
kubeadm-1.17.3-0.x86_64
kubectl-1.17.3-0.x86_64`

I've installed MetalLB (https://metallb.universe.tf) and then deployed Ingress-Nginx from URL https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/baremetal/deploy.yaml

When trying to deploy endpoint.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: test-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:

• http:
  paths:
  
  
  
  • path: /testpath
    
    pathType: Prefix
    
    backend:
    
    serviceName: hellok8s-service
    
    servicePort: 8080
  
  

I'm getting error as below:

[root@l000d01kms001 KubeHelm]# kubectl apply -f endpoint.yaml --validate=false
Error from server (InternalError): error when creating "endpoint.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: context deadline exceeded

---

`[root@l000d01kms001 KubeHelm]# kubectl logs pod/ingress-nginx-admission-create-44q78 -n ingress-nginx
^C
[root@l000d01kms001 KubeHelm]# kubectl logs pod/ingress-nginx-admission-patch-gqwbr -n ingress-nginx
{"level":"info","msg":"patching webhook configurations 'ingress-nginx-admission' mutating=false, validating=true, failurePolicy=Fail","source":"k8s/k8s.go:38","time":"2020-05-20T20:29:52Z"}
{"level":"info","msg":"Patched hook(s)","source":"k8s/k8s.go:91","time":"2020-05-20T20:29:52Z"}

[root@l000d01kms001 KubeHelm]# kubectl logs pod/ingress-nginx-controller-f8d756996-rhmjl -n ingress-nginx

NGINX Ingress controller
Release: 0.32.0
Build: git-446845114
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.17.10

---

I0520 20:29:56.807345 6 flags.go:204] Watching for Ingress class: nginx
W0520 20:29:56.807691 6 flags.go:249] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0520 20:29:56.807752 6 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0520 20:29:56.807947 6 main.go:220] Creating API client for https://10.96.0.1:443
I0520 20:29:56.815151 6 main.go:264] Running in Kubernetes cluster version v1.17 (v1.17.3) - git (clean) commit 06ad960bfd03b39c8310aaf92d1e7c12ce618213 - platform linux/amd64
I0520 20:29:57.013592 6 main.go:105] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem
I0520 20:29:57.018776 6 ssl.go:528] loading tls certificate from certificate path /usr/local/certificates/cert and key path /usr/local/certificates/key
I0520 20:29:57.043694 6 nginx.go:263] Starting NGINX Ingress controller
I0520 20:29:57.047238 6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"a204e9f1-6905-44c8-9fd0-e0aae4355e46", APIVersion:"v1", ResourceVersion:"2794479", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0520 20:29:58.244113 6 nginx.go:307] Starting NGINX process
I0520 20:29:58.244155 6 leaderelection.go:242] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I0520 20:29:58.244567 6 nginx.go:327] Starting validation webhook on :8443 with keys /usr/local/certificates/cert /usr/local/certificates/key
I0520 20:29:58.244740 6 controller.go:139] Configuration changes detected, backend reload required.
I0520 20:29:58.248112 6 leaderelection.go:252] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0520 20:29:58.248192 6 status.go:86] new leader elected: ingress-nginx-controller-f8d756996-rhmjl
I0520 20:29:58.306048 6 controller.go:155] Backend successfully reloaded.
I0520 20:29:58.306087 6 controller.go:164] Initial sync, sleeping for 1 second.
[root@l000d01kms001 KubeHelm]#`

[aledbf]

@luppie when you deploy the ingress controller for the first time two Jobs are executed in order to generate the SSL certificate for the validation webhook. For this reason, it can take up to 90 seconds the start of the controller. After you start you can run the command

kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=90s

when that finish the ingress controller webhook is ready and you should not see that error

[luppie]

@aledbf - this command returns
`[root@l000d01kms001 KubeHelm]# kubectl wait --namespace ingress-nginx \

--for=condition=ready pod \
--selector=app.kubernetes.io/component=controller \
--timeout=90s
pod/ingress-nginx-controller-f8d756996-rhmjl condition met
[root@l000d01kms001 KubeHelm]#`

Error still occurs
[root@l000d01kms001 KubeHelm]# kubectl apply -f endpoint.yaml --validate=false Error from server (InternalError): error when creating "endpoint.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: context deadline exceeded [root@l000d01kms001 KubeHelm]#

[aledbf]

this command returns
Error still occurs

Ok, that is strange.

pathType: Prefix

Checking the example and the versions of the packages you used, the only "error" I see in the field pathType. That field is available since k8s v1.18.

[aledbf]

@luppie I cannot reproduce this issue

kind create cluster --image=kindest/node:v1.17.5@sha256:ab3f9e6ec5ad8840eeb1f76c89bb7948c77bbf76bcebe1a8b59790b8ae9a283a

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/baremetal/deploy.yaml

kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=90s

echo "
 apiVersion: networking.k8s.io/v1beta1
 kind: Ingress
 metadata:
   name: test-ingress
   annotations:
     nginx.ingress.kubernetes.io/rewrite-target: /
 spec:
   rules:
   - http:
       paths:
       - path: /testpath
         pathType: Prefix
         backend:
           serviceName: hellok8s-service
           servicePort: 8080
" | kubectl apply -f -

error: error validating "STDIN": error validating data: ValidationError(Ingress.spec.rules[0].http.paths[0]): unknown field "pathType" in io.k8s.api.networking.v1beta1.HTTPIngressPath; if you choose to ignore these errors, turn validation off with --validate=false

In fact, the error I see is the one I mentioned in my previous comment

Removing the line pathType: Prefix I get
ingress.networking.k8s.io/test-ingress created

[luppie]

I've reinstalled Kubernetes with Calico network plugin and problem doesn't occur.

[adamxy]

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

[thirumurthis]

I got the same error:

Error from server (InternalError): error when creating "ingressresource.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

after deleting the kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission now the ingress rule got created.

[marcboivin]

@adamxy's solution worked for me to, but why? What was wrong in my config? I'm also using MetalLB. Will this be a recurrent problem?

Regards,

[boxcee]

Yeah, seems like an odd approach to just delete something.

We are using a private cluster on GKE. Not sure if related.

You can also disable the webhook: controller.admissionWebhooks.enabled.

I am still curious why this doesn't work.

[msgongora]

@boxcee explanation here: kubernetes/kubernetes/issues/79739

[hunzaboy]

Run the ingress controller again worked for me.

Related issue : ( might help others )
I also had the issue of not able to access the localhost. So I reset Kubernetes from Docker Dashboard and then it worked.