Ingress-nginx: Validating webhook validate.nginx.ingress.kubernetes.io: annotation key has invalid format

[phidlipus]

NGINX Ingress controller version: 0.33.0

Kubernetes version (use kubectl version): 1.16.9 & 1.18.3

Environment:

Bare-metal and minikube (for testing)
Installed from https://github.com/kubernetes/ingress-nginx/blob/master/deploy/static/provider/baremetal/deploy.yaml and from the Helm Chart (chart: ingress-nginx-2.6.0) - ingress-nginx 0.33.0 in both situations

What happened:
I tried to create invalid ingress object:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: k8s-debug
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      blah;
spec:
  rules:
  - host: blah.blah.test
    http:
      paths:
      - backend:
          serviceName: k8s-debug
          servicePort: 80

It was refused to create it as expected, but Kube-apiserver also logs fail message. Probably there is something broken in message exchange between validation webhook and kube-apiserver.
The important part is Failed to set admission audit annotation validate.nginx.ingress.kubernetes.io/nginx.ingress.kubernetes.io/error to .... for validating webhook validate.nginx.ingress.kubernetes.io: annotation key has invalid format, the right format is a DNS subdomain prefix and '/' and key name. (e.g. 'podsecuritypolicy.admission.k8s.io/admit-policy').

W0618 05:54:38.001390       1 dispatcher.go:227] Failed to set admission audit annotation validate.nginx.ingress.kubernetes.io/nginx.ingress.kubernetes.io/error to
-------------------------------------------------------------------------------
Error: exit status 1
2020/06/18 05:54:37 [emerg] 126#126: unknown directive "blah" in /tmp/nginx-cfg429015755:546
nginx: [emerg] unknown directive "blah" in /tmp/nginx-cfg429015755:546
nginx: configuration file /tmp/nginx-cfg429015755 test failed

-------------------------------------------------------------------------------
 for validating webhook validate.nginx.ingress.kubernetes.io: annotation key has invalid format, the right format is a DNS subdomain prefix and '/' and key name. (e.g. 'podsecuritypolicy.admission.k8s.io/admit-policy')
W0618 05:54:38.001533       1 dispatcher.go:141] rejected by webhook "validate.nginx.ingress.kubernetes.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"validate.nginx.ingress.kubernetes.io\" denied the request: \n-------------------------------------------------------------------------------\nError: exit status 1\n2020/06/18 05:54:37 [emerg] 126#126: unknown directive \"blah\" in /tmp/nginx-cfg429015755:546\nnginx: [emerg] unknown directive \"blah\" in /tmp/nginx-cfg429015755:546\nnginx: configuration file /tmp/nginx-cfg429015755 test failed\n\n-------------------------------------------------------------------------------\n", Reason:"", Details:(*v1.StatusDetails)(nil), Code:400}}

What you expected to happen:

There shouldn't be this part of message:

W0618 05:54:38.001390       1 dispatcher.go:227] Failed to set admission audit annotation validate.nginx.ingress.kubernetes.io/nginx.ingress.kubernetes.io/error to
-------------------------------------------------------------------------------
Error: exit status 1
2020/06/18 05:54:37 [emerg] 126#126: unknown directive "blah" in /tmp/nginx-cfg429015755:546
nginx: [emerg] unknown directive "blah" in /tmp/nginx-cfg429015755:546
nginx: configuration file /tmp/nginx-cfg429015755 test failed

-------------------------------------------------------------------------------
 for validating webhook validate.nginx.ingress.kubernetes.io: annotation key has invalid format, the right format is a DNS subdomain prefix and '/' and key name. (e.g. 'podsecuritypolicy.admission.k8s.io/admit-policy')

How to reproduce it:
start minikube

minikube start

deploy Ingress-nginx from static yaml file

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml

or from Helm Chart

kubectl create ns ingress-nginx
helm -n ingress-nginx install my-release ingress-nginx/ingress-nginx

check the logs of the Kube-apiserver and Ingress-nginx in two separate consoles

kubectl -n kube-system logs -f kube-apiserver-minikube
kubectl -n ingress-nginx logs -f ingress-nginx-controller-pod

try to create invalid ingress (yaml is above)

kubectl apply -f invalid-ingress.yaml

/kind bug

[kundan2707]

How to reproduce it:
start minikube

    ```minikube start```

deploy Ingress-nginx from static yaml file

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml

Method to deploy ingress-nginx in minikube is describes in deployement doc is followoing.
minikube addons enable ingress

I tried same invalid-ingress.yaml on minikube and it got created without any error

vagrant@ubuntu-xenial:~$ minikube addons enable ingress 🌟 The 'ingress' addon is enabled vagrant@ubuntu-xenial:~$ kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-66bff467f8-8sj7t 1/1 Running 2 56d kube-system coredns-66bff467f8-rd4s2 1/1 Running 2 56d kube-system etcd-minikube 1/1 Running 1 56d kube-system kindnet-nn4pw 1/1 Running 2 56d kube-system kube-apiserver-minikube 1/1 Running 1 56d kube-system kube-controller-manager-minikube 1/1 Running 1 56d kube-system kube-proxy-mjwwx 1/1 Running 1 56d kube-system kube-scheduler-minikube 1/1 Running 1 56d kube-system nginx-ingress-controller-6d57c87cb9-zn9b7 1/1 Running 2 56d kube-system storage-provisioner 1/1 Running 2 56d vagrant@ubuntu-xenial:~$ kubectl apply -f invalid-ingress.yaml ingress.networking.k8s.io/k8s-debug created vagrant@ubuntu-xenial:~$ kubectl get Ingress -A NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE default k8s-debug <none> blah.blah.test 172.17.0.2 80 15m

[phidlipus]

@kundan2707 thank you.
This is not about deployment method. I used both methods (deploy.yaml and Helm Chart) on Minikube just for testing of validating webhook.

Do you have validating webhook correctly enabled? I don't think so, because in that case the invalid Ingress objects should be rejected and only valid Ingress objects will be allowed.

[phidlipus]

@kundan2707 which version of Minikube are you using?
I just tested minikube addons enable ingress on minikube version: v1.11.0 and it rejects invalid ingress object too and there is same fail in Kube-apiserver's logs. So it's really not about deployment method.

I don't know if you understand me. The validating webhook working fine - it rejects invalid objects and allows valid objects. It's working fine for me. This issue is about the fail message in Kube-apiserver log which say that there is some mistake in webhook's configuration or code. This is not fatal and webhook is working, but (probably) there is some incompatibility with Kube-apiserver's way of working with log messages.

[kundan2707]

@kundan2707 thank you.
This is not about deployment method. I used both methods (deploy.yaml and Helm Chart) on Minikube just for testing of validating webhook.

Do you have validating webhook correctly enabled? I don't think so, because in that case the invalid Ingress objects should be rejected and only valid Ingress objects will be allowed.

@filip-havlicek Yes, I checked validating webhoot was not correctly enabled

[kundan2707]

@kundan2707 which version of Minikube are you using?
I just tested minikube addons enable ingress on minikube version: v1.11.0 and it rejects invalid ingress object too and there is same fail in Kube-apiserver's logs. So it's really not about deployment method.

I don't know if you understand me. The validating webhook working fine - it rejects invalid objects and allows valid objects. It's working fine for me. This issue is about the fail message in Kube-apiserver log which say that there is some mistake in webhook's configuration or code. This is not fatal and webhook is working, but (probably) there is some incompatibility with Kube-apiserver's way of working with log messages.
@filip-havlicek I am using minikube version: v1.11.0
Yes i understand. After corretly configuring validating webhook I am also getting same message

[vsxen]

API group is err in doc

https://github.com/kubernetes/ingress-nginx/blob/nginx-0.30.0/docs/deploy/validating-webhook.md
```yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
name: check-ingress
webhooks:

• name: validate.nginx.ingress.kubernetes.io
  rules:
  
  
  
  • apiGroups:
  
  
  • extensions #1.16 -
  
  
  • networking.k8s.io #1.16+
    
    apiVersions:
  
  
  • v1beta1
    
    operations:
  
  
  • CREATE
  
  
  • UPDATE
    
    resources:
  
  
  • ingresses
    
    failurePolicy: Fail
    
    clientConfig:
    
    service:
    
    namespace: ingress-nginx
    
    name: ingress-validation-webhook
    
    path: /networking.k8s.io/v1beta1/ingress
    
    caBundle:
    
    ````
  
  

https://github.com/kubernetes/ingress-nginx/commit/0dce5be743ae012dc5b22c1cf4b91dcdb8b64d88

[kilianc]

I am getting the above message

> kubectl apply -f ingress.yaml

Error from server: error when creating "ingress.yaml": admission webhook "validate.nginx.ingress.kubernetes.io"
denied the request: rejecting admission review because the request does not contains an Ingress resource but
networking.k8s.io/v1, Resource=ingresses with name example-ingress in namespace default

What does this mean?

[aledbf]

Please update to v0.41.0 (chart version 3.8.0).

[aledbf]

I am getting the above message

> kubectl apply -f ingress.yaml

Error from server: error when creating "ingress.yaml": admission webhook "validate.nginx.ingress.kubernetes.io"
denied the request: rejecting admission review because the request does not contains an Ingress resource but
networking.k8s.io/v1, Resource=ingresses with name example-ingress in namespace default

What does this mean?

This issue is fixed in 0.40.0. Before this version ingresses networking.k8s.io/v1 (you are running k8s 1.19) were not supported in the validation webhook