Cloud-on-k8s: All-in-one installation using k8s 1.19 fails because of a removed k8s API (v1beta1/ValidatingWebhookConfiguration)

[maorkh4]

Deploying ECK operator 1.3.0 using the all-in-one YAML with k8s 1.19 fails because it tries to use the removed admissionregistration.k8s.io/v1beta1/ValidatingWebhookConfiguration API. The error message is:

kubernetes:admissionregistration.k8s.io/v1beta1:ValidatingWebhookConfiguration (elastic-webhook.k8s.elastic.co):
  error: apiVersion "admissionregistration.k8s.io/v1beta1/ValidatingWebhookConfiguration" was removed in Kubernetes 1.19. Use "admissionregistration.k8s.io/v1/ValidatingWebhookConfiguration" instead.
  See https://git.k8s.io/kubernetes/CHANGELOG/CHANGELOG-1.19.md#deprecation-1 for more information.

kubectl version returns:

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

Downgrading to k8s 1.18.8 fixes the problem but produces the warning:

kubernetes:admissionregistration.k8s.io/v1beta1:ValidatingWebhookConfiguration (elastic-webhook.k8s.elastic.co):
  warning: admissionregistration.k8s.io/v1beta1/ValidatingWebhookConfiguration is deprecated by admissionregistration.k8s.io/v1/ValidatingWebhookConfiguration and not supported by Kubernetes v1.19+ clusters.

An issue was already opened regarding the warning on k8s 1.16+ (#3886).

[alongubkin]

This error prevents us from using K8s 1.19. It's also pretty annoying in local dev because we can't update our Docker for Desktop versions (which are now K8s 1.19 by default).

Is there any solution to this (even a temporary one)?

[sebgl]

Temporary solution: I'm pretty sure (not tested) you can manually modify the all-in-one YAML to use the ValidatingWebhookConfiguration v1 instead of v1beta1.

[alongubkin]

We tried this, it didn't work, since some of the fields in ValidatingWebhookConfiguration were changed

[charith-elastic]

I think this is a little bit more involved than just updating the webhook definition. I am getting the following error when using v1 with ECK 1.3.0 on Kubernetes 1.18:

Manifest

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: elastic-webhook.k8s.elastic.co
  labels:

    app.kubernetes.io/name: elastic-operator
    app.kubernetes.io/instance: elastic-operator
    app.kubernetes.io/version: "1.4.0-SNAPSHOT"
    helm.sh/chart: eck-operator-1.4.0-SNAPSHOT
    app.kubernetes.io/managed-by: Helm
webhooks:
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-apm-k8s-elastic-co-v1-apmserver
  failurePolicy: Ignore
  name: elastic-apm-validation-v1.k8s.elastic.co  
  admissionReviewVersions: [v1, v1beta1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - apm.k8s.elastic.co
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - apmservers
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-apm-k8s-elastic-co-v1beta1-apmserver
  failurePolicy: Ignore
  name: elastic-apm-validation-v1beta1.k8s.elastic.co  
  admissionReviewVersions: [v1, v1beta1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - apm.k8s.elastic.co
    apiVersions:
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - apmservers
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-beat-k8s-elastic-co-v1beta1-beat
  failurePolicy: Ignore
  name: elastic-beat-validation-v1beta1.k8s.elastic.co  
  admissionReviewVersions: [v1, v1beta1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - beat.k8s.elastic.co
    apiVersions:
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - beats
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-enterprisesearch-k8s-elastic-co-v1beta1-enterprisesearch
  failurePolicy: Ignore
  name: elastic-ent-validation-v1beta1.k8s.elastic.co  
  admissionReviewVersions: [v1, v1beta1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - enterprisesearch.k8s.elastic.co
    apiVersions:
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - enterprisesearches
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-elasticsearch-k8s-elastic-co-v1-elasticsearch
  failurePolicy: Ignore
  name: elastic-es-validation-v1.k8s.elastic.co  
  admissionReviewVersions: [v1, v1beta1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - elasticsearch.k8s.elastic.co
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - elasticsearches
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-elasticsearch-k8s-elastic-co-v1beta1-elasticsearch
  failurePolicy: Ignore
  name: elastic-es-validation-v1beta1.k8s.elastic.co  
  admissionReviewVersions: [v1, v1beta1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - elasticsearch.k8s.elastic.co
    apiVersions:
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - elasticsearches
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-kibana-k8s-elastic-co-v1-kibana
  failurePolicy: Ignore
  name: elastic-kb-validation-v1.k8s.elastic.co  
  admissionReviewVersions: [v1, v1beta1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - kibana.k8s.elastic.co
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - kibanas
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-kibana-k8s-elastic-co-v1beta1-kibana
  failurePolicy: Ignore
  name: elastic-kb-validation-v1beta1.k8s.elastic.co  
  admissionReviewVersions: [v1, v1beta1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - kibana.k8s.elastic.co
    apiVersions:
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - kibanas

Internal error occurred: failed calling webhook "elastic-es-validation-v1beta1.k8s.elastic.co": expected response.uid="54d0440e-83a6-4f87-817c-0e6819ed7060", got ""

It looks like controller-runtime cannot decode the request:

{"log.level":"error","@timestamp":"2020-11-19T12:46:47.257Z","log.logger":"controller-runtime.webhook.webhooks","message":"unable to decode the request","service.version":"1.3.0+6db1914b","service.type":"eck","ecs.version":"1.4.0","webhook":"/validate-elasticsearch-k8s-elastic-co-v1-elasticsearch","error":"no kind \"AdmissionReview\" is registered for version \"admission.k8s.io/v1\" in scheme \"pkg/runtime/scheme.go:101\"","error.stack_trace":"sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:79\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1\n\t/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2042\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1\n\t/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:100\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2042\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2\n\t/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2042\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2417\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2843\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1925"}

This looks like an issue with controller-runtime but I couldn't find anyone else running into the problem. We need to investigate further.

We also manipulate the webhook in code when the operator is configured to manage webhook certificates. We may have to upgrade those API calls to v1 as well but I am not sure how that will affect the operator when run on older versions of Kubernetes.

A temporary solution for those on Kubernetes 1.19 would be to disable the webhook. The operator will still validate the manifests and any validation errors will be emitted as events associated with the resource.

[charith-elastic]

It is an issue with controller-runtime: https://github.com/kubernetes-sigs/controller-runtime/issues/1161

[charith-elastic]

I found a workaround. The trick is to set v1beta1 first in the admissionReviewVersions array.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: elastic-webhook.k8s.elastic.co
  labels:

    app.kubernetes.io/name: elastic-operator
    app.kubernetes.io/instance: elastic-operator
    app.kubernetes.io/version: "1.4.0-SNAPSHOT"
    helm.sh/chart: eck-operator-1.4.0-SNAPSHOT
    app.kubernetes.io/managed-by: Helm
webhooks:
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-apm-k8s-elastic-co-v1-apmserver
  failurePolicy: Ignore
  name: elastic-apm-validation-v1.k8s.elastic.co  
  admissionReviewVersions: [v1beta1, v1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - apm.k8s.elastic.co
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - apmservers
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-apm-k8s-elastic-co-v1beta1-apmserver
  failurePolicy: Ignore
  name: elastic-apm-validation-v1beta1.k8s.elastic.co  
  admissionReviewVersions: [v1beta1, v1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - apm.k8s.elastic.co
    apiVersions:
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - apmservers
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-beat-k8s-elastic-co-v1beta1-beat
  failurePolicy: Ignore
  name: elastic-beat-validation-v1beta1.k8s.elastic.co  
  admissionReviewVersions: [v1beta1, v1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - beat.k8s.elastic.co
    apiVersions:
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - beats
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-enterprisesearch-k8s-elastic-co-v1beta1-enterprisesearch
  failurePolicy: Ignore
  name: elastic-ent-validation-v1beta1.k8s.elastic.co  
  admissionReviewVersions: [v1beta1, v1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - enterprisesearch.k8s.elastic.co
    apiVersions:
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - enterprisesearches
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-elasticsearch-k8s-elastic-co-v1-elasticsearch
  failurePolicy: Ignore
  name: elastic-es-validation-v1.k8s.elastic.co  
  admissionReviewVersions: [v1beta1, v1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - elasticsearch.k8s.elastic.co
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - elasticsearches
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-elasticsearch-k8s-elastic-co-v1beta1-elasticsearch
  failurePolicy: Ignore
  name: elastic-es-validation-v1beta1.k8s.elastic.co  
  admissionReviewVersions: [v1beta1, v1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - elasticsearch.k8s.elastic.co
    apiVersions:
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - elasticsearches
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-kibana-k8s-elastic-co-v1-kibana
  failurePolicy: Ignore
  name: elastic-kb-validation-v1.k8s.elastic.co  
  admissionReviewVersions: [v1beta1, v1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - kibana.k8s.elastic.co
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - kibanas
- clientConfig:
    caBundle: Cg==
    service:
      name: elastic-webhook-server
      namespace: elastic-system
      path: /validate-kibana-k8s-elastic-co-v1beta1-kibana
  failurePolicy: Ignore
  name: elastic-kb-validation-v1beta1.k8s.elastic.co  
  admissionReviewVersions: [v1beta1, v1]  
  sideEffects: "None"
  rules:
  - apiGroups:
    - kibana.k8s.elastic.co
    apiVersions:
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - kibanas

[barkbay]

Hi @maorkh4, can you give us more information about the kind of cluster your are using (self managed, Azure, EKS...) ? admissionregistration.k8s.io/v1beta1 is supposed to be removed in 1.22 only: https://github.com/kubernetes/kubernetes/issues/82021#issuecomment-636873001

I did a test on Kind 1.19.3 (kind create cluster --image kindest/node:v1.19.3) and it is still working :

~ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-11T13:17:17Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-11-13T02:48:43Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

~ kubectl get nodes
NAME                 STATUS   ROLES    AGE     VERSION
kind-control-plane   Ready    master   6m23s   v1.19.3

~ kubectl apply -f https://download.elastic.co/downloads/eck/1.3.0/all-in-one.yaml

namespace/elastic-system created
serviceaccount/elastic-operator created
secret/elastic-webhook-server-cert created
configmap/elastic-operator created
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/apmservers.apm.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/beats.beat.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticsearches.elasticsearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/enterprisesearches.enterprisesearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/kibanas.kibana.k8s.elastic.co created
clusterrole.rbac.authorization.k8s.io/elastic-operator created
clusterrole.rbac.authorization.k8s.io/elastic-operator-view created
clusterrole.rbac.authorization.k8s.io/elastic-operator-edit created
clusterrolebinding.rbac.authorization.k8s.io/elastic-operator created
service/elastic-webhook-server created
statefulset.apps/elastic-operator created
Warning: admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
validatingwebhookconfiguration.admissionregistration.k8s.io/elastic-webhook.k8s.elastic.co created

~ kubectl get validatingwebhookconfiguration
NAME                             WEBHOOKS   AGE
elastic-webhook.k8s.elastic.co   8          28s

[maorkh4]

@charith-elastic @barkbay Thanks for your response.

@barkbay, regarding your question, I tried to deploy the ECK operator locally using docker-desktop cluster and pulumi.
I tried to run the same commands as you did with the docker-desktop cluster and it worked for me also. The error is reproduced only when using pulumi - so it seems like an issue in that SDK.

It worth noting that @charith-elastic's workaround helps when deploying with pulumi, so we might use it.

[maorkh4]

Opened an issue for pulumi: https://github.com/pulumi/pulumi-kubernetes/issues/1388

[pebrc]

Given that the issue seems to be with pulumi I am closing this issue for now. Please reopen if needed.