Cert-manager: Cannot install webhook with helm

[snstanton]

I am unable to install due to a validation failure:

missing required field "caBundle" in io.k8s.api.admissionregistration.v1beta1.WebhookClientConfig

The docs say to specify --validate=false, but that is not an option to helm.

I would expect the template to be set up properly to support the k8s v1.12 api since that's what's available on EKS.

Environment details::
kubectl client v1.14.1 and server v1.12.6-eks-d69f1b
Amazon EKS cluster
helm v2.14.0
cert-manager-v0.7.2

/kind bug

[sc0ttwad3]

I'm having the same issue trying to install cert-manager and getting the missing required field "caBundle" error.

$ helm install --name cert-manager --namespace cert-manager --version v0.7.2 jetstack/cert-manager

Error: validation failed: error validating "": error validating data: [ValidationError(ValidatingWebhookConfiguration.webhooks[0].clientConfig): missing required field "caBundle" in io.k8s.api.admissionregistration.v1beta1.WebhookClientConfig, ValidationError(ValidatingWebhookConfiguration.webhooks[1].clientConfig): missing required field "caBundle" in io.k8s.api.admissionregistration.v1beta1.WebhookClientConfig, ValidationError(ValidatingWebhookConfiguration.webhooks[2].clientConfig): missing required field "caBundle" in io.k8s.api.admissionregistration.v1beta1.WebhookClientConfig]

Environment:

Helm v2.14.0

kubectl
    Client "v1.10.11", GoVersion:"go1.9.3", Platform:"windows/amd64"}
    Server "v1.10.11", GoVersion:"go1.9.3", Platform:"linux/amd64"}

Server: Docker Engine - Community
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  OS/Arch:          linux/amd64
Kubernetes:
  Version:          v1.10.11
  StackAPI:         v1beta2

Client: Docker Engine - Community
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.8
 OS/Arch:           windows/amd64

Windows 10 Professional
Insiders Build: 18895
Go version: go1.12.4 windows/amd64

[caiohasouza]

I have the same error on helm 2.14, if i downgrade helm to 2.13 works fine.

[caiohasouza]

In helm 2.14 the schema validation errors not is silenced anymore (https://github.com/helm/helm/pull/5576), there are a helm PR to fix (https://github.com/helm/helm/issues/5640).

[sc0ttwad3]

@caiohasouza Thanks, I was starting to pull my hair out. ;-)

[snstanton]

It seems like a version check around an empty caBundle property in the validating-webhook.yaml template would be a permanent fix for this problem.

[caiohasouza]

@snstanton sounds good, do you have some example or PR?

[Ogala]

I have the same error on helm 2.14, if i downgrade helm to 2.13 works fine.

This just happened to me after I upgraded to 2.14. @caiohasouza what's the best way to downgrade?

[snstanton]

I just went through this process. I uninstalled kubernetes-helm and then ran this command to install 2.13.1 directly from git:

brew install  https://raw.githubusercontent.com/Homebrew/homebrew-core/1e8e12c73c465f15ae25ed8bc5f321018b9107b1/Formula/kubernetes-helm.rb

[caiohasouza]

@Ogala i didn't "downgrade" the tiller, I just create a new temporarily tiller on 2.13 to run test.

[caiohasouza]

I created a PR (https://github.com/jetstack/cert-manager/pull/1694) to fix this problem, i tested locally and works fine.

[jhnferraris]

@caiohasouza Hello I'm new here in Helm. can i ask how to install cert-manager and point it to use your PR?

[caiohasouza]

@jhnferraris in my local environment i cloned the cert-manager repository, make the both changes how to PR and deploy with helm using the local chart, do you understand?

[spacecat]

I have the same error on helm 2.14, if i downgrade helm to 2.13 works fine.

This just happened to me after I upgraded to 2.14. @caiohasouza what's the best way to downgrade?

You could try the following (Windows):

1. Download version 2.13.0 of helm from releases in the helm GitHub repo.

2. Check current version:

helm version
Client: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}

1. Replace helm.exe and tiller.exe with the older versions.

2. Downgrade helm:

helm init --upgrade
$HELM_HOME has been configured at C:\Users\homer\.helm.
Error: error when upgrading: current Tiller version is newer, use --force-upgrade to downgrade

helm init --upgrade --force-upgrade
$HELM_HOME has been configured at C:\Users\homer\.helm.

Tiller (the Helm server-side component) has been upgraded to the current version.

helm version
Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}

[WebSpider]

Yeah, I'm seeing the same on:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:23:09Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7", GitCommit:"6f482974b76db3f1e0f5d24605a9d1d38fad9a2b", GitTreeState:"clean", BuildDate:"2019-03-25T02:41:57Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

$ helm version
2019/05/29 02:32:48.168301 main.go:220: WARNING: cannot create syslog logger
Client: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}

[CanKattwinkel]

In order to extend the Windows answer for bash systems:

Download and make executable (chmod +x) https://raw.githubusercontent.com/helm/helm/master/scripts/get

$ DESIRED_VERSION=v2.13.1 ./helm.sh 
$ helm init --upgrade --force-upgrade

[iceydee]

Just ran into this and when I went to download the older version of helm I noticed that 2.14.1 has a fix for validation of manifests.

This is fixed in helm now.

[alhafoudh]

Just ran into this and when I went to download the older version of helm I noticed that 2.14.1 has a fix for validation of manifests.

This is fixed in helm now.

I can confirm! Be sure to upgrade tiller also.

[caiohasouza]

Yes, i commented in my PR (https://github.com/jetstack/cert-manager/pull/1694).

[filoe]

How to upgrade to help 2.14.1.
helm init --upgrade only upgrades to 2.14.0

[alhafoudh]

@filoe did you upgrade you local helm binary first?

[filoe]

My fault @alhafoudh.
Ran the install script and everything works as expected.

[Grishas]

Just ran into this and when I went to download the older version of helm I noticed that 2.14.1 has a fix for validation of manifests.

This is fixed in helm now.

thanks for the point

solution update helm and tiller to 2.14.1

curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh
chmod +x get_helm.sh
./get_helm.sh
helm init --service-account tiller
helm init --service-account tiller --upgrade
helm version
helm install --name cert-manager --namespace cert-manager --version v0.8.1 jetstack/cert-manager

[munnerz]

Sounds like this is fixed in the latest version of Helm! I'm going to close this now to keep the board tidy 😄

[get2arun]

Hi Team,
I am using the lastet version of helm v3.0.0 rc2, I am facing the same issue.
But the issue is with kubernetes v 1.12 and caBundle is a required field.

[FabriceMk]

Same, we are using helm v3.0.1 and got the issue with Kubernetes 1.12
The bug appeared with 2.14, was fixed in 2.14.1

But it seems there may be a regression with the 3.x branch

We had to downgrade to 2.14.1 to get it working.

[galingit]

+1
I am also facing the same issue when I try to install vault-helm with 3.x version.
Any workaround for current helm version?

[vfiset]

@galingit did you find a solution ?

[galingit]

No, we fixed the issue by deploying manually vault and exporting all yaml files to a new custom helm chart.