Jx: Create Cluster with Helm 3 failing at Ingress creation

[tdcox]

Summary

Trying to create a cluster using helm3 fails with an RBAC issue

Steps to reproduce the behavior

➜ jx create cluster gke \
--cluster-name='d2' \
--default-admin-password='............' \
--enable-autoupgrade=false \
--environment-git-owner='.........' \
--git-username='............' \
--helm3=true \
--http='true' \
--labels='demo=true' \
--machine-type='n1-standard-2' \
--max-num-nodes='3' \
--min-num-nodes='3' \
--project-id='jx-demo-215821' \
--prow=true \
--skip-login=true \
--tls-acme='false' \
--zone='us-west1-b'
Updated property [core/project].
Lets ensure we have container and compute enabled on your project via: gcloud services enable container compute
Waiting for async operation operations/tmo-acf.c2f84bc1-95d9-4556-bf7b-54193270c07f to complete...
Operation finished successfully. The following command can describe the Operation details:
 gcloud services operations describe operations/tmo-acf.c2f84bc1-95d9-4556-bf7b-54193270c07f
Waiting for async operation operations/tmo-acf.0d862801-3d4c-40e8-a74e-08af84cc02e6 to complete...
Operation finished successfully. The following command can describe the Operation details:
 gcloud services operations describe operations/tmo-acf.0d862801-3d4c-40e8-a74e-08af84cc02e6
Creating cluster...
Initialising cluster ...
Git configured for user: .............. and email .............
Trying to create ClusterRoleBinding ............-cluster-admin-binding for role: cluster-admin for user ..............
: clusterrolebindings.rbac.authorization.k8s.io "...............-cluster-admin-binding" not foundCreated ClusterRoleBinding ................-cluster-admin-binding
Using helm3
helm installed and configured
? No existing ingress controller found in the kube-system namespace, shall we install one? Yes
Installing using helm binary: helm
Installing using helm binary: helm
Installing using helm binary: helm
Installing using helm binary: helm
Failed to install ingress chart: failed to run 'helm install --wait --name jxing --namespace kube-system stable/nginx-ingress --set rbac.create=true' command in directory '', output: 'Error: release jxing failed: namespaces "kube-system" is forbidden: User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "kube-system": Unknown user "system:serviceaccount:kube-system:default"': exit status 1
ingress init failed: deployments.extensions "jxing-nginx-ingress-controller" not found
error creating cluster failed to initialize the jx: deployments.extensions "jxing-nginx-ingress-controller" not found
error: failed to initialize the jx: deployments.extensions "jxing-nginx-ingress-controller" not found

Trying the same with helm 2 works fine...

➜ jx create cluster gke \
--cluster-name='d2' \
--default-admin-password='............' \
--enable-autoupgrade=false \
--environment-git-owner='............' \
--git-username='............' \
--helm3=false \
--http='true' \
--labels='demo=true' \
--machine-type='n1-standard-2' \
--max-num-nodes='3' \
--min-num-nodes='3' \
--project-id='jx-demo-215821' \
--prow=true \
--skip-login=true \
--tls-acme='false' \
--zone='us-west1-b'
Updated property [core/project].
Lets ensure we have container and compute enabled on your project via: gcloud services enable container compute
Waiting for async operation operations/tmo-acf.16ff48c1-ed14-4647-8ffe-99f2899313e4 to complete...
Operation finished successfully. The following command can describe the Operation details:
 gcloud services operations describe operations/tmo-acf.16ff48c1-ed14-4647-8ffe-99f2899313e4
Waiting for async operation operations/tmo-acf.c4bab553-6871-45da-9907-32acd7a6d06b to complete...
Operation finished successfully. The following command can describe the Operation details:
 gcloud services operations describe operations/tmo-acf.c4bab553-6871-45da-9907-32acd7a6d06b
Creating cluster...
Initialising cluster ...
Git configured for user: ............ and email ............
Trying to create ClusterRoleBinding ............-cluster-admin-binding for role: cluster-admin for user ............
: clusterrolebindings.rbac.authorization.k8s.io "............-cluster-admin-binding" not foundCreated ClusterRoleBinding ............-cluster-admin-binding
Using helm2
Created ServiceAccount tiller in namespace kube-system
Trying to create ClusterRoleBinding tiller for role: cluster-admin and ServiceAccount: kube-system/tiller
Created ClusterRoleBinding tiller
Initialising helm using ServiceAccount tiller in namespace kube-system
helm installed and configured
? No existing ingress controller found in the kube-system namespace, shall we install one? Yes
Installing using helm binary: helm
Waiting for external loadbalancer to be created and update the nginx-ingress-controller service in kube-system namespace
Note: this loadbalancer will fail to be provisioned if you have insufficient quotas, this can happen easily on a GKE free account. To view quotas run: gcloud compute project-info describe
External loadbalancer created
Waiting to find the external host name of the ingress controller Service in namespace kube-system with name jxing-nginx-ingress-controller
You can now configure a wildcard DNS pointing to the new loadbalancer address ............

Looks like Tiller is installed in both cases but in the former no ServiceAccount is created. It also appears that the helm binary is being called instead of the helm3 binary?

Jx version

The output of jx version is:

NAME               VERSION
jx                 1.3.235
jenkins x platform 0.0.2267
kubernetes cluster v1.9.7-gke.6
kubectl            v1.11.2
helm client        v2.10.0+g9ad53aa
helm server        v2.10.0+g9ad53aa
git                git version 2.18.0

Kubernetes cluster

GKE

Operating system / Environment

macOS 10.13.6

[salaboy]

I am hitting the same issue

Installing using helm binary: helm
Failed to install ingress chart: failed to run 'helm install --wait --name jxing --namespace kube-system stable/nginx-ingress --set rbac.create=true' command in directory '', output: 'Error: release jxing failed: namespaces "kube-system" is forbidden: User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "kube-system"': exit status 1
ingress init failed: deployments.apps "jxing-nginx-ingress-controller" not found
error: failed to initialize the jx: deployments.apps "jxing-nginx-ingress-controller" not found

[salaboy]

This fixed it for me:

kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller --upgrade

[ccojocar]

@tdcox I am going to close this issue. The fix is to create the cluster admin role binding for tiller.