Jx: Helm fails to install in a brand new cluster in AKS and Minikube

[carlosignaciom]

Summary

Steps to reproduce the behavior

jx create cluster aks -c mycluster -n myresourcename -v 1.11.9 -o 2 --verbose

Expected behavior

Jenkins x up and running

Actual behavior

Upgrading Chart 'upgrade --namespace jx --install --timeout 6000 --version 0.0.3832 --values /home/carlos/.jx/cloud-environments/env-aks/myvalues.yaml --values /home/carlos/.jx/adminSecrets.yaml --values /home/carlos/.jx/extraValues.yaml --values /home/carlos/.jx/cloud-environments/env-aks/secrets.yaml jenkins-x jenkins-x/jenkins-x-platform'
error creating cluster installing the Jenkins X platform: failed to install/upgrade the jenkins-x platform chart: failed to run 'helm upgrade --namespace jx --install --timeout 6000 --version 0.0.3832 --values /home/carlos/.jx/cloud-environments/env-aks/myvalues.yaml --values /home/carlos/.jx/adminSecrets.yaml --values /home/carlos/.jx/extraValues.yaml --values /home/carlos/.jx/cloud-environments/env-aks/secrets.yaml jenkins-x jenkins-x/jenkins-x-platform' command in directory '/home/carlos/.jx/cloud-environments/env-aks', output: 'Release "jenkins-x" does not exist. Installing it now.
NAME: jenkins-x
Error: getting deployed release "jenkins-x": release: "jenkins-x" not found'

Jx version

The output of jx version is:

NAME               VERSION
jx                 1.3.1096
Kubernetes cluster v1.11.9
kubectl            v1.8.0
helm client        v2.13.1+g618447c
helm server        v2.13.1+g618447c
git                git version 2.17.1
Operating System   Ubuntu 18.04.2 LTS

Jenkins type

• [x] Classic Jenkins
• [ ] Serverless Jenkins

Kubernetes cluster

Brand New AKS cluster with jenkins-x

Operating system / Environment

[jcalavia]

Same behaviour in minikube:

Steps

jx create cluster minikube

Actual behaviour

...
Installing jenkins-x-platform version: 0.0.3832
Adding values file /Users/user/.jx/cloud-environments/env-minikube/myvalues.yaml
Adding values file /Users/user/.jx/adminSecrets.yaml
Adding values file /Users/user/.jx/extraValues.yaml
Adding values file /Users/user/.jx/cloud-environments/env-minikube/secrets.yaml
Upgrading Chart 'upgrade --namespace services --install --timeout 6000 --version 0.0.3832 --values /Users/user/.jx/cloud-environments/env-minikube/myvalues.yaml --values /Users/user/.jx/adminSecrets.yaml --values /Users/user/.jx/extraValues.yaml --values /Users/user/.jx/cloud-environments/env-minikube/secrets.yaml jenkins-x jenkins-x/jenkins-x-platform'



error creating cluster installing the Jenkins X platform: failed to install/upgrade the jenkins-x platform chart: failed to run 'helm upgrade --namespace services --install --timeout 6000 --version 0.0.3832 --values /Users/user/.jx/cloud-environments/env-minikube/myvalues.yaml --values /Users/user/.jx/adminSecrets.yaml --values /Users/user/.jx/extraValues.yaml --values /Users/user/.jx/cloud-environments/env-minikube/secrets.yaml jenkins-x jenkins-x/jenkins-x-platform' command in directory '/Users/user/.jx/cloud-environments/env-minikube', output: 'Release "jenkins-x" does not exist. Installing it now.
NAME:   jenkins-x
Error: getting deployed release "jenkins-x": release: "jenkins-x" not found'

JX Version

jx version 

NAME             VERSION
jx               1.3.1096
git              git version 2.20.1
Operating System Mac OS X 10.14.4 build 18E226

[krishnakumar6893]

@carlosignaciom @jcalavia Yup, issue occurs from yesterday. Not able to install Jenkins-X locally using minikube. Also facing issue with chart upgrades

[desolatorxxl]

Same issue with my cluster set up "the hard way", I already got it working about two weeks ago on the same cluster, though.

Edit: Here's a gist with the full log https://gist.github.com/desolatorxxl/d70eadd02f44a7180d943ed957a161ca.

[desolatorxxl]

So, tiller reports that the configmap is too large:

[tiller] 2019/04/10 20:21:26 performing install for jenkins-x
[tiller] 2019/04/10 20:21:26 executing 3 crd-install hooks for jenkins-x
[tiller] 2019/04/10 20:21:26 hooks complete for crd-install jenkins-x
[tiller] 2019/04/10 20:21:26 executing 3 pre-install hooks for jenkins-x
[tiller] 2019/04/10 20:21:26 hooks complete for pre-install jenkins-x
[storage] 2019/04/10 20:21:26 getting release history for "jenkins-x"
[storage] 2019/04/10 20:21:27 creating release "jenkins-x.v1"
[storage/driver] 2019/04/10 20:21:27 create: failed to create: ConfigMap "jenkins-x.v1" is invalid: []: Too long: must have at most 1048576 characters
[tiller] 2019/04/10 20:21:27 warning: Failed to record release jenkins-x: ConfigMap "jenkins-x.v1" is invalid: []: Too long: must have at most 1048576 characters
[kube] 2019/04/10 20:21:27 building resources from manifest
[kube] 2019/04/10 20:21:27 creating 108 resource(s)

It seems the resources are created anyway but the release won't appear in helm ls because of the above warning.

Im not a helm expert, but 108 resources are probably quite a bunch of yaml to store into one configmap. Maybe we are hitting https://github.com/helm/helm/issues/1413?
The release configmap contents are compressed with gzip but maybe it's still too big.

[GingerBuffalo]

I think mine is the same repro:
@desolatorxxl Thanks for the point to this thread. Do you or anyone else have an idea on a workaround for the configmap issue?

I also noticed it looks like all resources are deployed, despite helm ls not showing a deployment. Unfortunately it cuts short all the other pieces jx install does to set up the gitops repos for pipelines. It just quits out when it hits this.

[desolatorxxl]

@GingerBuffalo The latest releases of jenkins-x/jenkins-x-platform seem to be broken. I was able to "recover" the latest working state by checking wich version was used the last time:

[root@master01 ~]# ls -lisa ~/.helm/cache/archive/
total 1140
256029   4 drwxr-xr-x. 2 root root   4096 Apr 11 02:01 .
256028   4 drwxr-xr-x. 4 root root   4096 Mar 19 20:31 ..
263744  80 -rw-r--r--. 1 root root  78266 Mar 19 22:07 jenkins-x-platform-0.0.3553.tgz
263740  80 -rw-r--r--. 1 root root  79495 Mar 31 18:49 jenkins-x-platform-0.0.3698.tgz
263861 828 -rw-r--r--. 1 root root 845803 Apr 11 02:04 jenkins-x-platform-0.0.3832.tgz
...

Which according to the date was jenkins-x-platform-0.0.3698. So I hacked the version back into ~/.jx/jenkins-x-versions/charts/jenkins-x/jenkins-x-platform.yml:

gitUrl: https://github.com/jenkins-x/jenkins-x-platform
version: 0.0.3698

Than I ran jx install ... as usual and voila, everything worked again! 🎉

I also tested the latest version v0.0.3847, but that one fails with "Error: getting deployed release" aswell.

[krishnakumar6893]

@desolatorxxl I tried changing the platform value and ran jx install but getting below error.

Error: release jenkins-x failed: secrets "jenkins-x-chartmuseum" already exists'

[desolatorxxl]

@krishnakumar6893 I think you just have to change the version number in ~/.jx/jenkins-x-versions/charts/jenkins-x/jenkins-x-platform.yml to make it work, of course it is only a workaround.

[pmuir]

Fix coming soon.

[GingerBuffalo]

@GingerBuffalo The latest releases of jenkins-x/jenkins-x-platform seem to be broken. I was able to "recover" the latest working state by checking wich version was used the last time:

[root@master01 ~]# ls -lisa ~/.helm/cache/archive/
total 1140
256029   4 drwxr-xr-x. 2 root root   4096 Apr 11 02:01 .
256028   4 drwxr-xr-x. 4 root root   4096 Mar 19 20:31 ..
263744  80 -rw-r--r--. 1 root root  78266 Mar 19 22:07 jenkins-x-platform-0.0.3553.tgz
263740  80 -rw-r--r--. 1 root root  79495 Mar 31 18:49 jenkins-x-platform-0.0.3698.tgz
263861 828 -rw-r--r--. 1 root root 845803 Apr 11 02:04 jenkins-x-platform-0.0.3832.tgz
...

Which according to the date was jenkins-x-platform-0.0.3698. So I hacked the version back into ~/.jx/jenkins-x-versions/charts/jenkins-x/jenkins-x-platform.yml:

gitUrl: https://github.com/jenkins-x/jenkins-x-platform
version: 0.0.3698

Than I ran jx install ... as usual and voila, everything worked again! 🎉

I also tested the latest version v0.0.3847, but that one fails with "Error: getting deployed release" aswell.

I had a feeling a rollback to previous helm chart version could work. I didn't know about this file: ~/.jx/jenkins-x-versions/charts/jenkins-x/jenkins-x-platform.yml though. So I was trying to manually run the helm chart, referencing a previous version. Kept hitting the error message @krishnakumar6893 reports "Error: release jenkins-x failed: secrets "jenkins-x-chartmuseum" already exists'".

I reinstalled jx, and deleted my ~/.jx folder. Then started a jx install, but paused at the last prompt about which buildpack, went and found the file you point to, made your update, and it's all working now.

Thanks for the pointer!

[ccojocar]

@jstrachan We can close also this one when you fix for config map is merged.

[jstrachan]

fix has been released - we moved the pod templates into separate ConfigMaps