Jx: Jenkins Build fails with the error - failed to read the Helm version: failed to run 'helm version --short' command in directory '.', output: 'Client: v2.10.0+g9ad53a a

[harpresing]

Summary

Jenkins build fails with the message

+ jx step helm release
No $CHART_REPOSITORY defined so using the default value of: http://jenkins-x-chartmuseum:8080
Using helmBinary helm with feature flag: none
error: failed to build dependencies for chart from directory '.': failed to read the Helm version: failed to run 'helm version --short' command in directory '.', output: 'Client: v2.10.0+g9ad53aa
Error: forwarding ports: error upgrading connection: the server could not find the requested resource': exit status 1

Steps to reproduce the behavior

• Create a Kubernetes cluster in GKE
  jx create cluster gke --skip-login --default-admin-password=mySecretPassWord123 -n myclustername

  • Create a Spring boot service- jx create spring -d web -d actuator

Jx version

The output of jx version is:

NAME               VERSION
jx                 1.3.439
jenkins x      platform 0.0.2771
Kubernetes  cluster v1.9.7-gke.6
kubectl         v1.10.7
helm client   v2.11.0+g2e55dbe
helm server  v2.11.0+g2e55dbe
git                git version 2.11.0

Kubernetes cluster

Using a Kubernetes cluster with GKE which was created via the command
jx create cluster gke --skip-login --default-admin-password=password -n myclustername

Operating system / Environment

Google cloud shell

Repository created by jx

https://github.com/etSingh/data_processor

[omermizrahi]

happened to me today too, the only thing that helped me was to delete jx namespace and reinstall jx on the cluster.

[mikaelmh]

Same here on GKE. Happened to me yesterday as well. I re-installed, deleted the cluster and re-installed several times, but still stuck.

After downgrading helm to 2.10, it still fails with a slightly different error;

[app-name] Running shell script
+ jx step helm release
No $CHART_REPOSITORY defined so using the default value of: http://jenkins-x-chartmuseum:8080
Using helmBinary helm with feature flag: none
Adding missing Helm repo: jenkins-x https://chartmuseum.build.cd.jenkins-x.io
Successfully added Helm repository jenkins-x.
Adding missing Helm repo: releases http://jenkins-x-chartmuseum:8080
Successfully added Helm repository releases.
No $CHART_REPOSITORY defined so using the default value of: http://jenkins-x-chartmuseum:8080
Uploading chart file app-name-1.0.136.tgz to http://jenkins-x-chartmuseum:8080/api/charts
Received 401 response: {"error":"unauthorized"}
error: Failed to post chart to http://jenkins-x-chartmuseum:8080/api/charts due to response 401: {"error":"unauthorized"}
script returned exit code 1

(it seems to essentially be the same error as I think it is not able to connect with local chart museum, though I can navigate to the local chart museum URL in the browser).

Anyone able to solve this?

[hgajjar]

Same here, I think its related to https://github.com/helm/helm/issues/4031
Jenkins X installs helm v2.10.0, which has this issue and it looks like its fixed in v2.11.0

How can we upgrade helm version in Jenkins X?

[jonathanbeber]

same with version 2.11

❯ jx version
Using helmBinary helm with feature flag: none
NAME               VERSION
jx                 1.3.445
jenkins x platform 0.0.2771
Kubernetes cluster v1.10.7-gke.6
kubectl            v1.12.1
helm client        v2.11.0+g2e55dbe
helm server        v2.11.0+g2e55dbe
git                git version 2.14.1

[bryanlarsen]

The upgrade to 2.11 works for me when I manually run the step.

1. download helm 2.11
2. helm init --upgrade
3. cd charts/jxtest
4. CHART_REPOSITORY=http://chartmuseum.jx.example.com jx step helm release

The problem is that the helm binary in the nodejs container on my jenkins is still 2.10, even though my cluster has the 2.11 tiller. How do I upgrade my container's helm client?

[VinodEgallapati]

I am getting the same issue, though I upgrade helm to 2.11.0 in the container jenkinsxio/builder-go

I have done the below steps,

Downloaded jenkinsxio/builder-go image.
Downloaded helm 2.11.0 and performed the below steps,

wget https://storage.googleapis.com/kubernetes-helm/helm-v2.11.0-linux-amd64.tar.gz | tar xzv && \
mv linux-amd64/helm /usr/bin/ && \
mv linux-amd64/tiller /usr/bin/ && \
rm -rf linux-amd64

helm init --client-only

And committed and tagged the docker image to use it for pipeline job

• jx step helm release
  Using helmBinary helm with feature flag: none
  error: failed to build dependencies for chart from directory '.': failed to read the Helm version: failed to run 'helm version --short' command in directory '.', output: 'Client: v2.11.0+g2e55dbe
  Error: forwarding ports: error upgrading connection: the server could not find the requested resource': exit status 1

Can you please help me with the fix.

[luca010]

I had the same error. For some reason, I didn't have the global env TILLER_NAMESPACE configured. I setted it to kube-system and now everything works fine.

[Marthijs-Berfelo]

@luca010 : thanks for the hint. I solved the same error in our environment the same way!

[VinodEgallapati]

Thank you for the hint, I see some other error now,

• jx step helm release
  No $CHART_REPOSITORY defined so using the default value of: http://jenkins-x-chartmuseum:8080
  Using helmBinary helm with feature flag: none
  error: failed to build dependencies for chart from directory '.': failed to add chart repositories: failed to retrieve the install charts: failed to list repositories: failed to run 'helm repo list' command in directory '.', output: 'Error: Couldn't load repositories file (/home/jenkins/.helm/repository/repositories.yaml).
  You might need to run helm init (or helm init --client-only if tiller is already installed)': exit status 1

Using image jenkinsxio/builder-go , version: 0.1.33.

Is it due to helm version incompatibility between client and server ?

[wangxiang5050]

@VinodEgallapati I have the same error. Have you resolved it?

[ysaakpr]

I had the same error. For some reason, I didn't have the global env TILLER_NAMESPACE configured. I setted it to kube-system and now everything works fine.

[M1123581E]

jenkins configured global env TILLER_NAMESPACE but it doesn't work. so we need to configure it manually. is it a bug?

[sboardwell]

I had the same problem. Adding TILLER_NAMESPACE to the Jenkinsfile environment block solved the problem.

However, when I checked, the global variable was also empty. I've checked the helm chart and the TILLER_NAMESPACE is set everywhere. Really does seem like a bug. I will keep an eye on the global variable when updating. Has everyone else checked the global variable value?

[ghost]

For some reason TILLER_NAMESPACE was defined in our global Jenkins (X) config but it had no value; removing it solved this issue for us.

[polothy]

We are experiencing the same problem. Setting TILLER_NAMESPACE in our Jenkinsfile fixes it:

  environment {
    TILLER_NAMESPACE = "kube-system"
  }

Found two references to TILLER_NAMESPACE in the jenkins-x-platform project:

https://github.com/jenkins-x/jenkins-x-platform/blob/e2819eea110d5936a489a00969667ba5b044db5a/values.yaml#L807

https://github.com/jenkins-x/jenkins-x-platform/blob/e2819eea110d5936a489a00969667ba5b044db5a/templates/jenkins-x-pod-templates.yaml#L78-L79

My gut says that's where it is coming from, but not 100% sure. The odd thing is that it looks like Helm should be ignoring TILLER_NAMESPACE if it is empty:

https://github.com/helm/helm/blob/de5a6a931bfc47c1fa3a72db038002cbf68315ff/cmd/tiller/tiller.go#L238-L240

Anyways, it certainly is causing a problem for several people and for us.

EDIT: here is the commit in JX platform that introduced it: https://github.com/jenkins-x/jenkins-x-platform/commit/5ceae96e41d1c2a7d1f9583e1ee68af717ed0718

EDIT2: And the PR for the commit: https://github.com/jenkins-x/jenkins-x-platform/pull/4052

[polothy]

Just confirmed that if the TILLER_NAMESPACE env var is unset, then it also works. I _think_ Jenkins protects env vars between step commands, so I had to do it in a single command, like so:

            sh 'unset TILLER_NAMESPACE && jx step helm build'
            sh 'unset TILLER_NAMESPACE && jx step helm apply'

[polothy]

Proposed a fix in https://github.com/jenkins-x/jenkins-x-platform/pull/4864

[sboardwell]

As a side note, perhaps having either

1. an installation of the jenkins chart prior to this commit
2. an older custom config map from a time prior to the commit

may be the root cause of the global variable TILLER_NAMESPACE being empty in Jenkins itself.

[LeoQuote]

I got Error: Couldn't load repositories file (/home/jenkins/.helm/repository/repositories.yaml

+ jx step helm release
No $CHART_REPOSITORY defined so using the default value of: http://jenkins-x-chartmuseum:8080

error: failed to build dependencies for chart from directory '.': failed to add chart repositories: failed to retrieve the install charts: failed to list repositories: failed to run 'helm repo list' command in directory '.', output: 'Error: Couldn't load repositories file (/home/jenkins/.helm/repository/repositories.yaml).
You might need to run `helm init` (or `helm init --client-only` if tiller is already installed)'

jx version

jx                 1.3.1048
jenkins x platform 0.0.3657
Kubernetes cluster v1.13.3
kubectl            v1.13.3
helm client        v2.13.1+g618447c
helm server        v2.13.1+g618447c
git                git version 1.8.3.1
Operating System   CentOS Linux release 7.2.1511 (Core)