Terraform-provider-helm: error: failed to download <chart> (hint:running helm repo update may help)

[sean-ersw]

⚠️ Note from maintainers about this issue, please read

This issue is attracting a lot of thumbs because it is not one specific issue. This error message comes from the helm package and is a catch-all error that is displayed when the chart fails to download for any reason. The real error is suppressed and debugging has to be enabled to surface it.

Please add HELM_DEBUG=1 to your terraform apply and run again to see if the error message is more specific. If the error message shows something different please open a new issue with the output.

Original Issue

Terraform Version and Provider Version

Terraform v0.12.26

Provider Version

• provider.google v3.27.0
• provider.google-beta v3.27.0
• provider.helm v1.2.3
• provider.random v2.2.1

Affected Resource(s)

• helm_release

Terraform Configuration Files

Provider in root module

provider "helm" {
  kubernetes {
    config_context = "name-of-my-context"
  }
}

Basics of the module

resource "helm_release" "gitlab-runner" {
  name                    = "gitlab-runner"
  repository            = "https://charts.gitlab.io"
  chart                    = "gitlab-runner"
  version                 = "0.18.0"
  namespace          = "gitlab-runner"
  create_namespace   = true
...
}

Execution of the root module

module "gitlab_runner" {
  source = "../modules/helm_gitlab_runner"
  tags = "test, test2"
  registration_token = "removed"
}

Expected Behavior

Chart should be successfully downloaded and the plan applied

Actual Behavior

helm_release.gitlab-runner: Creating...

Error: failed to download "https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.18.0.tgz" (hint: running `helm repo update` may help)

  on main.tf line 1, in resource "helm_release" "gitlab-runner":
   1: resource "helm_release" "gitlab-runner" {

Steps to Reproduce

1. terraform init
2. terraform plan
3. terraform apply -auto-approve

Important Factoids

The target cluster is a GKE instance and the active kubectl ctx is correct and functions fine.

I have tried all kinds of variations here such as setting `chart = "https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.18.0.tgz" , removing version, older versions etc etc

I have done all kinds of resets of helm cache, remove and add repos, update etc etc

Have also used the terraform docker image to ensure that its not an environmental problem of some sort

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[S0meth1ng85]

It is strange. But today I have the same problem but with this repo: repository = "https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/"

Two days ago my code worked as expected. And today I am testing AKS K8S with RBAC enabled and all is ok except this helm_release...

Don` know how to resolve it

Terraform v0.12.28

• provider.azurerm v2.18.0
• provider.helm v1.2.3
• provider.kubernetes v1.11.3

[comerford]

I was having this exact problem with an agones install, and was able to reproduce it while attempting a direct install too, same failure text. So I added a debug to the manual install:

helm --debug install agones https://agones.dev/chart/stable/agones-1.7.0.tgz -n agones-system

That told me that the issue was that there was a missing file:

Error: no cached repo found. (try 'helm repo update'): open ~/.cache/helm/repository/local-index.yaml: no such file or directory

I took a guess that this was actually the file in ~/.helm/repository/local/index.yaml that probably got missed in a helm 2to3 migration. So, I copied it over:

cp ~/.helm/repository/local/index.yaml ~/.cache/helm/repository/local-index.yamll

Now when I run my terraform apply, no errors (at least because of this)

[ikethecoder]

I had a similar issue when I add verify = true included in my helm_release. When I set verify to false the error went away. I'd consider this a workaround as I don't know why verification of the package fails.

[matutetandil]

Same here!!! I have two resources, both failing.

resource "helm_release" "nginx-ingress" {
  name  = "nginx-ingress-lb"
  chart = "stable/nginx-ingress"

  set {
    name  = "controller.publishService.enabled"
    value = "true"
  }
}

and

resource "helm_release" "cert-manager" {
  name       = "cert-manager"
  repository = "https://charts.jetstack.io" 
  chart      = "jetstack/cert-manager"
  version    = "v0.15.2"
  namespace  = "cert-manager"
}

Both giving me the same error:

Error: failed to download "<chart_name>" (hint: running `helm repo update` may help)

My Terraform --version output:

$ terraform --version
Terraform v0.12.28
+ provider.digitalocean v1.21.0
+ provider.helm v1.2.3
+ provider.kubernetes v1.11.4
+ provider.local v1.4.0
+ provider.null v2.1.2

[manoj8928]

Similar issue while deploying cluster autoscaler using helm

$ terraform apply -auto-approve -input=false $PLAN module.eks_helm_charts.helm_release.metric-server: Creating... module.eks_helm_charts.helm_release.cluster_autoscaler: Creating... module.eks_helm_charts.helm_release.metric-server: Still creating... [10s elapsed] module.eks_helm_charts.helm_release.metric-server: Creation complete after 10s [id=metric-server-release] Error: failed to download "stable/cluster-autoscaler" (hint: runninghelm repo updatemay help) on eks-helm-charts/helm.tf line 14, in resource "helm_release" "cluster_autoscaler": 14: resource "helm_release" "cluster_autoscaler" {

I am using below version
` terraform -v

• Terraform v0.12.28
• provider.aws v2.70.0
• provider.helm v1.2.4
• provider.kubernetes v1.12.0
• provider.local v1.4.0
• provider.null v2.1.2
• provider.template v2.1.2
  Your version of Terraform is out of date! The latest version
  is 0.12.29. You can update by downloading from https://www.terraform.io/downloads.html`

[llnformer]

same thing with terraform0.13
with the following resource definition:

resource "helm_release" "aws_node_termination_handler" {
   name       = "aws-node-termination-handler"
   chart      = "eks/aws-node-termination-handler"
   repository = "https://aws.github.io/eks-charts"
   namespace  = "kube-system"
   version    = "0.9.2"
   verify     = false
}

```terraform --version
Terraform v0.13.0-beta3

• provider example.com/tonkean/kubectl v1.5.1
• provider registry.terraform.io/hashicorp/aws v3.0.0
• provider registry.terraform.io/hashicorp/helm v1.2.3
• provider registry.terraform.io/hashicorp/kubernetes v1.12.0
• provider registry.terraform.io/hashicorp/local v1.4.0
• provider registry.terraform.io/hashicorp/null v2.1.2
• provider registry.terraform.io/hashicorp/random v2.3.0
• provider registry.terraform.io/hashicorp/template v2.1.2```

[bugok]

I'm also encountering this, with a private repo I have in jfrog.
I'm getting a similar error:

Error: failed to download "local://mychart-0.1.0.tgz" (hint: running `helm repo update` may help)

The local:// part is what shows up in the repos index.yaml file, which works for other project, so I don't think that's the source of the problem, but I may be wrong.

I'm using terraform 0.12.20 and helm provider 1.2.4.

[bugok]

Running the same terraform stack with HELM_DEBUG=1 environment variable gave a different error:

Error: non-absolute URLs should be in form of repo_name/path_to_chart, got: local://mychart-0.1.0.tgz

Not sure what this means though...

[bugok]

I have figured this out, at least from my end. For future readers:

I was using a local jfrog helm-chart repository. The problem with using such a repo directly is that the index.yaml file is being created with references of local://mychart-1.0.1.tgz to the chart. Whenever the helm provider uses that repo, it can't parse the local:// part, and fails.

The solution for me was to create a virtual repository (also using jfrog) which is fronting the local repository. The virtual repository's index.yaml file is configured well with the full path to the chart in the local repository.

So, if you're encountering a local:// related error and you're using a local jfrog repo, what worked for me was to define a virtual repo which is fronting the local one, and configure the helm_release resource to use the virtual repository.

[jrhouston]

Hi @sean-ersw thanks for opening this. Can you share all of the attributes set on the helm_release resource? I am not able to reproduce this just with what's in the issue.

However, if I add a verify = true then I do get the error:

Error: failed to download "https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.18.0.tgz" (hint: running `helm repo update` may help)

  on main.tf line 3, in resource "helm_release" "gitlab-runner":
   3: resource "helm_release" "gitlab-runner" {

If I add HELM_DEBUG=1 then helm tell us this:

Error: failed to fetch provenance "https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.18.0.tgz.prov"

  on main.tf line 3, in resource "helm_release" "gitlab-runner":
   3: resource "helm_release" "gitlab-runner" {

I get the same error if I try to use this repository and install this chart directly with helm and set --verify to true.

HELM_DEBUG=1 helm install gitlab-runner --verify https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.18.0.tgz                                                                         
install.go:159: [debug] Original chart version: ""
Error: failed to fetch provenance "https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.18.0.tgz.prov"
# ... trace

This somewhat points to a problem with the repository, as these errors are coming from the helm package not the provider itself.

[jrhouston]

@matutetandil

resource "helm_release" "cert-manager" {
  name       = "cert-manager"
  repository = "https://charts.jetstack.io" 
  chart      = "jetstack/cert-manager"
  version    = "v0.15.2"
  namespace  = "cert-manager"
}

This doesn't work because you have specified jetstack/cert-manager as the chart name. It should just be cert-manager as you have set the repository attribute to use the jetstack repository.

The part before the / in the chart name refers to a repository you have added with helm repo add <name> <url> which isn't the case here.

[jrhouston]

@llnformer As above, if you change:

chart = "eks/aws-node-termination-handler" to chart = "aws-node-termination-handler"

The release should work.

[sean-ersw]

@jrhouston here is the full version with some stuff redacted

resource "helm_release" "gitlab-runner" {
  name                    = "gitlab-runner"
  repository              = "https://charts.gitlab.io"
  chart                   = "gitlab-runner"
  namespace               = "gitlab"
  create_namespace        = true
  timeout                 = 600
  reuse_values            = true
  recreate_pods           = true
  cleanup_on_fail         = true
  wait                    = true
  verify                  = true
  max_history             = 10

  set {
    name  = "imagePullPolicy"
    value = "always"
  }

  set {
    name  = "gitlabUrl"
    value = var.gitlab_url
  }

  set {
    name  = "runners.tags"
    value = var.tags
  }

  set {
    name  = "runnerRegistrationToken"
    value = var.registration_token
  }

  set {
    name  = "concurrent"
    value = var.max_concurrent_jobs
  }

  set {
    name  = "unregisterRunners"
    value = true
  }

  set {
    name  = "terminationGracePeriodSeconds"
    value = 3600
  }

  set {
    name  = "checkInterval"
    value = 10
  }

  set {
    name  = "rbac.enabled"
    value = true
  }

  set {
    name  = "rbac.create"
    value = true
  }

  set {
    name  = "rbac.clusterWideAccess"
    value = false
  }

  set {
    name  = "metrics.enabled"
    value = true
  }

  set {
    name  = "runners.image"
    value = "alpine:latest"
  }

  set {
    name  = "runners.locked"
    value = false
  }

  set {
    name  = "runners.runUntagged"
    value = true
  }

  set {
    name  = "runners.protected"
    value = false
  }

  set {
    name  = "runners.privileged"
    value = true
  }

  set {
    name  = "securityContext.fsGroup"
    value = 65533
  }

  set {
    name  = "securityContext.runAsUser"
    value = 100
  }

}

[jrhouston]

Thanks @sean-ersw. I see that verify is indeed set to true for the helm_release. When I set it to false the error goes away.

The verify flag should be used when you want to verify the integrity of a chart before you install it. This seems like an obvious thing to set to true at first glance, however the repository also needs to host a provenance file alongside the chart called $chartname.tgz.prov that is used to do the verification. The charts.gitlab.io repository doesn't seem to do this, so the download fails at the verification step.

There's a couple of things here:

1. The error message is totally confusing. We have to add HELM_DEBUG=1 to actually find out what went wrong. This error message comes from the helm ChartDownloader and you get the same error message if you use helm install --verify at the CLI.

   This is an issue we can open on the upstream repository to get a more actionable message, as the suggestion to run helm repo update in the non-debug error message is useless in this case. The problem is that the install action swallows the error from the ChartDownloader unless debug is switched on, returning a generic failed to download error instead. See here.

2. It is not obvious what the verify flag does. The provider documentation just says that it verifies the chart - which seems like a pretty sensible thing to switch on. We should update this documentation to explain further what this flag does and link through to the helm documentation on provenance.

References

https://helm.sh/docs/topics/provenance/

[steebchen]

I'm running into similar issues:

$ export HELM_DEBUG=1
$ tf apply
[...]
helm_release.cassandra: Creating...

Error: chart "bitnami/cassandra" not found in https://charts.bitnami.com/bitnami repository

  on k8s-cassandra.tf line 5, in resource "helm_release" "cassandra":
   5: resource "helm_release" "cassandra" {



Process finished with exit code 1

with tf

resource "helm_release" "cassandra" {
  name       = "cassandra"
  repository = "https://charts.bitnami.com/bitnami"
  chart      = "bitnami/cassandra"
}

I can't get it working regardless of setting verify = false, trying different chart names (chart key) such as cassandra instead of bitnami/cassandra. It all results in the same error. Manually deploying via helm install cassandra bitnami/cassandra works fine.

---

Somehow I got it working one time, then destroyed the helm deployment, and when I tried re-creating it with tf apply the same error occurred. It's super weird.

I also have another helm deployment where I had the same issue. Somehow I got it working, but when increasing the version, the same error may appear.

[jrhouston]

Hi @steebchen changing your configuration to this worked for me:

resource "helm_release" "cassandra" {
  name       = "cassandra"
  repository = "https://charts.bitnami.com/bitnami"
  chart      = "cassandra"
}

To make this more obvious, we are going to add some validation to the schema so that using the / in the chart name when the repository attribute is set becomes invalid.

[steebchen]

Thanks, but I tried that as well and it didn't change anything. I still get the same error.

[vakkur]

It worked when I used terraform 0.13 but when switching back to terraform 0.12.28 it is failing

resource "helm_release" "cluster_autoscaler" {
name = "autoscaler"
repository = "https://kubernetes.github.io/autoscaler"
chart = "cluster-autoscaler-chart"
namespace = "cluster-autoscaler"

set {
name = "autoDiscovery.clusterName"
value = var.cluster_name
}

set {
name = "rbac.create"
value = true
}

set {
name = "rbac.serviceAccountAnnotations.eks.amazonaws.com/role-arn"
value = aws_iam_role.argocd_nodegroup_role.arn
}

depends_on = [aws_eks_cluster.argocd_cluster, aws_eks_node_group.argocd_nodegroup, kubernetes_namespace.cluster_autoscaler]

}

[vakkur]

Looks like the issue is resolved I was able to deploy using helm, i did not make any changes

This is my provider configuration,

provider "helm" {
version = "1.0.0"
kubernetes {
load_config_file = true
}
}
Helm Version
version.BuildInfo{Version:"v3.2.3", GitCommit:"8f832046e258e2cb800894579b1b3b50c2d83492", GitTreeState:"clean", GoVersion:"go1.13.12"}

[mnoseworthy]

Adding a helm_repository to feed into the release resolved the issue for me... I know helm_repository is being deprecated but, I just need this working!

data "helm_repository" "incubator" {
  name = "incubator"
  url  = "https://kubernetes-charts-incubator.storage.googleapis.com"
}

resource "helm_release" "my_cache" {
  name       = "my-cache"
  repository = data.helm_repository.incubator.metadata[0].name
  chart      = "redis-cache"
}

[aareet]

Hi folks, I'm pinning this issue for visibility. This issue is attracting a lot of thumbs because it is not one specific issue. This error message comes from the helm package and is a catch-all error that is displayed when the chart fails to download for any reason. The real error is suppressed and debugging has to be enabled to surface it.

Please add HELM_DEBUG=1 to your terraform apply and run again to see if the error message is more specific. If the error message shows something different please open a new issue with the output.

I'm going to close this issue now to avoid further confusion.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!