Terraform-provider-helm: Plugin crushes after upgrading to 1.3.1

[gkocur]

Terraform Version and Provider Version

v0.13.3

Provider Version

v1.3.1

Terraform Configuration Files

provider "helm" {
  kubernetes {
    load_config_file = false
    token = data.google_client_config.current.access_token
    host = module.k8s.cluster_endpoint
    client_certificate     = module.k8s.client_certificate
    client_key             = module.k8s.cluster_endpoint
    cluster_ca_certificate = module.k8s.cluster_ca_certificate
  }
  version = "1.3.1"
}

Expected Behavior

terraform plan and terraform apply should work as expected

Actual Behavior

Plugin crushes with panic: runtime error: invalid memory address or nil pointer dereference

Steps to Reproduce

1. terraform plan

Important Factoids

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

As in the title - after upgrading to v1.3.1 running plan or apply makes the plugin crush.
Works fine with v1.3.0

[sennerholm]

Hi!
When I tried to upgrade the provider in my setup with already created resources it seems to work. Are you starting with a clean (empty state?)
Sincerely
Mikael

[jrhouston]

Hi @gkocur can you add a gist with the full debug log of your run?

[evgmoskalenko]

Hi,

I faced the same problem when my terraform CI automatically updated the terraform provider version. But on my agent installed helm version: 3.2.4. In the Changelog release helm should be 3.3.4. I will try to update on my CI/CD agent.

Error: rpc error: code = Unavailable desc = transport is closing

Error: rpc error: code = Canceled desc = context canceled

panic: runtime error: invalid memory address or nil pointer dereference
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: [signal SIGSEGV: segmentation violation code=0x1 addr=0xa8 pc=0x17c6522]
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: 
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: goroutine 39 [running]:
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: k8s.io/cli-runtime/pkg/genericclioptions.(*ConfigFlags).ToRawKubeConfigLoader(0x0, 0x5fcb23d81f2c9ebf, 0x1)
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/k8s.io/cli-runtime/pkg/genericclioptions/config_flags.go:136 +0x22
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: helm.sh/helm/v3/pkg/cli.(*EnvSettings).Namespace(0xc0005c2aa0, 0xc000445bf0, 0x20b3ae7)
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/cli/environment.go:131 +0x2f
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: helm.sh/helm/v3/pkg/cli.(*EnvSettings).EnvVars(0xc0005c2aa0, 0x160)
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/cli/environment.go:116 +0x553
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: helm.sh/helm/v3/pkg/plugin.SetupPluginEnv(0xc0005c2aa0, 0xc000b8992e, 0x2, 0xc00014e400, 0x30)
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/plugin/plugin.go:260 +0x40
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: helm.sh/helm/v3/pkg/getter.(*pluginGetter).Get(0xc000a5a6c0, 0xc000612340, 0x40, 0xc000b14ff0, 0x4, 0x6, 0xc000a5a6c0, 0x0, 0x0)
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/getter/plugingetter.go:73 +0x290
2020-10-01T12:52:25.246Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: helm.sh/helm/v3/pkg/downloader.(*ChartDownloader).DownloadTo(0xc0006cf398, 0xc00014e200, 0x31, 0xc0003dcfa0, 0x5, 0xc0005cd200, 0x21, 0x1, 0x0, 0x1d036c0, ...)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/downloader/chart_downloader.go:97 +0x13b
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: helm.sh/helm/v3/pkg/action.(*ChartPathOptions).LocateChart(0xc000154ea0, 0xc00014e200, 0x31, 0xc0005c2aa0, 0x0, 0x0, 0xc0003dcfa0, 0x0)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/helm.sh/helm/v3/pkg/action/install.go:669 +0x4bb
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: github.com/hashicorp/terraform-provider-helm/helm.getChart(0x24a59e0, 0xc000af9880, 0xc000445620, 0xc00014e200, 0x31, 0xc000154ea0, 0x0, 0x0, 0x0, 0x0, ...)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/helm/resource_release.go:799 +0xdc
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: github.com/hashicorp/terraform-provider-helm/helm.resourceDiff(0xc000af9880, 0x1e887a0, 0xc000445620, 0xc000b11b00, 0xc000af9880)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/helm/resource_release.go:663 +0x25c
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: github.com/hashicorp/terraform-plugin-sdk/helper/schema.schemaMap.Diff(0xc000282a50, 0xc000793cc0, 0xc000a56b10, 0x2187160, 0x1e887a0, 0xc000445620, 0x0, 0x24f9940, 0xffffffffffffffff, 0x0)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/schema.go:518 +0xac2
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).simpleDiff(0xc000154900, 0xc000793cc0, 0xc000a56b10, 0x1e887a0, 0xc000445620, 0xc000a56b01, 0x0, 0x0)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/resource.go:356 +0x85
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).SimpleDiff(0xc00017ea80, 0xc0007d9a28, 0xc000793cc0, 0xc000a56b10, 0xc0006493e0, 0xc000a56b10, 0x0)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/provider.go:321 +0x99
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).PlanResourceChange(0xc0000d4890, 0x24f8340, 0xc000649380, 0xc000775420, 0xc0000d4890, 0x0, 0xc0007c8b30)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin/grpc_provider.go:633 +0x793
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_PlanResourceChange_Handler(0x1ffb4e0, 0xc0000d4890, 0x24f8340, 0xc000649380, 0xc000790240, 0x0, 0x24f8340, 0xc000649380, 0xc000a06000, 0x7fb1)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5/tfplugin5.pb.go:3287 +0x217
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0004df080, 0x2520fa0, 0xc0001a6180, 0xc0002dcc00, 0xc000173e00, 0x35809a8, 0x0, 0x0, 0x0)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:1024 +0x4f4
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: google.golang.org/grpc.(*Server).handleStream(0xc0004df080, 0x2520fa0, 0xc0001a6180, 0xc0002dcc00, 0x0)
2020-10-01T12:52:25.247Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:1313 +0xd97
2020-10-01T12:52:25.248Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0004ae750, 0xc0004df080, 0x2520fa0, 0xc0001a6180, 0xc0002dcc00)
2020-10-01T12:52:25.248Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:722 +0xbb
2020-10-01T12:52:25.248Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-10-01T12:52:25.248Z [DEBUG] plugin.terraform-provider-helm_v1.3.1_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-helm/vendor/google.golang.org/grpc/server.go:720 +0xa1
2020/10/01 12:52:25 [ERROR] module.prometheus: eval: *terraform.EvalDiff, err: rpc error: code = Unavailable desc = transport is closing
2020/10/01 12:52:25 [ERROR] module.prometheus: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/10/01 12:52:25 [TRACE] [walkPlan] Exiting eval tree: module.prometheus.helm_release.prometheus[0]
2020/10/01 12:52:25 [TRACE] vertex "module.prometheus.helm_release.prometheus[0]": visit complete
2020/10/01 12:52:25 [TRACE] vertex "module.prometheus.helm_release.prometheus": dynamic subgraph encountered errors
2020/10/01 12:52:25 [TRACE] vertex "module.prometheus.helm_release.prometheus": visit complete
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "module.prometheus.output.status_prometheus" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "helm_release.alb_ingress_controller" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "output.alb_ingress_controller_status" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "helm_release.k8s_spot_termination_handler" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "output.k8s_spot_termination_handler_status" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "helm_release.external_dns" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "output.external_dns_status" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "output.prometheus_operator_status" errored, so skipping
2020-10-01T12:52:25.255Z [DEBUG] plugin: plugin process exited: path=/godata/pipelines/***/***/***/.terraform/plugins/linux_amd64/terraform-provider-helm_v1.3.1_x4 pid=86063 error="exit status 2"
2020/10/01 12:52:25 [TRACE] Re-validating config for "module.argocd.helm_release.argocd[0]"
2020/10/01 12:52:25 [TRACE] GRPCProvider: ValidateResourceTypeConfig
2020/10/01 12:52:25 [ERROR] module.argocd: eval: *terraform.EvalDiff, err: rpc error: code = Canceled desc = context canceled
2020/10/01 12:52:25 [ERROR] module.argocd: eval: *terraform.EvalSequence, err: rpc error: code = Canceled desc = context canceled
2020/10/01 12:52:25 [TRACE] [walkPlan] Exiting eval tree: module.argocd.helm_release.argocd[0]
2020/10/01 12:52:25 [TRACE] vertex "module.argocd.helm_release.argocd[0]": visit complete
2020/10/01 12:52:25 [TRACE] vertex "module.argocd.helm_release.argocd": dynamic subgraph encountered errors
2020/10/01 12:52:25 [TRACE] vertex "module.argocd.helm_release.argocd": visit complete
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "module.argocd.output.status_argocd" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "output.argocd_status" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "provider.helm (close)" errored, so skipping
2020/10/01 12:52:25 [TRACE] dag/walk: upstream of "root" errored, so skipping
2020/10/01 12:52:25 [INFO] backend/local: plan operation completed
2020-10-01T12:52:25.364Z [DEBUG] plugin: plugin exited



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

SECURITY WARNING: the "crash.log" file that was created may contain 
sensitive information that must be redacted before it is safe to share 
on the issue tracker.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

[aareet]

Thanks @evgmoskalenko, @gkocur do you have the same issue after upgrading your helm version?

[jrhouston]

@evgmoskalenko @gkocur Are either of you using a helm plugin for your chart repository? I've managed to reproduce this crash with the google cloud storage plugin.

[evgmoskalenko]

I use only these plugins:

• https://github.com/hypnoglow/helm-s3.git
• https://github.com/totango/helm-ssm.git
• https://github.com/databus23/helm-diff

[gkocur]

Sorry for late reply.
Yes, I use gcs plugin with helm.
Here is gist with full crush.log: https://gist.github.com/gkocur/7d9b2e982ec068b73b171c03ada7b3c8

[jrhouston]

OK I figured out what's going on here.

When we upgraded to the latest helm package we brought in this change. This code relies upon the private config field of EnvSettings being set. When we initialize the provider we just initialize an EnvSettings struct by hand instead of using the New() function in the helm/cli package, which is what sets the config field. This code only gets hit when helm has to setup the environment for a plugin, because it calls EnvVars() on the EnvSettings struct, which in turn calls the above code.

In the commit I linked above you can see initializations of the EnvSettings struct were changed to use the New() function so we we should do the same. PR incoming. 🚀

[bergbrains]

@evgmoskalenko @gkocur Are either of you using a helm plugin for your chart repository? I've managed to reproduce this crash with the google cloud storage plugin.

I hit this over the weekend, and I'm using the S3 Helm plugin.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!