Helmfile: Error: Failed to render chart: exit status 1: Error: EOF

[andrewnazarov]

From time to time (usually couple of times per day) I get the following error while doing helmfile apply (actually its terraform apply with helmfile provider):

helm exited with status 1:
   Error: Failed to render chart: exit status 1: Error: EOF

It's quite difficult to diagnose and impossible to replicate as a subsequent run almost always helps to solve the problem. And I'm a bit confused where to start.

[andrewnazarov]

Sometimes a message is a bit different:

helm exited with status 1:
   Error: Failed to render chart: exit status 1: Error: unexpected EOF

A word unexpected is a difference.

[mumoshu]

@andrewnazarov Thanks for reporting!

Could you add --log-level=debug so that it looks like helmfile --log-level=debug apply? Then logs would include the exact helm commands being run, which may help further debugging this.

[mumoshu]

For now, it looks like it's rather helm-diff or helm issues.

[mcabrito]

I've teh same problem.

``` ..Successfully got an update from the "jenkins-x" chart repository
Update Complete. ⎈ Happy Helming!⎈

worker 1/1 started
worker 1/1 finished
worker 1/1 started
Comparing release=istio, chart=istio.io/istio
exec: helm diff upgrade --reset-values --allow-unreleased istio istio.io/istio --version 1.5.2 --namespace istio-system --values /tmp/values544762042 --detailed-exitcode
exec: helm diff upgrade --reset-values --allow-unreleased istio istio.io/istio --version 1.5.2 --namespace istio-system --values /tmp/values544762042 --detailed-exitcode: ******

    Release was not present in Helm.  Diff will show entire contents as new.

---

worker 1/1 finished
err: release "istio" in "helmfile-istio.yml" failed: helm exited with status 1:
Error: Failed to render chart: exit status 1: Error: failed to download "istio.io/istio" (hint: running helm repo update may help)

Error: plugin "diff" exited with error
in ./helmfile-istio.yml: helm exited with status 1:
Error: Failed to render chart: exit status 1: Error: failed to download "istio.io/istio" (hint: running helm repo update may help)

Error: plugin "diff" exited with error
$ helm version
version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}```

[mumoshu]

@mcabrito Thanks for reporting!

Could you please run helm diff upgrade --reset-values --allow-unreleased istio istio.io/istio --version 1.5.2 --namespace istio-system --values somevalues.yaml as logged in order to diagnose the root cause of this?

Sharing the content of the values yaml would also be helpful for debugging. Try running helmfile --log-level=debug apply --retain-values-files to retain rendered/temporary values.yaml files that are used for running helm upgrade.

[andrewnazarov]

We stopped getting this by setting concurrency = 1. The error happened eventually, so it was kinda difficult to catch. Usually, the error was about .tgz archives described in a separate issue.

[mumoshu]

@andrewnazarov Thanks - that's great info. I heard that helm has a race issue that gives you strange errors when you tried to fetch/use the same chart concurrently. Maybe you are also affected by that and that's why disabling the concurrency solves the issue.

I thought we already had both helm and helmfile issues for that. But I couldn't find them right now.

[lebedik]

We stopped getting this by setting concurrency = 1. The error happened eventually, so it was kinda difficult to catch. Usually, the error was about .tgz archives described in a separate issue.

Could you please provide a hint how to achieve this?

[andrewnazarov]

@lebedik
helmfile apply --concurrency 1

[lebedik]

1319