helmfile -f ... apply throws error

[kody-abe]

When running helmfile apply I get the following after the diff:

Error: identified at least one change, exiting with non-zero exit code (detailed-exitcode parameter enabled)
identified at least one change, exiting with non-zero exit code (detailed-exitcode parameter enabled)
Error: plugin "diff" exited with error

err: exit status 1

[mumoshu]

@kody-abe Hey! Do you mean that (1) helmfile apply emits an extraneous error message like err: exit status 1 before prompting you for sync after the diff or (2) helmfile apply exits immediately after diff?

[kody-abe]

@mumoshu It exits immediately with that error after the diff. Never get to the part where it asks for sync. The diff command on its own worked fine.

[mumoshu]

@kody-abe Makes sense. That's probably due to that your helm doesn't include https://github.com/helm/helm/pull/4367. Would you mind upgrading your helm to v2.10 and try again?

[davidovich]

I also have this problem, but it is not as easy as upgrading helm, as we have live clusters with a 2.9.x tiller.

This is compounded by the fact that helm diff plugin embeds helm, which is shabby: consider:

helm v. | helm diff v. | cluster v. | works
---------|-------|----------|--------
2.9.x | 2.10.x | 2.9.x | no
2.10.x| 2.10.x | 2.9.x | no
2.10.x|2.10.x|2.10.x| yes

Then helm 2.9.x drives a plugin which embeds helm at a different version 2.10, which wants to diff a cluster with another 2.9.

The only compatible guarantee is to have all the same versions. Anyway, as I write this, it is the tight client-server version coupling of helm-tiller which is at cause. But having separate installations brings confusion.

[mumoshu]

I believe that I understand the pain about that, but am unsure how we could reliably fix it.

Is it possible for you to use helm-tiller so that you can run helmfile connected to a local tiller, that connects to your k8s cluster? This allows you to run a newer version of tiller locally, while running an older one in your cluster.
But not sure if tiller 2.9.x in the cluster is able to read/write tiller-related resource objects written by local tiller 2.10? Does tiller have a forward-compatibility?

[kody-abe]

@mumoshu Just like @davidovich We are running 2.9 on our cluster. So, we are still blocked.

[gianrubio]

The flag --detailed-exitcode shouldn't be passed to helm diff plugin when the chart is not installed yet.

On the diff plugin command we have
--detailed-exitcode return a non-zero exit code when there are changes

That means the helm diff command will always fail during an installation because there are changes involved.

You will be able to reproduce the error running these commands in a fresh cluster (no helm chart installed).

helmfile.yaml

repositories:
  - name: stable
    url: https://kubernetes-charts.storage.googleapis.com

releases:
- name: metricbaet
  chart: stable/metricbeat

helmfile diff command success

➜  helmfile diff
Comparing metricbaet stable/metricbeat
********************

        Release was not present in Helm.  Diff will show entire contents as new.

********************
default, metricbaet-metricbeat-daemonset-modules, Secret (v1) has been added:
....
➜  echo $?
0

helmfile diff command error

➜ helmfile diff --detailed-exitcode
Error: identified at least one change, exiting with non-zero exit code (detailed-exitcode parameter enabled)
identified at least one change, exiting with non-zero exit code (detailed-exitcode parameter enabled)
Error: plugin "diff" exited with error

err: exit status 2
exit status 2

➜  helmfile apply
Adding repo stable https://kubernetes-charts.storage.googleapis.com
"stable" has been added to your repositories

Updating repo
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
...Successfully got an update from the "chartmuseum" chart repository
...Successfully got an update from the "jenkins-x" chart repository
Update Complete. ⎈ Happy Helming!⎈

Comparing metricbaet stable/metricbeat
********************

        Release was not present in Helm.  Diff will show entire contents as new.

********************
default, metricbaet-metricbeat-daemonset-modules, Secret (v1) has been added:
-
+ # Source: metricbeat/templates/secret.yaml
+ apiVersion: v1
....

Error: identified at least one change, exiting with non-zero exit code (detailed-exitcode parameter enabled)
identified at least one change, exiting with non-zero exit code (detailed-exitcode parameter enabled)
Error: plugin "diff" exited with error

Upgrading stable/metricbeat
Error: failed to download "stable/metricbeat" (hint: running `helm repo update` may help)

err: release "metricbaet" in "helmfile.yaml" failed: exit status 1
exit status 1

➜  bla helm version
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
➜  helmfile --version
helmfile version v0.40.1
➜  helm plugin list
NAME    VERSION         DESCRIPTION
diff    2.11.0+1        Preview helm upgrade changes as a diff
tiller  0.5.4           Start a Tiller server locally

[yurrriq]

Is there a fix for this? I'm experiencing the same behaviour @gianrubio described.

λ helm version
Client: &version.Version{SemVer:"v2.11.0+unreleased", GitCommit:"", GitTreeState:""}
Server: &version.Version{SemVer:"v2.10.0", GitCommit:"9ad53aac42165a5fadc6c87be0dea6b115f93090", GitTreeState:"clean"}

λ helmfile --version
helmfile version 0.40.1

λ helm diff version
2.11.0+2

Edit: I actually just upgraded helm-diff and now it's mad about my client/server Helm version mismatch.

Edit: I've upgraded the server and still can't use helmfile.

λ helm version
Client: &version.Version{SemVer:"v2.11.0+unreleased", GitCommit:"", GitTreeState:""}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

[rrajadevops]

@gianrubio I am also getting same error. Please find below my configurations.

• helm version
  Client: &version.Version{SemVer:"v2.12.1", GitCommit:"02a47c7249b1fc6d8fd3b94e6b4babf9d818144e", GitTreeState:"clean"}
  Server: &version.Version{SemVer:"v2.12.1", GitCommit:"02a47c7249b1fc6d8fd3b94e6b4babf9d818144e", GitTreeState:"clean"}
• helmfile --version
  helmfile version v0.41.0

Error: identified at least one change, exiting with non-zero exit code (detailed-exitcode parameter enabled)
identified at least one change, exiting with non-zero exit code (detailed-exitcode parameter enabled)
Error: plugin "diff" exited with error

[boisde]

same here.

λ helm version
Client: &version.Version{SemVer:"v2.12.2", GitCommit:"7d2b0c73d734f6586ed222a567c5d103fed435be", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.2", GitCommit:"7d2b0c73d734f6586ed222a567c5d103fed435be", GitTreeState:"clean"}

λ helmfile --version
helmfile version v0.41.0

λ helm plugin list
NAME    VERSION     DESCRIPTION
diff    2.11.0+3    Preview helm upgrade changes as a diff

[danielyang990]

same here.

λ helm version
Client: &version.Version{SemVer:"v2.12.2", GitCommit:"7d2b0c73d734f6586ed222a567c5d103fed435be", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.2", GitCommit:"7d2b0c73d734f6586ed222a567c5d103fed435be", GitTreeState:"clean"}

λ helmfile --version
helmfile version v0.41.0

λ helm plugin list
NAME      VERSION     DESCRIPTION
diff      2.11.0+3    Preview helm upgrade changes as a diff

I have the save version , have you solved this question?

[mumoshu]

@gianrubio Hey!

Sorry but I'm pretty confused and don't understand what your actual issues are.

Error: identified at least one change, exiting with non-zero exit code (detailed-exitcode parameter enabled)
identified at least one change, exiting with non-zero exit code (detailed-exitcode parameter enabled)
Error: plugin "diff" exited with error

Upgrading stable/metricbeat

This is a kind of the expected output from helmfile. This basically means helmfile CORRECTLY found there are deltas in the desired state, and therefore running helm upgrade(hence Upgrading stable/metricbeat.

The upgrade error followed by that is suspicious, though:

Error: failed to download "stable/metricbeat" (hint: running `helm repo update` may help)

err: release "metricbaet" in "helmfile.yaml" failed: exit status 1
exit status 1

So this isn't an issue arose from helm-diff. More likely your helm upgrade, or the local charts cache is broken.

Can you try running helm upgrade --install stable/metricbeat to verify the issue?

@yurrriq @rrajadevops @boisde @danielyang990 Could you share your exact command and the output from it? So that it gets more clear whether you're talking abount helmfile diff or helmfile apply, or about (confusing) logs when helmfile runs helm-diff internally, or the helm upgrade errors like @gianrubio is encountering.

[grebois]

workaround:

helmDefaults:
  verify: false

no idea why

[yurrriq]

I gave up on helmfile for a few months and then came back. I'm no longer having these issues.

[mumoshu]

@yurrriq Glad to hear it finally worked for you ☺️ Thanks for reporting!

[NaffanDroo]

We see a similar error, though it skips over it, with helm v2.13.1 - is it all down to compatibility with helm diff then?

We are running: helmfile version v0.54.2

[mumoshu]

@NaffanDroo Hey!

I'm not aware of any compatibility issue related to the error atm.

But the latest version of helmfile fixes confusing error messages and hopefully let you see what's the actual error is(if any).

Would you mind trying the latest ver., and the the full log and the pointer to which line of it seems problematic?

[Constantin07]

Having the same issue with helfile v0.63.0, helm v2.14.0 when trying to helmfile sync:

$ helmfile sync
Upgrading stable/nginx-ingress

List of releases in error :
RELEASE
nginx-ingress
in ./helmfile.yaml: failed processing release nginx-ingress: helm exited with status 1:
  Error: failed to download "stable/nginx-ingress" (hint: running `helm repo update` may help)

With @grebois workaround I don't get this error.

[RiceBowlJr]

@Constantin07 try with the flag --log-level debug, it is very verbose but might help you find the error.

[Constantin07]

@RiceBowlJr Thanks, with the latest v0.78.0 I cannot repoduce this issue anymore.

[rustycl0ck]

I faced this issue in v0.79.3, @grebois' response above fixed my issue.

[ArturCC]

Exists in v0.90.9 on clean install, verify: false helps.

[mumoshu]

This thread turned out to be a mix of many kinds of errors:

• If the release being diffed has some kind of errro verify: false might be needed
• If the release being diffed has some custom resources for CRDs that aren't installed onto the cluster yet, you'll need disableValidation: true.
• If you're on Helm 2, you should upgrade to Helm 3.

Closing as resolved!