Helmfile: `needs` doesn't work for upgrades.

[RobertoUa]

needs doesn't work for already deployed releases and also for selected releases using --selector.

If I make changes to my-release and kubernetes-external-secrets and then run helmfile apply I get this error:

syncing 2 groups of releases in this order: consul, elk, external-secrets, kube-system/kubernetes-external-secrets, kube2iam, redis, storage -> default/my-release
err: "default/my-release" needs "consul", but it must be one of kube-system/kubernetes-external-secrets, default/my-release

So dependecies without changes are not taken into account. For the same reason --selector doesn't work as well.

The issue is critical, since needs is usable only for clean installation.

https://github.com/roboll/helmfile/pull/914

[mumoshu]

@RobertoUa Hey! Thanks for reporting. I think I've managed to fix this in #922.

Before merging it, I'm going to leave it for one night and review it myself with fresh brain tomorrow morning.

But I'd greatly appreciate it if you could give it a try by building it yourself with make install ☺️

[mumoshu]

In nutshell #922 changes it so that the DAG is computed before computing the diff between the desired state and the current state. For helmfile apply, I've changed it so that (1) if there're one or more releases that should be deleted, we traverse the full DAG containing all the releases in the reverse order and if a release is marked for deletion, do delete. Skip otherwise. (2) Do the same for upgrades, by traversing the full DAG in a normal order.

[RobertoUa]

Awesome! Can't wait to try it! I will report any issues I find.

[RobertoUa]

I'm getting a new error

Comparing release=external-secrets, chart=...
Comparing release=my-release, chart=...
"default/external-secrets" has dependency to inexistent release "kubernetes-external-secrets"

when run with hemlfile --environment ... --selector my-release apply
helmfile:

releases:
- name: kubernetes-external-secrets
  namespace: kube-system
- name: external-secrets
  namespace: default
  needs:
  - kubernetes-external-secrets
- name: my-release
  namespace: default
  needs:
  - external-secrets

(actually kubernetes-external-secrets & external-secrets defined in a separate helmfiles loaded as bases, but I'm not sure it makes any difference)

(By the way, I always run with --skip-deps locally because it downloads repos event after helmfile deps & helmfile repos, btw Comparin release=..., chart=... also takes a long time for some reason

[mumoshu]

@RobertoUa I think it's working as expected this time.

When a namespace is specified in the needed release, it becomes a part of the ID.

That is,

needs:
- kubernetes-external-secrets

should be changed to:

needs:
- kube-system/kubernetes-external-secrets

Maybe helmfile can do a better job on the error message, at least.

Perhaps something like "default/external-secrets" has dependency to inexistent release "kubernetes-external-secrets: Did you mean "kube-system/kubernetes-external-secrets"? should be feasible.

[RobertoUa]

When I use namespace (kube-system/kubernetes-external-secrets) it doesn't work as well. That's the problem.

[RobertoUa]

I resolved the issue. The problem was that external-secrets.yaml had

{{ readFile "base.yaml" }}

---

...

and base.yaml had environments and helmDefaults

Not sure why it would fail like that, but removing readFile helped.

[mumoshu]

@RobertoUa Glad it worked for you!

Just curious, what did you see when you've used the correct ID like kube-system/kubernetes-external-secrets in needs?

I think the error message wouldn't have been like "default/external-secrets" has dependency to inexistent release "kubernetes-external-secrets" if you've used the correct ID.

[RobertoUa]

I though specifying namaspace was optional. It seems that it's not, even when charts are in the same namespace and even for charts from default namespace.
It's fine, but it wasn't obvious.

Just curious, what did you see when you've used the correct ID like kube-system/kubernetes-external-secrets in needs?

Chart diff

[RobertoUa]

Okay, I don't know what's going on, but I still get this error sometimes. I specified namespaces everywhere. I use --selector app=test. Label app: test has both external-secrets and my-release. And kubernetes-external-secrets doesn't have this label.
in ./helmfile.yaml: "default/external-secrets" has dependency to inexistent release "kube-system/kubernetes-external-secrets"

[mumoshu]

@RobertoUa Thanks for trying! Hmm, would you mind enabling debug logs with helmfile --log-level debug apply and see if some message like this one is shown?

processing 2 groups of releases in this order:
GROUP RELEASES
1     bar, baz
2     foo

As long as the release is listed here, it can be referenced from needs.

[mumoshu]

Just curious, what did you see when you've used the correct ID like kube-system/kubernetes-external-secrets in needs?

Chart diff

Are you saying that you saw helm diff detected changes on your release?

[mumoshu]

but I still get this error sometimes

Does "sometimes" here mean that you do encounter an error sometimes but not always, even though you have not changed your configs, right?

[mumoshu]

If possible, a minimal example to reproduce the problem would be super helpful. This "sounds like" a bug but I'm not entirely sure what can cause that.

[mumoshu]

I though specifying namaspace was optional. It seems that it's not, even when charts are in the same namespace and even for charts from default namespace.
It's fine, but it wasn't obvious.

I've introduced the restriction to simplify the first implementation. Probably it can be improved.

[RobertoUa]

Are you saying that you saw helm diff detected changes on your release?

yes

Does "sometimes" here mean that you do encounter an error sometimes but not always, even though you have not changed your configs, right?

At first I thought it worked but now it doesn't, I don't know that changed. I think I wasn't using --selector but now I do. Or probably my releases didn't have any changes (helmfile apply result was No affected releases). And now my-release has a change.

Example is almost the same as above:

releases:
- name: kubernetes-external-secrets
  namespace: kube-system

- name: external-secrets
  namespace: default
  labels:
    app: test
  needs:
  - kubernetes-external-secrets

- name: my-release
  namespace: default
  labels:
    app: test
  needs:
  - external-secrets

helmfile --environment production --selector app=test apply --skip-deps

I could check the code, but I don't write Go. There should be a reason why releases could be filtered and I'm getting inexistent release.

Running with --log-level debug helmfile doesn't output any processing N groups of releases in this order

This might be useful:

<Diff ....>
identified at least one change, exiting with non-zero exit code (detailed-exitcode parameter enabled)

worker 1/1 finished
Affected releases are:
  my-release (./my-release) UPDATED

err: "default/external-secrets" has dependency to inexistent release "kube-system/kubernetes-external-secrets"
in ./helmfile.yaml: "default/external-secrets" has dependency to inexistent release "kube-system/kubernetes-external-secrets"

[mumoshu]

@RobertoUa Your info was so helpful! I managed to spot the cause. #934 should fix it.

[dududko]

@mumoshu Hi! Since the PR with the bugfix code is already merged to master, can you please tag a new release so that the binaries with the fix will be available for download ? Thank you

[mumoshu]

Hey! v0.90.6 tagged. Probably it will be released in a few minutes.