Flux: Flux chart upgrade failed with `--reuse-values`

[mar1n3r0]

Describe the bug
Flux upgrade failed

To Reproduce

1. flux 1 Wed Jul 31 11:46:18 2019 DEPLOYED flux-0.11.0 1.13.2 flux
2. helm upgrade --reuse-values flux \
   --set image.tag=1.14.2 \
   weaveworks/flux
3. Upgrade failed

Expected behavior
Upgrade to complete.
Logs
Error: render error in "flux/templates/deployment.yaml": template: flux/templates/deployment.yaml:172:35: executing "flux/templates/deployment.yaml" at <.Values.sync.state>: nil pointer evaluating interface {}.state
Error: UPGRADE FAILED: render error in "flux/templates/deployment.yaml": template: flux/templates/deployment.yaml:172:35: executing "flux/templates/deployment.yaml" at <.Values.sync.state>: nil pointer evaluating interface {}.state

Additional context

• Flux version: 1.13.2
• Helm Operator version: 2.14.3
• Kubernetes version: 1.13.7
• Git provider: bitbucket
• Container registry provider: dockerhub

[hiddeco]

--reuse-values does not merge with any new values introduced in the newer version of the chart. It will simply take the old values, and add any additional --set arguments to it.

You can read more about this here: https://medium.com/@kcatstack/understand-helm-upgrade-flags-reset-values-reuse-values-6e58ac8f127e

[mar1n3r0]

Thanks @hiddeco that was helpful and I can see the release in helm list although it fails with the below:

helm upgrade --reuse-values flux --set image.tag=1.14.2 --set sync.state=git weaveworks/flux
UPGRADE FAILED
Error: failed to create patch: merging an object in json but data type is not struct, instead is: map
Error: UPGRADE FAILED: failed to create patch: merging an object in json but data type is not struct, instead is: map

Think the reason for that is right after I provided sync state it deleted my old git access key but then I stumble upon the recent reported error #2386

Since the new flux instance is still in crash loop the fluxctl identity reports ambiguous 2 pods with label app=flux.

I rolled back but it keeps spawning the second pod with same label.

2 | flux-7745c4b9c7 | CrashLoopBackOff | flux: docker.io/fluxcd/flux:1.14.2 | 26 Sep 2019, 16:49:52 | 1/1
-- | -- | -- | -- | -- | --
1 | flux-7c658c8c8b | OK | flux: docker.io/fluxcd/flux:1.13.2 | 31 Jul 2019, 11:46:18 | 1/1

[hiddeco]

@mar1n3r0 isn't this because there are other values that changed from type, and you are still passing along the old type for that value?

[mar1n3r0]

The thing is if there is a value missing it just gives me the error with the value missing or wrong. But what's more interesting is that after the rollback it keeps the new pod in the crash loop instead of deleting it. Right now I am back to 1.13.2 but the 1.14.2 pod keeps restarting itself.
This is rendering rollback impossible for me for now and I am in the middle between two releases although helm list reports successful rollback.

[hiddeco]

@mar1n3r0 I think your best option in this case is to just start fresh. Given your cluster state is declared in git, this should be a safe operation and will be much quicker than trying to fix your faulty Helm release state.

[mar1n3r0]

Yep that sounds like the sanest option. I am only unsure which is the latest get started since I have 3 different versions open.

Can you please confirm if it's this one ?

https://github.com/fluxcd/flux/blob/master/docs/tutorials/get-started.md

[hiddeco]

We have published docs these days with a stable version, based on the latest semver release of Flux. The Helm get started guide for this version can be found here: https://docs.fluxcd.io/en/stable/tutorials/get-started-helm.html

[mar1n3r0]

Thanks a bunch I will report back once going through it.

[mar1n3r0]

Solved, the reinstall went smoothly.