Flux: Helm operator fails to upgrade due to existing release

[pmquang]

Hi,

I have a Helm Release look like ( and it installed successfully before ) :

---
apiVersion: flux.weave.works/v1beta1
kind: HelmRelease
metadata:
  annotations:
    flux.weave.works/automated: "false"
    #flux.weave.works/tag.chart-image: glob:stg-*
  name: profile-v2-api
  namespace: dev
spec:
  chart:
    git: [email protected]:xxx/flux.git
    path: charts/common-service
    ref: master  
  releaseName: profile-v2-api

After upgrading K8S cluster ( GKE ), I have some error log like:

ts=2019-03-22T04:12:40.446368403Z caller=release.go:186 component=release error="Chart release failed: profile-v2-api: &status.statusError{Code:2, Message:\"a release named profile-v2-api already exists.\\nRun: helm ls --all profile-v2-api; to check the status of the release\\nOr run: helm del --purge profile-v2-api; to delete it\", Details:[]*any.Any(nil)}"

Why this error happened ? How to manually mark this release belong to the Helm Operator ( to make this error gone ) ?

How Helm Operator recognize which releases it release or not ? ( it get some information in memcached ) ?

Regards

[hiddeco]

The operator will attempt to resolve the Helm release by consulting Tiller, in case it is found and the release status is DEPLOYED it will perform an upgrade, otherwise it will do an install.

In your case it attempted to install, so the release was there but did not have a DEPLOYED status (you can retrieve the status of a release by running helm status <release name>, it is listed under STATUS:). The operator was also unable to automatically recover from this (by removing the release) as the release history has more than 1 revision.

Rolling back (by performing a helm rollback <release name> <revision number>) or ensuring in another way the release gets back to a healthy DEPLOYED state will resolve the issue.

[pmquang]

Hi,

Could you add feature like:

forceUpgrade: 1

to make Helm operator always use helm upgrade -i

Regards

[hiddeco]

The forceUpgrade is actually already implemented, the problem however is that this flag is only used when performing an upgrade and the operator attempts to install.

@squaremo / @stefanprodan: is there a specific reason we only attempt to perform an upgrade when the state of a release is DEPLOYED? Can it be widened to DEPLOYED || FAILED to allow enforced upgrades to happen?

[pmquang]

Hmm,

Thanks for your info, but Im not sure your option is helm upgrade --force or helm upgrade -i

--force seems dangerous because it actually delete the release and install --replace the new one.

Actually, I can work around this by deleting the configmap of failed Helm release. After that, Helm Operator can upgrade normally.

Regards

[hiddeco]

It is helm upgrade --force, helm upgrade -i will not do anything in this scenario as it instructs to run an install _if a release by this name doesn't already exist_.

[pmquang]

LVNHCMAC0000177:flux 123491$ cd flux/ LVNHCMAC0000177:flux 123491$ ll total 88 -rw-r--r-- 1 123491 754940789 8227 Mar 1 10:06 CHANGELOG.md -rw-r--r-- 1 123491 754940789 478 Mar 1 10:06 Chart.yaml -rwxr-xr-x 1 123491 754940789 14383 Mar 1 10:06 README.md drwxr-xr-x 15 123491 754940789 510 Mar 1 10:06 templates -rw-r--r-- 1 123491 754940789 9758 Mar 22 16:14 values.yaml LVNHCMAC0000177:flux 123491$ vim values.yaml LVNHCMAC0000177:flux 123491$ helm ls | grep flux flux 5 Fri Mar 22 16:15:02 2019 DEPLOYED flux-0.6.3 1.10.1 flux LVNHCMAC0000177:flux 123491$ vim values.yaml LVNHCMAC0000177:flux 123491$ helm upgrade -i flux --namespace flux ./ Error: UPGRADE FAILED: v1.Deployment.Spec: v1.DeploymentSpec.Replicas: readUint32: unexpected character: �, error found in #10 byte of ...|eplicas":"1-3","revi|..., bigger context ...|"spec":{"progressDeadlineSeconds":600,"replicas":"1-3","revisionHistoryLimit":10,"selector":{"matchL|... LVNHCMAC0000177:flux 123491$ helm ls | grep flux flux 6 Tue Mar 26 20:50:33 2019 FAILED flux-0.6.3 1.10.1 flux LVNHCMAC0000177:flux 123491$ vim values.yaml LVNHCMAC0000177:flux 123491$ helm upgrade -i flux --namespace flux ./ Release "flux" has been upgraded. Happy Helming! LAST DEPLOYED: Tue Mar 26 20:51:18 2019 NAMESPACE: flux STATUS: DEPLOYED

Thank you again, seems I got wrong but I want to clarify this.

In a normal operation like above, helm can upgrade even the last release FAILED without forceUpgrade option. ( of course exclude the first failed release )

So, Helm Operation can simply upgrade the same way like that. Why it marks the release existed and can not upgrade ?

Regards

[squaremo]

@pmquang Did you get past this problem? (-> can we close the issue?)

[pmquang]

Yes, seems it's due to helm, not flux or helm operator. We can close this. Thanks