Helm-operator: Upgrading deprecated deployment apiVersion results in: rendered manifests contain a new resource that already exists. Unable to continue with update: existing resource conflict

[sjdweb]

Describe the bug
I am trying to upgrade some old deprecated apiVersions for Deployment objects before GKE forces upgrade to v1.16 next week.

I've tried to follow various steps from issues like:

• https://github.com/helm/helm/issues/7697#issuecomment-613535044
• https://helm.sh/docs/topics/kubernetes_apis/

To Reproduce
Steps to reproduce the behaviour:

1. Create PR with apiVersion changes
2. Merge in and wait for everything to sync
3. See that HelmRelease upgrade failed
4. Check helm-operator logs (pasted below)

Expected behavior
Deployment object should be upgraded.

Logs
If applicable, please provide logs. In a standard stand-alone installation, you'd get this by running kubectl logs -n default deploy/flux-helm-operator.

ts=2020-10-02T22:16:17.232323206Z caller=release.go:289 component=release release=debug-apiupgrade targetNamespace=default resource=default:helmrelease/debug-apiupgrade helmVersion=v3 info="running dry-run upgrade to compare with release version '2'" action=dry-run-compare
ts=2020-10-02T22:16:17.247836575Z caller=helm.go:69 component=helm version=v3 info="preparing upgrade for debug-apiupgrade" targetNamespace=default release=debug-apiupgrade
ts=2020-10-02T22:16:17.275182755Z caller=helm.go:69 component=helm version=v3 info="resetting values to the chart's original version" targetNamespace=default release=debug-apiupgrade
ts=2020-10-02T22:16:17.694274526Z caller=helm.go:69 component=helm version=v3 info="performing update for fluxcloud" targetNamespace=flux release=fluxcloud
ts=2020-10-02T22:16:17.77400089Z caller=helm.go:69 component=helm version=v3 info="performing update for nginx-ingress" targetNamespace=nginx-ingress release=nginx-ingress
ts=2020-10-02T22:16:17.780156961Z caller=helm.go:69 component=helm version=v3 info="performing update for debug-apiupgrade" targetNamespace=default release=debug-apiupgrade
ts=2020-10-02T22:16:17.781069376Z caller=helm.go:69 component=helm version=v3 info="performing update for cert-manager" targetNamespace=cert-manager release=cert-manager
ts=2020-10-02T22:16:17.817347708Z caller=helm.go:69 component=helm version=v3 info="dry run for fluxcloud" targetNamespace=flux release=fluxcloud
ts=2020-10-02T22:16:17.87814247Z caller=release.go:311 component=release release=fluxcloud targetNamespace=flux resource=flux:helmrelease/fluxcloud helmVersion=v3 info="no changes" phase=dry-run-compare
ts=2020-10-02T22:16:17.912832985Z caller=release.go:79 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v3 info="starting sync run"
ts=2020-10-02T22:16:17.940786673Z caller=release.go:294 component=release release=debug-apiupgrade targetNamespace=default resource=default:helmrelease/debug-apiupgrade helmVersion=v3 error="dry-run upgrade for comparison failed: rendered manifests contain a new resource that already exists. Unable to continue with update: existing resource conflict: namespace: default, name: debug-apiupgrade, existing_kind: apps/v1, Kind=Deployment, new_kind: apps/v1, Kind=Deployment" phase=dry-run-compare

HelmRelease:

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  annotations:
    flux.weave.works/automated: "false"
    fluxcd.io/sync-checksum: 7a40b34f8eaef548ce48adcf141ffc4069ac90f4
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"helm.fluxcd.io/v1","kind":"HelmRelease","metadata":{"annotations":{"flux.weave.works/automated":"false","fluxcd.io/sync-checksum":"7a40b34f8eaef548ce48adcf141ffc4069ac90f4"},"labels":{"fluxcd.io/sync-gc-mark":"sha256.sLTQxfXX0DcIp0NWAp8q-oSEl1jyqn6JgEEThk0JGoQ"},"name":"debug-apiupgrade","namespace":"default"},"spec":{"chart":{"git":"[email protected]:xxx/xxxx","path":"charts/debug-apiupgrade"},"helmVersion":"v3","releaseName":"debug-apiupgrade","rollback":{"enable":true,"force":true,"maxRetries":3,"retry":true}}}
  creationTimestamp: "2020-10-02T21:23:13Z"
  generation: 1
  labels:
    fluxcd.io/sync-gc-mark: sha256.sLTQxfXX0DcIp0NWAp8q-oSEl1jyqn6JgEEThk0JGoQ
  name: debug-apiupgrade
  namespace: default
  resourceVersion: "11226252"
  selfLink: /apis/helm.fluxcd.io/v1/namespaces/default/helmreleases/debug-apiupgrade
  uid: 1e55cfb6-2085-4a5c-a89a-b8a2fa39e70e
spec:
  chart:
    git: [email protected]:xxxx/xxxx
    path: charts/debug-apiupgrade
  helmVersion: v3
  releaseName: debug-apiupgrade
  rollback:
    enable: true
    force: true
    maxRetries: 3
    retry: true
status:
  conditions:
  - lastTransitionTime: "2020-10-02T21:32:28Z"
    lastUpdateTime: "2020-10-02T22:17:31Z"
    message: Chart fetch was successful for Helm release 'debug-apiupgrade' in 'default'.
    reason: ChartFetched
    status: "True"
    type: ChartFetched
  - lastTransitionTime: "2020-10-02T22:17:34Z"
    lastUpdateTime: "2020-10-02T22:17:34Z"
    message: Installation or upgrade succeeded for Helm release 'debug-apiupgrade'
      in 'default'.
    reason: Deployed
    status: "True"
    type: Deployed
  - lastTransitionTime: "2020-10-02T22:17:34Z"
    lastUpdateTime: "2020-10-02T22:22:18Z"
    message: Release was successful for Helm release 'debug-apiupgrade' in 'default'.
    reason: Succeeded
    status: "True"
    type: Released
  lastAttemptedRevision: 24ea28b391213f6639bd489d302ba8baae062710
  observedGeneration: 1
  phase: Succeeded
  releaseName: debug-apiupgrade
  releaseStatus: deployed
  revision: 24ea28b391213f6639bd489d302ba8baae062710

You can see I even tried the applying the recommended annotations to help Helm along:

$k get deployment debug-apiupgrade -o yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    helm.fluxcd.io/antecedent: default:helmrelease/debug-apiupgrade
    meta.helm.sh/release-name: debug-apiupgrade
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2020-10-02T21:23:14Z"
  generation: 4
  labels:
    app: debug-apiupgrade
    app.kubernetes.io/managed-by: Helm
    chart: debug-apiupgrade-0.1.0
    component: api
    release: debug-apiupgrade
  name: debug-apiupgrade
  namespace: default

Additional context
Add any other context about the problem here, e.g

• Helm Operator version: 1.2.0
• Targeted Helm version: v3
• Kubernetes version: v1.15.x
• Git provider: private
• Container registry provider:

[avielb]

faced the same issue like this few month ago, from what I saw unfortunately it's not a helm operator issue but helm/kubernetes issue which is not getting the attention it needs.

[sjdweb]

I managed to work around this issue, instructions below:

Steps to manually upgrade release

Have a PR of the updated chart (apiVersion) ready to go, but not yet merged.

Add the following annotations and labels:

$ k annotate deployment <your-release> meta.helm.sh/release-name=<your-release>
$ k annotate deployment <your-release> meta.helm.sh/release-namespace=default
$ k label deployment <your-release> app.kubernetes.io/managed-by=Helm

Grab the release values:

$ helm get values <your-release> > /tmp/<your-release>.values.yaml

Manually upgrade the release on the cluster:

$ helm upgrade -f /tmp/<your-release>.values.yaml <your-release> ./charts/<your-chart> # we reference the local path of the chart from above PR.
Release "<your-release>" has been upgraded. Happy Helming!

Confirm the apiVersion is now correct:

$ helm get manifest <your-release>
---
# Source: <your-release>/templates/deployment.yaml
kind: Deployment
apiVersion: **apps/v1**
metadata:
  name: <your-release>
  namespace: default
  labels:
    # ...
spec:
  selector:
    matchLabels:
      # ...
      release: <your-release>
  replicas: 1

Now you can go ahead and merge in your PR that updates the chart. Allow FluxCD a few minutes to poll for this change, and the same for helm-operator. You can always follow the logs for helm-operator if you're feeling impatient:

$ kubectl -n flux logs -f helm-operator-xxx-xxx

Eventually you'll see some good news:

ts=2020-10-05T16:54:56.344324383Z caller=release.go:79 component=release release=debug-apiupgrade targetNamespace=default resource=default:helmrelease/debug-apiupgrade helmVersion=v3 info="starting sync run"
ts=2020-10-05T16:54:56.798917608Z caller=release.go:289 component=release release=debug-apiupgrade targetNamespace=default resource=default:helmrelease/debug-apiupgrade helmVersion=v3 info="running dry-run upgrade to compare with release version '4'" action=dry-run-compare
ts=2020-10-05T16:54:56.805087255Z caller=helm.go:69 component=helm version=v3 info="preparing upgrade for debug-apiupgrade" targetNamespace=default release=debug-apiupgrade
ts=2020-10-05T16:54:56.820154722Z caller=helm.go:69 component=helm version=v3 info="resetting values to the chart's original version" targetNamespace=default release=debug-apiupgrade
ts=2020-10-05T16:54:57.233925333Z caller=helm.go:69 component=helm version=v3 info="performing update for debug-apiupgrade" targetNamespace=default release=debug-apiupgrade
ts=2020-10-05T16:54:57.33001111Z caller=helm.go:69 component=helm version=v3 info="dry run for debug-apiupgrade" targetNamespace=default release=debug-apiupgrade
ts=2020-10-05T16:54:57.364945462Z caller=release.go:311 component=release release=debug-apiupgrade targetNamespace=default resource=default:helmrelease/debug-apiupgrade helmVersion=v3 info="no changes" phase=dry-run-compare

Again, verify the helm release is correct:

$ helm get manifest debug-apiupgrade
---
# Source: <your-release>/templates/deployment.yaml
kind: Deployment
apiVersion: **apps/v1**

Done! Downtime avoided.

[rafaelgaspar]

This has been fixed in Helm v3.2, any reason why the helm-operator is still using v3.1.2?