Helm-operator: Helm-operator CrashLoopBackOff after error="failed to prepare chart for release: chart not ready: no existing git mirror found"

[kamilswiec]

Describe the bug
After migration from helmv2 to helmv3 helmreleases worked fine. But after deletion of one helmrealeases helm-operator started to crash with logs about waiting to clone or finding existing mirror for charts. We tried to reproduce error on newly created cluster, with just one helm release which uses our chart. Helm operator failed immediately.

• Cloning repo from helm-operator pod works.
• Installing chart manually with helmv3 works.

To Reproduce
Steps to reproduce the behaviour:

1. Kubernetes cluster
2. Flux + GitLab setup + Helm-operator
3. helm-operator \
   --enabled-helm-versions=v3 \
   --kubeconfig=/root/.kube/config \
   --log-format=fmt \
   --git-timeout=20s \
   --git-poll-interval=30s \
   --charts-sync-interval=3m \
   --status-update-interval=30s \
   --update-chart-deps=true \
   --log-release-diffs=false \
   --workers=4 \
   --tiller-namespace=kube-system
4. Add helmrealease with chart source in gitlab
5. kubectl logs helm-operator

Expected behavior
Applying charts to namespace.

Logs

ts=2020-09-08T08:47:29.914163697Z caller=operator.go:86 component=operator info="setting up event handlers"
ts=2020-09-08T08:47:29.915861744Z caller=operator.go:107 component=operator info="event handlers set up"
ts=2020-09-08T08:47:29.916934303Z caller=main.go:300 component=helm-operator info="waiting for informer caches to sync"
ts=2020-09-08T08:47:30.017847147Z caller=main.go:305 component=helm-operator info="informer caches synced"
ts=2020-09-08T08:47:30.038617114Z caller=operator.go:119 component=operator info="starting operator"
ts=2020-09-08T08:47:30.039914825Z caller=operator.go:121 component=operator info="starting workers"
ts=2020-09-08T08:47:30.041374016Z caller=release.go:79 component=release release=gnet-app-phone-services-authz targetNamespace=gnet resource=gnet:helmrelease/app-phone-services-authz helmVer
sion=v3 info="starting sync run"
ts=2020-09-08T08:47:30.041922813Z caller=git.go:311 component=gitchartsync info="started mirroring new remote" [email protected]:reef/gitops-helm [email protected]
.pl:reef/gitops-helm
ts=2020-09-08T08:47:30.043325727Z caller=git.go:104 component=gitchartsync info="starting sync of git chart sources"
ts=2020-09-08T08:47:30.044549378Z caller=server.go:42 component=daemonhttp info="starting HTTP server on :3030"
ts=2020-09-08T08:47:30.091892421Z caller=release.go:85 component=release release=gnet-app-phone-services-authz targetNamespace=gnet resource=gnet:helmrelease/app-phone-services-authz helmVer
sion=v3 error="failed to prepare chart for release: chart not ready: no existing git mirror found"
ts=2020-09-08T08:47:30.925668979Z caller=checkpoint.go:24 component=checkpoint msg="up to date" latest=0.10.1
ts=2020-09-08T08:47:33.190826565Z caller=release.go:79 component=release release=gnet-app-phone-services-authz targetNamespace=gnet resource=gnet:helmrelease/app-phone-services-authz helmVer
sion=v3 info="starting sync run"
ts=2020-09-08T08:47:34.078453178Z caller=logwriter.go:28 info="2020/09/08 08:47:34 Found 0 tiller pods"

Sample logs from our cluster which also occured:

ts=2020-09-08T09:18:20.272673231Z caller=release.go:85 component=release release=gnet-app-phone-services-licensing targetNamespace=gnet resource=gnet:helmrelease/app-phone-services-licensing helmVersion=v3 error="failed to prepare chart for release: chart not ready: git repo has not been cloned yet"

Additional context
Add any other context about the problem here, e.g

• Helm Operator version: 1.2.0
• Targeted Helm version: 3.3.0
• Kubernetes version: v1.17.4
• Git provider: GitLab
• Container registry provider: docker-registry

helm-operator.yaml

---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: helm-operator
  namespace: flux
#  annotations:
#    helm.fluxcd.io/migrate: "true"
spec:
  helmVersion: v3
  releaseName: helm-operator
  chart:
    repository: https://charts.fluxcd.io
    name: helm-operator
    version: 1.2.0
  values:
    helm:
      versions: v3
    createCRD: false
    LogReleaseDiffs: true
    git:
      pollInterval: 30s
      chartsPath: charts
      ssh:
        known_hosts: |-
          git.example.pl [redacted]
          git.example.pl [redacted]
          git.example.pl [redacted]
    resources:
      requests:
        cpu: 50m
        memory: 64Mi
      limits:
        cpu: 1
        memory: 2Gi

helmrealease.yaml

---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: app-phone-services-authz
  namespace: gnet
  annotations:
    helm.fluxcd.io/migrate: "true"
spec:
  helmVersion: v3
  chart:
    git: git@githost:reef/gitops-helm
    ref: master
    path: charts/cloud-app
  values:
    replicaCount: 1
    fullnameOverride: app-phone-services-authz
    image:
      repository: [redacted]
      tag: 0.40.0

[ttarczynski]

I work with @kamilswiec in one team. We've digged more into this issue, and it seems that there are two separate errors (bugs?) that we hit:

1. "failed to prepare chart for release: chart not ready: no existing git mirror found"
   This is logged with log level error, but I'm not sure if it breaks something or not
2. Found 0 tiller pods
   This causes the helm-operator crash. And I think it's caused by the helm.fluxcd.io/migrate annotation – see #528

[varac]

I propose to continue with the failed to prepare chart for release: chart not ready: no existing git mirror found error in this issue (pls create a dedicated issue for the Found 0 tiller pods one).

I experience the same no existing git mirror found error which is misleading because helm-opereator could fetch from the same, unmodified git repo a day before. So the release was installed successfully, but got into a Failed state the day after.
This is pretty annoying and it would be good to have more debug information to investigate the root cause of this.

I have as an example this chart definition:

  chart:
    git: https://open.greenhost.net/openappstack/wordpress-helm
    ref: 0.1.2
    path: .

and helm-operator complains like this:

root@v0-4:~# kubectl -n oas logs helm-operator-86c5869dbc-fpdxq  | grep wordp
ts=2020-09-10T11:21:56.194437501Z caller=release.go:75 component=release release=wordpress targetNamespace=oas-apps resource=oas-apps:helmrelease/wordpress helmVersion=v3 info="starting sync run"
ts=2020-09-10T11:21:56.194503791Z caller=git.go:311 component=gitchartsync info="started mirroring new remote" remote=https://open.greenhost.net/openappstack/wordpress-helm mirror=https://open.greenhost.net/openappstack/wordpress-helm
ts=2020-09-10T11:21:56.311750643Z caller=release.go:81 component=release release=wordpress targetNamespace=oas-apps resource=oas-apps:helmrelease/wordpress helmVersion=v3 error="failed to prepare chart for release: chart not ready: no existing git mirror found"
ts=2020-09-10T11:22:11.472357869Z caller=release.go:75 component=release release=wordpress targetNamespace=oas-apps resource=oas-apps:helmrelease/wordpress helmVersion=v3 info="starting sync run"
ts=2020-09-10T11:23:08.418587016Z caller=release.go:105 component=release release=wordpress targetNamespace=oas-apps resource=oas-apps:helmrelease/wordpress helmVersion=v3 error="failed to determine sync action for release: status 'failed' of release does not allow a safe upgrade"

The helm-operator pod is able to clone this chart with git just fine:

root@v0-4:~# kubectl -n oas exec -it helm-operator-86c5869dbc-fpdxq -- git clone -b 0.1.2 https://open.greenhost.net/openappstack/wordpress-helm
Cloning into 'wordpress-helm'...
warning: redirecting to https://open.greenhost.net/openappstack/wordpress-helm.git/
remote: Enumerating objects: 122, done.
remote: Counting objects: 100% (122/122), done.
remote: Compressing objects: 100% (41/41), done.
remote: Total 508 (delta 98), reused 103 (delta 81), pack-reused 386
Receiving objects: 100% (508/508), 117.78 KiB | 4.36 MiB/s, done.
Resolving deltas: 100% (279/279), done.
Note: checking out 'a4be8ab8dbe753054f47f033681d3b3f1ae362f5'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

[varac]

Btw, I experienced the above with helm-operator 1.1.0 (chart, helm-operator and docker image version) unfortunately, because #412 and #423 suggest it would have been fixed in the 1.1.0 chart.

[ttarczynski]

Yes, I've already created a separate issue for the Found 0 tiller pods error:

• #528 – Helm-operator CrashLoopBackOff when helm.fluxcd.io/migrate annotation is set and no Tiller in the cluster

[Bujail]

I'm also facing same issue with helm operator 1.1.0.

    spec:
      containers:
      - args:
        - --allow-namespace=myns
        - --enabled-helm-versions=v3
        - --git-timeout=20s
        - --git-poll-interval=5m
        - --charts-sync-interval=3m
        - --status-update-interval=30s
        - --update-chart-deps=true
        - --log-release-diffs=true
        - --workers=4
        image: docker.io/fluxcd/helm-operator:1.1.0

Any leads on this issue?

[stretchnate]

I'm experiencing the same issue on helm operator v1.2.0 and helm v3. I see error="failed to prepare chart for release: chart not ready: no existing git mirror found" but no subsequent error and when I sh into the pod I can see the mirror has been cloned and can even re-clone it in a different dir but no indication as to why HO can't find the mirror.

If you need more details on my setup I'm happy to provide them.