Helm-operator: Dry run upgrade check fails

[qvmedvedev]

Describe the bug
After successful chart installation/upgrade dry-run upgrade check fails

To Reproduce
Create namespace

kubectl create namespace mon

Create custom values file

---
allowNamespace: mon
logReleaseDiffs: true
helm:
  versions: "v3"
configureRepositories:
  enable: true
  secretName: flux-mon-helm-repositories
  repositories:
    - name: stable
      url: https://kubernetes-charts.storage.googleapis.com
clusterRole:
  create: false
extraEnvs:
  - name: CHECKPOINT_DISABLE
    value: "true"

Install helm-operator using that file

helm3 install helm-operator-mon --namespace mon fluxcd/helm-operator --values ./helm-operator-mon.yaml

Create HelmRelease

cat <<EOF | kubectl apply -f -
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: prometheus-pushgateway
  namespace: mon
spec:
  releaseName: prometheus-pushgateway
  targetNamespace: mon
  chart:
    repository: https://kubernetes-charts.storage.googleapis.com
    version: 1.2.6
    name: prometheus-pushgateway
EOF

Get helmreleases

kubectl get hr -n mon
NAME                     RELEASE                  STATUS     MESSAGE                       AGE
prometheus-pushgateway   prometheus-pushgateway   deployed   Helm release sync succeeded   1m

Get helmreleases after 5 minutes

kubectl get hr -n mon
NAME                     RELEASE                  STATUS     MESSAGE                                                                                                                                                                                                                                          AGE
prometheus-pushgateway   prometheus-pushgateway   deployed   failed to upgrade chart for release [prometheus-pushgateway]: query: failed to query with labels: secrets is forbidden: User "system:serviceaccount:mon:helm-operator-mon" cannot list resource "secrets" in API group "" at the cluster scope   7m

Expected behavior
Dry-run checks pass successfully

Logs

W0211 10:57:47.978488       8 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
ts=2020-02-11T10:57:49.434392675Z caller=repository.go:125 component=helm version=v3 info="successfully imported repository" name=stable url=https://kubernetes-charts.storage.googleapis.com
ts=2020-02-11T10:57:55.411810624Z caller=operator.go:98 component=operator info="setting up event handlers"
ts=2020-02-11T10:57:55.411846408Z caller=operator.go:117 component=operator info="event handlers set up"
ts=2020-02-11T10:57:55.411864741Z caller=main.go:280 component=helm-operator info="waiting for informer caches to sync"
ts=2020-02-11T10:57:55.512013002Z caller=main.go:285 component=helm-operator info="informer caches synced"
ts=2020-02-11T10:57:55.512280068Z caller=server.go:42 component=daemonhttp info="starting HTTP server on :3030"
ts=2020-02-11T10:57:55.512990719Z caller=operator.go:129 component=operator info="starting operator"
ts=2020-02-11T10:57:55.513043096Z caller=operator.go:131 component=operator info="starting workers"
ts=2020-02-11T10:57:55.513036971Z caller=git.go:105 component=gitchartsync info="starting sync of git chart sources"
ts=2020-02-11T11:06:26.801234111Z caller=release.go:317 component=release release=prometheus-pushgateway targetNamespace=mon resource=mon:helmrelease/prometheus-pushgateway helmVersion=v3 info="no existing release" action=install
ts=2020-02-11T11:06:27.226818291Z caller=helm.go:66 component=helm version=v3 info="creating 3 resource(s)"
ts=2020-02-11T11:06:27.338713398Z caller=helm.go:66 component=helm version=v3 info="beginning wait for 3 resources with timeout of 5m0s"
ts=2020-02-11T11:06:29.371602997Z caller=helm.go:66 component=helm version=v3 info="Deployment is not ready: mon/prometheus-pushgateway. 0 out of 1 expected pods are ready"
ts=2020-02-11T11:06:31.348830144Z caller=helm.go:66 component=helm version=v3 info="Deployment is not ready: mon/prometheus-pushgateway. 0 out of 1 expected pods are ready"
ts=2020-02-11T11:06:33.348114436Z caller=helm.go:66 component=helm version=v3 info="Deployment is not ready: mon/prometheus-pushgateway. 0 out of 1 expected pods are ready"
ts=2020-02-11T11:06:35.348056737Z caller=helm.go:66 component=helm version=v3 info="Deployment is not ready: mon/prometheus-pushgateway. 0 out of 1 expected pods are ready"
ts=2020-02-11T11:06:37.348748732Z caller=helm.go:66 component=helm version=v3 info="Deployment is not ready: mon/prometheus-pushgateway. 0 out of 1 expected pods are ready"
ts=2020-02-11T11:06:39.351615029Z caller=helm.go:66 component=helm version=v3 info="Deployment is not ready: mon/prometheus-pushgateway. 0 out of 1 expected pods are ready"
ts=2020-02-11T11:06:41.349898237Z caller=helm.go:66 component=helm version=v3 info="Deployment is not ready: mon/prometheus-pushgateway. 0 out of 1 expected pods are ready"
ts=2020-02-11T11:06:43.531438508Z caller=release.go:251 component=release release=prometheus-pushgateway targetNamespace=mon resource=mon:helmrelease/prometheus-pushgateway helmVersion=v3 info="Helm release sync succeeded" revision=1.2.6
ts=2020-02-11T11:06:55.420687496Z caller=operator.go:307 component=operator info="enqueuing release" resource=mon:helmrelease/prometheus-pushgateway
ts=2020-02-11T11:06:55.515811277Z caller=release.go:342 component=release release=prometheus-pushgateway targetNamespace=mon resource=mon:helmrelease/prometheus-pushgateway helmVersion=v3 info="performing dry-run upgrade to see if release has diverged"
ts=2020-02-11T11:06:55.51881448Z caller=helm.go:66 component=helm version=v3 info="preparing upgrade for prometheus-pushgateway"
ts=2020-02-11T11:06:55.528157835Z caller=release.go:185 component=release release=prometheus-pushgateway targetNamespace=mon resource=mon:helmrelease/prometheus-pushgateway helmVersion=v3 error="failed to determine if the release should be synced" err="failed to upgrade chart for release [prometheus-pushgateway]: query: failed to query with labels: secrets is forbidden: User \"system:serviceaccount:mon:helm-operator-mon\" cannot list resource \"secrets\" in API group \"\" at the cluster scope"

Additional context

• Helm operator version: 1.0.0-rc8
• Targeted Helm version: 3.0.2
• Kubernetes version: 1.15.5

[hiddeco]

I do not think this is an issue with the operator but an issue with the chart you are using, doing a helm upgrade -i dry-run upgrade with the exact same values as specified in the HelmRelease will likely yield the same result.

[stefanprodan]

@hiddeco this seems like a RBAC issue, Helm Operator was installed with a role and on upgrade Helm v3 tries to query secrets at cluster level instead of namespace level.

[hiddeco]

@stefanprodan you are absolutely right. Sorry about that @qvmedvedev, I am not having my brightest day today.

We used to not pass along the namespace for dry-run operations, this has been fixed in https://github.com/fluxcd/helm-operator/commit/6f0daff9c9815059cd7182e24126d56a44c9a830 but not made it to a release yet. Can you please try the following image from our prerelease images, and report back if this resolves the issue for you @qvmedvedev? fluxcd/helm-operator-prerelease:master-3f64a324

[qvmedvedev]

Yeah, fluxcd/helm-operator-prerelease:master-3f64a324 has fixed my issue. Great, thanks!