Charts: [stable/prometheus-operator] chart does not work with helm diff plugin

[mojochao]

Describe the bug

After installing the chart, immediately running helm diff upgrade command returns a diff. All other charts return empty diffs when run as such.

Version of Helm and Kubernetes:

helm v3.0.0
helm-diff plugin v3.0.0-rc.7
AWS EKS cluster running Kubernetes 1.14

Which chart:

stable/prometheus-operator

What happened:

After successfully installing the chart, immediately running a helm diff upgrade command returns a diff, although the installation works as expected.

Start by installing the chart with no values provided.

$ helm install prometheus-operator stable/prometheus-operator -n monitoring
manifest_sorter.go:175: info: skipping unknown hook: "crd-install"
manifest_sorter.go:175: info: skipping unknown hook: "crd-install"
manifest_sorter.go:175: info: skipping unknown hook: "crd-install"
manifest_sorter.go:175: info: skipping unknown hook: "crd-install"
manifest_sorter.go:175: info: skipping unknown hook: "crd-install"
NAME: prometheus-operator
LAST DEPLOYED: Mon Dec  2 20:27:57 2019
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
The Prometheus Operator has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=prometheus-operator"

Visit https://github.com/coreos/prometheus-operator for instructions on how
to create & configure Alertmanager and Prometheus instances using the Operator.

It shows up in the chart release listing.

$ helm list -n monitoring
NAME                NAMESPACE   REVISION    UPDATED                                 STATUS      CHART                       APP VERSION
prometheus-operator monitoring  1           2019-12-02 20:27:57.042011 -0600 CST    deployed    prometheus-operator-8.3.0   0.34.0

Everything looks normal in the cluster.

$ kubectl get all -n monitoring
NAME                                                          READY   STATUS    RESTARTS   AGE
pod/alertmanager-prometheus-operator-alertmanager-0           2/2     Running   0          38m
pod/prometheus-operator-grafana-fc55ccf5b-257lx               2/2     Running   0          38m
pod/prometheus-operator-kube-state-metrics-56bbc68694-7gkl5   1/1     Running   0          38m
pod/prometheus-operator-operator-7795b9bdc8-wxv9w             2/2     Running   0          38m
pod/prometheus-operator-prometheus-node-exporter-nzf2j        1/1     Running   0          38m
pod/prometheus-prometheus-operator-prometheus-0               3/3     Running   1          38m

NAME                                                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/alertmanager-operated                          ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   38m
service/prometheus-operated                            ClusterIP   None             <none>        9090/TCP                     38m
service/prometheus-operator-alertmanager               ClusterIP   10.100.231.37    <none>        9093/TCP                     38m
service/prometheus-operator-grafana                    ClusterIP   10.100.60.199    <none>        80/TCP                       38m
service/prometheus-operator-kube-state-metrics         ClusterIP   10.100.65.217    <none>        8080/TCP                     38m
service/prometheus-operator-operator                   ClusterIP   10.100.234.34    <none>        8080/TCP,443/TCP             38m
service/prometheus-operator-prometheus                 ClusterIP   10.100.171.127   <none>        9090/TCP                     38m
service/prometheus-operator-prometheus-node-exporter   ClusterIP   10.100.224.139   <none>        9100/TCP                     38m

NAME                                                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/prometheus-operator-prometheus-node-exporter   1         1         1       1            1           <none>          38m

NAME                                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prometheus-operator-grafana              1/1     1            1           38m
deployment.apps/prometheus-operator-kube-state-metrics   1/1     1            1           38m
deployment.apps/prometheus-operator-operator             1/1     1            1           38m

NAME                                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/prometheus-operator-grafana-fc55ccf5b               1         1         1       38m
replicaset.apps/prometheus-operator-kube-state-metrics-56bbc68694   1         1         1       38m
replicaset.apps/prometheus-operator-operator-7795b9bdc8             1         1         1       38m

NAME                                                             READY   AGE
statefulset.apps/alertmanager-prometheus-operator-alertmanager   1/1     38m
statefulset.apps/prometheus-prometheus-operator-prometheus       1/1     38m

Running a helm diff upgrade command shows new manifests in the diff, however.

$ helm diff upgrade prometheus-operator stable/prometheus-operator -n monitoring
monitoring, prometheus-operator-k8s-coredns, ConfigMap (v1) has been added:
-
+ # Source: prometheus-operator/templates/grafana/dashboards-1.14/k8s-coredns.yaml
+ # Added manually, can be changed in-place.
+ apiVersion: v1
+ kind: ConfigMap
+ metadata:
+   name: prometheus-operator-k8s-coredns
+   labels:
+     grafana_dashboard: "1"
+     app: prometheus-operator-grafana
+
+     chart: prometheus-operator-8.3.0
+     release: "prometheus-operator"
+     heritage: "Helm"
... lots of additional diff content elided, can provide if needed

What you expected to happen:

I expected no diff. No chart values were changed (or provided for that matter), but new manifests are present in the diff output.

Note that no other chart installed in my cluster exhibits this behavior with the helm-diff plugin.

How to reproduce it (as minimally and precisely as possible):

1. Install the helm-diff plugin with the helm plugin install https://github.com/databus23/helm-diff --version 3.0.0-rc.7 command
2. Create a monitoring namespace with the kubectl create namespace monitoring command
3. Install chart with the helm install prometheus-operator stable/prometheus-operator -n monitoring command
4. Observe new manifests in the helm diff upgrade prometheus-operator stable/prometheus-operator -n monitoring command output.

Even though no other chart installs exhibit this problem with the helm-diff plugin, I did report it in their issues (https://github.com/databus23/helm-diff/issues/168), but have seen no comments on the issue thus far.

[RaymondKYLiu]

I ran into the same problem as yours, my environment is the same.

helm v3.0.0
helm-diff plugin v3.0.0-rc.7
AWS EKS cluster running Kubernetes 1.14

I found an abnormal thing that the ConfigMap prometheus-operator-k8s-coredns isn't created ...

prometheus-operator-apiserver                           1      9d
prometheus-operator-cluster-total                       1      9d
prometheus-operator-controller-manager                  1      9d
prometheus-operator-etcd                                1      9d
prometheus-operator-grafana                             2      9d
prometheus-operator-grafana-config-dashboards           1      9d
prometheus-operator-grafana-datasource                  1      9d
prometheus-operator-grafana-test                        1      9d
prometheus-operator-k8s-resources-cluster               1      9d
prometheus-operator-k8s-resources-namespace             1      9d
prometheus-operator-k8s-resources-node                  1      9d
prometheus-operator-k8s-resources-pod                   1      9d
prometheus-operator-k8s-resources-workload              1      9d
prometheus-operator-k8s-resources-workloads-namespace   1      9d
prometheus-operator-kubelet                             1      9d
prometheus-operator-namespace-by-pod                    1      9d
prometheus-operator-namespace-by-workload               1      9d
prometheus-operator-node-cluster-rsrc-use               1      9d
prometheus-operator-node-rsrc-use                       1      9d
prometheus-operator-nodes                               1      9d
prometheus-operator-persistentvolumesusage              1      9d
prometheus-operator-pod-total                           1      9d
prometheus-operator-pods                                1      9d
prometheus-operator-prometheus                          1      9d
prometheus-operator-proxy                               1      9d
prometheus-operator-scheduler                           1      9d
prometheus-operator-statefulset                         1      9d
prometheus-operator-workload-total                      1      9d
prometheus-prometheus-operator-prometheus-rulefiles-0   23     9d

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[TBeijen]

Similar situation.

helm v3.0.1
helm-diff plugin v3.0.0-rc.7
chart prometheus-operator-8.3.3
AWS EKS v1.14.9-eks

Strange enough, if I template, extract the core-dns cm and apply directly, it installs just fine and is picked up by Grafana. 🤔

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[sshishov]

Also have this issue

helm: v3.1.0
helm-diff: 3.0.0-rc.7
prometheus-operator: 8.9.2
AWS EKS: v1.14

[dogzzdogzz]

Not sure if it is because the secret of helm release almost reach etcd size limit (1 mb) on object ? If so, then I think we should use kubectl apply instead of helm to install such "big" chart.

[early-breakfast]

i do have same issue

[snedi]

I also bumped into the same issue. Expecting to see my own changes only but instead I see lots of other unknown diffs.

[davidham]

Same issue, the chart installs and upgrades, but the prometheus-operator-k8s-coredns dashboard ConfigMap is not created, and always shows as a new addition.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.