Charts: [stable/prometheus-operator] Chart not working with helm 2.12.0

[viggeh]

Is this a request for help?: No

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-04T07:48:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:05:37Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
$ helm version
Client: &version.Version{SemVer:"v2.12.0", GitCommit:"d325d2a9c179b33af1a024cdb5a4472b6288016a", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.0", GitCommit:"d325d2a9c179b33af1a024cdb5a4472b6288016a", GitTreeState:"clean"}

Which chart:
stable/prometheus-operator

What happened:

$ helm install stable/prometheus-operator
Error: release wonderful-ant failed: customresourcedefinitions.apiextensions.k8s.io "alertmanagers.monitoring.coreos.com" already exists

What you expected to happen:
Prometheus-operator chart to be installed.

How to reproduce it (as minimally and precisely as possible):

1. Have helm at 2.11.0 installed on the cluster and try to install the prometheus-operator chart. Observe that installation works.
2. Delete prometheus-operator chart. Delete all CRDs. Upgrade helm to 2.12.0 installed on the cluster and try to install the prometheus-operator chart. Observe that installation fails.

Anything else we need to know:
Basically, the chart works fine with helm 2.11.0 but is broken in 2.12.0

[rafi]

Using --force on the second run works for me:

helm upgrade --install --force monitor -f values-override.yaml --namespace mon stable/prometheus-operator --timeout=900

[philoserf]

I have seen the problem. Works in 2.11.0. I have not yet identified change that broke the CRD install.

[paskal]

Only PR with CRD mentioned in releases is https://github.com/helm/helm/pull/4709.

[philoserf]

Seeing reports on Stackoverflow too.

https://stackoverflow.com/questions/53694728/prometheus-operator-alertmanagers-monitoring-coreos-com-already-exists

I have only seen reports for this operator so far. I will try another chart that installs a CRD to rule out the chart as the source of the problem.

[philoserf]

I see it is already confirmed with cert-manager at #9166

I will open an issue with helm/helm unless I find one there already.

[vsliouniaev]

The crd-install hook has been an awful experience in my experience. To run this in production across multiple clusters we have a separate chart to provision CRDs and then another step to validate that they have been created because of : https://github.com/helm/charts/issues/9241, https://github.com/helm/helm/issues/4925

Would it be worth simply removing the crd-install hooks as the default option and asking folks to provision these by hand?

[gzur]

While this issue is still in limbo - I discovered the following (horribad) workaround that might be of use to someone:

1. Try installing the operator as normal:
   helm install stable/prometheus-operator --name monitoring

2. When this fails with the error mentioned above, go into your values.yaml and set prometheusOperator.createCustomResource to false

3. Uninstall the operator
   helm delete --purge monitoring

4. (Optional) Verify that the CRD’s are still there: kubectl get crd

5. With prometheusOperator.createCustomResource still set to false - try re-installing the operator:
   helm install stable/stable/prometheus-operator --name monitoring

6. Hopefully PROFIT $$$

[vsliouniaev]

@gzur you are correct. The idea is that you deploy the 4 CRDs first, then the rest of the chart.

1. Deploy the 4 CRDs from here: https://github.com/coreos/prometheus-operator/tree/master/example/prometheus-operator-crd. You can do this using:

kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheusrule.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/servicemonitor.crd.yaml

2.

prometheusOperator.createCustomResource= false

1. Install the chart

helm install stable/prometheus-operator

[gzur]

@vsliouniaev but set prometheusOperator.createCustomResource is set to false first - right?

[WilliamCassanova]

After a few iterations I believe I have gotten this to work...It's hackish.

helm delete prometheus --purge
kubectl get crd | grep monitoring | awk 'BEGIN { FS = " " } { print $1 }' | ( while read line
  do
    kubectl delete crd ${line}
  done
  )

helm install --namespace monitoring --name prometheus stable/prometheus-operator -f ../helm-charts/monitoring/prometheus-operator-overrides.yaml -f ../secrets/secrets.yaml --no-crd-hook

The secret sauce is --no-crd-hook

[gzur]

Yeah, the --no-crd-hook flag works for me as well on v2.12.0 - which lends further credence to the idea that this was introduced by https://github.com/helm/helm/pull/4709

[philoserf]

The helm revert pull request is in flight.
https://github.com/helm/helm/pull/5067

[paskal]

v2.12.1 is released, could someone confirm it works now? @viggeh?

[viggeh]

Can confirm that this is fixed in v2.12.1.

[RickS-C137]

I'm running on v2.12.2 and still got the problem. So is this a regression?

[mazzy89]

In my case you need to run the command twice. at the first time fails because the CRDs are not there. at the second time succeed

[RickS-C137]

I also can not confirm that. I've run it several times, the result is everytime the same. Trying the above mentioned workaround now

[tim-beatson]

I too, on helm v2.12.2 am seeing this issue and running the command twice does not succeed.

[vsliouniaev]

This issue appears to be the culprit:
https://github.com/helm/helm/pull/5112

[tim-beatson]

@vsliouniaev, then I guess all we can do is wait for helm 2.12.3?

[afirth]

@vsliouniaev, then I guess all we can do is wait for helm 2.12.3?

2.13 according to the milestone :(

[tim-beatson]

@afirth, I've just upgraded to 2.12.3 and CRDs are being handled correctly again

[RickS-C137]

I can't confirm that for my env. Just upgraded to 2.12.3 and nothing changed 😕

[RickS-C137]

When trying the workaround described by vsliouniaev above I got another error. I followed the steps and create the CRDs first. But when running the following:

$ helm install -n prometheus-operator --namespace monitoring -f values.yml stable/prometheus-operator --set prometheusOperator.createCustomResource=false
Error: release prometheus-operator failed: clusterroles.rbac.authorization.k8s.io "psp-prometheus-operator-prometheus-node-exporter" is forbidden: attempt to grant extra privileges: [{[use] [extensions] [podsecuritypolicies] [prometheus-operator-prometheus-node-exporter] []}] user=&{system:serviceaccount:kube-system:tiller aa0fd7ff-996b-11e8-adca-0a9e2bcf0986 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[]

Here's my values.yml:

global:
  rbac:
    create: false
    pspEnabled: false

grafana:
  rbac:
    create: false
    pspUseAppArmor: false

kube-state-metrics:
  rbac:
    create: false
  podSecurityPolicy:
    enabled: false

nodeExporter:
  rbac:
    create: false
    pspEnabled: false

As you guys can see, I disabled rbac for the node-exporter but it still fails. So can anybody please explain to me what the issue is? Am I missing something?

EDIT: Layer 8 issue, I got the name of the prometheus-node-exporter chart wrong. Sorry guys. Was able to deploy it :)

[badarou]

Alors que ce problème est toujours dans les limbes - j'ai découvert la solution de contournement suivante (horribad) qui pourrait être utile à quelqu'un:

1. Essayez d'installer l'opérateur normalement:
   helm install stable/prometheus-operator --name monitoring
2. Lorsque cela échoue avec l'erreur mentionnée ci-dessus, accédez à votre fichier values.yaml et définissez-le prometheusOperator.createCustomResourcesurfalse
3. Désinstaller l'opérateur
   helm delete --purge monitoring
4. (Facultatif) Vérifiez que les CRD sont toujours présents: kubectl get crd
5. Avec prometheusOperator.createCustomResourcetoujours défini sur false- essayez de réinstaller l'opérateur:
   helm install stable/stable/prometheus-operator --name monitoring
6. Espérons que PROFIT $$$

I have a problem is quite different.
When I execute the command
kubectl port-forward -n monitoring prometheus-prometheus-operator-prometheus-0 9090 on my cluster Kubernetes (AKS) it gives me this result
Forwarding from 127.0.0.1.19090 -> 9090
Forwarding from [:: 1]: 9090 -> 9090
and when I put the URL http://127.0.0.1:9090/targets on my browser, I have no result.

 $ kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-prometheus-operator-alertmanager-0 2/2 Running 0 11h
prometheus-operator-grafana-8549568bc7-zntgz 2/2 Running 0 11h
prometheus-operator-kube-state-metrics-5fc876d8f6-v5zh7 1/1 Running 0 11h
prometheus-operator-operator-75d88ccc6-t4lt5 1/1 Running 0 11h
prometheus-operator-prometheus-node-export-zqk7t 1/1 Running 0 11h
prometheus-prometheus-operator-prometheus-0 3/3 Running 1 11h

What should be done ? or how to forwarding prometheus from my public cloud to my local machine.
Thank you

[diegomanso]

When trying the workaround described by vsliouniaev above I got another error. I followed the steps and create the CRDs first. But when running the following:

$ helm install -n prometheus-operator --namespace monitoring -f values.yml stable/prometheus-operator --set prometheusOperator.createCustomResource=false
Error: release prometheus-operator failed: clusterroles.rbac.authorization.k8s.io "psp-prometheus-operator-prometheus-node-exporter" is forbidden: attempt to grant extra privileges: [{[use] [extensions] [podsecuritypolicies] [prometheus-operator-prometheus-node-exporter] []}] user=&{system:serviceaccount:kube-system:tiller aa0fd7ff-996b-11e8-adca-0a9e2bcf0986 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[]

Here's my values.yml:

global:
  rbac:
    create: false
    pspEnabled: false

grafana:
  rbac:
    create: false
    pspUseAppArmor: false

kube-state-metrics:
  rbac:
    create: false
  podSecurityPolicy:
    enabled: false

nodeExporter:
  rbac:
    create: false
    pspEnabled: false

As you guys can see, I disabled rbac for the node-exporter but it still fails. So can anybody please explain to me what the issue is? Am I missing something?

EDIT: Layer 8 issue, I got the name of the prometheus-node-exporter chart wrong. Sorry guys. Was able to deploy it :)

Could you please give more details? I got the same error on the node exporter. Thanks in advance

EDIT: I found what you mean, I include this config:

prometheus-node-exporter:
rbac:
create: false
pspEnabled: false