Helm-charts: prometheus-kube-stack unknown fields probenamespaceselector and probeSelector

Created on 23 Oct 2020 · 15Comments · Source: prometheus-community/helm-charts

Describe the bug
Fresh installs of prometheus-kube-stack fail with

error validating "": error validating data: [ValidationError(Prometheus.spec): unknown field "probeNamespaceSelector" in com.coreos.monitoring.v1.Prometheus.spec, ValidationError(Prometheus.spec): unknown field "probeSelector" in com.coreos.monitoring.v1.Prometheus.spec]
helm.go:84: [debug] error validating "": error validating data: [ValidationError(Prometheus.spec): unknown field "probeNamespaceSelector" in com.coreos.monitoring.v1.Prometheus.spec, ValidationError(Prometheus.spec): unknown field "probeSelector" in com.coreos.monitoring.v1.Prometheus.spec]

Version of Helm and Kubernetes:

Helm Version:

$ helm version
version.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}```

Kubernetes Version:

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Which chart: prometheus-kube-stack
Which version of the chart: 10.1.2

What happened:
Attempted to do a fresh helm install of the chart, however it failed with this error:

error validating "": error validating data: [ValidationError(Prometheus.spec): unknown field "probeNamespaceSelector" in com.coreos.monitoring.v1.Prometheus.spec, ValidationError(Prometheus.spec): unknown field "probeSelector" in com.coreos.monitoring.v1.Prometheus.spec]
helm.go:84: [debug] error validating "": error validating data: [ValidationError(Prometheus.spec): unknown field "probeNamespaceSelector" in com.coreos.monitoring.v1.Prometheus.spec, ValidationError(Prometheus.spec): unknown field "probeSelector" in com.coreos.monitoring.v1.Prometheus.spec]

What you expected to happen:
Expected the chart to be deployed with default values

How to reproduce it (as minimally and precisely as possible):
helm install prom-kube -n <YOUR-NAMESPACE> -community/kube-prometheus-stack

Anything else we need to know:
The install works perfectly fine if you specify version 10.1.1, so this definitely seems to be specific to 10.1.2

bug

Source

ikegentz

👍39

Most helpful comment

Can you try to delete the crds before retrying to install the chart?
See https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#uninstall-chart

kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd thanosrulers.monitoring.coreos.com
kubectl delete crd probes.monitoring.coreos.com

irizzant on 26 Oct 2020

👍14 🎉4

All 15 comments

schneifi on 23 Oct 2020

👎2

LuChengTHU on 23 Oct 2020

👎1

I hit the same issue when I tried to upgrade my kube-prometheus-stack production deployment from v10.1.1 to v10.1.2.
But v10.1.2 works in case of fresh installation.
Finally the upgrade passed after updating CRDs on my production deployment

matofeder on 23 Oct 2020

👍2

@matofeder I also tried to create the CRD's manually according to the instructions however I get the same error. Did you create them in a different way?

ikegentz on 23 Oct 2020

As i understood, the problem with the prometheuses.monitoring.coreos.com CRD. I guess yours one doesn't contain the probeNamespaceSelector field. I faced with the same problem. To fix it I updated all the *.monitoring.coreos.com CRDs.

vterdunov on 23 Oct 2020

Can you try to delete the crds before retrying to install the chart?
See https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#uninstall-chart

kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd thanosrulers.monitoring.coreos.com
kubectl delete crd probes.monitoring.coreos.com

irizzant on 26 Oct 2020

👍14 🎉4

Can you try to delete the crds before retrying to install the chart?
See https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#uninstall-chart

kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd thanosrulers.monitoring.coreos.com
kubectl delete crd probes.monitoring.coreos.com

As described in your shared link it is required to import probes CRD and it fixes the issue:

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.42/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml

Just a small note, I am migrated from the old chart prometheus-operator to new prometheus-kube-stack and experienced an issue then my new rules wasn't loaded by Prometheus Operator. To fix this it required to recreate all Prometheus CRDs.

kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd thanosrulers.monitoring.coreos.com
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.42/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.42/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.42/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.42/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.42/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.42/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml

That fixes the observed issue after migrating from the old chart to the new as well as migrating to the new chart version.

maxutlvl on 27 Oct 2020

👍12 🎉5

@maxutlvl Thanks! It's working for me.

vnatarov-mn on 28 Oct 2020

Closing as it's solved...

monotek on 28 Oct 2020

I think this problem still exists. I had a similar issue when upgrading to chart version 10.2.0 kube-prometheus-stack. I solved the problem by following the suggestions made here, i-e deleting the CRDs and recreating them. However today after making some configuration changes to the chart values why I ran the helm upgrade command I got the same error. I had to again delete the CRDs and recreate them and the upgrade command worked after that.
The question is why probeNamespaceSelector field not there when the upgrade command is run? I think this is a bug and the solution suggested in this page is a workaround. There should be no need to delete and create CRDs before every helm upgrade command.

mohammadasim on 29 Oct 2020

👍3

You don't need to delete the CRDs, you can just upgrade CRDs without data loss:

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.42.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.42.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.42.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.42.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.42.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.42.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.42.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml

Viperoo on 2 Nov 2020

👍17

Thanks @Viperoo
I have now written a simple python script that automates this process. https://github.com/mohammadasim/crd_script/blob/master/apply_crd.py

mohammadasim on 12 Nov 2020

I see this issue is closed as fixed but it really isn't.

I had moved to prometheus-kube-stack from helm stable repo (prometheus-operator) and I had issues with CRD's as described here. So while I was upgrading cluster from old to new version this solution works fine. I mean delete old CRD's.

Unfortunately this problem exists even in brand new deployment. I have a code which spins up EKS cluster along with all needed basic resources. This code deploys this helm chart and each time I create brand new EKS cluster my deployment code fails to install Prometheus Operator.

Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Prometheus.spec): unknown field "probeNamespaceSelector" in com.coreos.monitoring.v1.Prometheus.spec, ValidationError(Prometheus.spec): unknown field "probeSelector" in com.coreos.monitoring.v1.Prometheus.spec]

If I then go and delete CRD's in this brand new cluster which helm chart just have failed to deploy (fresh deployment not upgrade) and rerun code it will work like a charm and prometheus will work.

This is weird due to the fact that same helm chart is being used and on first run where there is no CRD's and helm installs them installation will fail. Then delete CRD's which code just have deployed to cluster and run exact same helm chart installation and it's working.

szpuni on 23 Nov 2020

👍3 😕1

I just faced the same issue as @szpuni.

I tried to run the helm chart with:

helm install prometheus-operator prometheus-community/kube-prometheus-stack

And the error occurs:

Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Prometheus.spec): unknown field "probeNamespaceSelector" in com.coreos.monitoring.v1.Prometheus.spec, ValidationError(Prometheus.spec): unknown field "probeSelector" in com.coreos.monitoring.v1.Prometheus.spec]

After deleting all the CRDs:

kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd thanosrulers.monitoring.coreos.com

Then, running helm install command again everything works without any problem.

andre-lx on 8 Dec 2020

I sloved this problem by deleting the old crds,

kubectl delete crd alertmanagerconfigs.monitoring.coreos.com 
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com
kubectl delete crd probes.monitoring.coreos.com
kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd thanosrulers.monitoring.coreos.com

and then use
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack
everything is ok