Charts: [table/prometheus-operator] Cannot upgrade after rules change ("prometheusrulemutate.monitoring.coreos.com": the server could not find the requested resource)

[fbueno]

Describe the bug
I'm trying to upgrade my prometheus-operator release with only a few (for time changes) on helm. Every time I run helm upgrade --namespace monitoring prometheus-operator stable/prometheus-operator -f ./values.yaml

UPGRADE FAILED
ROLLING BACK
Error: failed to create resource: Internal error occurred: failed calling admission webhook "prometheusrulemutate.monitoring.coreos.com": the server could not find the requested resource
Error: UPGRADE FAILED: failed to create resource: Internal error occurred: failed calling admission webhook "prometheusrulemutate.monitoring.coreos.com": the server could not find the requested resource

If I keep the values.yml file unchanged I get no error.

Version of Helm and Kubernetes:

helm version
Client: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"ab91afd7062d4240e95e51ac00a18bd58fddd365", GitTreeState:"clean", BuildDate:"2019-02-26T12:59:46Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.10-eks-825e5d", GitCommit:"825e5de08cb05714f9b224cd6c47d9514df1d1a7", GitTreeState:"clean", BuildDate:"2019-08-18T03:58:32Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}



md5-66a4134fe98ac188413d0102963030ec



-          for: 30m
+          for: 10m



md5-6f877353238d4275bc8703a26308f9cf



-          for: 20m
+          for: 5m



md5-2731db5eadbaa6827b8cc3abe5300b10



prometheus-operator     74              Mon Oct  7 19:48:29 2019        FAILED          prometheus-operator-6.7.3       0.31.1          monitoring

Change alert rules in your values.yml file.
Apply the upgrade.

Anything else we need to know:
I tried setting --set prometheusOperator.admissionWebhooks.enabled=false and even removing or editting ValidatingWebhookConfiguration prometheus-operator-admission and the prometheus-prometheus-operator-prometheus-rulefiles-0 configmap.
I always got the same.

[bood]

Try the --force flag to let helm recreate necessary resources

[fbueno]

@bood It didn't help.
I got the same message.
I tried downloading the chart (my current version, previous version or latest version) removing the webhooks templates hard setting enable: false to prometheus-operator and the webhook. I got the same message every time.

[fbueno]

Ok, more information, I deleted --purge it and installed it again (with the same PVCs).
helm upgrade --install --namespace monitoring prometheus-operator stable/prometheus-operator -f ./values.yaml --install --version 6.13.0

Then I changed only one expression on a custom rule. It was > 20 I added > 1.
And got the same message:

helm upgrade --namespace monitoring prometheus-operator stable/prometheus-operator -f ./values.yaml --version 6.13.0
UPGRADE FAILED
ROLLING BACK
Error: Internal error occurred: failed calling admission webhook "prometheusrulemutate.monitoring.coreos.com": the server could not find the requested resource
Error: UPGRADE FAILED: Internal error occurred: failed calling admission webhook "prometheusrulemutate.monitoring.coreos.com": the server could not find the requested resource

[bood]

I'm using helm3 for prometheus-operator now. I do remember seeing some kind of error on helm2 when upgrading too (I was using helm v2.11.0 but I didn't record the error message).

I suggest you try: upgrade helm2 to latest version or use helm3 instead.

[fbueno]

Hmm. It's still beta, right?
Plus, I would have to upgrade several installations (releases). Anyway, I'm gonna take that into consideration (thank you). But will keep this open for a while.

[bobbui]

still happening even upgrade to helm 2.16

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[BahmaniAlireza]

if you have run before without disabling the webhook, you must manually delete the following kinds:

kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io //delete all objects
kubectl get MutatingWebhookConfiguration //delete all objects

and after that run:

helm install --name prometheus-operator stable/prometheus-operator
--set prometheusOperator.admissionWebhooks.enabled=false
--set prometheusOperator.admissionWebhooks.patch.enabled=false
--set prometheusOperator.tlsProxy.enabled=false