Helm-charts: [kube-prometheus-stack] helm upgrade --install runs with errors after first time

[yosefy]

Describe the bug
A clear and concise description of what the bug is.

Version of Helm and Kubernetes:

Helm Version:

$ helm version
version.BuildInfo{Version:"v3.3.1", GitCommit:"249e5215cde0c3fa72e27eb7a30e8d55c9696144", GitTreeState:"dirty", GoVersion:"go1.15"}

Kubernetes Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T23:42:50Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.12-gke.2", GitCommit:"fb7add51f767aae42655d39972210dc1c5dbd4b3", GitTreeState:"clean", BuildDate:"2020-06-01T22:20:10Z", GoVersion:"go1.12.17b4", Compiler:"gc", Platform:"linux/amd64"}

Which chart:
prometheus-kube-stack
Which version of the chart:

What happened:
running
helm --debug --stderrthreshold 1 upgrade --install prometheus prometheus-community/kube-prometheus-stack -f values.yaml

first time runs ok but when running second time, with or without changes, we get this type of errors about rules

client.go:436: [debug] Looks like there are no changes for Service "prometheus-kube-prometheus-alertmanager"
client.go:436: [debug] Looks like there are no changes for Service "prometheus-kube-prometheus-kube-controller-manager"
client.go:436: [debug] Looks like there are no changes for Service "prometheus-kube-prometheus-kube-etcd"
client.go:436: [debug] Looks like there are no changes for Service "prometheus-kube-prometheus-kube-proxy"
client.go:436: [debug] Looks like there are no changes for Service "prometheus-kube-prometheus-kube-scheduler"
client.go:436: [debug] Looks like there are no changes for Service "prometheus-kube-prometheus-operator"
client.go:436: [debug] Looks like there are no changes for Service "prometheus-kube-prometheus-prometheus"
client.go:436: [debug] Looks like there are no changes for DaemonSet "prometheus-prometheus-node-exporter"
client.go:436: [debug] Looks like there are no changes for Deployment "prometheus-kube-prometheus-operator"
client.go:205: [debug] error updating the resource "prometheus-kube-prometheus-alertmanager.rules":
cannot patch "prometheus-kube-prometheus-alertmanager.rules" with kind PrometheusRule: Timeout: request did not complete within requested timeout 30s
client.go:205: [debug] error updating the resource "prometheus-kube-prometheus-etcd":
cannot patch "prometheus-kube-prometheus-etcd" with kind PrometheusRule: Timeout: request did not complete within requested timeout 30s
client.go:205: [debug] error updating the resource "prometheus-kube-prometheus-general.rules":
cannot patch "prometheus-kube-prometheus-general.rules" with kind PrometheusRule: Timeout: request did not complete within requested timeout 30s
client.go:205: [debug] error updating the resource "prometheus-kube-prometheus-k8s.rules":
cannot patch "prometheus-kube-prometheus-k8s.rules" with kind PrometheusRule: Timeout: request did not complete within requested timeout 30s
client.go:205: [debug] error updating the resource "prometheus-kube-prometheus-kube-apiserver-availability.rules":

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

helm --debug --stderrthreshold 1 upgrade --install prometheus prometheus-community/kube-prometheus-stack -f values.yaml
<~--

rules are only default from the chart

thanks

[fktkrt]

I've encountered a similar situation with the legacy chart earlier. Are your admissionWebhooks enabled?
Can you try to delete the existing MutatingWebhookConfiguration and ValidatingWebhookConfiguration related to this chart, and try the upgrade again?

[yashmehrotra]

Facing the same issue, any help would be appreciated

[fktkrt]

@yashmehrotra, @yosefy could you please check whether the comment above is helpful in solving your issue?

[yashmehrotra]

@fktkrt Sorry for the late reply, so by default prometheusOperator.admissionWebhooks.enabled was set to true. And with this config, it was timing out.

So, I changed that to false, removed the existing MutatingWebhookConfiguration and ValidatingWebhookConfiguration and it worked as expected 😃

Do you know the reason for this though ? Could this be caused by a race condition ? Seems pretty weird that this is the default behavior

Thanks for your help though !

[ddides]

I was seeing the same errors and the workaround also allowed me to successfully redeploy an updated configuration.

[fktkrt]

I'm glad it helped.

@yashmehrotra Unfortunately I'm not sure about the root cause. Maybe when you created a a new PrometheusRule the admissionWebhook was trying to communicate with the removed prometheus-operator?

[yosefy]

i can confirm that it helps

On Wed, Oct 7, 2020 at 3:38 PM fktkrt [e-mail] wrote:

I'm glad it helped.

@yashmehrotra https://github.com/yashmehrotra Unfortunately I'm not
sure about the root cause. Maybe when you created a a new PrometheusRule
the admissionWebhook was trying to communicate with the removed
prometheus-operator?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/prometheus-community/helm-charts/issues/142#issuecomment-704906634,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADCXF3M4O6NU52SYGJEUS7LSJRONFANCNFSM4RYIKJ7Q
.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[morremeyer]

I verified that this is happening when deploying the chart with the default configuration, so this is definitely an issue that we need to look into.

[gw0]

I can confirm this is happening on GKE private clusters with Helm v3. With default values the initial deployment works, but subsequent upgrades fail with OP timeout issue (unless you disable the webhook with prometheusOperator.admissionWebhooks.enabled=false as suggested by @fktkrt). With the disabled webhook the initial deployment timeouts waiting for tls-secret volume (unless you provide it manually).

I found an analysis of this issue (from an older version where tls-proxy was a separate container). The author identified and I can confirm that prometheus-operator webhook is using port 8443 and it needs to be allowed on the firewall between GKE masters and GKE pods in order to not timeout.

• https://managedkube.com/prometheus-operator/gke/update/2020/04/12/prometheus-operator-gke-update.html

A known workaround (used by some Helm charts) is to allow specifying the webhook port and set it to 10250 which is the only non-privileged port allowed by the GKE firewall by default.

I prepared a PR: https://github.com/prometheus-community/helm-charts/pull/400. This issue should be fixed if you set prometheusOperator.tls.internalPort=10250.

[fktkrt]

I guess this is not cloud specific, I have seen multiple similar reports from EKS clusters as well.

[gw0]

I managed to recreate this issue in a GKE private cluster by forcing the change of PrometheusRules by changing labels.

Steps to reproduce on a GKE private cluster:

$ helm version
version.BuildInfo{Version:"v3.1.3", GitCommit:"0a9a9a88e8afd6e77337a3e2ad744756e191429a", GitTreeState:"clean", GoVersion:"go1.13.10"}
$ kubectl version
hClient Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.15", GitCommit:"2adc8d7091e89b6e3ca8d048140618ec89b39369", GitTreeState:"clean", BuildDate:"2020-09-02T11:40:00Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-gke.404", GitCommit:"81e2afbf51b876d15f0af546998c670207fdd592", GitTreeState:"clean", BuildDate:"2020-10-30T23:52:44Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

## initial install SUCCEEDS
$ helm upgrade --install tmp-monitoring4 prometheus-community/kube-prometheus-stack --version 12.3.0 --set prometheusOperator.enabled=true --set prometheus.enabled=false --set grafana.enabled=false --set alertmanager.enabled=false --set defaultRules.create=true --set defaultRules.labels.foo=bar1 --debug

## upgrade FAILS if you change a PrometheusRule (e.g. add a label)
$ helm upgrade --install tmp-monitoring4 prometheus-community/kube-prometheus-stack --version 12.3.0 --set prometheusOperator.enabled=true --set prometheus.enabled=false --set grafana.enabled=false --set alertmanager.enabled=false --set defaultRules.create=true --set defaultRules.labels.foo=bar2 --debug
...
client.go:195: [debug] error updating the resource "tmp-monitoring4-kube-prome-alertmanager.rules":
     cannot patch "tmp-monitoring4-kube-prome-alertmanager.rules" with kind PrometheusRule: Timeout: request did not complete within requested timeout 30s
client.go:195: [debug] error updating the resource "tmp-monitoring4-kube-prome-etcd":
     cannot patch "tmp-monitoring4-kube-prome-etcd" with kind PrometheusRule: Timeout: request did not complete within requested timeout 30s
...

## same happens with --force

## upgrade SUCCEEDS with PR #400 and internalPort=10250
$ helm upgrade --install tmp-monitoring4 /prometheus-helm-charts/charts/kube-prometheus-stack/ --set prometheusOperator.enabled=true --set prometheusOperator.tls.internalPort=10250 --set prometheus.enabled=false --set grafana.enabled=false --set alertmanager.enabled=false --set defaultRules.create=true --set defaultRules.labels.foo=bar3 --debug

My hypothesis is that this occurred spontaneously due to upgrading Kubernetes to 1.16, upgrading from an older kube-prometheus-stack chart 10.x to 11.x, or enabling/disabling the creation of PrometheusRules.

[viorelflorian]

Had the same issue on upgrade (install worked fine).

@gw0 is somewhat correct and pointed us in the right direction.

If you intend to deploy the stack behind a reverse proxy (with tls termination) and operate only within the Kubernetes cluster, then you should not have any tls enabled within the default values, but unfortunately for us (and some of you) the default values file has this value set -> _prometheusOperator.tls.enabled=true_

Set that to false and keep upgrading to your hearts content.

Happy helming ! :)