Kube-prometheus: function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused

Created on 22 Apr 2020  ·  14Comments  ·  Source: prometheus-operator/kube-prometheus

kubectl logs -n monitoring prometheus-k8s-0 prometheus-config-reloader
ts=2020-04-22T06:23:32.511769584Z caller=main.go:85 msg="Starting prometheus-config-reloader version '0.34.0'."
level=info ts=2020-04-22T06:23:32.512471468Z caller=reloader.go:127 msg="started watching config file for changes" in=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml
level=error ts=2020-04-22T06:23:32.514839653Z caller=runutil.go:87 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused"
level=info ts=2020-04-22T06:23:37.57771175Z caller=reloader.go:258 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=

k8s v1.16.0
prome-operator：release-0.4
什么都没改，默认creae的
kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-main ClusterIP 10.110.73.218 9093/TCP 137m
alertmanager-operated ClusterIP None 9093/TCP,9094/TCP,9094/UDP 135m
grafana NodePort 10.97.230.151 3000:30300/TCP 137m
kube-state-metrics ClusterIP None 8443/TCP,9443/TCP 137m
node-exporter ClusterIP None 9100/TCP 137m
prometheus NodePort 10.102.36.249 9090:30011/TCP 102d
prometheus-adapter ClusterIP 10.108.218.180 443/TCP 137m
prometheus-k8s ClusterIP 10.106.158.92 9090/TCP 137m
prometheus-node-exporter ClusterIP None 9100/TCP 102d
prometheus-operated ClusterIP None 9090/TCP 135m
prometheus-operator ClusterIP None 8080/TCP 137m

kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 2/2 Running 0 153m
alertmanager-main-1 2/2 Running 0 153m
alertmanager-main-2 2/2 Running 0 153m
grafana-58dc7468d7-9zpl6 1/1 Running 0 155m
kube-state-metrics-765c7c7f95-gjlpc 3/3 Running 0 155m
node-exporter-qmvcv 2/2 Running 0 155m
node-exporter-sh5j4 2/2 Running 0 155m
node-exporter-xzwtn 2/2 Running 0 155m
prometheus-adapter-5cd5798d96-wd4sv 1/1 Running 0 155m
prometheus-k8s-0 3/3 Running 1 153m
prometheus-k8s-1 3/3 Running 1 153m
prometheus-operator-99dccdc56-nrh7q 1/1 Running 0 155m

help help me

picture SirBlues
👍2

All 14 comments

Can you share the entire logs, because it seems to happen only once?

lilic picture lilic on 22 Apr 2020

@lilic This is the entire log, I can't use Nodeport port access my promethues service, but use the proxy server 9090 command can access, then I from version 0.4 to version 0.3, is ok, I am v1.16.0 k8s version, but Prometheus Prometheus - config - reloader or there is an error, I found 0.3, 0.4, 0.5 version has this problem, but version 0.4 in k8s: use 1.16.0 Nodeport cannot access

这个就是整个日志，我无法用Nodeport端口访问我的promethues服务，但是使用代理9090命令可以访问，后来我从0.4版本降到0.3版本，就可以了，我的k8s版本是v1.16.0，但是prometheus的prometheus-config-reloader还是有错误，我发现0.3,0.4,0.5版本都有这个问题，但是0.4版本在k8s：1.16.0使用nodeport无法访问

kubectl logs -n monitoring prometheus-k8s-0 prometheus-config-reloader
ts=2020-04-22T10:46:39.513852854Z caller=main.go:85 msg="Starting prometheus-config-reloader version '0.34.0'."
level=info ts=2020-04-22T10:46:39.514095179Z caller=reloader.go:127 msg="started watching config file for changes" in=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml
level=error ts=2020-04-22T10:46:39.516670324Z caller=runutil.go:87 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2020-04-22T10:46:44.516202904Z caller=runutil.go:87 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused"
level=info ts=2020-04-22T10:46:49.570987493Z caller=reloader.go:258 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=
level=info ts=2020-04-23T01:58:27.63198799Z caller=reloader.go:258 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=
level=info ts=2020-04-23T02:01:49.630752242Z caller=reloader.go:258 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=
level=info ts=2020-04-23T02:07:49.628832432Z caller=reloader.go:258 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=
level=info ts=2020-04-23T02:13:49.633836103Z caller=reloader.go:258 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=

release-0.3 prometheus-config-reloader all log

SirBlues picture SirBlues on 23 Apr 2020

@SirBlues I meet same issue too, did you fixed it ?

ne1000 picture ne1000 on 26 Aug 2020
👍1

-config-reloader还是有错误，我发现0.3,0.4,0.5版本都有这个问题，但是0.4版本在k8s：1.16.0使用nodeport无法访问

kubectl logs -n monitoring prometheus-k8s-0 prometheus-config-reloader

Can you describe your problem with English?

feixiaohuijava picture feixiaohuijava on 9 Sep 2020
👍1

I'm experiencing the same issue. I tried deploying various versions of https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack chart. Every time I end up with the prometheus statefulset pod dying in a loop.

kc get pods

alertmanager-kube-prometheus-kube-prome-alertmanager-0   0/2     Terminating         0          4s
filebeat-filebeat-75k9g                                  1/1     Running             0          20h
filebeat-filebeat-7ndgr                                  1/1     Running             0          20h
filebeat-filebeat-8j44v                                  1/1     Running             0          20h
filebeat-filebeat-bwtxf                                  1/1     Running             0          20h
kube-prometheus-grafana-66b78d469c-xffmr                 2/2     Running             0          2m31s
kube-prometheus-kube-prome-operator-578494b6cb-fbx7b     2/2     Running             0          2m31s
kube-prometheus-kube-state-metrics-78d7cbdccd-6vdzr      1/1     Running             0          2m32s
kube-prometheus-prometheus-node-exporter-79m7q           1/1     Running             0          2m31s
kube-prometheus-prometheus-node-exporter-phprq           1/1     Running             0          2m31s
kube-prometheus-prometheus-node-exporter-svd5v           1/1     Running             0          2m32s
kube-prometheus-prometheus-node-exporter-wtdln           1/1     Running             0          2m31s
prometheus-kube-prometheus-kube-prome-prometheus-0       0/3     ContainerCreating   0          1s  # This keeps restarting

I also noticed that one of the configmaps keeps recreating
kc get configmap

kube-prometheus-grafana                                        1      7m48s
kube-prometheus-grafana-config-dashboards                      1      7m48s
kube-prometheus-grafana-test                                   1      7m48s
kube-prometheus-kube-prome-apiserver                           1      7m48s
kube-prometheus-kube-prome-cluster-total                       1      7m48s
kube-prometheus-kube-prome-controller-manager                  1      7m48s
kube-prometheus-kube-prome-etcd                                1      7m48s
kube-prometheus-kube-prome-grafana-datasource                  1      7m48s
kube-prometheus-kube-prome-k8s-coredns                         1      7m48s
kube-prometheus-kube-prome-k8s-resources-cluster               1      7m48s
kube-prometheus-kube-prome-k8s-resources-namespace             1      7m48s
kube-prometheus-kube-prome-k8s-resources-node                  1      7m48s
kube-prometheus-kube-prome-k8s-resources-pod                   1      7m48s
kube-prometheus-kube-prome-k8s-resources-workload              1      7m48s
kube-prometheus-kube-prome-k8s-resources-workloads-namespace   1      7m48s
kube-prometheus-kube-prome-kubelet                             1      7m48s
kube-prometheus-kube-prome-namespace-by-pod                    1      7m48s
kube-prometheus-kube-prome-namespace-by-workload               1      7m48s
kube-prometheus-kube-prome-node-cluster-rsrc-use               1      7m48s
kube-prometheus-kube-prome-node-rsrc-use                       1      7m48s
kube-prometheus-kube-prome-nodes                               1      7m48s
kube-prometheus-kube-prome-persistentvolumesusage              1      7m48s
kube-prometheus-kube-prome-pod-total                           1      7m48s
kube-prometheus-kube-prome-prometheus                          1      7m48s
kube-prometheus-kube-prome-proxy                               1      7m48s
kube-prometheus-kube-prome-scheduler                           1      7m48s
kube-prometheus-kube-prome-statefulset                         1      7m48s
kube-prometheus-kube-prome-workload-total                      1      7m48s
prometheus-kube-prometheus-kube-prome-prometheus-rulefiles-0   26     7m46s # Keeps getting recreated

After looking at the logs I found:

level=info ts=2020-11-20T10:08:19.006624264Z caller=main.go:147 msg="Starting prometheus-config-reloader" version="(version=0.43.2, branch=refs/tags/v0.43.2, revision=b86ab77239f2a11ee69ad05b24122958d8b2df5b)"
10:08:19.014
level=info ts=2020-11-20T10:08:19.006765857Z caller=main.go:148 build_context="(go=go1.14.10, user=simonpasquier, date=20201109-10:56:57)"
10:08:19.014
level=info ts=2020-11-20T10:08:19.00707087Z caller=main.go:182 msg="Starting web server for metrics" listen=:8080
10:08:19.107
level=error ts=2020-11-20T10:08:19.101732526Z caller=runutil.go:98 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://127.0.0.1:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
10:10:29.556
level=error ts=2020-11-20T10:10:29.554Z caller=main.go:290 msg="Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)" err="open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"

More logs:

10:28:10.806
level=info ts=2020-11-20T10:28:10.804Z caller=kubernetes.go:253 component="discovery manager notify" discovery=k8s msg="Using pod service account via in-cluster config"
10:28:10.988
level=info ts=2020-11-20T10:28:10.988Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
10:28:10.989
level=info ts=2020-11-20T10:28:10.988954918Z caller=reloader.go:289 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=
10:28:10.989
level=info ts=2020-11-20T10:28:10.989078889Z caller=reloader.go:157 msg="started watching config file and non-recursively rule dirs for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=
10:28:10.994
level=info ts=2020-11-20T10:28:10.993Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
10:28:11.000
level=info ts=2020-11-20T10:28:11.000Z caller=kubernetes.go:253 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
10:28:11.002
level=info ts=2020-11-20T10:28:11.002Z caller=kubernetes.go:253 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
10:28:11.005
level=info ts=2020-11-20T10:28:11.005Z caller=kubernetes.go:253 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
10:28:11.008
level=info ts=2020-11-20T10:28:11.007Z caller=kubernetes.go:253 component="discovery manager notify" discovery=k8s msg="Using pod service account via in-cluster config"
10:28:11.227
level=info ts=2020-11-20T10:28:11.223Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
10:28:11.228
level=info ts=2020-11-20T10:28:11.224345767Z caller=reloader.go:289 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=
10:43:47.611
level=info ts=2020-11-20T10:43:47.610Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.2, branch=HEAD, revision=a6600f564e3c483cc820bae6c7a551db701a22b3)"
10:43:47.611
level=info ts=2020-11-20T10:43:47.610Z caller=main.go:338 build_context="(go=go1.14.4, user=root@130a411dd4ff, date=20200609-09:05:58)"
10:43:47.611
level=info ts=2020-11-20T10:43:47.610Z caller=main.go:339 host_details="(Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 prometheus-kube-prometheus-kube-prome-prometheus-0 (none))"
10:43:47.611
level=info ts=2020-11-20T10:43:47.611Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)"
10:43:47.612
level=info ts=2020-11-20T10:43:47.611Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)"
10:43:47.616
level=info ts=2020-11-20T10:43:47.614Z caller=main.go:678 msg="Starting TSDB ..."
10:43:47.616
level=info ts=2020-11-20T10:43:47.614Z caller=web.go:523 component=web msg="Start listening for connections" address=0.0.0.0:9090
10:43:47.622
level=info ts=2020-11-20T10:43:47.621Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile"
10:43:47.624
level=info ts=2020-11-20T10:43:47.624Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
10:43:47.624
level=info ts=2020-11-20T10:43:47.624Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=2.293438ms
10:43:47.626
level=info ts=2020-11-20T10:43:47.626Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC
10:43:47.626
level=info ts=2020-11-20T10:43:47.626Z caller=main.go:695 msg="TSDB started"
10:43:47.626
level=info ts=2020-11-20T10:43:47.626Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
10:43:47.626
level=info ts=2020-11-20T10:43:47.626Z caller=main.go:547 msg="Stopping scrape discovery manager..."
10:43:47.626
level=info ts=2020-11-20T10:43:47.626Z caller=main.go:561 msg="Stopping notify discovery manager..."
10:43:47.626
level=info ts=2020-11-20T10:43:47.626Z caller=main.go:583 msg="Stopping scrape manager..."
10:43:47.626
level=info ts=2020-11-20T10:43:47.626Z caller=main.go:557 msg="Notify discovery manager stopped"
10:43:47.626
level=info ts=2020-11-20T10:43:47.626Z caller=main.go:543 msg="Scrape discovery manager stopped"
10:43:47.626
level=info ts=2020-11-20T10:43:47.626Z caller=main.go:577 msg="Scrape manager stopped"
10:43:47.626
level=info ts=2020-11-20T10:43:47.626Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..."
10:43:47.626
level=info ts=2020-11-20T10:43:47.626Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped"
10:43:47.632
level=info ts=2020-11-20T10:43:47.631Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..."
10:43:47.632
level=info ts=2020-11-20T10:43:47.631Z caller=main.go:749 msg="Notifier manager stopped"
10:43:47.632
level=error ts=2020-11-20T10:43:47.631Z caller=main.go:758 err="error loading config from \"/etc/prometheus/config_out/prometheus.env.yaml\": couldn't load configuration (--config.file=\"/etc/prometheus/config_out/prometheus.env.yaml\"): open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"
10:43:47.807
ts=2020-11-20T10:43:47.804259552Z caller=main.go:85 msg="Starting prometheus-config-reloader version '0.38.1'."
10:43:47.820
level=error ts=2020-11-20T10:43:47.819299697Z caller=runutil.go:98 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://127.0.0.1:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused"
10:43:48.073
2020/11/20 10:43:48 Watching directory: "/etc/prometheus/rules/prometheus-kube-prometheus-kube-prome-prometheus-rulefiles-0"
d-baranowski picture d-baranowski on 20 Nov 2020

@d-baranowski can you share logs from the prometheus-operator?

simonpasquier picture simonpasquier on 20 Nov 2020

Also if I describe the pod I get some interesting events...

kc describe pod prometheus-kube-prometheus-kube-prome-prometheus-0                                                                                                                                                        [10:50:07]
Name:                      prometheus-kube-prometheus-kube-prome-prometheus-0
Namespace:                 monitoring
Priority:                  0
Node:                      rks-test-03/10.0.15.33
Start Time:                Fri, 20 Nov 2020 11:00:46 +0000
Labels:                    app=prometheus
                           controller-revision-hash=prometheus-kube-prometheus-kube-prome-prometheus-654b8874c4
                           prometheus=kube-prometheus-kube-prome-prometheus
                           statefulset.kubernetes.io/pod-name=prometheus-kube-prometheus-kube-prome-prometheus-0
Annotations:               cni.projectcalico.org/podIP: 10.42.2.195/32
                           cni.projectcalico.org/podIPs: 10.42.2.195/32
Status:                    Terminating (lasts <invalid>)
Termination Grace Period:  600s
IP:                        
IPs:                       <none>
Controlled By:             StatefulSet/prometheus-kube-prometheus-kube-prome-prometheus
Containers:
  prometheus:
    Container ID:  
    Image:         quay.io/prometheus/prometheus:v2.18.2
    Image ID:      
    Port:          9090/TCP
    Host Port:     0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --storage.tsdb.retention.time=10d
      --web.enable-lifecycle
      --storage.tsdb.no-lockfile
      --web.external-url=http://kube-prometheus-kube-prome-prometheus.monitoring:9090
      --web.route-prefix=/
    State:          Terminated
      Exit Code:    0
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:      http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=120
    Environment:    <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-kube-prometheus-kube-prome-prometheus-rulefiles-0 from prometheus-kube-prometheus-kube-prome-prometheus-rulefiles-0 (rw)
      /prometheus from prometheus-kube-prometheus-kube-prome-prometheus-db (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-prometheus-kube-prome-prometheus-token-2gjdk (ro)
  prometheus-config-reloader:
    Container ID:  
    Image:         quay.io/coreos/prometheus-config-reloader:v0.38.1
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/prometheus-config-reloader
    Args:
      --log-format=logfmt
      --reload-url=http://127.0.0.1:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
    State:          Terminated
      Exit Code:    0
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  25Mi
    Requests:
      cpu:     100m
      memory:  25Mi
    Environment:
      POD_NAME:  prometheus-kube-prometheus-kube-prome-prometheus-0 (v1:metadata.name)
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-prometheus-kube-prome-prometheus-token-2gjdk (ro)
  rules-configmap-reloader:
    Container ID:  
    Image:         docker.io/jimmidyson/configmap-reload:v0.3.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      --webhook-url=http://127.0.0.1:9090/-/reload
      --volume-dir=/etc/prometheus/rules/prometheus-kube-prometheus-kube-prome-prometheus-rulefiles-0
    State:          Terminated
      Exit Code:    0
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  25Mi
    Requests:
      cpu:        100m
      memory:     25Mi
    Environment:  <none>
    Mounts:
      /etc/prometheus/rules/prometheus-kube-prometheus-kube-prome-prometheus-rulefiles-0 from prometheus-kube-prometheus-kube-prome-prometheus-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-prometheus-kube-prome-prometheus-token-2gjdk (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-kube-prometheus-kube-prome-prometheus
    Optional:    false
  tls-assets:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-kube-prometheus-kube-prome-prometheus-tls-assets
    Optional:    false
  config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  prometheus-kube-prometheus-kube-prome-prometheus-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-kube-prometheus-kube-prome-prometheus-rulefiles-0
    Optional:  false
  prometheus-kube-prometheus-kube-prome-prometheus-db:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-prometheus-kube-prome-prometheus-token-2gjdk:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kube-prometheus-kube-prome-prometheus-token-2gjdk
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age   From                  Message
  ----     ------     ----  ----                  -------
  Normal   Scheduled  3s    default-scheduler     Successfully assigned monitoring/prometheus-kube-prometheus-kube-prome-prometheus-0 to rks-test-03
  Normal   Pulled     1s    kubelet, rks-test-03  Container image "quay.io/prometheus/prometheus:v2.18.2" already present on machine
  Warning  Failed     1s    kubelet, rks-test-03  Error: cannot find volume "config-out" to mount into container "prometheus"
  Normal   Pulled     1s    kubelet, rks-test-03  Container image "quay.io/coreos/prometheus-config-reloader:v0.38.1" already present on machine
  Warning  Failed     1s    kubelet, rks-test-03  Error: cannot find volume "config" to mount into container "prometheus-config-reloader"
  Normal   Pulled     1s    kubelet, rks-test-03  Container image "docker.io/jimmidyson/configmap-reload:v0.3.0" already present on machine
  Warning  Failed     1s    kubelet, rks-test-03  Error: cannot find volume "prometheus-kube-prometheus-kube-prome-prometheus-rulefiles-0" to mount into container "rules-configmap-reloader"
d-baranowski picture d-baranowski on 20 Nov 2020

kc logs -f kube-prometheus-kube-prome-operator-578494b6cb-fbx7b -c kube-prometheus-stack 

E1120 10:33:47.682067       1 operator.go:322] Sync "monitoring/kube-prometheus-kube-prome-alertmanager" failed: synchronizing governing service failed: updating service object failed: Operation cannot be fulfilled on services "alertmanager-operated": the object has been modified; please apply your changes to the latest version and try again
level=info ts=2020-11-20T10:33:47.691183701Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
level=info ts=2020-11-20T10:33:47.715892267Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus-kube-prome-prometheus
level=info ts=2020-11-20T10:33:47.834049362Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
level=info ts=2020-11-20T10:33:47.939659874Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
level=info ts=2020-11-20T10:33:48.023284131Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
level=info ts=2020-11-20T10:33:48.303548464Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
level=info ts=2020-11-20T10:33:48.443056541Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
level=info ts=2020-11-20T10:33:48.451370776Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=cattle-prometheus/cluster-monitoring
level=info ts=2020-11-20T10:33:48.512164332Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
level=info ts=2020-11-20T10:33:48.587791834Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
E1120 10:33:48.636835       1 operator.go:322] Sync "monitoring/kube-prometheus-kube-prome-alertmanager" failed: synchronizing governing service failed: updating service object failed: Operation cannot be fulfilled on services "alertmanager-operated": the object has been modified; please apply your changes to the latest version and try again
level=info ts=2020-11-20T10:33:48.643000394Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
level=info ts=2020-11-20T10:33:48.762942419Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
level=info ts=2020-11-20T10:33:48.783722805Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus-kube-prome-prometheus
level=info ts=2020-11-20T10:33:48.81772102Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
level=info ts=2020-11-20T10:33:49.200282134Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
level=info ts=2020-11-20T10:33:49.339551889Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
level=info ts=2020-11-20T10:33:49.435894431Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
level=info ts=2020-11-20T10:33:49.50323647Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus-kube-prome-prometheus
level=info ts=2020-11-20T10:33:49.534289188Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
level=info ts=2020-11-20T10:33:49.614518759Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanag^Corbidden,Message:Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden,Field:spec,},},RetryAfterSeconds:0,UID:,}"
level=info ts=2020-11-20T10:33:49.876688224Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome-alertmanager
level=info ts=2020-11-20T10:33:49.914674476Z caller=operator.go:497 component=alertmanageroperator msg="resolving illegal update of Alertmanager StatefulSet" details="&StatusDetails{Name:alertmanager-kube-prometheus-kube-prome-alertmanager,Group:apps,Kind:StatefulSet,Causes:[]StatusCause{StatusCause{Type:FieldValueForbidden,Message:Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden,Field:spec,},},RetryAfterSeconds:0,UID:,}"
level=info ts=2020-11-20T10:33:49.937338258Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus-kube-prome
d-baranowski picture d-baranowski on 20 Nov 2020

Looks like the statefulset itself gets recreated by prometheus operator which matches the problem mentioned in this issue: https://github.com/helm/charts/issues/21828

It's because there were multiple prometheus operators interfering with each other

We've disabled rancher monitoring which deleted it's prometheus operator and now it's working fine

d-baranowski picture d-baranowski on 20 Nov 2020

As far as I can tell from the pasted logs, the error message is transient and the situation resolves itself. Feel free to reopen with more information if it isn't the case.

simonpasquier picture simonpasquier on 20 Nov 2020
👍1

As soon as the second prom operator is deleted the situation resolves itself

d-baranowski picture d-baranowski on 20 Nov 2020

@d-baranowski - How did you find out or check if there were 2 prometheus operators? I am having similar issue, and still looking into on how to resolve it

laurentiuspurba picture laurentiuspurba on 5 Apr 2021

@laurentiuspurba You can do something like this kubectl get pods --all-namespaces | grep operator to fetch all the pods in all namespaces and filter the result to lines which include the string 'operator' for example result could be

flux              helm-operator-5f7d7cc6f4-rmnss                           1/1     Running     0          31d
prometheus        kube-prometheus-kube-prome-operator-5d4c4986d5-vqvft     1/1     Running     0          24d

You can then see if there is more than one prometheus operator running on your cluster.

d-baranowski picture d-baranowski on 5 Apr 2021

@d-baranowski Yes, I got that before your response. I really appreciate your help.

I did something similar

▶ kubectl get pods --all-namespaces | grep prome
prometheus                                    cert-exporter-cert-manager-67c8ffb99d-wn6ps                       1/1     Running                 0          35d
prometheus                                    prometheus-prometheus-oper-operator-766c496857-cwlvm              1/1     Running                 0          19h
prometheus                                    prometheus-prometheus-prometheus-oper-prometheus-0                0/4     Terminating             0          4m41s
sumologic                                     collection-prometheus-oper-operator-7c5cc7749-bcjzg               1/1     Running                 0          34d

From here, I saw there were 2 prometheus-oper-operator, one was on prometheus ns, while the other on sumolgic ns.

laurentiuspurba picture laurentiuspurba on 5 Apr 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

tewing-riffyn picture tewing-riffyn  ·  4Comments
DimitrijeManic picture DimitrijeManic  ·  3Comments
KlavsKlavsen picture KlavsKlavsen  ·  3Comments
rupadhya1 picture rupadhya1  ·  3Comments
Aracki picture Aracki  ·  6Comments