Charts: [stable/prometheus] Error loading config couldn't load configuration

[kikulikov]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Version of Helm and Kubernetes:

Helm

Client: &version.Version{SemVer:"v2.9.0", GitCommit:"f6025bb9ee7daf9fee0026541c90a6f557a3e0bc", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.0", GitCommit:"f6025bb9ee7daf9fee0026541c90a6f557a3e0bc", GitTreeState:"clean"}

Kubernetes

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-13T22:29:03Z", GoVersion:"go1.9.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

Which chart:

stable/prometheus

What happened:

prometheus-server doesn't start (seems that it fails to read the config)

level=info ts=2018-04-27T11:49:24.130532394Z caller=main.go:220 msg="Starting Prometheus" version="(version=2.2.1, branch=HEAD, revision=bc6058c81272a8d938c05e75607371284236aadc)"
level=info ts=2018-04-27T11:49:24.130584536Z caller=main.go:221 build_context="(go=go1.10, user=root@149e5b3f0829, date=20180314-14:15:45)"
level=info ts=2018-04-27T11:49:24.130602575Z caller=main.go:222 host_details="(Linux 4.4.115-k8s #1 SMP Thu Feb 8 15:37:40 UTC 2018 x86_64 monitoring-prometheus-server-66b965544c-4sdnz (none))"
level=info ts=2018-04-27T11:49:24.130617119Z caller=main.go:223 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2018-04-27T11:49:24.13426692Z caller=main.go:504 msg="Starting TSDB ..."
level=info ts=2018-04-27T11:49:24.134278401Z caller=web.go:382 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-04-27T11:49:24.141836717Z caller=main.go:514 msg="TSDB started"
level=info ts=2018-04-27T11:49:24.141878766Z caller=main.go:588 msg="Loading configuration file" filename=/etc/config/prometheus.yml
level=info ts=2018-04-27T11:49:24.142813758Z caller=main.go:398 msg="Stopping scrape discovery manager..."
level=info ts=2018-04-27T11:49:24.142869442Z caller=main.go:411 msg="Stopping notify discovery manager..."
level=info ts=2018-04-27T11:49:24.142886024Z caller=main.go:432 msg="Stopping scrape manager..."
level=info ts=2018-04-27T11:49:24.142972217Z caller=manager.go:460 component="rule manager" msg="Stopping rule manager..."
level=info ts=2018-04-27T11:49:24.142994119Z caller=manager.go:466 component="rule manager" msg="Rule manager stopped"
level=info ts=2018-04-27T11:49:24.143003421Z caller=notifier.go:512 component=notifier msg="Stopping notification manager..."
level=info ts=2018-04-27T11:49:24.143030843Z caller=main.go:394 msg="Scrape discovery manager stopped"
level=info ts=2018-04-27T11:49:24.143049653Z caller=main.go:407 msg="Notify discovery manager stopped"
level=info ts=2018-04-27T11:49:24.143062485Z caller=main.go:426 msg="Scrape manager stopped"
level=info ts=2018-04-27T11:49:24.143078552Z caller=main.go:573 msg="Notifier manager stopped"
level=error ts=2018-04-27T11:49:24.143958459Z caller=main.go:582 err="Error loading config couldn't load configuration (--config.file=/etc/config/prometheus.yml): parsing YAML file /etc/config/prometheus.yml: yaml: line 160: mapping values are not allowed in this context"
level=info ts=2018-04-27T11:49:24.144010502Z caller=main.go:584 msg="See you next time!"

What you expected to happen:

prometheus-server reads the config and starts correctly

How to reproduce it (as minimally and precisely as possible):

helm install --namespace monitoring --name monitoring stable/prometheus

Anything else we need to know:

[georgepapas]

I encountered the same problem, looks like its caused by the latest helm update to 2.9.0

https://github.com/kubernetes/helm/issues/3981

[kikulikov]

@georgepapas Have you found any fix/workaround (except downgrading helm)?

[georgepapas]

@cyrillk No, I downgraded to the previous version. It is planned to be fixed in 2.9.1. See this PR

https://github.com/kubernetes/helm/pull/3988

[hoeghh]

The problem is that the configmap is missing a linebreak.

        target_label: kubernetes_pod_namealerting:
      alertmanagers:

needs to be

        target_label: kubernetes_pod_name
    alerting:
      alertmanagers:

Deploy prometheus, and then run

kubectl edit configmap prometheus-prometheus-server

Remember to add correct indentation. Then save, and prometheus will start.

The problem is that prometheus only scrape itself for some reason. It ignores the other targets in the configmap. I think its because prometheus doesnt have access rights (RBAC) to anything but itself.

[hoeghh]

The RBAC is broken.

To get around this, once you have prometheus started, do the following :

echo 'apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"] ' >  clusterRole.yaml 

echo 'apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: default
  namespace: default' > clusterRoleBinding.yaml 

kubectl create -f clusterRole.yaml
kubectl create -f clusterRoleBinding.yaml 

But, I think the rbac should be fixed in the chart. But this will get you up and running.

[ellisio]

This was resolved in https://github.com/kubernetes/charts/pull/5591.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[OpsKunal]

Hello Guys, I am facing following error while starting Prometheus in Ubuntu:

level=error ts=2020-01-12T12:20:07.420045815Z caller=main.go:582 err="Error loading config couldn't load configuration (--config.file=/etc/prometheus/prometheus.yml): parsing YAML file /etc/prometheus/prometheus.yml: yaml: line 14: did not find expected key"
level=info ts=2020-01-12T12:20:07.420173602Z caller=main.go:584 msg="See you next time!"
root@ip-172-31-35-124:/home/ubuntu/prometheus# nano /etc/systemd/system/prometheus.service
root@ip-172-31-35-124:/home/ubuntu/prometheus# chown prometheus:prometheus /etc/prometheus/prometheus.yml root@ip-172-31-35-124:/home/ubuntu/prometheus# sudo nano /etc/prometheus/prometheus.yml root@ip-172-31-35-124:/home/ubuntu/prometheus# sudo -u prometheus /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries
level=info ts=2020-01-12T12:25:44.997963659Z caller=main.go:220 msg="Starting Prometheus" version="(version=2.2.1, branch=HEAD, revision=bc6058c81272a8d938c05e75607371284236aadc)"
level=info ts=2020-01-12T12:25:44.998199918Z caller=main.go:221 build_context="(go=go1.10, user=root@149e5b3f0829, date=20180314-14:15:45)"
level=info ts=2020-01-12T12:25:44.998357855Z caller=main.go:222 host_details="(Linux 4.15.0-1051-aws #53-Ubuntu SMP Wed Sep 18 13:35:53 UTC 2019 x86_64 ip-172-31-35-124 (none))"
level=info ts=2020-01-12T12:25:44.998490269Z caller=main.go:223 fd_limits="(soft=1024, hard=1048576)"
level=info ts=2020-01-12T12:25:45.001843327Z caller=main.go:504 msg="Starting TSDB ..."
level=info ts=2020-01-12T12:25:45.008982307Z caller=web.go:382 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-01-12T12:25:45.010994669Z caller=main.go:514 msg="TSDB started"
level=info ts=2020-01-12T12:25:45.011093885Z caller=main.go:588 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2020-01-12T12:25:45.011283081Z caller=main.go:398 msg="Stopping scrape discovery manager..."
level=info ts=2020-01-12T12:25:45.011368819Z caller=main.go:411 msg="Stopping notify discovery manager..."
level=info ts=2020-01-12T12:25:45.011438927Z caller=main.go:432 msg="Stopping scrape manager..."
level=info ts=2020-01-12T12:25:45.01150996Z caller=main.go:407 msg="Notify discovery manager stopped"
level=info ts=2020-01-12T12:25:45.012957722Z caller=manager.go:460 component="rule manager" msg="Stopping rule manager..."
level=info ts=2020-01-12T12:25:45.013062961Z caller=manager.go:466 component="rule manager" msg="Rule manager stopped"
level=info ts=2020-01-12T12:25:45.013144431Z caller=notifier.go:512 component=notifier msg="Stopping notification manager..."
level=info ts=2020-01-12T12:25:45.013301202Z caller=main.go:394 msg="Scrape discovery manager stopped"
level=info ts=2020-01-12T12:25:45.013740977Z caller=main.go:426 msg="Scrape manager stopped"
level=info ts=2020-01-12T12:25:45.013846084Z caller=main.go:573 msg="Notifier manager stopped"
level=error ts=2020-01-12T12:25:45.01393102Z caller=main.go:582 err="Error loading config couldn't load configuration (--config.file=/etc/prometheus/prometheus.yml): parsing YAML file /etc/prometheus/prometheus.yml: yaml: line 10: did not find expected key"