Kube-prometheus: Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)

[Luis-TT]

What happened?
prometheus container in prometheus-k8s-0 pod

level=error ts=2021-04-20T09:19:24.092Z caller=main.go:347 msg="Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)" err="parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: yaml: unmarshal errors:\n  line 1318: field sourceLabels not found in type relabel.plain\n  line 1320: field targetLabel not found in type relabel.plain\n  line 1322: cannot unmarshal !!map into []*targetgroup.Group"

config-reloader container in prometheus-k8s-0 pod

level=info ts=2021-04-20T09:18:43.051672176Z caller=main.go:147 msg="Starting prometheus-config-reloader" version="(version=0.47.0, branch=refs/tags/pkg/client/v0.47.0, revision=539108b043e9ecc53c4e044083651e2ebfbd3492)"
level=info ts=2021-04-20T09:18:43.051731776Z caller=main.go:148 build_context="(go=go1.16.3, user=simonpasquier, date=20210413-15:46:43)"
level=info ts=2021-04-20T09:18:43.051899975Z caller=main.go:182 msg="Starting web server for metrics" listen=:8080
level=error ts=2021-04-20T09:18:43.054793173Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-20T09:18:48.057370279Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-20T09:18:53.057330776Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-20T09:18:58.057567168Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-20T09:19:03.057381991Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-20T09:19:08.059276105Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-20T09:19:13.055997718Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-20T09:19:18.059652246Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-20T09:19:23.057442178Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-20T09:19:28.055158124Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-20T09:19:33.057289918Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"

Same for prometheus-k8s-1 pod. Prometheus is not reachable.

How to reproduce it (as minimally and precisely as possible):
my .jsonnet

local ingress(name, namespace, rules) = {
  apiVersion: 'networking.k8s.io/v1',
  kind: 'Ingress',
  metadata: {
    name: name,
    namespace: namespace,
  },
  spec: { rules: rules },
};

local kp =
  (import 'kube-prometheus/main.libsonnet') +
  (import 'kube-prometheus/addons/windows.libsonnet') +
  (import 'kube-prometheus/addons/all-namespaces.libsonnet') +
  {
    values+:: {
      common+: {
        namespace: 'monitoring',
      },
      windowsScrapeConfig+:: {
        static_configs: {
          targets: ['****:9100'],
        },
      },
    },
    grafana+:: {
        config+: {
          sections+: {
            server+: {
              root_url: 'http://grafana.****',
            },
          },
        },
      },
      alertmanager+:: {
        alertmanager+: {
          spec+: {
            externalUrl: 'http://alertmanager.****',
          },
        },
      },
      prometheus+:: {
        prometheus+: {
          spec+: {
            externalUrl: 'http://prometheus.****'
          },
        },
      },
      ingress+:: {
        'alertmanager-main': ingress(
          'alertmanager-main',
          $.values.common.namespace,
          [{
            host: 'alertmanager.****',
            http: {
              paths: [{
                backend: {
                  service: {
                    name: 'alertmanager-main',
                    port: {
                      name: 'web',
                    },
                  },
                },
                pathType: 'Prefix',
                path: '/',
              }],
            },
          }]
        ),
        grafana: ingress(
          'grafana',
          $.values.common.namespace,
          [{
            host: 'grafana.****',
            http: {
              paths: [{
                backend: {
                  service: {
                    name: 'grafana',
                    port: {
                      name: 'http',
                    },
                  },
                },
                pathType: 'Prefix',
                path: '/',
              }],
            },
          }],
      ),
        'prometheus-k8s': ingress(
          'prometheus-k8s',
          $.values.common.namespace,
          [{
            host: 'prometheus.****',
            http: {
              paths: [{
                backend: {
                  service: {
                    name: 'prometheus-k8s',
                    port: {
                      name: 'web',
                    },
                  },
                },
                pathType: 'Prefix',
                path: '/',
              }],
            },
          }],
        ),
      },
   } + {
    // Create basic auth secret - replace 'auth' file with your own
    ingress+:: {
      'basic-auth-secret': {
        apiVersion: 'v1',
        kind: 'Secret',
        metadata: {
          name: 'basic-auth',
          namespace: $.values.common.namespace,
        },
        data: { auth: std.base64(importstr 'auth') },
        type: 'Opaque',
      },
    },
   };

{ 'setup/0namespace-namespace': kp.kubePrometheus.namespace } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor' && name != 'prometheusRule'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor and prometheusRule are separated so that they can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ 'prometheus-operator-prometheusRule': kp.prometheusOperator.prometheusRule } +
{ 'kube-prometheus-prometheusRule': kp.kubePrometheus.prometheusRule } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['blackbox-exporter-' + name]: kp.blackboxExporter[name] for name in std.objectFields(kp.blackboxExporter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['kubernetes-' + name]: kp.kubernetesControlPlane[name] for name in std.objectFields(kp.kubernetesControlPlane) }
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) }
{ [name + '-ingress']: kp.ingress[name] for name in std.objectFields(kp.ingress) }

• Prometheus Operator version:

  Image: quay.io/prometheus-operator/prometheus-operator:v0.47.0

• Kubernetes version information:

  Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:02:01Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

kubeadm

• Prometheus Operator Logs:

level=info ts=2021-04-20T09:18:26.337850887Z caller=operator.go:1163 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2021-04-20T09:18:26.46137697Z caller=operator.go:673 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=error ts=2021-04-20T09:18:27.536121886Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="Sync \"monitoring/k8s\" failed: creating config failed: loading additional scrape configs from Secret failed: secret prometheus-k8s-additional-scrape-config could not be found"
level=info ts=2021-04-20T09:18:27.536232284Z caller=operator.go:1163 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2021-04-20T09:18:27.716567917Z caller=operator.go:673 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=error ts=2021-04-20T09:18:27.911520493Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="Sync \"monitoring/k8s\" failed: creating config failed: loading additional scrape configs from Secret failed: secret prometheus-k8s-additional-scrape-config could not be found"
level=info ts=2021-04-20T09:18:28.29086696Z caller=operator.go:1163 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2021-04-20T09:18:28.32596174Z caller=operator.go:673 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=info ts=2021-04-20T09:18:28.714155318Z caller=operator.go:673 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=info ts=2021-04-20T09:18:28.722815413Z caller=operator.go:1163 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2021-04-20T09:18:29.598963Z caller=operator.go:1163 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2021-04-20T09:18:30.081521112Z caller=operator.go:1163 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2021-04-20T09:18:35.866472201Z caller=operator.go:673 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=info ts=2021-04-20T09:18:35.8670625Z caller=operator.go:1163 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2021-04-20T09:18:41.242753783Z caller=operator.go:673 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=info ts=2021-04-20T09:18:41.313973225Z caller=operator.go:673 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=info ts=2021-04-20T09:18:43.361357509Z caller=operator.go:1163 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2021-04-20T09:18:49.980846126Z caller=operator.go:673 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=info ts=2021-04-20T09:18:50.426330085Z caller=operator.go:673 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main

[paulfantom]

It looks like there might be an issue with this relabelling: https://github.com/prometheus-operator/kube-prometheus/blob/main/jsonnet/kube-prometheus/addons/windows.libsonnet#L13-L23. Can you try running it with following line commented out:

(import 'kube-prometheus/addons/windows.libsonnet') +

?

[Luis-TT]

Prometheus pods are now running successfully and everything else too.
As expected windows nodes are not being scraped.

[paulfantom]

It appears that your additional scrape configs are not correct. Possibly windowsScrapeConfig is invalid.

[Luis-TT]

How come? It's exactly like in the example https://github.com/prometheus-operator/kube-prometheus/blob/main/examples/windows.jsonnet#L9

[paulfantom]

Ah, sorry, I was wrong. It appears you didn't load secret with additional configs. This is reflected in prometheus-operator logs:

secret prometheus-k8s-additional-scrape-config could not be found

Because of this, prometheus config cannot be created correctly by the operator and prometheus will be in a crashloop.

Load that secret into your environment and it should be working. I just checked, secret is properly created from an addon.

[Luis-TT]

Didn't fix my issue. I loaded the secret before everything else and the Prometheus Operator didn't write the error but the prometheus pods are still failing with the same error.

[paulfantom]

Ok, looking closer at this, it seems that few lines in windows addon are incorrect. Could you test the patch below and report if this works? Also if it works, could you create a PR?

diff --git a/jsonnet/kube-prometheus/addons/windows.libsonnet b/jsonnet/kube-prometheus/addons/windows.libsonnet
index cbf9071..8992032 100644
--- a/jsonnet/kube-prometheus/addons/windows.libsonnet
+++ b/jsonnet/kube-prometheus/addons/windows.libsonnet
@@ -3,6 +3,7 @@ local windowsrules = import 'kubernetes-mixin/rules/windows.libsonnet';

 {
   values+:: {
+    // This needs to follow prometheus naming convention and not prometheus-operator one
     windowsScrapeConfig+:: {
       job_name: 'windows-exporter',
       static_configs: [
@@ -15,10 +16,10 @@ local windowsrules = import 'kubernetes-mixin/rules/windows.libsonnet';
           action: 'replace',
           regex: '(.*)',
           replacement: '$1',
-          sourceLabels: [
+          source_labels: [
             '__meta_kubernetes_endpoint_address_target_name',
           ],
-          targetLabel: 'instance',
+          target_label: 'instance',
         },
       ],
     },

[Luis-TT]

I changed the file "\vendor\github.com\prometheus-operator\kube-prometheus\jsonnet\kube-prometheus\addons\windows.libsonnet" accordingly and rebuilt and applied. Sadly same errors still occuring.

[paulfantom]

What is in prometheus pod/container logs after applying this change? You shouldn't have following error msg anymore:

level=error ts=2021-04-20T09:19:24.092Z caller=main.go:347 msg="Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)" err="parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: yaml: unmarshal errors:\n  line 1318: field sourceLabels not found in type relabel.plain\n  line 1320: field targetLabel not found in type relabel.plain\n  line 1322: cannot unmarshal !!map into []*targetgroup.Group"

Could you also paste the content of prometheus-k8s-additional-scrape-config Secret that is on a cluster?

[Luis-TT]

prometheus in prometheus-k8s-0

level=error ts=2021-04-27T10:13:40.122Z caller=main.go:347 msg="Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)" err="parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: yaml: unmarshal errors:\n  line 1322: cannot unmarshal !!map into []*targetgroup.Group"

config-loader in prometheus-k8s-0

level=info ts=2021-04-27T10:12:11.635359805Z caller=main.go:147 msg="Starting prometheus-config-reloader" version="(version=0.47.0, branch=refs/tags/pkg/client/v0.47.0, revision=539108b043e9ecc53c4e044083651e2ebfbd3492)"
level=info ts=2021-04-27T10:12:11.636037559Z caller=main.go:148 build_context="(go=go1.16.3, user=simonpasquier, date=20210413-15:46:43)"
level=info ts=2021-04-27T10:12:11.636385727Z caller=main.go:182 msg="Starting web server for metrics" listen=:8080
level=error ts=2021-04-27T10:12:11.719908684Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-27T10:12:16.65340276Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-27T10:12:21.653592917Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-27T10:12:26.649998289Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-27T10:12:31.651815862Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-27T10:12:36.650011858Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-27T10:12:41.653856934Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-27T10:12:46.653400458Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-27T10:12:51.654138078Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-27T10:12:56.649990294Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2021-04-27T10:13:01.650412143Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"

prometheus-k8s-additional-scrape-config Secret:

apiVersion: v1
kind: Secret
metadata:
  name: prometheus-k8s-additional-scrape-config
  namespace: monitoring
stringData:
  prometheus-additional.yaml: |-
    - "job_name": "windows-exporter"
      "relabel_configs":
      - "action": "replace"
        "regex": "(.*)"
        "replacement": "$1"
        "source_labels":
        - "__meta_kubernetes_endpoint_address_target_name"
        "target_label": "instance"
      "static_configs":
        "targets":
        - "*****:9100"

[paulfantom]

Ok, so the patch is fixing one issue but there is another one. It appears that following section is also incorrect:

      "static_configs":
        "targets":
        - "*****:9100"

We need to have:

      "static_configs":
       - "targets":
         - "*****:9100"

As static_configs expect a list and not a map.

You need to change this in your config:

      windowsScrapeConfig+:: {
        static_configs: {
          targets: ['****:9100'],
        },
      },   

To following (notice [ and ] in static_configs section):

      windowsScrapeConfig+:: {
        static_configs: [{
          targets: ['****:9100'],
        }],
      },

It appears we have an incorrect example in https://github.com/prometheus-operator/kube-prometheus/blob/main/examples/windows.jsonnet#L10-L12 which should be fixed alongside applying the patch from earlier.

[Luis-TT]

Now everything is working as expected. Thank you very much!
I will create the PR.