Lens: Documentation is incorrect for Prometheus troubleshooting

[sshishov]

Describe the bug
While configuring custom Prometheus according to troubleshooting guide, I was confused which setting should I change in Prometheus configuration

To Reproduce
Steps to reproduce the behavior:

1. Look at README file here: https://github.com/lensapp/lens/blob/936cbd53d400e8973a763373f05514f894a2578a/troubleshooting/custom-prometheus.md

Expected behavior
The description should be clear without confusion

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Lens Version: any
• OS: [e.g. OSX]
• Installation method (e.g. snap or AppImage in Linux):

Logs:
When you run the application executable from command line you will see some logging output. Please paste them here:

Your logs go here...

Kubeconfig:
Quite often the problems are caused by malformed kubeconfig which the application tries to load. Please share your kubeconfig, remember to remove any secret and sensitive information.

your kubeconfig here

Additional context
Add any other context about the problem here.

[alikhil]

I did everything as it written in readme.md but metrics in the applications still do not work

[elepedus]

I too am somewhat confused about the proper way to enable metric in a cluster which already has an existing Prometheus deployment. The documentation appears to pertain to deployments of the Prometheus operator, rather than the Prometheus Helm Chart.

I have a few clusters which were created using GitLab, and Prometheus was installed as a GitLab Managed App, which I believe uses the Helm chart. The metrics work well in GitLab, but unfortunately, I'm not familiar enough with Prometheus to translate the instructions from the readme to work with the Helm chart ConfigMap.

I'm happy to provide a sample of the values.yaml section from the ConfigMap, if it would help improve the readme :)

[RiFi2k]

@elepedus I'm trying to get this to work with gitlab-managed-apps/prometheus-prometheus-server:9090 Prometheus existing install. :grimacing:

I can get the metrics to spin like they are loading now initially which before it was just the metrics not being set up error from the start. Here is what I had to do to get it to where I'm at and maybe someone can figure out what I'm missing.

In the cluster settings...
Prometheus installation method set to helm
Prometheus service address set to gitlab-managed-apps/prometheus-prometheus-server:9090
This is the namespace the name of the service and the port as installed by GitLab in my cluster.

This is the modified config.yml file adding in the couple things they asked for https://github.com/lensapp/lens/blob/master/troubleshooting/custom-prometheus.md#kube-prometheus

global:
  evaluation_interval: 1m
  scrape_interval: 30s
  scrape_timeout: 10s
rule_files:
- /etc/config/rules
- /etc/config/alerts
scrape_configs:
- job_name: prometheus
  static_configs:
  - targets:
    - localhost:9090
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  job_name: kubernetes-cadvisor
  kubernetes_sd_configs:
  - role: node
  metric_relabel_configs:
  - regex: (.+)-.+-.+
    source_labels:
    - pod_name
    target_label: environment
  - action: replace
    target_label: instance
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - replacement: kubernetes.default.svc:443
    target_label: __address__
  - regex: (.+)
    replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
    source_labels:
    - __meta_kubernetes_node_name
    target_label: __metrics_path__
  - action: replace
    regex: (.*)
    replacement: $1
    source_labels:
    - __meta_kubernetes_pod_node_name
    target_label: kubernetes_node
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
- job_name: kubernetes-service-endpoints
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - action: keep
    regex: true
    source_labels:
    - __meta_kubernetes_service_annotation_prometheus_io_scrape
  - action: replace
    regex: (https?)
    source_labels:
    - __meta_kubernetes_service_annotation_prometheus_io_scheme
    target_label: __scheme__
  - action: replace
    regex: (.+)
    source_labels:
    - __meta_kubernetes_service_annotation_prometheus_io_path
    target_label: __metrics_path__
  - action: replace
    regex: (.+)(?::\d+);(\d+)
    replacement: $1:$2
    source_labels:
    - __address__
    - __meta_kubernetes_service_annotation_prometheus_io_port
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - action: replace
    source_labels:
    - __meta_kubernetes_namespace
    target_label: kubernetes_namespace
  - action: replace
    source_labels:
    - __meta_kubernetes_service_name
    target_label: kubernetes_name
  - action: replace
    regex: (.*)
    replacement: $1
    source_labels:
    - __meta_kubernetes_pod_node_name
    target_label: kubernetes_node
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  job_name: kubernetes-nodes
  kubernetes_sd_configs:
  - role: node
  metric_relabel_configs:
  - regex: (.+)-.+-.+
    source_labels:
    - pod_name
    target_label: environment
  - action: replace
    target_label: instance
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - replacement: kubernetes.default.svc:443
    target_label: __address__
  - regex: (.+)
    replacement: /api/v1/nodes/${1}/proxy/metrics
    source_labels:
    - __meta_kubernetes_node_name
    target_label: __metrics_path__
  - action: replace
    regex: (.*)
    replacement: $1
    source_labels:
    - __meta_kubernetes_pod_node_name
    target_label: kubernetes_node
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  job_name: kubernetes-pods
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - action: keep
    regex: "true"
    source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_scrape
  - action: replace
    regex: (.+)
    source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_path
    target_label: __metrics_path__
  - action: replace
    regex: ([^:]+)(?::[0-9]+)?;([0-9]+)
    replacement: $1:$2
    source_labels:
    - __address__
    - __meta_kubernetes_pod_annotation_prometheus_io_port
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - action: replace
    source_labels:
    - __meta_kubernetes_namespace
    target_label: kubernetes_namespace
  - action: replace
    source_labels:
    - __meta_kubernetes_pod_name
    target_label: kubernetes_pod_name
  - action: replace
    regex: (.*)
    replacement: $1
    source_labels:
    - __meta_kubernetes_pod_node_name
    target_label: kubernetes_node
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
- job_name: autoscaler
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - action: keep
    regex: knative-serving;autoscaler;metrics
    source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_label_app
    - __meta_kubernetes_pod_container_port_name
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - action: replace
    regex: (.*)
    replacement: $1
    source_labels:
    - __meta_kubernetes_pod_node_name
    target_label: kubernetes_node
  scrape_interval: 3s
  scrape_timeout: 3s
- job_name: activator
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - action: keep
    regex: knative-serving;activator;metrics-port
    source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_label_app
    - __meta_kubernetes_pod_container_port_name
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - action: replace
    regex: (.*)
    replacement: $1
    source_labels:
    - __meta_kubernetes_pod_node_name
    target_label: kubernetes_node
  scrape_interval: 3s
  scrape_timeout: 3s
- job_name: istio-mesh
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - action: keep
    regex: istio-system;istio-telemetry;prometheus
    source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_service_name
    - __meta_kubernetes_endpoint_port_name
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - action: replace
    regex: (.*)
    replacement: $1
    source_labels:
    - __meta_kubernetes_pod_node_name
    target_label: kubernetes_node
  scrape_interval: 5s
- job_name: istio-policy
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - action: keep
    regex: istio-system;istio-policy;http-monitoring
    source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_service_name
    - __meta_kubernetes_endpoint_port_name
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - action: replace
    regex: (.*)
    replacement: $1
    source_labels:
    - __meta_kubernetes_pod_node_name
    target_label: kubernetes_node
  scrape_interval: 5s
- job_name: istio-telemetry
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - action: keep
    regex: istio-system;istio-telemetry;http-monitoring
    source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_service_name
    - __meta_kubernetes_endpoint_port_name
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - action: replace
    regex: (.*)
    replacement: $1
    source_labels:
    - __meta_kubernetes_pod_node_name
    target_label: kubernetes_node
  scrape_interval: 5s
- job_name: istio-pilot
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - action: keep
    regex: istio-system;istio-pilot;http-monitoring
    source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_service_name
    - __meta_kubernetes_endpoint_port_name
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - action: replace
    regex: (.*)
    replacement: $1
    source_labels:
    - __meta_kubernetes_pod_node_name
    target_label: kubernetes_node
  scrape_interval: 5s

[wjrogers]

I have installed Prometheus Operator in the 'monitoring' namespace in my cluster via the bitnami helm chart. I set Lens to "Prometheus Operator" and service address monitoring/prometheus-operator-prometheus:9090. I have also followed the using custom prometheus instructions and verified that the changes have been applied to the ServiceMonitor specs.

Some metrics appear in Lens. I can see pod CPU, Memory, Network, and Filesystem usage. But the cluster overview says "Metrics not available at the moment," and the Nodes overview resource graphs are blank lines. Despite scouring related issues and the documentation, I can't get this to work at all.

Has anyone gotten this running?

[emrahcetiner]

Some metrics appear in Lens. I can see pod CPU, Memory, Network, and Filesystem usage. But the cluster overview says "Metrics not available at the moment," and the Nodes overview resource graphs are blank lines. Despite scouring related issues and the documentation, I can't get this to work at all.

+1 same issue

[RiFi2k]

Took a better look at the docs, just make sure you have installed kube-prometheus as explained in the quickstart or customizing the install explain below it. https://github.com/coreos/kube-prometheus#quickstart

Then depending on what you do, say you have it installed already go find the CRD generated for node-exporter and add the first snippet from the custom prometheus docs into the relabeling, the CRD comes from --> https://github.com/coreos/kube-prometheus/blob/master/jsonnet/kube-prometheus/node-exporter/node-exporter.libsonnet#L176

Then add the second to snippet to the kublet CRD generated from here https://github.com/coreos/kube-prometheus/blob/master/jsonnet/kube-prometheus/prometheus/prometheus.libsonnet#L265

Or use the jsonnet code they provided and compile kube-prometheus yourself and it will work out the box.

Also it looks like if you installed the prometheus-operator helm chart you want to add the snippets into nodeExporter.serviceMonitor.relabelings and kubelet.serviceMonitor.metricRelabelings. Don't quote me but probably like this

[{"action": "replace", "regex": "(.*)", "replacement": "$1", "sourceLabels": ["__meta_kubernetes_pod_node_name"], "targetLabel": "kubernetes_node"}]

and

[{"action": "replace", "sourceLabels": ["node"], "targetLabel": "instance"}]

Hopefully that gets anyone else stuck pointed in the right direction.

[nitrogear]

@RiFi2k your solution worked for me. I found confusing that https://github.com/lensapp/lens/blob/936cbd53d400e8973a763373f05514f894a2578a/troubleshooting/custom-prometheus.md doesn't clearly describe in what endpoint changes should be made in CRD node-exporter and kubelet.
Here is exactly what helped me:
in CRD node-exporter I had to add new endpoint "https" (I had only "metrics" endpoint):

- port: https
  relabelings:
    - action: replace
      regex: (.*)
      replacement: $1
      sourceLabels:
        - __meta_kubernetes_pod_node_name
      targetLabel: instance

And in kubelet CRD I added in relabelings of "https-metrics" endpoint:

- action: replace
  regex: (.+)
  replacement: $1
  sourceLabels:
    - __metrics_path__
  targetLabel: metrics_path

[mboutet]

I have installed Prometheus Operator in the 'monitoring' namespace in my cluster via the bitnami helm chart. I set Lens to "Prometheus Operator" and service address monitoring/prometheus-operator-prometheus:9090. I have also followed the using custom prometheus instructions and verified that the changes have been applied to the ServiceMonitor specs.

Some metrics appear in Lens. I can see pod CPU, Memory, Network, and Filesystem usage. But the cluster overview says "Metrics not available at the moment," and the Nodes overview resource graphs are blank lines. Despite scouring related issues and the documentation, I can't get this to work at all.

Has anyone gotten this running?

Same issue with the bitnami prometheus-operator chart. If I set to "Helm" instead, I can see the overall cluster metrics. However, the resources requests/limits for the pods don't show (they show for the node metrics though). I meticulously followed the instructions in custom-prometheus.md.

[mboutet]

@wjrogers and @emrahcetiner, I managed to make appear the limits/request for the pods by setting kube-state-metrics.serviceMonitor.honorLabels to true. I'm also using the bitnami prometheus-operator. However, I need to select "Helm" instead of "Prometheus Operator" for "Prometheus installation method." in Lens.

[theclue]

I've followed the solution from @nitrogear and the enanchment from @mboutet, setting as "Prometheus Operator" in Lens settings. It __almost__ works, but some metrics still missing:

• Overall cluster metrics, left panel: still shows "Metrics are not available at the moment"
• Overall cluster metrics, right panel: CPU and Ram Usage are N/A. Pods Usage is ok. Requests and Limits are ok

The same metrics are missing at nodes level, but they works at Pods level.

any idea? Oo'

[wenerme]

Here is mine helm values for bitnami/kube-prometheus, rember change installation from auto detect to Helm install, prometheus address is monitoring/kube-prometheus-prometheus:9090

node-exporter:
  serviceMonitor:
    relabelings:
    - action: replace
      regex: (.*)
      replacement: $1
      sourceLabels:
      - __meta_kubernetes_pod_node_name
      targetLabel: kubernetes_node

kubelet:
  serviceMonitor:
    metricRelabelings:
    - action: replace
      sourceLabels:
      - node
      targetLabel: instance

kube-state-metrics:
  serviceMonitor:
    honorLabels: true

[czunker]

@wenerme settings also work with kube-prometheus-stack.

Only the kube-state-metrics part is not needed, but set to the same values by the chart.

[peimanja]

hmm I could not get it to work with the latest kube-prometheus-stack installation.

[MRostanski]

@czunker alas, I am not able to reproduce your success - having GKE 1.17.9 and the values for kube-prometheus-stack in full are:

defaultRules:
  create: false
alertmanager:
  enabled: false
grafana:
  enabled: true
  serviceMonitor:
    enabled: false
  resources:
    limits:
      cpu: 100m
      memory: 128Mi
    requests:
      cpu: 100m
      memory: 128Mi
  persistence:
    type: pvc
    enabled: true
    storageClassName: ssd
    size: 10Gi
  adminPassword: ZAQ12wsx
kubeApiServer:
  enabled: false
kubelet:
  enabled: true
  serviceMonitor:
    metricRelabelings:
      - action: replace
        sourceLabels:
          - node
        targetLabel: instance
kubeControllerManager:
  enabled: false
coreDns:
  enabled: false
kubeEtcd:
  enabled: false
kubeScheduler:
  enabled: false
kubeStateMetrics:
  enabled: true
nodeExporter:
  enabled: true
  serviceMonitor:
    relabelings:
      - action: replace
        regex: (.*)
        replacement: $1
        sourceLabels:
          - __meta_kubernetes_pod_node_name
        targetLabel: kubernetes_node
prometheus:
  enabled: true
  resources:
    limits:
      cpu: 1000m
      memory: 1200Mi
    requests:
      cpu: 1000m
      memory: 1200Mi
prometheusOperator:
  admissionWebhooks:
    enabled: false

[czunker]

@MRostanski This is our config:

prometheus:
...

grafana:
...

additionalPrometheusRulesMap:
...

alertmanager:
...

kubelet:
  serviceMonitor:
    https: false
    metricRelabelings:
    - action: replace
      sourceLabels:
      - node
      targetLabel: instance

nodeExporter:
  serviceMonitor:
    relabelings:
      - action: replace
        regex: (.*)
        replacement: $1
        sourceLabels:
        - __meta_kubernetes_pod_node_name
        targetLabel: kubernetes_node

Looks like yours for the use of Lens.

But make sure Prometheus is using this config. Take a look at the current config via Web UI: https://your.host/config
You should see the relabelings there. Make sure the ServiceMonitor contains the relabelings and perhaps restart the Prometheus Pod.

In case you are using Network Policies, these might also prevent Lens from reaching Prometheus to get the metrics.

[peimanja]

In my case closing and reopening Lens fixed my issue after applying the config

[omninonsense]

After a lot of head-scratching, it turns out @peimanja was right that it _requires_ a restart for some of the settings to apply (at least the Prometheus one). Reloading (Ctrl+R on Linux) will not work, neither will disconnecting and reconnecting the cluster!

Also, for the people who're trying to get this to work with the GitLab installation of prometheus, you can select Helm as the installation method (that's what Gitlab uses) and put this into the service address field: gitlab-managed-apps/prometheus-prometheus-server:80, restart and voila.

[rafi-fisker]

Does anyone know how to get the cluster overview metrics working using the prometheus helm chart ?

EDIT: nevermind, somehow it started working after restarting prometheus server

[tonychoe]

In my case, the ingress path was the issue.

I set the ingress like this and Lens was not able to discover it automatically.

prometheus:
  ...
  ingress:
    enabled: true
    ...
    paths:
      - /prometheus

So I have updated the Lens Settings to Prometheus Operator and default/promkube-prometheus:9090/prometheus, then it worked.