Cadvisor: HELP - CAdvisor default in GKE cluster ?

[arjun-dandagi]

we are running our application clusters in GKE now and i want to understand if the nodes we by default are configured with cAdvisor ( i am beginner if it looks silly) , what i see in prometheus file is just
`job_name: kubernetes-cadvisor
scrape_interval: 5s
scrape_timeout: 5s
metrics_path: /metrics
scheme: https
kubernetes_sd_configs:

• api_server: null
  role: node
  namespaces:
  names: []
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  insecure_skip_verify: false
  relabel_configs:
• separator: ;
  regex: __meta_kubernetes_node_label_(.+)
  replacement: $1
  action: labelmap
• separator: ;
  regex: (.*)
  target_label: __address__
  replacement: kubernetes.default.svc:443
  action: replace
• source_labels: [__meta_kubernetes_node_name]
  separator: ;
  regex: (.+)
  target_label: __metrics_path__
  replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
  action: replace`

this config used by many, but I am unsure if I have to install into the cluster, i actually see the cAdvisor as "up" in Prometheus, I didn't install it explicitly. so where is it running 😭

[dashpole]

cAdvisor is currently built-in to the kubelet on each node. The prometheus metrics are exposed at /metrics/cadvisor from each kubelet. You need to install that config in the cluster to get metrics collected by cAdvisor.

[arjun-dandagi]

@dashpole so cAdvisor only exposes container metrics ( for Prometheus by default), what if I want to scrape the disk space available on each node? (should I install node_exporter on each of my GK instances in the cluster ? ) what I am trying to find is, I need only one exporter about entire cluster in GKE

2>> while trying out container_cpu_load_average_10s, I see 0 values all the time when referred to this link, I see that I need to pass the flag to cAdvisor since you say that it is builtin to Kubelet, how do I enable enable_load_reader=true

[dopey]

Hey, I found this issue while trying to figure out why my cadvisor metrics weren't making it into prometheus. I ended up finding a blog post from which I cribbed a configuration that worked for me (GKE v0.16.x). The following snippet is just the cadvisor and kubelet scrape configs:

    - job_name: kubernetes-kubelet
      honor_timestamps: true
      scrape_interval: 15s
      scrape_timeout: 10s
      metrics_path: /metrics
      scheme: https
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc.cluster.local:443
      - target_label: __metrics_path__
        source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        replacement: /api/v1/nodes/${1}/proxy/metrics
    - job_name: kubernetes-cadvisor
      honor_timestamps: true
      scrape_interval: 15s
      scrape_timeout: 10s
      metrics_path: /metrics/cadvisor
      scheme: https
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc.cluster.local:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
      metric_relabel_configs:
      - source_labels: [namespace]
        separator: ;
        regex: ^$
        replacement: $1
        action: drop
      - source_labels: [pod]
        separator: ;
        regex: ^$
        replacement: $1
        action: drop

The important bit for me was the address replacement.

[nitish-predera]

while trying out container_cpu_load_average_10s, I see 0 values all the time when referred to this link, I see that I need to pass the flag to cAdvisor since you say that it is builtin to Kubelet, how do I enable enable_load_reader=true

@arjun-dandagi Were you able to find a way to add this enable_load_reader=true flag in GKE?