Loki: Promtail extraScrapeConfigs not being picked up

Created on 12 May 2020 · 17Comments · Source: grafana/loki

Describe the bug
It seems that excluding logs in a pod using the configuration below does not work:

extraScrapeConfigs:
 - job_name: blacklist
   kubernetes_sd_configs:
   - role: pod
   relabel_configs:
   - action: drop
     regex: test
     source_labels:
     - __meta_kubernetes_pod_label_app

To Reproduce
Steps to reproduce the behavior:

Deployed helm loki-stack (0.36.2)
Attach grafana to loki datasource
Query: {app="test"} in Grafana Loki
See logs

Expected behavior
Not seeing any logs

Environment:

Infrastructure: kubernetes (1.15)
Deployment tool: helm

Maybe i'm missing something, help would be appreciated!

componenpackaging good first issue helm help wanted

Source

robbertvdg

Most helpful comment

Hi there!

I found this issue since I was facing with it, was trying to drop logs from a specific namespace..
What I found out that, to use "_keep_" instead of "_drop_" (because in my case I was curious just for one namespace)
Also I had to overwrite the default config in the helm chart values, here it is if someone wants to play with it:

Inserted code at:

    - job_name: kubernetes-pods-app
      pipeline_stages:
        - docker: {}
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: drop
        regex: .+
        source_labels:
        - __meta_kubernetes_pod_label_name
      ###
      - action: keep
        regex: kube-system
        source_labels:
          - __meta_kubernetes_namespace
          - __meta_kubernetes_pod_namespace
      ###

Full config code:

scrapeConfigs:
  - job_name: kubernetes-pods-name
    pipeline_stages:
      - docker: {}
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - source_labels:
      - __meta_kubernetes_pod_label_name
      target_label: __service__
    - source_labels:
      - __meta_kubernetes_pod_node_name
      target_label: __host__
    - action: drop
      regex: ''
      source_labels:
      - __service__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - action: replace
      replacement: $1
      separator: /
      source_labels:
      - __meta_kubernetes_namespace
      - __service__
      target_label: job
    - action: replace
      source_labels:
      - __meta_kubernetes_namespace
      target_label: namespace
    - action: replace
      source_labels:
      - __meta_kubernetes_pod_name
      target_label: pod
    - action: replace
      source_labels:
      - __meta_kubernetes_pod_container_name
      target_label: container
    - replacement: /var/log/pods/*$1/*.log
      separator: /
      source_labels:
      - __meta_kubernetes_pod_uid
      - __meta_kubernetes_pod_container_name
      target_label: __path__
  - job_name: kubernetes-pods-app
    pipeline_stages:
      - docker: {}
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - action: drop
      regex: .+
      source_labels:
      - __meta_kubernetes_pod_label_name
    - action: keep
      regex: kube-system
      source_labels:
        - __meta_kubernetes_namespace
        - __meta_kubernetes_pod_namespace
    - source_labels:
      - __meta_kubernetes_pod_label_app
      target_label: __service__
    - source_labels:
      - __meta_kubernetes_pod_node_name
      target_label: __host__
    - action: drop
      regex: ''
      source_labels:
      - __service__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - action: replace
      replacement: $1
      separator: /
      source_labels:
      - __meta_kubernetes_namespace
      - __service__
      target_label: job
    - action: replace
      source_labels:
      - __meta_kubernetes_namespace
      target_label: namespace
    - action: replace
      source_labels:
      - __meta_kubernetes_pod_name
      target_label: pod
    - action: replace
      source_labels:
      - __meta_kubernetes_pod_container_name
      target_label: container
    - replacement: /var/log/pods/*$1/*.log
      separator: /
      source_labels:
      - __meta_kubernetes_pod_uid
      - __meta_kubernetes_pod_container_name
      target_label: __path__
  - job_name: kubernetes-pods-direct-controllers
    pipeline_stages:
      - docker: {}
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - action: drop
      regex: .+
      separator: ''
      source_labels:
      - __meta_kubernetes_pod_label_name
      - __meta_kubernetes_pod_label_app
    - action: drop
      regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
      source_labels:
      - __meta_kubernetes_pod_controller_name
    - source_labels:
      - __meta_kubernetes_pod_controller_name
      target_label: __service__
    - source_labels:
      - __meta_kubernetes_pod_node_name
      target_label: __host__
    - action: drop
      regex: ''
      source_labels:
      - __service__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - action: replace
      replacement: $1
      separator: /
      source_labels:
      - __meta_kubernetes_namespace
      - __service__
      target_label: job
    - action: replace
      source_labels:
      - __meta_kubernetes_namespace
      target_label: namespace
    - action: replace
      source_labels:
      - __meta_kubernetes_pod_name
      target_label: pod
    - action: replace
      source_labels:
      - __meta_kubernetes_pod_container_name
      target_label: container
    - replacement: /var/log/pods/*$1/*.log
      separator: /
      source_labels:
      - __meta_kubernetes_pod_uid
      - __meta_kubernetes_pod_container_name
      target_label: __path__
  - job_name: kubernetes-pods-indirect-controller
    pipeline_stages:
      - docker: {}
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - action: drop
      regex: .+
      separator: ''
      source_labels:
      - __meta_kubernetes_pod_label_name
      - __meta_kubernetes_pod_label_app
    - action: keep
      regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
      source_labels:
      - __meta_kubernetes_pod_controller_name
    - action: replace
      regex: '([0-9a-z-.]+)-[0-9a-f]{8,10}'
      source_labels:
      - __meta_kubernetes_pod_controller_name
      target_label: __service__
    - source_labels:
      - __meta_kubernetes_pod_node_name
      target_label: __host__
    - action: drop
      regex: ''
      source_labels:
      - __service__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - action: replace
      replacement: $1
      separator: /
      source_labels:
      - __meta_kubernetes_namespace
      - __service__
      target_label: job
    - action: replace
      source_labels:
      - __meta_kubernetes_namespace
      target_label: namespace
    - action: replace
      source_labels:
      - __meta_kubernetes_pod_name
      target_label: pod
    - action: replace
      source_labels:
      - __meta_kubernetes_pod_container_name
      target_label: container
    - replacement: /var/log/pods/*$1/*.log
      separator: /
      source_labels:
      - __meta_kubernetes_pod_uid
      - __meta_kubernetes_pod_container_name
      target_label: __path__
  - job_name: kubernetes-pods-static
    pipeline_stages:
      - docker: {}
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - action: drop
      regex: ''
      source_labels:
      - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
    - action: replace
      source_labels:
      - __meta_kubernetes_pod_label_component
      target_label: __service__
    - source_labels:
      - __meta_kubernetes_pod_node_name
      target_label: __host__
    - action: drop
      regex: ''
      source_labels:
      - __service__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - action: replace
      replacement: $1
      separator: /
      source_labels:
      - __meta_kubernetes_namespace
      - __service__
      target_label: job
    - action: replace
      source_labels:
      - __meta_kubernetes_namespace
      target_label: namespace
    - action: replace
      source_labels:
      - __meta_kubernetes_pod_name
      target_label: pod
    - action: replace
      source_labels:
      - __meta_kubernetes_pod_container_name
      target_label: container
    - replacement: /var/log/pods/*$1/*.log
      separator: /
      source_labels:
      - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
      - __meta_kubernetes_pod_container_name
      target_label: __path__

Kubernetes version: v1.19.3

Hope it helps for someone,
Cheers

norbertbodo91 on 17 Nov 2020

👍2

All 17 comments

After a bit more research we came up with the following that did work:

scrapeConfigs:
    - job_name: kubernetes-pods-name
      [.. default .. ]
    - job_name: kubernetes-pods-app
      pipeline_stages:
        - docker: {}
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: drop
        regex: .+
        source_labels:
        - __meta_kubernetes_pod_label_name
      - action: drop
         regex: test
         source_labels:
        - __meta_kubernetes_pod_label_app
     [ .. default .. ]

Which feels very hacky and unsustainable, since we have to copy the default, modify it and then overwrite it. Is this intended behaviour?

robbertvdg on 13 May 2020

I guess it's because extraScrapeConfigs is at the end and not at the beginning ? It this was the problem I'm ok to receive a PR to move it to the top, because you're right if you want to drop targets this is useless.

Can you confirm that this was the problem ?

cyriltovena on 27 May 2020

I think that would solve the problem, will make a PR soon.

robbertvdg on 28 May 2020

You mean a PR ;)

cyriltovena on 29 May 2020

😄1

I tested yesterday with the extraScrapeConfigs on top, instead of at the bottom, but that did not seem to work. It feels to me like the kubernetes-pods-app job still picks it up. I am not too familiair with this, can you confirm this is the case?

robbertvdg on 29 May 2020

@robbertvdg

Were you able to exclude logs for a specific pod? I've tried with the following within the existing "kubernetes-pods-app" job but with no luck:

_- action: drop
regex: application-controller
source_labels:
- __meta_kubernetes_pod_label_app_

Maybe you can share with me how your "kubernetes-pods-app" looks like? Thanks.

djmilosev on 20 Aug 2020

Yes, it should work like this:

    - job_name: kubernetes-pods-app
      pipeline_stages:
        - docker: {}
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: drop
        regex: .+
        source_labels:
        - __meta_kubernetes_pod_label_name
     # custom code
      - action: drop
        regex: <pod app label>
        source_labels:
          - __meta_kubernetes_pod_label_app
     # custom code end
      - source_labels:
        - __meta_kubernetes_pod_label_app
        target_label: __service__
      - source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ''
        source_labels:
        - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - __meta_kubernetes_namespace
        - __service__
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: instance
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container_name
      - replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_uid
        - __meta_kubernetes_pod_container_name
        target_label: __path__

robbertvdg on 20 Aug 2020

@robbertvdg

Can you please help me to create configuration to exclude multiple namespaces (kube-system,default)

cloudcafetech on 18 Sep 2020

I haven't tested it, but the following code at the custom code block in my previous comment should work:

     - action: drop
        regex: kube-system
        source_labels:
          - __meta_kubernetes_namespace
     - action: drop
        regex: default
        source_labels:
          - __meta_kubernetes_namespace

robbertvdg on 18 Sep 2020

NOT working :(

Not sure if I missing anything, below my configuration ...

      - job_name: kubernetes-pods-app
        pipeline_stages:
          - docker: {}
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - action: drop
            regex: .+
            source_labels:
              - __meta_kubernetes_pod_label_name
          - action: drop
            regex: kube-system
            source_labels:
              - __meta_kubernetes_namespace
          - action: drop
            regex: logging
            source_labels:
              - __meta_kubernetes_namespace
          - action: drop
            regex: monitoring
            source_labels:
              - __meta_kubernetes_namespace
          - action: drop
            regex: tracing
            source_labels:
              - __meta_kubernetes_namespace
          - source_labels:
              - __meta_kubernetes_pod_label_app
            target_label: __service__
          - source_labels:
              - __meta_kubernetes_pod_node_name
            target_label: __host__
          - action: drop
            regex: ""
            source_labels:
              - __service__
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - action: replace
            replacement: $1
            separator: /
            source_labels:
              - __meta_kubernetes_namespace
              - __service__
            target_label: job
          - action: replace
            source_labels:
              - __meta_kubernetes_namespace
            target_label: namespace
          - action: replace
            source_labels:
              - __meta_kubernetes_pod_name
            target_label: pod
          - action: replace
            source_labels:
              - __meta_kubernetes_pod_container_name
            target_label: container
          - replacement: /var/log/pods/*$1/*.log
            separator: /
            source_labels:
              - __meta_kubernetes_pod_uid
              - __meta_kubernetes_pod_container_name
            target_label: __path__

cloudcafetech on 20 Sep 2020

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

stale[bot] on 7 Nov 2020

@cloudcafetech Did it worked for you? I am facing the same issue. My config does look like the same

dolisss on 12 Nov 2020

I think you should check for __meta_kubernetes_pod_namespace instead of __meta_kubernetes_namespace, since we are checking for pod logs in this job.

robbertvdg on 12 Nov 2020

@dolisss

Sorry for late reply.
No, my configuration didn't work, will try Robert latest configuration.
Meanwhile if you tried, please let me know.

Thanks in advance.

cloudcafetech on 13 Nov 2020

@robbertvdg @cloudcafetech I tried using __meta_kubernetes_pod_namespace but I cannot see any changes, I am attaching my promtail.yaml file for your reference.
promtail.zip

dolisss on 13 Nov 2020

@robbertvdg

Not working my end also.