Loki: Promtail: Cant get the value of filename in relabel_configs

[jarvisaoieong]

In relabel_configs, I would like to get the filename label value to rewrite to a specific label.

For instance, there are some log files in the directory.

• /logs/example-out.log
• /logs/example-error.log

I expect set a label called "service" equal to "example"

Then, I have the setting below in the relabel_configs of promtail

static_configs:
- targets:
    - localhost
  labels:
    job: test
    __path__: /logs/*.log
relabel_configs:
- source_labels: [filename]
  regex: '^.*\/(.+)-(out|error)\.log$'
  replacement: '${1}'
  target_label: service
  action: replace

I have tested "filename" and "__path__" into the source_labels.

There is no value return when source_labels set to "filename".
When the source_label set to "__path__", the value equals to "/logs/*.log"

How can i get the correct file path in relabel_configs?

[odinsy]

same problem here

scrapeConfigs:
- job_name: logs
  entry_parser: raw
  static_configs:
  - targets:
    - localhost
    labels:
      __path__: /var/log/services/*/service.log
  relabel_configs:
  - source_labels:
    - filename
    target_label: 'service'
    regex: '(services)\/?([^\/]+)\/'
    replacement: '$2'

[cyriltovena]

You should try __path__, also there is a page exposed by promtail to help you figure this out.

See: https://github.com/grafana/loki/blob/master/docs/troubleshooting.md#troubleshooting-targets

[odinsy]

I tried with __path__, but it's not working too.
There are only two discovered labels on service discovery page:

__address__="localhost"
__path__="/var/log/services/*/service.log"

[cyriltovena]

ok the expansion of the __path__ happens after the relabeling. This means this is something we don't support yet, though I think this is nice to have.

I'm switching this issue to a feature request, PR are welcome :).

[cyriltovena]

Actually you should use the pipeline stages instead that would be easier for you, to extract service from filename.

see: https://github.com/grafana/loki/blob/master/docs/logentry/processing-log-lines.md

[odinsy]

I tried to configure pipeline stage, but it doesn't work.
Is it what do u speaking about?

scrapeConfigs:
- job_name: logs
  entry_parser: raw
  static_configs:
  - targets:
    - localhost
    labels:
      __path__: /var/log/services/*/service.log

pipelineStages:
- match:
  selector: '{"__path__"="/var/log/services/*/service.log"}'
  stages:
  - regex:
    expression: (?:services)\/([^\/]+)\/
  - labels:
    service:

PS:
Output from promtail agent with relabel_configs below msg="Adding target" key={}:

level=warn ts=2019-07-22T16:43:32.38580011Z caller=filetargetmanager.go:96 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=info ts=2019-07-22T16:43:32.386044721Z caller=server.go:120 http=[::]:3101 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2019-07-22T16:43:32.38626894Z caller=main.go:49 msg="Starting Promtail" version="(version=master-4439828, branch=master, revision=4439828)"
level=info ts=2019-07-22T16:43:37.386312754Z caller=filetargetmanager.go:245 msg="Adding target" key={}

[jarvisaoieong]

So far, what i know is that the pipeline stage can only process the log entry.
It can't handle the filename/path. It would be a feature request.

[cyriltovena]

It can also process labels after discovery and relabeling

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[adityacs]

@cyriltovena Working on this. Will create a PR soon.

[rmgpinto]

@adityacs can you document how to use the filename on the pipeline_stages please?

I have this:

scrape_configs:
  - job_name: promtail
    static_configs:
      - targets:
        - localhost
        labels:
          job: promtail
          __path__: /var/log/pods/*/*/*.log
    pipeline_stages:
      - match:
          selector: '{job="promtail"}'
          stages:
            - regex:
                expression: "(?:pods)/(.*)_"
            - labels:
                first_dir:

The idea is to add a label (first_dir) with the name of the first directory of the path.

[cyriltovena]

You need to add the label __path__ as a source for the regex stage otherwise it parses the log line not the label. See https://github.com/grafana/loki/blob/master/docs/clients/promtail/stages/regex.md#schema

[randomcoww]

@rmgpinto I was trying to do roughly the same thing. I was able to get labels created from filename as follows:

scrape_configs:
- job_name: containers
  static_configs:
  - targets:
    - localhost
    labels:
      job: static_pods
      __path__: /var/log/pods/*/*/*.log
  pipeline_stages:
  - match:
      selector: '{job="static_pods"}'
      stages:
      - regex:
          source: filename
          expression: "(?:pods)/(?P<namespace>\\S+?)_(?P<pod>\\S+?)-(?P<host>\\S+?)_\\S+?/(?P<container>\\S+?)/"
      - labels:
          namespace:
          pod:
          host:
          container:

path or __path__ were not working for me as regex source fields, but filename appears to be generated and available for use.

[RobertHeim]

Any idea how that would work when using syslog and trying to rewrite the host label? This does not work:

scrape_configs:
  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:4000
      labels:
        job: "foo"
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: 'host'
    pipeline_stages:
    - match:
      selector: '{host="d.c5632d-1e15-c340-hf7a-7nbf41n31254"}'
      stages:
      - labels:
          host: 'staging'

We do not see any logs, but if we remove the pipeline_stages we see them.

[cyriltovena]

You need the regex see this example: https://gist.github.com/cyriltovena/7cc5359a692206782db889f4c73f78b1

[RobertHeim]

Thanks, we tried it before and it was not working - but actually the issue was that we did not properly escape the regex when using double quotes. Using single quotes and the following config works:

scrape_configs:
  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:4000
      labels:
        job: "foo"
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: 'env'
        regex: '(d\.c5632d-1e15-c340-hf7a-7nbf41n31254)'
        replacement: 'production'
      - source_labels: ['__syslog_message_hostname']
        target_label: 'env'
        regex: '(d\.a311e-n351-c171-ef2c-69e84121c441)'
        replacement: 'staging'

[analytically]

Trying to get this working to have Promtail drop it's own logs on Nomad:

scrape_configs:
- job_name: stdout
  pipeline_stages:
    - match:
      selector: '{filename="/alloc/logs/log-shipper.stdout.0"}'
      action: drop
  static_configs:
  - targets:
      - localhost
    labels:
      job: %s
      stream: stdout
      __path__: /alloc/logs/*.stdout.[0-9]*