Fluent-bit: Loki output plugin auto_kubernetes_labels error

Created on 20 Nov 2020 · 6Comments · Source: fluent/fluent-bit

Bug Report

Describe the bug
When using the Loki output plugin with the Tail input plugin and Kubernetes filter we're getting an error while using auto_kubernetes_labels on. We know that Loki is working correctly and that we can ship our logs there as we have a working Promtail deployment which we're trying to replace with this code.

To Reproduce
Using the following configuration on a K8s daemonset.

[INPUT]
  Name tail
  Path /var/log/containers/*.log
  Parser docker
  Tag kube.*
  Mem_Buf_Limit 5MB
  Skip_Long_Lines On
  Docker_Mode On
  Docker_Mode_Flush 4

[FILTER]
  Name kubernetes
  Match kube.*
  Merge_Log On
  Merge_Log_Key log_processed
  Keep_Log Off
  K8S-Logging.Parser On
  K8S-Logging.Exclude On

[OUTPUT]
  Name loki
  Match kube.*
  Host loki.logging.svc
  Port 3100
  Labels test=aaa
  auto_kubernetes_labels on

The error we're getting.

2020-11-20T12:39:52.544338589Z [2020/11/20 12:39:52] [ warn] [engine] failed to flush chunk '1-1605875992.200309438.flb', retry in 9 seconds: task_id=0, input=tail.0 > output=loki.1
2020-11-20T12:39:53.539678059Z [2020/11/20 12:39:53] [error] [output:loki:loki.1] loki.logging.svc:3100, HTTP status=400
2020-11-20T12:39:53.539713222Z error parsing labels: parse error at line 1, col 50: syntax error: unexpected ., expecting = or =~ or !~ or !=

Expected behavior
Logs should be shipped to Loki correctly.

Screenshots
n/a

Your Environment