Fluent-bit: Loki plugin - remove labels from Loki fields

[jcdauchy-moodys]

Bug Report

Describe the bug
Using line_format=key_value, labels are pushed as fields to LOKI

To Reproduce
Here is an example of parsed fields, the following attributes are "labels" :

• k8s_pod_name
• k8s_namespace_name
• k8s_container_name

k8s_pod_name="grafana-xxxxxx" k8s_namespace_name="monitoring" k8s_host="ip-xxxxxxxx.xxxxx-1.compute.internal" k8s_container_name="grafana" level="info" message="Request Completed" logger="context" userId="0" orgId="0" uname=null method="GET" path="/" status="302" remote_addr="172.17.0.1" time_ms="0" size="37" referer=null

• Steps to reproduce the problem:

Here is my configuration :

[OUTPUT]
    Name loki
    # For the moment, we send all containers kube
    Match kube.*
    # For fluent-bit native loki plugin
    #host loki-distributor.monitoring.svc.cluster.local
    host ${LOKI_IP}
    # Port 3100
    port ${LOKI_PORT}
    # List of keys to use as Labels in Loki
    labels component=k8s-containers
    # Which fields are used as labels : https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/record-accessor
    label_keys $k8s_namespace_name,$k8s_container_name,$k8s_pod_name
    # Send as logfmt format
    line_format key_value

Expected behavior

All labels should be removed from fields sent to LOKI

Screenshots

Your Environment

• Version used: 1.6.6
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?): Kubernetes 1.16
• Operating System and version: Linux AWS
• Filters and plugins: Loki plugin

[zhiguangwang]

Same here: using fluent-bit 1.6.8, can reproduce with either line_format=key_value or line_format=json.

It would be nice to provide a configuration parameter remove_label_fields, like vector.

[l2dy]

FYI, Loki's official Fluent Bit output plugin (written in Go) supports this and it's the default behavior. https://github.com/grafana/loki/tree/master/cmd/fluent-bit

[phin1x]

The loki team plans to deprecate their go plugin. From the loki 2.1 release notes:

"...In time our plan is to deprecate and eliminate our output plugin in favor of the native Loki support. However until then you can continue using the plugin with the following change..."

In my opinion the missing remove_label_fields parameter is the reason not to use the output plugin. I don't want to have the time or labels, which i already send as stream labels in the log line.
The loki go plugin also converts the log line from kv/json to a single plain log line, if only the log key remains. this is also a nice feature because we don't need to parse the log line in loki again.