Fluent-bit: [filter_kubernetes] enhancement: provide mechanism to exclude containers from fluent bit via annotations

[TheRealDwright]

Problem Statement
In the current implementation the most common way to get container logs parsed by fluentbit in a kubernetes cluster is to have a filter applied to containers using a log message like the one detailed here:

input-kubernetes.conf: |
  [INPUT]
      Name              tail
      Tag               kube.*
      Path              /var/log/containers/*.log
      Parser            docker
      DB                /var/log/flb_kube.db

This represents a bit of a problem with dev containers sometimes polluting our log platform with unwanted logs. We have a select few services we want to include, but the default rule is to exclude.

Describe the solution you'd like

A simple way to include logs in the opposite of how #555 was implemented where you can annotate containers with a fluentbit.io/include: "true" and the fluentbit daemonset will only pick up these logs.

Describe alternatives you've considered

I've updated my Path in the above config to /var/log/containers/*deployment*.log and ensured all my deployments I want to aggregate logs for have deployment in the name.

Additional context



I have a Kubernetes cluster set up with kops 1.10 and used https://github.com/fluent/fluent-bit-kubernetes-logging to set up fluentbit which then forwards to a fluentd service running the logz.io plugin.

[edsiper]

looks like I misunderstood the original requirement, you can try the following annotation:

fluentbit.io/exclude: "true"

[TheRealDwright]

Hi @edsiper thanks for replying, but I think I've poorly worded this enhancement sorry.

I want to have fluent-bit by default exclude, and annotate a select few deployments we can include.

[kaspernissen]

I'm searching for a similar thing. I want to e.g. say that all pods with label: something should be included, otherwise just discard the logs. Maybe something like:

 K8S-Logging.include label=something,label=anotherthing

[botzill]

Hi. is there any solution to this so far? I'm also looking for this. Will be really helpful.

Thx!

[slayerjain]

Hey, Has anyone made any progress here? I'm also trying to figure out a solution to this :)

[infa-ddeore]

this will be very useful feature and end users will have control on sending logs by adding annotations

[albertocsm]

+1
option to exclude by default and opt in include

[infa-ddeore]

sad that this issue is open since Sep 2018 and still no solution yet

[dmytroleonenko]

https://github.com/dmytroleonenko/fluent-bit/tree/v1.4.2-include
In case anybody interested. I'm not sure the way I injected the logic would be fine with the upstream dev team. Works for me. If the config has include mode enabled, only pods with

fluentbit.io/include: "true"

annotations are sent

[infa-ddeore]

https://github.com/dmytroleonenko/fluent-bit/tree/v1.4.2-include
In case anybody interested. I'm not sure the way I injected the logic would be fine with the upstream dev team. Works for me. If the config has include mode enabled, only pods with

fluentbit.io/include: "true"

annotations are sent

how to enable include mode and do you have docker image already built?

[dmytroleonenko]

https://github.com/dmytroleonenko/fluent-bit/tree/v1.4.2-include
In case anybody interested. I'm not sure the way I injected the logic would be fine with the upstream dev team. Works for me. If the config has include mode enabled, only pods with

fluentbit.io/include: "true"

annotations are sent

how to enable include mode and do you have docker image already built?

You can enable it the same way like exclude mode, just use "include" instead of "exclude" word in both config and annotation. Include should work with "exclude" in combo equally well if you want to exclude particular container from the pod or specific stream (think of stdout) off of a pod log.
I use https://github.com/aws/aws-for-fluent-bit.git to build an image for EKS logger. Slightly modified their Dockerfile to get fluent-bit sources from a zip file (based on my fork sources) instead of their git clone way
I think I can build/push an image to the Dockerhub.
Check it here https://hub.docker.com/r/melco/aws-for-fluent-bit once DockerHub manages to build it

[edsiper]

let me confirm the expectation from for the default behavior:

One of K8S-Logging.Exclude or K8S-Logging.Include must be enabled (not both), behaviors:

| K8S-Logging.Exclude | K8S-Logging.Include | Pod Annotation | Process Log ? |
| --|--|--|--|
| On | Off | exclude: "true" | No |
| On | Off | exclude: "false" | Yes |
| Off | Off | any | Yes |
| Off | On | include: "true"| Yes |
| Off | On | include: "false" | No |

comments ?

[TheRealDwright]

@edsiper I think your approach makes the most sense. We've taken another path but it sounds like there are others who would like this enhancement.

[dmytroleonenko]

let me confirm the expectation from for the default behavior:

One of K8S-Logging.Exclude or K8S-Logging.Include must be enabled (not both), behaviors:

K8S-Logging.Exclude K8S-Logging.Include Pod Annotation Process Log ?
On Off exclude: "true" No
On Off exclude: "false" Yes
Off Off any Yes
Off On include: "true" Yes
Off On include: "false" No
comments ?

If the "false" annotation is a default behavior (no annotation == false annotation). Like if K8S-Logging.Include On then if I don't have any annotations on any pods what would happen?

[edsiper]

@dmytroleonenko the proposal above says: if K8S-Logging.Include is turned on, only the Pods that have an annotation fluentbit.io/include: "true" will be included in the pipeline, otherwise discarded.