Splunk-connect-for-kubernetes: splunk-kubernetes-logging - allow pod and container exclusion

[stiller-leser]

Hi everybody,

first off: Great work. This is really easy to use!

What would you like to be added:

I'd like to be able to limit the collected pod logs, in order to limit what is send to Splunk. Currently I am doing the following:

global:
  fluentd:
    # path of logfiles, default /var/log/containers/*.log
    path: /var/log/containers/*.log
    # paths of logfiles to exclude. object type is array as per fluentd specification:
    # https://docs.fluentd.org/input/tail#exclude_path
    exclude_path:
     - /var/log/containers/prometheus*.log
     - /var/log/containers/istio*.log
     - /var/log/containers/kube*.log

This works if I want to exclude pods (even though it is cumbersome), but this won't affect side car containers. So for Istio for example, I might only be interested in the logs of my apps, but not of the istio-proxy sidecar.

Why is this needed:

Limit the input into Splunk and increase visibility

[chaitanyaphalak]

Hey @stiller-leser, thanks for the kudos.
I think that you might be able to add an exclude path for the pods you dont want.
For example, if your app's pod name is "my-awesome-pod-xxxxxxx" I think you would be able to ignore logs for that using "/var/log/containers/my-awesome-pod*.log".
I agree that this approach is cumbersome.
But I think if you really want it, you could do a pull request for us in the following way:
Approach A)

• In values.yaml file take a list of pod names you want to ignore
• Then add a function to https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/helm-chart/splunk-kubernetes-logging/templates/_helpers.tpl file to check for pod_name and check if its in the ignore list, if yes drop the record.

Approach B)

• use a custom filter https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/helm-chart/splunk-kubernetes-logging/values.yaml#L355 to drop the records using a similar logic/using another plugin.

Please try out some other ideas too, if you want and let me know if you need help. I would be happy to review a PR for the implementation.

Thanks, Happy Friday.

[matthewmodestino]

+1 for updating the configmap to write the proper regex in the path. path can also take comma separated lists:

<source>
      @id containers.log
      @type tail
      @label @SPLUNK
      tag tail.containers.*
      path /var/log/containers/*.log
      pos_file /var/log/splunk-fluentd-containers.log.pos
      path_key source
      read_from_head true
      <parse>
        @type json
        time_key time
        time_type string
        time_format %Y-%m-%dT%H:%M:%S.%NZ
        localtime false
      </parse>
    </source>

https://docs.fluentd.org/input/tail#path

exposing it in helm is definitely useful when getting started, but in practice, updating the config in gitmap then pushing with some CI/CD pipe is probably ideal anyways, or supporting annotations where you would set an exclude on the deployment level.

Or maybe for the future, a CRD that listens for the changes to the configmap, though thats probably more suited for a Splunk Operator.

[stiller-leser]

Hi @chaitanyaphalak,

thank you for the feedback. I am currently using the exclude_path to exclude pods. This works fairly well. However, my main point here was that I do want to ignore containers in pods. So I guess since the containers log into the same log file as the pods, I would need to additionaly filter the log files to exlude the lines that describe a different container.

For now, I think this is good enough for me.

Best regards,
stiller-leser

[chaitanyaphalak]

@stiller-leser I am glad this approach works for you, let me know what you explore for containers in a pod, possibly post an example so that people in the community can benefit from your work :) Possibly you can create a PR to support this use-case :)
Thanks.

[chaitanyaphalak]

Closing as resolved. Please reopen/create a new one if you need further support.

[fulvi0]

Hi @chaitanyaphalak, is it possible to use customerFilter for nodes or Namespaces? It will be nice to logging only the pods that are running in specific nodes or Namespace.

[chaitanyaphalak]

@fulvi0 yes, that was the purpose for adding it:)
For namespace you could do something similar to https://github.com/splunk/splunk-connect-for-kubernetes/issues/220#issuecomment-519156055
For nodes, you can use taints so that logging pods dont run on tainted nodes.

[PrabuddhaRaj]

HI @chaitanyaphalak : i am using exclude_path to remove logs of the sidecar pod running in my kubernetes cluster. But it doesnt work. i tried annotating the pod with splunk.com/exclude=true but that also doesnt work as i am facing issue in restarting the pod and secondly i am suing kubectl to annotate. it should be done via script such values.yaml file. I only want logs of my application pod
any help is appreciated