Splunk-connect-for-kubernetes: Fluentd tuning for logging pods on big nodes.

[MikeTomlin19]

What happened:
We have deployed this to all of our OpenShift 4 bare metal clusters and the clusters with the most activity and pods/node are not able to send all container logs to splunk and/or have a long (30 min.) queue/delay in becoming visible in the splunk indexers.

some logs from a splunk-kubernetes-logging pod with info-level logs filtered out:
2020-11-06 14:03:32 +0000 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=92.02205770404544 slow_flush_log_threshold=20.0 plugin_id="object:2ade063e1c3c"
2020-11-06 14:04:40 +0000 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=67.76551360095618 slow_flush_log_threshold=20.0 plugin_id="object:2ade063e1c3c"
2020-11-06 14:05:36 +0000 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=56.591755325032864 slow_flush_log_threshold=20.0 plugin_id="object:2ade063e1c3c"
2020-11-06 14:06:35 +0000 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=58.183909379993565 slow_flush_log_threshold=20.0 plugin_id="object:2ade063e1c3c"
2020-11-06 14:07:42 +0000 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=67.70138586801477 slow_flush_log_threshold=20.0 plugin_id="object:2ade063e1c3c"
2020-11-06 14:08:38 +0000 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=55.92357094195904 slow_flush_log_threshold=20.0 plugin_id="object:2ade063e1c3c"
2020-11-06 14:09:47 +0000 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=68.77938993298449 slow_flush_log_threshold=20.0 plugin_id="object:2ade063e1c3c"
2020-11-06 14:10:42 +0000 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=54.5333183449693 slow_flush_log_threshold=20.0 plugin_id="object:2ade063e1c3c"
2020-11-06 14:12:06 +0000 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=83.83802559098694 slow_flush_log_threshold=20.0 plugin_id="object:2ade063e1c3c"
2020-11-06 14:13:16 +0000 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=70.05958420498064 slow_flush_log_threshold=20.0 plugin_id="object:2ade063e1c3c"

What you expected to happen:
We expect the splunk-kubernetes-logging pods to be able to send all of the container logs from the host to the splunk HEC in real time.

How to reproduce it (as minimally and precisely as possible):
Use the default settings in the 1.4.3 branch of this repo, and run over 100 pods/node.

Anything else we need to know?:
Our application nodes are 384 GB/RAM. Our average pod size is 1.5-2 GB/RAM.
RH CoreOS uses cri-o, not docker, so the log file locations are different.
We've tried changing the buffer type to "file" and the "retry_forever=true" parameter. This results in less dropped logs, but a delay of at least 30 minutes for them to show in splunk.

Environment:

• Kubernetes version (use kubectl version):
  $ kubectl version
  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2-0-g52c56ce", GitCommit:"d7f3ccf9a5bdc96ba92e31526cf014b3de4c46aa", GitTreeState:"clean", BuildDate:"2020-09-16T15:25:59Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.3+47c0e71", GitCommit:"47c0e71", GitTreeState:"clean", BuildDate:"2020-09-17T23:10:07Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g: cat /etc/os-release):
  $ cat /etc/os-release
  NAME="Red Hat Enterprise Linux CoreOS"
  VERSION="45.82.202009181447-0"
  VERSION_ID="4.5"
  OPENSHIFT_VERSION="4.5"
  RHEL_VERSION="8.2"
  PRETTY_NAME="Red Hat Enterprise Linux CoreOS 45.82.202009181447-0 (Ootpa)"
  ID="rhcos"
  ID_LIKE="rhel fedora"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos"
  HOME_URL="https://www.redhat.com/"
  BUG_REPORT_URL="https://bugzilla.redhat.com/"
  REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
  REDHAT_BUGZILLA_PRODUCT_VERSION="4.5"
  REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
  REDHAT_SUPPORT_PRODUCT_VERSION="4.5"
  OSTREE_VERSION='45.82.202009181447-0'

• Splunk version: 8.0.3

• Splunk Connect for Kubernetes helm chart version: 1.4.3
• Others:

[rockb1017]

Hello,

your heavy nodes seem to be hitting the fluentd bottleneck. its limit is around 3,000 events per seconds.
I would suggest spreading apps with high load of logs across different nodes. Also using Universal Forwarder is an option several others are using for high throughput.
Let @matthewmodestino know if you are interested in UF option.

[MikeTomlin19]

Thank you, @rockb1017.
@matthewmodestino is the universal forwarder compatible with CRI-O, OpenShift 4's container runtime? If so, are there helm charts etc.. for deploying it?
I'd also be happy to share more about our existing setup to ensure that we are truly not able to use fluentd for our scale.

[dilipbailwal]

@rockb1017, would you know how to check the limitation on fluentd or any splunk query where we can see data hiting that limitation ..?

[matthewmodestino]

@MikeTomlin19 yes the UF should in Openshift under cri-o. I run under containerd in Microk8s without issue. here is a sample test.

https://mattymo.io/code/mattymo/ta-k8s-logging/-/snippets/2

basic sample config here: https://mattymo.io/code/mattymo/ta-k8s-logging - still work in progress as we explore multiline and metadata enrichment with ingest time lookups and evals

No helm charts, likely destined for splunk operator -> https://github.com/splunk/splunk-operator would you mind opening an issue there and requesting UF node agent and/or sidecar there?

[rohit482]

Hi @matthewmodestino,

Where we have to setup Splunk Agent(UF)? In container host or CoreOs ?

Below the setting I am looking for to set index name etc.
basic sample config here: https://mattymo.io/code/mattymo/ta-k8s-logging - still work in progress as we explore multiline and metadata enrichment with ingest time lookups and evals

[MikeTomlin19]

@matthewmodestino what do you think about having 2-3 logging daemonsets? Could we use the exclude_path parameter or filters to distribute the container logs across the 3 logging fluentd instances on each node? For example, logging_container_1 grabs all logs from containers whose name starts with the letter "a-h", logging_container_2 for all containers "i-p" and logging_container_3 ""q-z".

[dengalebr]

@MikeTomlin19 we use 4 DaemonSet in our production clusters to meet the high EPS requirement. We have divided with application (from pod names using regex), we have first 3 tiers where we specified regex to read from. Last tier we included everything from /var/log/containers/*.log but exclude first 3 which are in another DS.

[matthewmodestino]

I think this is where we need to start thinking about convenience versus scale with node agent vs sidecar pattern.

https://kubernetes.io/docs/concepts/cluster-administration/logging/

If you have to run multiple daemonsets anyways, at what point does simply bolting application specific sidecars, or perhaps start looking at a model where we automate sidecar as part of app deploy? Even if just for the "crown jewels".

@dengalebr have you flipped over to UF? I have been able to use settings to get the parsing almost identical to fluentd, minus the labels...

I had the pleasure of talking to an internet scale customer back in 2017 and they were happily bolting sidecars to deploys, despite the rage from the "12 factor app" folks...but now looking back, they looking real smart :), of course they had to take care of the middleware to bolt on the sidecars, but last i heard they had 7K+ ufs running across all nodes. but no data problems they mentioned.

[dengalebr]

@matthewmodestino yes we tried UF but stuck on multi-line support. We observed UF is better than fluentd but almost equal to fluent-bit with multi-line for performance. We are working with fluent-bit contributors to fine tune further and add multi threading there so we can replace with fluentd.

Do let us know if you get any luck with multi-line without DSP in UF.

[matthewmodestino]

@dengalebr was any tuning done on UF? (parallelPipelines & Queue tuning? maxKBps??)

It looks like there is a way to possibly configure Splunk props/transforms to resubmit events to the "aggQueue" in Splunk to allow linebreak. It was PoC'd in the slack chat for CRI-O but we should take a closer look.

[dilipbailwal]

@matthewmodestino is there any documentation for the UF for CRI-O
We were not much successful with your gitlab