Splunk-connect-for-kubernetes: Support fluentd configuration for containerd logging format

[nisc-acooper]

What would you like to be added:
Support for running containerd directly instead of docker as a runtime

Why is this needed:
Currently, this project is expecting the docker json logging format to exist for container log output. We attempted to move towards only using containerd/cri-containerd as a runtime.
our splunk logging agents now spam "parsing errors"

This seems to have been somewhat solved at the fluentd level in this related issue
https://github.com/cri-o/cri-o/issues/897

I am not sure if the splunk-bundled fluentd supports the "multi-parsing" plugin used here. I can attempt to try this manually by adjusting config maps provided by existing helm chart.

[matthewmodestino]

Hi @nisc-acooper!

Please check out the develop branch and the confguration option for CRI-O. https://github.com/splunk/splunk-connect-for-kubernetes/blob/3470aa431ae875fd3bd581de8ac9db21c7065eba/helm-chart/splunk-kubernetes-logging/values.yaml#L51

We got it working on Openshift 4.2 but I believe the config needs a small update to ensure the fields we want remain extracted, will link to issue as i get it created.

We do provide the concat plugin in the images and the manifests provide examples you can use.
https://github.com/splunk/splunk-connect-for-kubernetes/issues/255

[nisc-acooper]

Thanks @matthewmodestino for the quick reply. Good to hear that this is already being worked on. It does not appear that the multi_format plugin is available in the splunk images. The benefit of the multi_format approach is to allow for a clean migration to containerd from docker in the same cluster (rolling node-at-a-time). Based on the helm chart settings you are planning for, it would be a deployment-wide change.

Could you consider passing multiple options to "Log format type" to allow for an easier transition?

[matthewmodestino]

Interesting...are you using Openshift or OSS k8s with CRI-O?

Is this a supported migration path drawn up somewhere by the OSS project or Red Hat, etc?

In theory adding a plugin isnt too hard to prove out, and from what I am hearing might really only be needed during the transition?

I’d love to hear if this is a common practice...

[nisc-acooper]

We have been using OSS k8s with docker via kubeadm installer. k8s supports containerd as a runtime, we were looking to bypass docker and only install containerd (docker bundles older versions of it and uses it under the covers) to allow for faster releases of containerd, reducing complexity/unneeded components of docker, and performance gains on the kube instances.

https://kubernetes.io/docs/setup/production-environment/container-runtimes/

The cri-o issue link was just an example of a way to allow for both logging types at the same time (I think cri-o and containerd both have the same logging format).

[matthewmodestino]

So you are not considering just building a new cluster with the new container runtime and migrating the workloads there, then turning down nodes as you upgrade? Why/How are you using the mix mode?

[nisc-acooper]

I think a new cluster could be an option in some cases. We have bare-metal cluster installs in some cases which would be a bit overkill to greenfield for runtime changeout. Runtimes are a node-local construct, there are no requirements to only run a single runtime across all cluster instances.

If this doesn't fit your roadmap, we can definitely look into this on our end. So far, this logging migration appears to be our biggest hurdle.

Thanks

[chaitanyaphalak]

I dont think implementing this feature is a generic use case for everyone.

[matthewmodestino]

it would definitely be an Advanced option. But I do think this is better solved for by ensuring that the right daemonset scheduling is done at the k8s layer with taint or annotations etc. I am happy to review in deeper detail over on Slack, or if you will be at Kubecon.

[nisc-acooper]

I will be at kubecon! I think the daemonset with taints makes good sense based on the design of your helm chart. I assume you have no other concerns of running multiple deployments of the chart in the same cluster scheduled to different node sets?

[chaitanyaphalak]

With SCK 1.3.0 we support containerd. Closing the issue.