Splunk-connect-for-kubernetes: Splunk connect for Kubernetes and Universal Forwarder

[Joerg-L]

Hi there
we are running kubernetes on prem and want to collect the logs with splunk connect for kubernetes and move the logs to the splunk cloud.

As the systems on prem can't connect externally, I would think that we can setup an universal forwarder on a machine, which has external access, and set the host in the config of splunk connect for kubernetes to the host name of the splunk forwarder.

Isn't it like that?
Thanks
Joerg

[dbaldwin-splunk]

There is the option of sending to HEC in Splunk Cloud - See docs.
http://dev.splunk.com/view/event-collector/SP-CAAAE7G#cloud
HEC setup is also listed in the pre-reqs and the following steps.
https://github.com/splunk/splunk-connect-for-kubernetes#prerequisites

Did you want to use a UF in your architecture?

[Joerg-L]

I have found the solution, I think, right now as I have configured the Universal Forwarder with an HEC Input, described like here http://dev.splunk.com/view/event-collector/SP-CAAAE6Q and just pointed the conenct for kubernetes to the UF-Machine.

Seams to work, more or less :D

[chaitanyaphalak]

Thanks @Joerg-L for getting back. You might have possible throughput and scalability implications by using this topology. Let us know if you need more help working this out.

[Joerg-L]

@chaitanyaphalak
I would think that we will have some "topics" like troughput or scale ability in any case.

I expect that the final setup will be to have:

• several Universal Forwarder Machines
• Loadbalancer like our Nginx Cluster in front of
• connect for kubernetes accessing the nginx cluster with single name

[matthewmodestino]

This is definitely a valid architecture, however, if you do not properly deploy or scale that intermediate tier (HF or UF), you will run into issues with throughput and indexing delays.

We do try and advise customers to go straight from the cluster to splunk cloud for highest performance, but we understand some orgs have a requirements to never send direct from the cluster to the internet.

I generally recommend running that layer in a container enviro, so you can actually scale it. Anecdotally, I would start with 3-4 HFs in each datacenter where your clusters live. Then scale it up based on indexing rate/throughput.

Customers usually drop 1 or 2 HFs or UFs and come back to us wondering why 100 fluentd collectors going through a single box is a causing delays.... don't be that guy ;)

Check out see scenario 3 here:

http://dev.splunk.com/view/event-collector/SP-CAAAE73

For max performance, you can want to terminate the connection from fluentd on the ingress of your loadbalancer, then allow the LB to balance the individual hec requests that come via said connection. Splunk Connect for Kubernetes uses persistent connections, which can cause the connection to stick to a single HF/UF. with sufficient collectors and HFs/UFs...you can manage it, but you can increase the ability to spread your data efficiently across your indexers.

If you have any questions let me know!

[Joerg-L]

@matthewmodestino thanks for your feedback.

We will monitor that in deep and we will see, what will happen on peformanc/delay.

We observe now a more "simpler" problem:
Our logs on-prem are almost "big" and now the system trys to import all historic log lines which will "blow" our daily index limit.

Is there any setting I can set to avoid import of the historic logs?

[matthewmodestino]

fluentd gives a ton of control on what you will pick up with the tail plugin. Ideally your container platform should be rolling logs (via docker or logrotate), or purging...otherwise I will check to see if they have a setting to only pick up files.

Will check fluentd docs for settings like Splunk ignoreOlderThan

[Joerg-L]

Such a setting would make sense to be available when deploying the connect for kubernetes

[chaitanyaphalak]

Closing as resolved. Open a new issue if you need further assistance.