Splunk-connect-for-kubernetes: Last event is not pushed to Splunk until new event occurs

[vbottu]

What happened:
Last log event is not pushed to splunk
What you expected to happen:
Everything should be pushed to Splunk
How to reproduce it (as minimally and precisely as possible):
This is my filter in output.conf
<filter tail.containers.var.log.containers.sb-*.log> @type concat key log timeout_label @SPLUNK stream_identity_key stream multiline_start_regexp /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}|^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}|^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\s-\s-/ multiline_end_regexp /\\n$/ separator "" flush_interval 2s use_first_timestamp true </filter>
Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):1.15.5
• Ruby version (use ruby --version):
• OS (e.g: cat /etc/os-release):NAME="Red Hat Enterprise Linux Server"
  VERSION="7.7 (Maipo)"
• Splunk version:7.3.3
• Others:

[matthewmodestino]

hmm, perhaps issue with the regex??

Can you provide some sample events that match this, and validate in rubular.com?

[vbottu]

I mean last event is not pushed to splunk, until a new event is there, when a new event is there the last but one will be pushed , sorry for the confusion

[vbottu]

Please see the sample event

2020-05-08T10:22:57,380-05:00 [influx-metrics-publisher] ERROR i.m.influx.InfluxMeterRegistry - failed to send metrics to influx java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:706) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at io.micrometer.core.ipc.http.HttpUrlConnectionSender.send(HttpUrlConnectionSender.java:60) at io.micrometer.core.ipc.http.HttpSender$Request$Builder.send(HttpSender.java:284) at io.micrometer.influx.InfluxMeterRegistry.publish(InfluxMeterRegistry.java:135) at io.micrometer.core.instrument.push.PushMeterRegistry.publishSafely(PushMeterRegistry.java:48) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

[vbottu]

324

[rockb1017]

Hello, which version of SCK are you using?
fix for related issues were released to 1.4.1.

[rockb1017]

https://github.com/splunk/splunk-connect-for-kubernetes/pull/369

[chaitanyaphalak]

Hey @vbottu as @rockb1017 mentioned a fix for this issue is already merged in the release 1.4.1. We request you to try out that release and get back to us. Thanks.

[vbottu]

Sure @chaitanyaphalak ,I will try out the 1.4.1 version

[vbottu]

I have tried installing version 1.4.1 , indexRouting is not working now, I am not seeing any of the logs pushed to splunk. I see the below error
2020-05-11 14:36:40 +0000 [error]: #0 Fluent::Plugin::SplunkHecOutput: Failed POST to https://xxxxxxxxxx:8088/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}

[vbottu]

splunk-kube-logging.txt
Attaching my logging chart values

[chaitanyaphalak]

Hey @vbottu, you will have to reconfigure index routing using https://github.com/splunk/splunk-connect-for-kubernetes#managing-sck-log-ingestion-by-using-annotations

[vbottu]

Ok, I’ve annotated my namespaces and I am seeing logs in Splunk now. What about the namespaces that does not have annotations and do we also need to ser indexRouting to true in addition to adding annotations . Thanks

[chaitanyaphalak]

For namespaces without annotations, add the appropriate annotations if you want to index logs from those namespaces. The older configs for indexRouting are now deprecated. Closing the issue as resolved for now. If you have another issue, please feel free to open up a new one. Thanks.

[matthewmodestino]

Hey I think I may know whats up here..

We are setting the concat flush_interval to 5s and the concat repo says it should be an integer.

I wonder if the 5s is ignored, thus setting 0 aka “wait forever”.

flush_interval (integer) (optional)

The number of seconds after which the last received event log will be flushed. If specified 0, wait for next line forever.

@vbottu can you try setting flush_interval 2 not flush_interval 2s when setting log concat/multiline rules?

I believe this may have been carried over/confused with fluentd output buffer setting which also is called flush_interval but takes 5s

[rockb1017]

one user have tried it with it https://github.com/splunk/splunk-connect-for-kubernetes/issues/243 but it didn't change anything. but i think this concat fix you developed fixed the flow.