Splunk-connect-for-kubernetes: Log loss observed at 15MB/s@3000 events/second

[bhattchaitanya]

Team,

I notice that at around 5000 bytes per message and 15MB/second the fluentd Splunk connect forwarder fails to forward nearly 2/3rd of the generated logs.

We don't see any error logs generated from the daemonset.
We don't see any 5xx from Splunk HEC endpoint.
Splunk HEC load balancer does not indicate any latency problem.
The overall latency of logs that successfully make to Splunk has less than 10 seconds of TP99 latency.
Buffer flush retry_count at the source is less than 5
Buffer size at the source is less than 40mb. Provisioned 600mb.
Queue size less than 2.
CPU less than 70%

Observation:
-JQ plugin tends to utilize 30% of a single CPU core at peak load.
-Overall CPU across all cores is less than 70%

Configuration delta from default:
```

@type memory
chunk_limit_records 100000
chunk_limit_size 200m
flush_interval 5s
flush_thread_count 5
overflow_action throw_exception
retry_max_times 3
total_limit_size 600m

[chaitanyaphalak]

What kind of filters are you using?

[bhattchaitanya]

We are using default filters. Nothing special.
Here is our configMap https://gist.github.com/bhattchaitanya/63df0aa02709232d3839d648136213f0

[chaitanyaphalak]

Buffer flush retry_count at the source is less than 5
Buffer size at the source is less than 40mb. Provisioned 600mb.
Queue size less than 2.
CPU less than 70%

Observation:
-JQ plugin tends to utilize 30% of a single CPU core at peak load.
-Overall CPU across all cores is less than 70%

From the above quotes and the observation are you saying the settings from the helm charts are not being picked up by fluentd?

[bhattchaitanya]

I think the configuration looks correct. I don't see any red flags unless you say based on the gist link I provided that something is in fact wrong.

What we are concerned about is the log loss and we want to know if this is a known issue.
Also, it would be great to know if you guys have run any benchmark tests that we could refer to.

[matthewmodestino]

The two most common causes for missing logs I see in environments is due to the log rotation settings being too small for high velocity logging. Please ensure docker or logrotate is set to sane production/high volume size. 10MB is not enough (default). More like 1GB/10gb, is probably more reasonable.

The other issue could be filling the buffer, though you would see that in the pod logs. Our customers who have stressed test it generally opt to use a fluentd file buffer to ensure they dont lose data if network is down or splunk is overloaded.

this is just a sample, exact configs will vary.

 <buffer>
          @type file
          path  /var/log/splunk-fluentd/
          chunk_limit_size 8MB
          total_limit_size 1GB
          flush_interval 5s
          flush_thread_count 1
          overflow_action block
          retry_forever true
        </buffer>

https://docs.fluentd.org/buffer/file

Let me know if these get you sorted. If not, if you open a support ticket and we can gather info about your enviro and assist!

[bhattchaitanya]

We are trying to change docker logOpt so that we do less frequent log rotation. We will update this thread once we have the result.

[bhattchaitanya]

We did observe that the tail plugin was not able to catch up when the frequency of log rotation was high. We are running more tests to confirm.

[matthewmodestino]

I think there are some advanced settings you can use to ensure rotated files are handled accordingly.

https://docs.fluentd.org/input/tail#enable_watch_timer

also, I believe I have seen users create inputs that account for the rolled logs of each pod, instead of just the log being written to.

[bhattchaitanya]

We did see significant improvement by increasing our log rotation file size limits in the Kubernetes cluster.spec file.
Thanks for hinting the enable_watch_time flag! We will play around with that.
Can you elaborate more on the "inputs that account for rolled logs". What does this actually mean?

[kyaparla]

Hi All,

We noticed some similar behaviour, but after enabling below settings, we are not seeing data loss.

retry_forever: true
retry_type: periodic
retry_wait: 5s

But, there is delay in getting data in to splunk, and wondering how to minimize the delay. Here are our buffer settings.

"@type": memory
total_limit_size: 500m
chunk_limit_size: 100m
chunk_limit_records: 20000
flush_interval: 5s
flush_thread_count: 8
queued_chunks_limit_size: 8
overflow_action: drop_oldest_chunk
#retry_max_times: 10
retry_forever: true
retry_type: periodic
retry_wait: 5s

Can someone please help understand these better?

1. does fluentd send each chunk as one request to splunk?
2. does each chunk, hold data from different sources?
3. is flush thread count, to flush data from buffer or write data to buffer?
4. Our splunk hec has max incoming request size set to 20MB, so what should we use for chunk_limit_size?

[rockb1017]

Hello,

1. yes
2. yes
3. both. The number of threads to flush/write chunks in parallel
4. it should be something smaller than 20MB.

[rockb1017]

Hello,@kyaparla

Do you still see data delay?
Could you also set flush_mode to interval just to be sure?
https://docs.fluentd.org/configuration/buffer-section

edit:
oh i see that you are sending high EPS. fluentd has limit around 3000 eps. so data delay is caused by fluentd not tailing log files fast enough. I would recommend spreading the applications that generate lots of logs across multiple nodes.