Splunk-connect-for-kubernetes: "400 Bad Request Event field cannot be blank" when empty line is logged

[szymonpk]

I am running latest version of logging module of splunk connect for k8s (1.0.1).

Recently I have stumbled on a strange issue, whole batches of logs (fluentd tries to send logs in batches) were discarded by HEC with 400 response and message Bad Request Event field cannot be blank. Looks like all the processing done with jq transforms new line only log messages (\n) to empty events. HEC drops the whole batch even if the only one message it empty. In result, we were missing huge chunks of the logs. I know it is partially due to bad logging on our side. However, I think fluentd/Splunk should do a better job to protect the operator against such issues. As a workaround I have added:

      # ensure we do not have empty line logs, they cannot be ingested by Splunk and result in 400 response from
      # the Splunk HEC
      <filter tail.containers.**>
        @type jq_transformer
        jq 'if .record.log == "\n" then .record.log = "E" else .record.log = .record.log end | .record'
      </filter>

right before <filter tail.containers.**>. It does the job, we do not lose logs anymore, developers see they application is pushing garbage (empty logs are shown as E in Splunk).

Is there a better way of solving this?

[matthewmodestino]

Hey Szymon!

Thanks for sharing this!

I ran into this with another user as well while setting up an input for nix audit logs.

Out of curiosity, is the log that is generating the empty lines using the default source and filters in the configmap of fluentd? Or any other parsing/rules happening?

Definitely need to add a fail safe for this.

[szymonpk]

It was log directly from docker, in json it was something like "message": "\n", I didn't had any additional rules.

[dancb10]

Hello,
We have the same issue on our side, the 2019-01-16 13:22:27 +0000 [error]: #0 Failed POST to https://splunk-hec:8088/services/collector, response: {"text":"Event field cannot be blank","code":13,"invalid-event-number":XX} error is thrown continuously. @szymonpk how did you add that line? did you fork the helm chart and added the filter in the configmap directly or did you managed to use the values override.
I think it would be nice to have a way of adding custom filters within the chart.

[szymonpk]

@dancb10 we have own fork. I didn't get any clear response if I should go with PR with my 'hack' (pushing records just with 'E' isn't the nicest idea) or if maintainers would like to implement something more general like mentioned custom filters.

[matthewmodestino]

We could look at using the null output plugin instead of sending events with E.

https://docs.fluentd.org/v1.0/articles/out_null

Would that be preferable?

[szymonpk]

For sure. I would like to throw them out.

[dancb10]

@szymonpk following your advice I've forked the helm and added the following in the configMap.yaml, in the output.conf section:

      {{- if and .Values.customFilters }}
      {{- range $name, $filterDef := .Values.customFilters }}
      <filter {{ $filterDef.tag }}>
        @type {{ $filterDef.type }}
        {{ $filterDef.body }}
      </filter>
      {{- end }}
      {{- end }}

The section is placed just above the section '# extract index fields and sourcetype for container logs
'
In the values.yaml file I've defined a default filter as follows:

customFilters:
  EmptyEventFilter:
    tag: tail.containers.**
    type: jq_transformer
    body: >
        jq 'if .record.log == "\n" then .record.log = "EmptyEvent" else .record.log = .record.log end | .record'

So now all events with the empty field can be seen under the "EmptyEvent" section

[h0ppyf33t]

Exclude empty lines in your configMap.yaml. The below helped us resolve this issue.

      # filter to  remove empty lines
      <filter tail.containers.**>
        @type grep
        <exclude>
         key log
         pattern ^$
        </exclude>
      </filter>

[matthewmodestino]

would be interested in the performance impacts in the various options. Will test a few as part of making this a default filter.

[chaitanyaphalak]

Closing due to inactivity

[matthewmodestino]

@rockb1017 @chaitanyaphalak can we revisit this? Ran into it again today. Should have a safeguard in the filters that protects against chunk loss due to empty events.

For now I’m going to add this at the end. Thanks @h0ppyf33t

 # filter to  remove empty lines
      <filter tail.containers.**>
        @type grep
        <exclude>
         key log
         pattern ^$
        </exclude>
      </filter>

And try to ensure a concat filter is added to any offenders

Thanks!

Matt

[matthewmodestino]

Root cause was Springboot and Istio Sidecar injector pods sending “\n” chars to break walls of text in cli.

We edited the configmap to insert the above rule. We should ship this in custom filters or default filters imo.

[rockb1017]

Thank you Matt.
So does pattern ^$ captured and exclude \n ?

[matthewmodestino]

yeah it stops capturing at the end of line, without \n i believe...keep me honest @h0ppyf33t

[h0ppyf33t]

The regex honors the EOL \n (CRLF). But, excludes all standalone \n s.
Here is an example:
https://regex101.com/r/SAbI4N/2

[rockb1017]

Thank you. this change has been staged for next release.