Splunk-connect-for-kubernetes: CR/LF in the logs --> header field value cannot include CR/LF

[tioumen]

Version of the components

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

2018-09-25 08:20:34 +0000 [info]: starting fluentd-1.2.0 pid=5 ruby="2.5.1"
2018-09-25 08:20:35 +0000 [info]: gem 'fluent-plugin-concat' version '2.2.2'
2018-09-25 08:20:35 +0000 [info]: gem 'fluent-plugin-jq' version '0.5.1'
2018-09-25 08:20:35 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.0.1'
2018-09-25 08:20:35 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.0.1'
2018-09-25 08:20:35 +0000 [info]: gem 'fluent-plugin-systemd' version '0.3.1'
2018-09-25 08:20:35 +0000 [info]: gem 'fluentd' version '1.2.0'

I'm getting CR/LF in the logs evrywhere which avoid pushing the logs to Splunk instance.

Error I'm getting

2018-09-25 08:20:41 +0000 [warn]: #0 bad chunk is moved to /tmp/fluentd/backup/worker0/object_3fca9345192c/576adc7bf8bbd67733d4d9a92e8bbde2.log
2018-09-25 08:20:46 +0000 [warn]: #0 got unrecoverable error in primary and no secondary error_class=ArgumentError error="header field value cannot include CR/LF"

Did you already deal with this error ?

Thanks.

[matthewmodestino]

Hi tioumen!

I actually ran into this with another customer, and it turned out to be that there was an erroneous character when pasting in the HEC token into the secret.

Please review the edits you made to the manifests/yaml files, specifically when you pasted your HEC token in, and ensure there are no extra characters trailing the token you copied in.

Did you deploy the manifests manually or using helm?

notepad++ or vi with :set list should do the trick.

[tioumen]

hey matthew!
Thanks for replying to this.

Actually I'm not using helm, directly used the manifest. For the token I used the secret to encore it in base64. I checked directly in the container from 'echo $SPLUNK_HEC_TOKEN' and the value doesn't contain any trailing character.

Any idea where I've done something wrong ?

Thanks again for your help

[matthewmodestino]

Can you verify the actual yamls you used using vi or notepad++?

Without seeing them it will be hard to advise, but it definitely sounds like there is an issue with the changes made to the yaml.

It has to be either the HEC token or any of the editing for the output in the configmaps

[dtregonning]

Hi @tioumen - did you have any luck with @matthewmodestino 's suggestions?

[tioumen]

Hey, sorry for the late answer, yeap finally I got it fixed. I didn't found the issue but resetting our git repo with fresh version we got it working fine. Maybe related to typo error ... somewhere.

[tioumen]

I close the issue.

[mightaswelby]

I know this issue is closed, but we've run into this a few times recently. It wasn’t a bad copy/paste, it was running echo without -n to prevent IT from inserting a crlf to give you the next prompt.

So when we base64 encoded “mytoken”

[root@mytest_bot]# echo "mytoken"
mytoken - there’s a newline here inserted by echo to move you to the next line, which got included in the | base64
[root@mytest_bot]#

But if you run it with the -n:

[root@mytest_bot]# echo -n "mytoken"
mytoken[root@mytest_bot]#

No newline, so that encodes ok. This goes for ANY secret you try to create with a manual base64 encode, so something to watch out for in the future as well or any time someone is getting an unexpected error with a secret.

[trustarun]

I was facing the same issue, while connecting with QB on Ruby on Rails development platefrom.

I have to pass authorization header after Base64 encoding of credentials. I realized that the ruby library returning n in the encoded string

Base64.encode64("ghhjkl")
=> "NTZnaGhqa2w=n"

I have to use strict encode, to get the decoding without n

Base64.strict_encode64("ghhjkl")
=> "NTZnaGhqa2w="