Fluent-bit: "Length of content has execeded limit" moving from 1.2.1 -> 1.3.0

Created on 2 Oct 2019 · 11Comments · Source: fluent/fluent-bit

Bug Report

Describe the bug
When moving from fluent-bit 1.2.1 to 1.3.0 it now errors on startup "Length of content has exceeded limit"

To Reproduce
Running fluent-bit with same config below on 1.3.0

Logs on startup

[2019/10/02 15:02:28] [  Error] File /fluent-bit/etc/fluent-bit.conf
[2019/10/02 15:02:28] [  Error] Error in line 118: Length of content has execeded limit

Steps to reproduce the problem:

By using same configuration file used with 1.2.1 and trying it with 1.3.0.

Expected behavior
Expected to be able to move the same config that we were using with fluent-bit 1.2.1 and use it in 1.3.0 without issues.

Your Environment

Version used: 1.3.0
Configuration:

fluent-bit.conf

[SERVICE]
    Flush           2
    Daemon          off
    Log_Level       ${LOG_LEVEL}
    Config_Watch    on
    Parsers_File parsers.conf
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

[INPUT]
    Name             tail
    Path             /var/log/containers/*.log
    Parser           docker
    Tag              kube.*
    Refresh_Interval 5
    Skip_Long_Lines  On
    DB               /tail-db/tail-containers-state.db
    DB.Sync          Normal

[INPUT]
    Name             tail
    Path             /var/lib/origin/audit-ocp.log
    Tag              audit
    Refresh_Interval 5
    DB               /tail-db/tail-audit-ocp-state.db
    DB.Sync          Normal
    Parser           json

[INPUT]
    Name            systemd
    Tag             host.*
    Systemd_Filter  _SYSTEMD_UNIT=docker.service
    Systemd_Filter  _SYSTEMD_UNIT=kubelet.service
    Systemd_Filter  _SYSTEMD_UNIT=node-problem-detector.service
    Systemd_Filter  _SYSTEMD_UNIT=atomic-openshift-node.service
    Systemd_Filter  _SYSTEMD_UNIT=sshd.service
    Systemd_Filter  _SYSTEMD_UNIT=auditd.service
    Max_Entries     2000
    Read_From_Tail  false
    DB               /tail-db/systemd-state.db
    DB.Sync          Normal
    Strip_Underscores On


[FILTER]
    Name                kubernetes
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Match               kube.*
    Kube_Tag_Prefix     kube.var.log.containers.
    Merge_Log           On
    Keep_Log            Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On


[FILTER]
    Name record_modifier
    Match host.*
    Whitelist_key HOSTNAME
    Whitelist_key SYSLOG_IDENTIFIER
    Whitelist_key MESSAGE
    Whitelist_key PID
    Whitelist_key EXE

# nest the record under the 'event' key
[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard *
    Nest_under event


[FILTER]
    Name    lua
    Match   kube.*
    script  hostname.lua
    call    cb_replace

[FILTER]
    Name      modify
    Match     host.*
    Add index ${SPLUNK_OPERATIONS_INDEX}
    Add host ${NODE_NAME}
    Add source systemd

[FILTER]
    Name      modify
    Match     audit
    Add index ${SPLUNK_OPERATIONS_INDEX}
    Add host  ${NODE_NAME}
    Add source audit

[OUTPUT]
    Name        splunk
    Match       host.*
    Host        ${SPLUNK_HOST}
    Port        ${SPLUNK_PORT}
    Splunk_Token ${SPLUNK_TOKEN}
    Splunk_Send_Raw On
    tls        On
    tls.verify Off
    Retry_Limit False

[OUTPUT]
    Name        splunk
    Match       audit
    Host        ${SPLUNK_HOST}
    Port        ${SPLUNK_PORT}
    Splunk_Token ${SPLUNK_TOKEN}
    Splunk_Send_Raw On
    tls        On
    tls.verify Off
    Retry_Limit False

parsers.conf

[PARSER]
    Name json
    Format json
    Decode_Field_As json log
[PARSER]
    Name docker
    Format json
    Decode_Field_as json log try_next
    Decode_Field_as escaped_utf8 log

Environment name and version (e.g. Kubernetes? What version?): Openshift 3.11/ Kubernetes 1.11
Filters and plugins: kubernetes, splunk

fixed question

Source

strantalis

Most helpful comment

@edsiper we run in the same issue - however we use the correct helm chart notation, fluentd still fails:

  parsers.conf: |
    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    [PARSER]
        Name   laravel
        Format regex
        Regex ^\[(?<time>[^\]]*)\]\[(?<logid>[^\]]*)\]\[(?<file>[^\]]*)\]\[(?<line>[^\]]*)\]\[(?<method>[^\]]*)\]\[(?<env>[^\]]*)\]\[(?<loglevel>[^\]]*)\]: (?<message>.*)$
        Time_Key time
        Time_Format %Y-%m-%d %H:%M:%S %z

Fluent-Bit Error:


Fluent Bit v1.3.2
Copyright (C) Treasure Data
 [2019/10/11 15:11:14] [  Error] File /fluent-bit/etc/parsers_custom.conf
[2019/10/11 15:11:14] [  Error] Error in line 11: Length of content has execeded limit

1.2.1 runs without problems

discostur on 11 Oct 2019

👍4

All 11 comments

Does your last line contains a n ?

edsiper on 2 Oct 2019

We were able to figure it out. In the configmap we changed it from |- to | to handle the multiline yaml differently.

apiVersion: v1
data:
  fluent-bit.conf: |-
    [SERVICE]
        Flush           2
        Daemon          off
        Log_Level       ${LOG_LEVEL}
        Parsers_File parsers.conf
        Plugins_File plugins.conf
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020

apiVersion: v1
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush           2
        Daemon          off
        Log_Level       ${LOG_LEVEL}
        Parsers_File parsers.conf
        Plugins_File plugins.conf
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020

Appreciate the help. It pointed us in the right direction.

strantalis on 2 Oct 2019

👍1

u're welcome.

I've added a new entry in the Kubernetes Filter docs about this possible issue:

https://docs.fluentbit.io/manual/filter/kubernetes#helm-charts-and-yaml

thanks again.

edsiper on 2 Oct 2019

@edsiper we run in the same issue - however we use the correct helm chart notation, fluentd still fails:

  parsers.conf: |
    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    [PARSER]
        Name   laravel
        Format regex
        Regex ^\[(?<time>[^\]]*)\]\[(?<logid>[^\]]*)\]\[(?<file>[^\]]*)\]\[(?<line>[^\]]*)\]\[(?<method>[^\]]*)\]\[(?<env>[^\]]*)\]\[(?<loglevel>[^\]]*)\]: (?<message>.*)$
        Time_Key time
        Time_Format %Y-%m-%d %H:%M:%S %z

Fluent-Bit Error:


Fluent Bit v1.3.2
Copyright (C) Treasure Data
 [2019/10/11 15:11:14] [  Error] File /fluent-bit/etc/parsers_custom.conf
[2019/10/11 15:11:14] [  Error] Error in line 11: Length of content has execeded limit

1.2.1 runs without problems

discostur on 11 Oct 2019

👍4

Same issue on Windows as well. No new line characters at the end.

Update: Parser file as well as conf file needs to end with a new line and the Windows td-agent-bit 1.3.2 just works perfectly.

xPB12 on 13 Oct 2019

@xPB12 can you please tell me how you force a new line at the end?

discostur on 14 Oct 2019

👍1

here explains the differences between | and |+ block chomping
basically |+ trims the n at the end of the file and | puts a n there

rmacian on 29 Oct 2019

https://docs.fluentbit.io/manual/filter/kubernetes#helm-charts-and-yaml

edsiper on 29 Oct 2019

my config is already like this:

  parsers.conf: |
    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    [PARSER]
        Name   laravel
        Format regex
        Regex ^\[(?<time>[^\]]*)\]\[(?<logid>[^\]]*)\]\[(?<file>[^\]]*)\]\[(?<line>[^\]]*)\]\[(?<method>[^\]]*)\]\[(?<env>[^\]]*)\]\[(?<loglevel>[^\]]*)\]: (?<message>.*)$
        Time_Key time
        Time_Format %Y-%m-%d %H:%M:%S %z

It still does not work ...

discostur on 7 Nov 2019

@discostur I think use latest stable/fluent-bit charts and adjust your values.yaml will help