Fluent-bit: Error io after upgrading to fluent-bit 1.6.7 from 1.6.6: [filter:kubernetes:kubernetes.0] upstream connection error

Created on 3 Dec 2020 · 14Comments · Source: fluent/fluent-bit

Bug Report

Describe the bug
After upgrading, got lot of error message about upstream connection error.

To Reproduce

Example log message if applicable:

[2020/12/03 10:17:36] [error] [filter:kubernetes:kubernetes.0] upstream connection error
[2020/12/03 10:17:36] [error] [io] connection #49 failed to: kubernetes.default.svc:443
[2020/12/03 10:17:36] [error] [filter:kubernetes:kubernetes.0] upstream connection error
[2020/12/03 10:17:36] [error] [io] connection #49 failed to: kubernetes.default.svc:443
[2020/12/03 10:17:36] [error] [filter:kubernetes:kubernetes.0] upstream connection error
[2020/12/03 10:17:36] [error] [io] connection #49 failed to: kubernetes.default.svc:443
[2020/12/03 10:17:36] [error] [filter:kubernetes:kubernetes.0] upstream connection error
[2020/12/03 10:17:36] [error] [io] connection #49 failed to: kubernetes.default.svc:443
[2020/12/03 10:17:36] [error] [filter:kubernetes:kubernetes.0] upstream connection error
[2020/12/03 10:17:36] [error] [io] connection #49 failed to: kubernetes.default.svc:443
[2020/12/03 10:17:36] [error] [filter:kubernetes:kubernetes.0] upstream connection error
[2020/12/03 10:17:36] [error] [io] connection #49 failed to: kubernetes.default.svc:443
[2020/12/03 10:17:36] [error] [filter:kubernetes:kubernetes.0] upstream connection error

Steps to reproduce the problem:

Expected behavior
No error

Your Environment

Version used: 1.6.7
Configuration:

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_Tag_Prefix     kube.var.log.containers.
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Merge_Log           On
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On
[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Parser            docker
    Tag               kube.*
    Refresh_Interval  5
    Mem_Buf_Limit     6MB
    Skip_Long_Lines   On
    read_from_head    on
    DB                /tail-db/tail-containers-state.db
    DB.Sync           Off
    DB.locking        true
[OUTPUT]
    Name  es
    Match *
    Host  XXXX
    Port  443
    Logstash_Format On
    Retry_Limit 5
    Type  _doc
    Trace_Error true
    Time_Key @timestamp-flb
    Replace_Dots On

    HTTP_User XXX
    HTTP_Passwd XXXX

    tls on
    tls.verify on

Environment name and version (e.g. Kubernetes? What version?): 1.18
Server type and version: Amazon AMI Linux 2
Operating System and version:
Filters and plugins: input tail, output es

bug fixed

Source

mtparet

👍6

All 14 comments

I can confirm I am also seeing this issue on AWS EKS after upgrading to the version released a few hours ago. The URL is correct.

Environment name and version (e.g. Kubernetes? What version?): 1.17
Server type and version: Amazon AMI Linux 2
Operating System and version:
Filters and plugins: input tail, filter kubernetes, output http

Problem does not exist in 1.6.6.
Problem presents in 1.6.7.

    [FILTER]
      Name           record_modifier
      Match          *
      Record         cluster_name ${CLUSTER_NAME}

    [FILTER]
      Name                kubernetes
      Match               kube.*
      Kube_URL            https://kubernetes.default.svc.cluster.local:443
      Merge_Log           On
      Merge_Log_Trim      On
      Keep_Log            Off
      K8S-Logging.Parser  On
      K8S-Logging.Exclude Off  
      Labels              On
      Annotations         On
      Buffer_Size         1m

    [FILTER]
      Name    lua
      Match   kube.*
      script  /fluent-bit/etc/dedot.lua
      call    dedot

    [FILTER]
      Name modify
      Match kube.*
      Condition Key_exists kubernetes.labels.app
      Rename kubernetes.labels.app kubernetes.labels.app_name

cdobbyn on 3 Dec 2020

EKS 1.18 can confirm. Release 1.6.7 is bugged.

shinebayar-g on 3 Dec 2020

Taking a look at this.

edsiper on 3 Dec 2020

👍1

troubleshooting:

filter_kubernetes code is the same, no changes
socket error handling was modified, possible issue is here.

edsiper on 3 Dec 2020

root cause of the problem;

regression that did not handle errno properly for connections in progress

Fixes:

33be5a4d
03daeebc

v1.6.8 is under release process.

edsiper on 3 Dec 2020

🎉1

Container images for v1.6.8 are already available, tags:

fluent/fluent-bit:1.6.8
fluent/fluent-bit:latest
fluent/fluent-bit:1.6
fluent/fluent-bit:1.6.8-debug
fluent/fluent-bit:1.6-debug

edsiper on 3 Dec 2020

👍1

seeing a larger number of http healthcheck failures to / after updating to 1.6.8, could there be a socket leak from this?