Fluent-bit: [error] [http_client] broken connection to

[gajus]

Bug Report

Describe the bug

[2019/01/09 19:57:27] [debug] [input tail.0] [mem buf] size = 6905737
[2019/01/09 19:57:27] [debug] [in_tail] file=/var/log/containers/fluent-bit-rwbn6_logging_fluent-bit-e32fc22fe5b4c545e47b91b2fd3c87091f477955d341214552c752fee606f850.log read=32694 lines=196
[2019/01/09 19:57:27] [error] [src/flb_http_client.c:802 errno=32] Broken pipe
[2019/01/09 19:57:27] [ warn] [out_es] http_do=-1
[2019/01/09 19:57:27] [error] [http_client] broken connection to es-2c3f419e-applaudience.aivencloud.com:21291 ?
[2019/01/09 19:57:27] [ warn] [out_es] http_do=-1
[2019/01/09 19:57:27] [debug] [retry] new retry created for task_id=0 attemps=1
[2019/01/09 19:57:27] [debug] [sched] retry=0x7f4043017320 0 in 5 seconds
[2019/01/09 19:57:27] [error] [http_client] broken connection to es-2c3f419e-applaudience.aivencloud.com:21291 ?
[2019/01/09 19:57:27] [ warn] [out_es] http_do=-1
[2019/01/09 19:57:27] [debug] [input tail.0] [mem buf] size = 7018547

To Reproduce

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit
data:
  fluent-bit.conf: |
    [SERVICE]
        Daemon        off
        Flush         1
        Log_Level     debug
        Parsers_File  parsers.conf
        tls On
        tls.verify Off
    [INPUT]
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     50MB
        Name              tail
        Parser            docker
        Path              /var/log/containers/*.log
        Refresh_Interval  10
        Skip_Long_Lines   On
        Tag               kube.*
    [FILTER]
        K8S-Logging.Exclude On
        K8S-Logging.Parser  On
        Match               kube.*
        Merge_Log           On
        Merge_Log_Trim      On
        Merge_Log_Key       roarr
        Name                kubernetes
    [OUTPUT]
        Host es-2c3f419e-applaudience.aivencloud.com
        Port 21291
        HTTP_User avnadmin
        HTTP_Passwd ags4c2ajld3fxyr5
        Logstash_Format On
        Match *
        Name es
  parsers.conf: |
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
        Decode_Field_As escaped log

Note: Credentials exposed intentionally. Feel free to debug using these credentials. Will be changed after 48 hours.

Expected behavior

Storing logs to the ES cluster.

Your Environment

• Version used: 1.0.1

Additional context

Launched using https://github.com/helm/charts/tree/master/stable/fluent-bit

fullConfigMap: true
existingConfigMap: fluent-bit
dnsPolicy: Default

I am able to reach the ES cluster after SSHing to the Pod (rebuilt with SSH support). So it is not a firewall issues as such.

[edsiper]

Your AWS elasticsearch service is closing the TCP connection because is expecting a TLS/SSL handshake. Your TLS session is not enabled in the output section, add "tls on" to the output section that connects to your service.