Fluentd: Fluentd in_forward behind NGINX TCP stream proxy

[Antiarchitect]

Check CONTRIBUTING guideline first and here is the list to help us investigate the problem.

Describe the bug
In my EFK I have multiple node log collectors (one per node) and n log aggregators (that performs log parsing and pushing it to Elastic). Collectors out_forward ---> in_forward log_aggregators. Log Aggregators are hidden behind K8s Service that points to them and for internal (in cluster) node log collectors and internal log aggregators service there is no error. But for external nodes I use ingress-nginx TCP stream module and node log collectors have pretty the same config file as internal ones.
So in NGINX log I see rare but annoying errors for this case:

2020/04/19 21:11:42 [error] 2742#2742: *2383018 recv() failed (104: Connection reset by peer) while proxying and reading from upstream, client: <external white IP>, server: 0.0.0.0:7443, upstream: "10.32.11.237:7443", bytes from/to client:2441/3491, bytes from/to upstream:3491/2441

Bytest from/to stays the same for every error it's 3491/2441.

To Reproduce

Expected behavior
No errors on the proxy of such kind

Your Environment

fluentd 1.10.2 (both node log collector and log aggregator)
K8s 1.17.5
Centos 7 
Kernel: 5.5.13-1.el7.elrepo.x86_64
docker 19.03.8

Your Configuration

    <system>
      log_level warn
      <log>
        format json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </log>
    </system>

    # Kubernetes and Hosts Logs Source
    <source>
      @type forward
      port 7443
      bind 0.0.0.0

      <transport tls>
        ca_path /fluentd/etc/tls/CAcert.pem
        cert_path /fluentd/etc/tls/cert.pem
        client_cert_auth true
        private_key_path /fluentd/etc/tls/key.pem
      </transport>
    </source>

Your Error Log

2020/04/19 21:11:42 [error] 2742#2742: *2383018 recv() failed (104: Connection reset by peer) while proxying and reading from upstream, client: <external white IP>, server: 0.0.0.0:7443, upstream: "10.32.11.237:7443", bytes from/to client:2441/3491, bytes from/to upstream:3491/2441

[repeatedly]

We are not familiar with ingress-nginx behavior, so we need help from users.

• This problem happens with ingress nginx or not. For example, pure nginx works without issue?
• What is the differences between nginx and other proxy softwares
• Other information to find the root cause.

[aaugustinas]

I am facing the same issue. Did you manage to solve it?

[Antiarchitect]

@aaugustinas Nope, still exists. One-two appearances in 5 minutes for me

[simonasr]

We also are facing ~this~ similar problem 😞
Up to 3% error rate (with readv() failed (104: Connection reset by peer) while reading upstream)

UPDATE: Same problem, different input plugin 🙈
Just noticed, that this thread is about in_forward plugin 🤦 but we have same nginx error messages with plugin in_http, might be related.. 🤔

Our fluentd in_http configuration is very simple:

<source>
  @type http
  port 9880
  bind 0.0.0.0
  body_size_limit 32m
  keepalive_timeout 10s
</source>

and there is nginx forwarding requests before fluentd.

[repeatedly]

Does this problem happen with heartbeat_type none?

[simonasr]

I updated my comment ⬆️ - we have same symptoms with in_http plugin.

[simonasr]

Looks like fluentd v1.11.1 has fixed the problem we had with in_http plugin 👏

[Antiarchitect]

It seems like problem is gone for me too! :) Thank you all for help. Closing.