fluentd dies with Errno::ECONNRESET error="Connection reset by peer - SSL_accept"

[dmnfortytwo]

2018-09-21 12:07:44 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.10.3'
2018-09-21 12:07:44 +0000 [info]: gem 'fluent-plugin-flowcounter' version '1.3.0'
2018-09-21 12:07:44 +0000 [info]: gem 'fluent-plugin-kafka' version '0.7.3'
2018-09-21 12:07:44 +0000 [info]: gem 'fluent-plugin-multiprocess' version '0.2.2'
2018-09-21 12:07:44 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.0.1'
2018-09-21 12:07:44 +0000 [info]: gem 'fluent-plugin-record-modifier' version '1.1.0'
2018-09-21 12:07:44 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.1.0'
2018-09-21 12:07:44 +0000 [info]: gem 'fluent-plugin-s3' version '1.1.3'
2018-09-21 12:07:44 +0000 [info]: gem 'fluent-plugin-td' version '1.0.0'
2018-09-21 12:07:44 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4'
2018-09-21 12:07:44 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.2.3'
2018-09-21 12:07:44 +0000 [info]: gem 'fluentd' version '1.2.5

td-agent runs as a forwarder to elasticsearch and sometimes dies with this stacktrace:

2018-09-21 12:07:44 +0000 [warn]: #0 thread exited by unexpected error plugin=Fluent::Plugin::ForwardInput title=:event_loop error_class=Errno::ECONNRESET error="Connection reset by peer - SSL_accept"
2018-09-21 12:07:44 +0000 [error]: #0 unexpected error error_class=Errno::ECONNRESET error="Connection reset by peer - SSL_accept"
  2018-09-21 12:07:44 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/server.rb:680:in `accept_nonblock'
  2018-09-21 12:07:44 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/server.rb:680:in `try_handshake'
  2018-09-21 12:07:44 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/server.rb:696:in `try_tls_accept'
  2018-09-21 12:07:44 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/server.rb:723:in `on_readable'
  2018-09-21 12:07:44 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:186:in `on_readable'
  2018-09-21 12:07:44 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
  2018-09-21 12:07:44 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
  2018-09-21 12:07:44 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2018-09-21 12:07:44 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2018-09-21 12:07:44 +0000 [error]: #0 unexpected error error_class=Errno::ECONNRESET error="Connection reset by peer - SSL_accept"
  2018-09-21 12:07:44 +0000 [error]: #0 suppressed same stacktrace
2018-09-21 12:07:44 +0000 [info]: Worker 0 finished unexpectedly with status 1

  <source>
    @type forward
    tag forwarded
    port 34228
    bind "0.0.0.0"
    source_address_key "forwarder_ip"
    source_hostname_key "forwarder_hostname"
    <transport tls>
      ca_cert_path "/etc/td-agent/server.pem"
      ca_private_key_path "/etc/td-agent/server.key"
    </transport>
    <security>
      shared_key xxxxxx
      self_hostname "harvester-gce-be-6"
    </security>
  </source>

We have multiprocess with 16 children and a haproxy in front, if it makes sense.

[ashishtilara]

+1

[Wolodija]

+1

[repeatedly]

Here is simple patch for it.

https://github.com/fluent/fluentd/pull/2243