Fluentd: SSL_accept SYSCALL returned=5 errno=0

[venky999]

Describe the bug

I am seeing this warning continuously and causing logs lost..
[warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"

To Reproduce
Run td-agent with the following config
Expected behavior
The warning should not occur

Your Environment

• Fluentd or td-agent version: fluentd --version or td-agent --version
  td-agent 1.10.0
• Operating system: cat /etc/os-release
  RED HAT 7.7
• Kernel version: uname -r
  3.10.0-1062.12.1.el7.x86_64
• Ruby
  2.0.0p648 (2015-12-16) [x86_64-linux]
• Openssl
  1.0.2k-fips 26 Jan 2017

Your Configuration

<source>
  @type forward
  port 52466
  <security>
      self_hostname XXX
      shared_key yyyyyyyy
  </security>
  <transport tls>
      cert_path /etc/td-agent/certs/test.crt
      private_key_path /etc/td-agent/certs/test.key
  </transport>
</source>

Your Error Log

2020-04-14 17:21:44 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:44 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:46 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:46 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:46 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:46 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"

Additional context

[ganmacs]

2.0.0p648 (2015-12-16) [x86_64-linux]

we don't support ruby2.0.0. please update ruby version.

[kritisingh]

i have same issue
Fluentd or td-agent version: fluentd --version or td-agent --version
td-agent 1.10.0
Operating system: cat /etc/os-release
Debian GNU/Linux 10 (buster)
Kernel version: uname -r
4.19.0-8-cloud-amd64
Ruby
ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux-gnu]
Openssl
OpenSSL 1.1.1d 10 Sep 2019

I followed link https://bundler.io/v2.0/guides/rubygems_tls_ssl_troubleshooting_guide.html#troubleshooting-certificate-errors and it says below

ruby -ropen-uri -e 'eval open("https://git.io/vQhWq").read'
--> output
Here's your Ruby and OpenSSL environment:

Ruby: 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux-gnu]
RubyGems: 2.7.6.2
Compiled with: OpenSSL 1.1.1d 10 Sep 2019
Loaded version: OpenSSL 1.1.1d 10 Sep 2019
SSL_CERT_FILE: /usr/lib/ssl/cert.pem
SSL_CERT_DIR: /usr/lib/ssl/certs

With that out of the way, let's see if you can connect to rubygems.org...

Bundler connection to rubygems.org: failed ❌ (uninitialized constant Bundler)
RubyGems connection to rubygems.org: success ✅
Ruby net/http connection to rubygems.org: success ✅

Although your Ruby installation and RubyGems can both connect to rubygems.org, Bundler is having trouble. The most likely way to fix this is to upgrade Bundler by running gem install bundler. Run this script again after doing that to make sure everything is all set. If you're still having trouble, check out the troubleshooting guide at http://ruby.to/ssl-check-failed 📦
(eval):136: warning: constant OpenSSL::SSL::SSLContext::METHODS is deprecated

/opt/td-agent/embedded/bin/fluent-gem list
--> output
* LOCAL GEMS *

addressable (2.7.0)
async (1.24.2)
async-http (0.50.7)
async-io (1.27.6)
async-pool (0.2.0)
aws-eventstream (1.0.3)
aws-partitions (1.288.0)
aws-sdk-core (3.92.0)
aws-sdk-kms (1.30.0)
aws-sdk-s3 (1.61.1)
aws-sdk-sqs (1.24.0)
aws-sigv4 (1.1.1)
bigdecimal (default: 1.3.2)
bundler (1.16.6)
concurrent-ruby (1.1.6)
console (1.8.2)
cool.io (1.6.0)
did_you_mean (1.1.0)
digest-crc (0.5.1)
elasticsearch (6.8.1)
elasticsearch-api (6.8.1)
elasticsearch-transport (6.8.1)
excon (0.73.0)
faraday (0.17.3)
ffi (1.12.2)
fluent-config-regexp-type (1.0.0)
fluent-logger (0.8.2)
fluent-plugin-elasticsearch (4.0.7, 3.0.0)
fluent-plugin-input-gelf (0.3.1)
fluent-plugin-kafka (0.13.0)
fluent-plugin-prometheus (1.7.3)
fluent-plugin-prometheus_pushgateway (0.0.2)
fluent-plugin-record-modifier (2.1.0)
fluent-plugin-rewrite-tag-filter (2.2.0)
fluent-plugin-s3 (1.3.0)
fluent-plugin-systemd (1.0.2)
fluent-plugin-td (1.1.0)
fluent-plugin-td-monitoring (0.2.4)
fluent-plugin-webhdfs (1.2.4)
fluentd (1.10.0)
gelfd2 (0.4.1)
hirb (0.7.3)
http_parser.rb (0.6.0)
httpclient (2.8.2.4)
io-console (default: 0.4.6)
ipaddress (0.8.3)
jmespath (1.4.0)
json (default: 2.0.4)
ltsv (0.1.2)
mini_portile2 (2.4.0)
minitest (5.10.1)
mixlib-cli (1.7.0)
mixlib-config (2.2.4)
mixlib-log (1.7.1)
mixlib-shellout (2.2.7)
msgpack (1.3.3)
multi_json (1.14.1)
multipart-post (2.1.1)
net-telnet (0.1.1)
nio4r (2.5.2)
nokogiri (1.10.9)
ohai (6.20.0)
oj (3.8.1)
openssl (default: 2.0.9)
parallel (1.19.1)
power_assert (0.4.1)
prometheus-client (0.9.0)
protocol-hpack (1.4.2)
protocol-http (0.15.1)
protocol-http1 (0.10.3)
protocol-http2 (0.11.6)
psych (default: 2.2.2)
public_suffix (4.0.3)
quantile (0.2.1)
rake (12.3.3, 12.0.0)
rdkafka (0.7.0)
rdoc (default: 5.0.1)
ruby-kafka (1.0.0)
ruby-progressbar (1.10.1)
rubyzip (1.3.0)
serverengine (2.2.1)
sigdump (0.2.4)
strptime (0.2.3)
systemd-journal (1.3.3)
systemu (2.5.2)
td (0.16.8)
td-client (1.0.7)
td-logger (0.3.27)
test-unit (3.2.3)
timers (4.3.0)
tzinfo (2.0.1)
tzinfo-data (1.2019.3)
webhdfs (0.9.0)
xmlrpc (0.2.1)
yajl-ruby (1.4.1)
zip-zip (0.3)

[kritisingh]

@venky999 did you find a solution yet?

[venky999]

@kritisingh no..seeing same issue

[johnkazmerzak]

This log message was being logged as a trace but is now logged as a warn. #2861

[repeatedly]

Yes. Above change shows this warning logs.
But we can't judge this issue is fluentd bug or certificate/openssl setting mismatch.
We need more information and reproducible step, e.g. how to create certificates, old fluentd works or not and more.

[kritisingh]

I get warning from section, even if there is no section,

<source>
    @type forward
    <transport tls>
        cert_path /etc/ssl/xx/xx.crt
        private_key_path /etc/ssl/xx/xx.key
        private_key_passphrase xx
    </transport>
</source>

I downgraded to td-agent 1.9.2 version and the ssl warning isn't visible anymore, probably because it is being logged as a trace in this version.

[konstantin-kornienko]

@venky999 , do you use fluentd behind load balancer? we're experiencing same errors on td-agents behind AWS ELB.

[Minipada]

@konstantin-kornienko I am seeing the same behaviour. If i test a tls server using the Load balancer port, it instantly complains:

shutting down SSL
CONNECTION CLOSED
SSL_accept:before SSL initialization
SSL_accept:error in before SSL initialization

But if i change the port, everything is fine. Could you find a solution?

[Phlak106]

Same behavior on kubernetes:
fluent/fluentd:v1.11.5-debian-1.0

Fluentd config:

<source>
  @type forward
  <transport tls>
    cert_path /fluentd/etc/fluentd.crt
    private_key_path /fluentd/etc/fluentd.key
    private_key_passphrase *******************
  </transport>
</source>

<match **>
  @type stdout
</match>

K8s spec:

apiVersion: v1
kind: Namespace
metadata:
  name: fluent
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fluent-deployment
  namespace: fluent
  labels:
    app: fluent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fluent
  template:
    metadata:
      labels:
        app: fluent
    spec:
      containers:
      - name: fluentd
        image: "myImageWithCertificatesBasedOnFluent1.15"
        imagePullPolicy: Always
        ports:
        - containerPort: 24224
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
---
kind: Service
apiVersion: v1
metadata:
  name: fluent-svc
  namespace: fluent
spec:
  type: LoadBalancer
  ports:
  - port: 24224
    targetPort: 24224
    name: forward
  selector:
    app: fluent

[robermar23]

I am experiencing the same issue in kubernetes.

Same specs as @Phlak106

I have also tried versions 1.11 and 1.12