Logstash: OpenSSL::SSL::SSLError: certificate verify failed

Created on 23 Jul 2015 · 6Comments · Source: elastic/logstash

Hi,

i am running logstash 1.5.2 as a server and my clients are using logstash 1.4.3-1-7e387fb (not logstash-forwarder) on Ubuntu 14.04. When I upgrade my client to the latest version 1.4.4-1-5608c19 it cant connect anymore with following error:

{:timestamp=>"2015-07-23T07:22:33.150000+0000", :message=>"All hosts unavailable, sleeping", :hosts=>["x.xxx.xxx.xx"], :e=>#<OpenSSL::SSL::SSLError: certificate verify failed>, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:180:in `connect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.22/lib/lumberjack/client.rb:84:in `connection_start'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.22/lib/lumberjack/client.rb:77:in `initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.22/lib/lumberjack/client.rb:36:in `connect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.22/lib/lumberjack/client.rb:25:in `initialize'", "/opt/logstash/lib/logstash/outputs/lumberjack.rb:53:in `connect'", "/opt/logstash/lib/logstash/outputs/lumberjack.rb:22:in `register'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/lib/logstash/pipeline.rb:220:in `outputworker'", "/opt/logstash/lib/logstash/pipeline.rb:152:in `start_outputs'"], :level=>:error, :file=>"logstash/outputs/lumberjack.rb", :line=>"56"}

In the changelog of 1.4.4 it says "Improved default security for SSL", but what exactly does that mean?
I use a self-signed certificate, but I connect through a FQDN and the CN is configured for that.
Why should that fail?

Thanks,
bert2002

bug

Source

bert2002

Most helpful comment

Since I wasted 6 hours on tracking why this error appeared in 2.3.0 for me, I hope this note helps another frustrated developer out there:
the OpenSSL::SSL::SSLError: certificate verify failed error can also be caused if you are using different certs on the sender and receiver (Logstash as shipper and another Logstash as receiver). In my case I generated a self signed cert and configured the LS sender to use it but forgot the place them as well in the Logstash receiver.
I discovered that only when I tried using Logstash Forwarder as sender while trying to isolate the error. Fortunately LSFWD printed a more correct error: Failed to tls handshake with 172.17.0.17 x509: certificate signed by unknown authority. This led me to check if the same certs are used on both ends and only then I saw they are not. Fixed it to be the self signed ones on both ends and it worked.

Mind you I'm using self signed certs only during Integration testing and not on production cases.

asafm on 3 May 2016

👍2

All 6 comments

The changes were to, for all Logstash plugins, explicitly reject SSLv3 and also deny any attempts to use weak SSL ciphers. I'm not sure how these would affect SSL certificate validation.

Can you attach your Logstash config from both servers?

jordansissel on 23 Jul 2015

Hi, the configuration is very simple and no custom configuration:
server:

input {
  lumberjack {
    port => 5000
    ssl_certificate => "/etc/pki/tls/certs/logstash.crt"
    ssl_key => "/etc/pki/tls/private/logstash.key"
    codec => "json"
  }
}
output {
  elasticsearch { host => localhost }
}

client:

input {
 [...]
}

filter {
 [...]
}

output {
  lumberjack {
    codec => "json"
    hosts => [ "my.fqdn.com" ]
    port => 5000
    ssl_certificate => "/etc/logstash/logstash.crt"
  }
  stdout { codec => json }
}

bert2002 on 24 Jul 2015

explicit sslv3 gets rejected:

$ openssl s_client -connect xxx.xxx.xxx.xxx:5000 -ssl3
CONNECTED(00000003)
139883376961168:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:598:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv3
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1437732175
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

tls1 is working:

~$ openssl s_client -connect xxx.xxx.xxx.xxx:5000 -tls1
CONNECTED(00000003)
depth=0 CN = my.fqdn.com
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = my.fqdn.com
verify return:1
---
Certificate chain
 0 s:/CN=my.fqdn.com
   i:/CN=my.fqdn.com
---
Server certificate
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
subject=/CN=my.fqdn.com
issuer=/CN=my.fqdn.com
---
No client certificate CA names sent
---
SSL handshake has read 1598 bytes and written 393 bytes
---
New, TLSv1/SSLv3, Cipher is EDH-RSA-DES-CBC3-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : EDH-RSA-DES-CBC3-SHA
    Session-ID: 55B20DB11D5D43704D7E8A0DF96A45325FCD69D64E24B664F48B107D199A0A96
    Session-ID-ctx: 
    Master-Key: 141B2B5A113CDA12F6343F77F48A4AC9F3922C328A2C2DB5654592EBBADD08DD9315BB5C430AD5B7A741ADDBCD0221E3
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1437732209
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---

bert2002 on 24 Jul 2015