Virtual-environments: Ubuntu - missing cert file - OpenSSL::X509::DEFAULT_CERT_FILE, ENV['SSL_CERT_FILE']

Created on 28 Aug 2019 · 10Comments · Source: actions/virtual-environments

macOS & Windows are okay, but Ubuntu is missing a cert file. Normally, one of the below should point to a valid cert file.

OpenSSL::X509::DEFAULT_CERT_FILE
ENV['SSL_CERT_FILE']

On macOS, the value of OpenSSL::X509::DEFAULT_CERT_FILE is
/usr/local/etc/openssl/cert.pem,

On Windows, the file is included in the Ruby distro.

bug

Source

MSP-Greg

Most helpful comment

We've not found any domain that fails, but I agree theoretically it may happen so there's a plan to fix it.
In the meantime, anyone facing any issue should be good by downloading the Mozilla CA cert into their path (Ubuntu only):

sudo wget -O /usr/lib/ssl/cert.pem "https://curl.haxx.se/ca/cacert.pem"

Thanks @MSP-Greg for the report.

pablonete on 30 Aug 2019

👍2

All 10 comments

We're taking a look. Will report back when we have more info.

kaylangan on 28 Aug 2019

This is correct.
In Ubuntu OpenSSL::X509::DEFAULT_CERT_FILE is set to /usr/lib/ssl/cert.pem, which doesn't exist.
Although so far I've not been able to reproduce any issue about this. Both installing gems and fetching https resources work successfully. Do you have any fail case?

pablonete on 29 Aug 2019

Both installing gems and fetching https resources work successfully.

RubyGems & Bundler ship there own certs, so that wouldn't be affected.

Do you have any fail case?

Not yet. Then again, it's been a while since I've worked with a system where the file was missing...

MSP-Greg on 29 Aug 2019

sudo wget -O /usr/lib/ssl/cert.pem "https://curl.haxx.se/ca/cacert.pem"

Thanks @MSP-Greg for the report.

pablonete on 30 Aug 2019

👍2

This is still an issue. I believe any software compiled with the system OpenSSL gets the setting for DEFAULT_CERT_FILE from the system library.

Also, at least as far as Ruby, the default is OpenSSL::SSL::VERIFY_NONE, not OpenSSL::SSL::VERIFY_PEER, so failing domains may not be an issue unless OpenSSL::SSL::SSLContext#verify_mode is set to OpenSSL::SSL::VERIFY_PEER.

Even then, to really test it, one might need to check for a failure to a URI at https://www.ssllabs.com/...

MSP-Greg on 19 Jan 2020

I've been trying to find out more about this, by debugging in Docker for the ubundu:18.04 image.

First, certificates are installed with apt-get install ca-certificates on Ubuntu and they end up in /etc/ssl/certs and the main file is /etc/ssl/certs/ca-certificates.crt.
All files under /etc/ssl/certs except ca-certificates.crt are symlinks and point to files under /usr/share/ca-certificates/mozilla.

Installing ruby with apt-get install ruby, it shows:

$ ruby -ropenssl -e 'p OpenSSL::X509::DEFAULT_CERT_FILE'
"/usr/lib/ssl/cert.pem"

$ ls -l /usr/lib/ssl   
total 4
lrwxrwxrwx. 1 root root   14 Nov 12 16:58 certs -> /etc/ssl/certs
drwxr-xr-x. 2 root root 4096 Jan 25 13:06 misc
lrwxrwxrwx. 1 root root   20 Nov 12 16:58 openssl.cnf -> /etc/ssl/openssl.cnf
lrwxrwxrwx. 1 root root   16 Nov 12 16:58 private -> /etc/ssl/private

So there is no /usr/lib/ssl/cert.pem.

OTOH, DEFAULT_CERT_DIR exists:

root@87d5668b20b2:/# ruby -ropenssl -e 'p OpenSSL::X509::DEFAULT_CERT_DIR'
"/usr/lib/ssl/certs"
root@87d5668b20b2:/# ls -l /usr/lib/ssl/certs
lrwxrwxrwx. 1 root root 14 Nov 12 16:58 /usr/lib/ssl/certs -> /etc/ssl/certs

OpenSSL::X509::DEFAULT_CERT_FILE is X509_get_default_cert_file() which is X509_CERT_FILE which is OPENSSLDIR "/cert.pem".
OpenSSL::X509::DEFAULT_CERT_DIR is X509_get_default_cert_dir() which is X509_CERT_DIR which is OPENSSLDIR "/certs".

Trying it in practice works fine:

$ ruby -ropen-uri -e 'puts open(%{https://expired.badssl.com/}) { |f| f.read(1024) }'    
...
/usr/lib/ruby/2.5.0/net/protocol.rb:44:in `connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired) (OpenSSL::SSL::SSLError)

$ ruby -ropen-uri -e 'puts open(%{https://rubygems.org/}) { |f| f.read(1024) }'
<!DOCTYPE html>
<html lang="en-us">
  <head>
    <title>RubyGems.org | your community gem host</title>
...

So in conclusion, on Ubuntu OpenSSL::X509::DEFAULT_CERT_FILE never exists, but it seems to never cause any problem, and OpenSSL::X509::DEFAULT_CERT_DIR exists and seems correct.

I'd guess OpenSSL is fine if it finds just a "cert dir", and probably looks at more locations than just those two:

$ strace ruby -ropen-uri -e 'puts open(%{https://rubygems.org/}) { |f| f.read(1024) }' |& grep cert
openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib/ssl/certs/b85455c4.0", 0x7ffd80766890) = -1 ENOENT (No such file or directory)
stat("/usr/lib/ssl/certs/5ad8a5d6.0", {st_mode=S_IFREG|0644, st_size=1261, ...}) = 0
openat(AT_FDCWD, "/usr/lib/ssl/certs/5ad8a5d6.0", O_RDONLY) = 10
stat("/usr/lib/ssl/certs/5ad8a5d6.1", 0x7ffd80766890) = -1 ENOENT (No such file or directory)

$ ls -l /usr/lib/ssl/certs/5ad8a5d6.0
lrwxrwxrwx. 1 root root 22 Jan 25 13:07 /usr/lib/ssl/certs/5ad8a5d6.0 -> GlobalSign_Root_CA.pem

@MSP-Greg So I think we can close this issue.
If there any issue due to X509_get_default_cert_file() it should probably be reported to Ubuntu or OpenSSL anyway.

eregon on 25 Jan 2020

@eregon

Thanks for looking into this. I've wanted to look at this more, as things do seem odd.

Re expired.badssl.com, that really isn't a verification issue, as I believe 'expired' is independent of the 'cert chain', kind of like a drivers license. Some of the other sites might fail.

But:

Looking at https://github.com/MSP-Greg/use-ruby-action-info/runs/398040668, the Ubuntu jobs >= 2.4, check the 'SSL Verify' line in the 'Ruby Info' step. All are 'Success'. For these, DEFAULT_CERT_DIR exists, but DEFAULT_CERT_FILE does not.

Looking at https://ci.appveyor.com/project/MSP-Greg/appveyor-ruby, all the old RubyInstaller/MSYS builds are '* FAILURE *'. For these, neither DEFAULT_CERT_DIR or DEFAULT_CERT_FILE exist.

So, I'm not sure why the Ubuntu builds are passing. I'll close for now and reopen if I find any helpful info.

As mentioned above, Ruby often defaults to VERIFY_NONE, so if someone is actually using VERIFY_PEER, one would hope it works correctly...

EDIT: I am mistaken about the above, see

https://github.com/ruby/ruby/blob/master/lib/open-uri.rb#L320-L323

MSP-Greg on 25 Jan 2020

Working locally, without a proper DEFAULT_CERT_FILE setting, I cannot open a valid https site, I receive an error string of 'unable to get local issuer certificate`.

But, Config::DEFAULT_CONFIG_FILE exists at /usr/lib/ssl/openssl.cnf. That must be setting cert info correctly? I haven't worked with OpenSSL config files much...

MSP-Greg on 25 Jan 2020

This is an issue on Ubuntu 18.04.02 with OpenSSL 1.1.0g. I have installed a 'local' CA certificate that is valid and is listed in /etc/ssl/certs/ca-certificates.crt. I can use the certificate if I include --ca-certificate=/etc/ssl/certs/ca-certificates.crt in the wget command. Using curl & wget without pointing to ca-certificates.crt does not work.

The missing /usr/lib/ssl/cert.pem file is a problem.

The solution I found (as my workaround) is symbolically link /usr/lib/ssl/cert.pem to /etc/ssl/certs/ca-certificates.crt