Truffleruby: OpenSSL compilation fails on macOS

[nateberkopec]

Continuing discussion from https://github.com/oracle/truffleruby/issues/1785#issuecomment-558560788

Failing part of the build log:

compiling ossl_ssl_session.c
ossl_ssl_session.c:104:9: warning: implicit
      declaration of function
      'SSL_SESSION_get_protocol_version' is invalid
      in C99 [-Wimplicit-function-declaration]
    if (SSL_SESSION_get_protocol_version(a...
        ^
1 warning generated.
compiling ossl_x509.c
compiling ossl_x509attr.c
compiling ossl_x509cert.c
ossl_x509cert.c:87:5: warning: implicit declaration
      of function 'X509_up_ref' is invalid in C99
      [-Wimplicit-function-declaration]
    X509_up_ref(x509);
    ^
ossl_x509cert.c:334:31: warning: implicit
      declaration of function
      'X509_get0_tbs_sigalg' is invalid in C99
      [-Wimplicit-function-declaration]
    if (!i2a_ASN1_OBJECT(out, X509_get0_tbs...
                              ^
ossl_x509cert.c:334:59: error: member reference
      type 'int' is not a pointer
  ...X509_get0_tbs_sigalg(x509)->algorithm)) {
     ~~~~~~~~~~~~~~~~~~~~~~~~~~  ^
ossl_x509cert.c:424:22: warning: implicit
      declaration of function 'X509_get0_notBefore'
      is invalid in C99
      [-Wimplicit-function-declaration]
    if (!(asn1time = X509_get0_notBefore(x509))) {
                     ^
ossl_x509cert.c:443:10: warning: implicit
      declaration of function 'X509_set1_notBefore'
      is invalid in C99
      [-Wimplicit-function-declaration]
    if (!X509_set1_notBefore(x509, asn1time)) {
         ^
ossl_x509cert.c:463:22: warning: implicit
      declaration of function 'X509_get0_notAfter'
      is invalid in C99
      [-Wimplicit-function-declaration]
    if (!(asn1time = X509_get0_notAfter(x509))) {
                     ^
ossl_x509cert.c:482:10: warning: implicit
      declaration of function 'X509_set1_notAfter'
      is invalid in C99
      [-Wimplicit-function-declaration]
    if (!X509_set1_notAfter(x509, asn1time)) {
         ^
6 warnings and 1 error generated.
make: *** [ossl_x509cert.o] Error 1
!!! Post-install tasks failed!

Other relevant lines:

Warning: [email protected] 1.1.1d is already installed and up-to-date
To reinstall 1.1.1d, run `brew reinstall [email protected]`

checking for OpenSSL version is 1.0.1 or later... yes

Not seeing anything else related to OpenSSL version numbers. Full log here.

More relevant info:

$ brew --prefix openssl
/Users/nateberkopec/.homebrew/opt/[email protected]

$ /opt/local/include/openssl
-bash: /opt/local/include/openssl: No such file or directory

[nateberkopec]

19.2.0 now fails in the same place for me, so this is definitely my local environment.

[nateberkopec]

Setting OPENSSL_PREFIX to /Users/nateberkopec/.homebrew/opt/[email protected] doesn't change the outcome :|

[nateberkopec]

I do also have libressl installed which may be messing with things somehow? My SSL setup is definitely weird thanks to all the stuff I have to do with Puma maintenance so I'm guessing the answer is in which SSL is being used where.

[eregon]

Thanks for creating a separate bug report, definitely easier to track stuff :)

Can MRI be installed in your environment with ruby-install ruby [SOME_VERSION] ?

Yes, this looks like it picks the headers from one place for the extconf.rb checks but then picks another header when compiling with make.
X509_get0_notBefore should be available in OpenSSL 1.1, so I guess somehow LibreSSL headers get picked up during compilation.

[eregon]

Could you also show your /Users/nateberkopec/.rubies/truffleruby-19.3.0/src/main/c/openssl/mkmf.log ?
And the generated Makefile /Users/nateberkopec/.rubies/truffleruby-19.3.0/src/main/c/openssl/Makefile ?

[nateberkopec]

Oh, look at that. MRI is failing too in the same spot. OK, so this is probably not your fault. Looks like I have something that needs to be fixed by a particular incantation of CONFIGURE_OPTS.

[eregon]

Right, we use almost the same extconf.rb for OpenSSL as MRI.

A wild guess here would be that pkg-config returns values inconsistent with what's found by default by the C compiler.

Having Homebrew in a non-standard location might also mean headers might not be found automatically.
I would also check if CPATH or any C compiler-related env var is set in your environment.
I'm happy to help you debug this if you want, but it would likely be much more convenient if we can do it interactively on Slack.

[CorbanR]

The issue that I ran into, was that I had some packages installed via brew that required an older version of openssl to be installed. So I have both [email protected] and [email protected] installed.

adding the following environment variable
export PKG_CONFIG_PATH="/usr/local/opt/[email protected]/lib/pkgconfig" allowed openssl compilation to succeed.

I also have export SDKROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" set in my ~/.bash_profile. Although for this particular issue, im not sure that is required.

[thbar]

@CorbanR thanks for sharing this! I had to also use the export PKG_CONFIG_PATH incantation like you did. Otherwise I saw Recompiling the OpenSSL C extension (against the installed libssl) in the early logs, then the failure mentioned in this issue.

I wonder if it would be relevant, if the default libssl is used, to issue a warning with those guidelines somewhere in the logs? In all cases, thanks!

[eregon]

In the future we are thinking to ship openssl with TruffleRuby, since it's so complicated to install and find a decent OpenSSL on macOS.

@thbar To help finding out the issue, could you show some of your configuration?
In your case what does brew --prefix openssl return?
Do you have both [email protected] and [email protected] installed?
Which pkg-config is it, i.e., which pkg-config ?

I would guess this line is to blame: https://github.com/oracle/truffleruby/blob/3ea67e46fee059cf3a173c193bfc6503d5b3356a/src/main/c/openssl/extconf.rb#L51
Could you try to replace that line with result = have_header("openssl/ssl.h") in truffleruby/src/main/c/openssl/extconf.rb (in your installed TruffleRuby) and then run truffleruby/lib/truffle/post_install_hook.sh again and see if that succeeds?

[thbar]

In the future we are thinking to ship openssl with TruffleRuby, since it's so complicated to install and find a decent OpenSSL on macOS.

On the other hand, while complicated, I appreciate that I can apply OpenSSL security fixes on my end (at the cost of reinstalling TruffleRuby), too!

Now to your questions.

brew --prefix openssl gives /usr/local/opt/[email protected].

On the versions, it's a bit complicated and I could not fully investigate yet :

• brew info [email protected] shows it's installed under /usr/local/opt/[email protected].
• brew info [email protected] indicates shopify/shopify/[email protected] is Not installed
• yet I can see:

$ ls -la /usr/local/opt/openssl*
lrwxr-xr-x  1 thbar  admin  24 Jul 14  2019 /usr/local/opt/openssl -> ../Cellar/openssl/1.0.2s
lrwxr-xr-x  1 thbar  admin  24 Jul 14  2019 /usr/local/opt/[email protected] -> ../Cellar/openssl/1.0.2s
lrwxr-xr-x  1 thbar  admin  28 Sep 30 17:32 /usr/local/opt/[email protected] -> ../Cellar/[email protected]/1.1.1d

Presumably [email protected] has been installed by another mean (in 2019).

which pkg-config gives /usr/local/bin/pkg-config.

Regarding the patch you provided: I will still get Recompiling the OpenSSL C extension (against the installed libssl) and then way more errors than before, for some reason (full logs at https://gist.github.com/thbar/340cedb6a5d234f0f301ad1cbd9113ad).

Presumably (presumably) something could be botched on my install anyway (although I have up-to-date MRI installed for all recent supported versions!).

That's it for what I can provide for now!

[eregon]

@thbar Thank you for providing this data.
/usr/local/opt/openssl -> ../Cellar/openssl/1.0.2s might be part of the issue in this case.
We'll specifically look for [email protected] to avoid that, and hope Homebrew doesn't remove that package anytime soon (they did for llvm@4 :/).
OpenSSL 1.0.2 should also work but probably it results in a mix in your case, and anyway 1.0.2 is end-of-life now so it shouldn't be used anymore.

I will adapt the logic for finding OpenSSL to also set PKG_CONFIG_PATH on macOS.
That way it should work out of the box.

[eregon]

This should be fixed with 2706c09e57c4c27fb6177a9e420a87c0befb9e70 which automatically sets PKG_CONFIG_PATH on macOS. It will be in the upcoming 20.0 release.

[eregon]

@nateberkopec @CorbanR @thbar TruffleRuby 20.0.0 was released and includes this fix. It would be useful if any of you can confirm it installs without troubles on macOS now.

[nateberkopec]

Successfully installed truffleruby 20.0.0 into /Users/nateberkopec/.rubies/truffleruby-20.0.0

💥

[thbar]

@eregon works for me as well:

$ rbenv install truffleruby-20.0.0
ruby-build: using openssl from homebrew
SNIP
TruffleRuby was successfully installed in /Users/thbar/.rbenv/versions/truffleruby-20.0.0
Installed truffleruby-20.0.0 to /Users/thbar/.rbenv/versions/truffleruby-20.0.0

Thanks!