Truffleruby: RVM installation with Macports

[ylluminarious]

Hi,

I'm really excited to try out Truffle Ruby via RVM. Unfortunately, when I run rvm install truffleruby, I get the following error:

Searching for binary rubies, this might take some time.
No binary rubies available for: osx/10.13/x86_64/truffleruby-1.0.0-rc2.
Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.
Checking requirements for osx_port.
Certificates bundle '/opt/local/etc/openssl/cert.pem' is already up to date.
Requirements installation successful.
TruffleRuby requires LLVM to be installed to run native extensions.
For more details and for setup instructions for your system, please see:
https://github.com/oracle/truffleruby/blob/master/doc/user/installing-llvm.md

I am using Macports instead of Homebrew, and my rvm autolibs preference is set accordingly:

$ rvm autolibs status
---
value: macports
number: 4
runner: osx_port
description: Allow RVM to use package manager if found, install missing dependencies, install package manager (only OS X).

I have the llvm-4.0, llvm-5.0, & llvm-6.0 ports installed and activated, so it seems that Truffle should have been built just fine. I guess that the build process just hasn't been set up to work with Macports yet.

[eregon]

Hello,

Thank you for the report.
Indeed, MacPorts is not really tested currently, so there are likely some rough edges.
This specific error is from a check I added in RVM:
https://github.com/rvm/rvm/blob/528e707b/scripts/functions/manage/truffleruby#L5-L11

The error is because the opt executable is not found (in PATH), and that's part of the executables LLVM provides.
I could remove this check in RVM, since TruffleRuby checks if LLVM's clang and opt are available too, in a more flexible way, when it needs to compile C extensions.

@ylluminarious Could you tell me in which bin directory the opt executable is available for llvm-4.0 in MacPorts?

A workaround in the meantime would be to add the bin directory to your PATH when installing and using TruffleRuby.

I think finding OpenSSL also needs some adjustments for MacPorts.

[ylluminarious]

@eregon Thanks for the quick response! The directory for opt in Macports is:

/opt/local/libexec/llvm-4.0/bin/opt

This is the same for llvm-5.0 and llvm-6.0.

Also, thanks for the idea of a workaround! It seems to have worked (somewhat). All I had to do was run:

$ PATH=$PATH:/opt/local/libexec/llvm-4.0/bin rvm install truffleruby

And the installation was able to proceed. Unfortunately, there were some errors in the build process:

Searching for binary rubies, this might take some time.
No binary rubies available for: osx/10.13/x86_64/truffleruby-1.0.0-rc2.
Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.
Checking requirements for osx_port.
Certificates bundle '/opt/local/etc/openssl/cert.pem' is already up to date.
Requirements installation successful.
truffleruby-1.0.0-rc2 - #downloading truffleruby-1.0.0-rc2-macos-amd64, this may take a while depending on your connection...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   630    0   630    0     0   1769      0 --:--:-- --:--:-- --:--:--  1764
100 42.0M  100 42.0M    0     0   265k      0  0:02:42  0:02:42 --:--:--  264k
truffleruby-1.0.0-rc2 - #extracting truffleruby-1.0.0-rc2-macos-amd64 to /Users/my_username/.rvm/src/truffleruby-1.0.0-rc2 - please wait
truffleruby-1.0.0-rc2 - #installing to /Users/my_username/.rvm/rubies/truffleruby-1.0.0-rc2 - please wait
truffleruby-1.0.0-rc2 - #compiling c-extensions - please wait
Error running './lib/truffle/post_install_hook.sh',
please read /Users/my_username/.rvm/log/1531188146_truffleruby-1.0.0-rc2/cext.log
truffleruby-1.0.0-rc2 - #making binaries executable - please wait
truffleruby-1.0.0-rc2 - #gemset created /Users/my_username/.rvm/gems/truffleruby-1.0.0-rc2@global
truffleruby-1.0.0-rc2 - #importing gemset /Users/my_username/.rvm/gemsets/global.gems - please wait
Error running 'command gem install /Users/my_username/.rvm/gem-cache/gem-wrappers-1.4.0.gem --local --no-ri --no-rdoc',
please read /Users/my_username/.rvm/log/1531188146_truffleruby-1.0.0-rc2/gem.install.gem-wrappers->=1.4.0.log
there was an error installing gem gem-wrappers
Error running 'command gem install rubygems-bundler --no-ri --no-rdoc',
please read /Users/my_username/.rvm/log/1531188146_truffleruby-1.0.0-rc2/gem.install.rubygems-bundler.log
there was an error installing gem rubygems-bundler
Error running 'command gem install /Users/my_username/.rvm/gem-cache/rvm-1.11.3.9.gem --local --no-ri --no-rdoc',
please read /Users/my_username/.rvm/log/1531188146_truffleruby-1.0.0-rc2/gem.install.rvm->=1.11.3.9.log
there was an error installing gem rvm
Error running 'command gem install bundler --no-ri --no-rdoc',
please read /Users/my_username/.rvm/log/1531188146_truffleruby-1.0.0-rc2/gem.install.bundler.log
there was an error installing gem bundler
truffleruby-1.0.0-rc2 - #generating global wrappers - please wait
Error running 'command gem install /Users/my_username/.rvm/gem-cache/gem-wrappers-1.4.0.gem --local --no-ri --no-rdoc',
please read /Users/my_username/.rvm/log/1531188146_truffleruby-1.0.0-rc2/gem.install.gem-wrappers->=1.4.0.log
Error running 'run_gem_wrappers regenerate',
please read /Users/my_username/.rvm/log/1531188146_truffleruby-1.0.0-rc2/gemset.wrappers.global.log
truffleruby-1.0.0-rc2 - #gemset created /Users/my_username/.rvm/gems/truffleruby-1.0.0-rc2
truffleruby-1.0.0-rc2 - #importing gemsetfile /Users/my_username/.rvm/gemsets/default.gems evaluated to empty gem list
truffleruby-1.0.0-rc2 - #generating default wrappers - please wait
Error running 'command gem install /Users/my_username/.rvm/gem-cache/gem-wrappers-1.4.0.gem --local --no-ri --no-rdoc',
please read /Users/my_username/.rvm/log/1531188146_truffleruby-1.0.0-rc2/gem.install.gem-wrappers->=1.4.0.log
Error running 'run_gem_wrappers regenerate',
please read /Users/my_username/.rvm/log/1531188146_truffleruby-1.0.0-rc2/gemset.wrappers.default.log

And ~/.rvm/gems/truffleruby-1.0.0-rc2/bin is empty. Perhaps this relates to the OpenSSL issues you mentioned? Yet, rvm list shows this:

Warning! PATH is not properly set up, /Users/my_username/.rvm/gems/truffleruby-1.0.0-rc2/bin is not available.
         Usually this is caused by shell initialization files. Search for PATH=... entries.
         You can also re-add RVM to your profile by running: rvm get stable --auto-dotfiles
         To fix it temporarily in this shell session run: rvm use truffleruby-1.0.0-rc2
         To ignore this error add rvm_silence_path_mismatch_check_flag=1 to your ~/.rvmrc file.
   jruby-1.7.19 [ x86_64 ]
   jruby-9.0.5.0 [ x86_64 ]
   ruby-2.2.3 [ x86_64 ]
 * ruby-2.5.1 [ x86_64 ]
=> truffleruby-1.0.0-rc2 [ x86_64 ]

# => - current
# =* - current && default
#  * - default

So it seems almost as though the installation only partially failed... ? It's a bit of a head-scratcher.

[eregon]

@ylluminarious Yes, this looks like the step truffleruby-1.0.0-rc2 - #compiling c-extensions - please wait failed, which seem extremely likely due to TruffleRuby failing to find the OpenSSL headers from MacPorts.
Could you post the contents of /Users/my_username/.rvm/log/1531188146_truffleruby-1.0.0-rc2/cext.log ?

And ~/.rvm/gems/truffleruby-1.0.0-rc2/bin is empty.

Right, the gems added by RVM all failed to install as RubyGems requires openssl and the extension couldn't be compiled.

BTW, are you using RVM stable ? The latest release of TruffleRuby is 1.0.0-rc3.
Not that it would help since there is no MacPorts support yet of course.

[ylluminarious]

@eregon Yes, I am using RVM stable. I will switch to master if that improves things.

Here are the contents of cext.log:

Compiling the OpenSSL C extension
checking for t_open() in -lnsl... no
checking for socket() in -lsocket... no
checking for openssl/ssl.h... no

Yes, the root of the problem is quite clear.

[eregon]

@ylluminarious I fixed the search for openssl and LLVM with MacPorts in https://github.com/oracle/truffleruby/commit/2bed55f9ffdeaf83b7bac848d64122bdf4b8d3df.
We also need https://github.com/rvm/rvm/pull/4427 in RVM to remove the too strict check for LLVM.

[ylluminarious]

@eregon Wow, thanks! So should this issue actually be solved now? How can I test your changes?

[eregon]

@ylluminarious Yes, this should be solved now and will be in the next release, 1.0.0-rc4 (planned around the end of the month).

Unfortunately, it seems difficult to test before the 1.0.0-rc4 release, because we use binary archives for RVM and those are only produced for releases currently.
We'd also like to test this on our side in an automated or mostly-automated way, but I'm not what's a good way to achieve that.
I had a colleague building TruffleRuby locally with MacPorts and that worked fine, but it's not the same as installing from RVM.
Obviously Docker is not enough to emulate a macOS system.
Maybe by using Vagrant with a macOS image with MacPorts in it? I have no experience with it unfortunately.

I'll keep this issue open until it's confirmed to work as expected :)

[ylluminarious]

@eregon Hmm, yes, what you describe is a bit of a predicament. I maybe could try to test things in a virtual machine that I have lying around, but I can't make any promises. I'm a bit strapped for time currently, so I may just have to wait till the next binary release is available.

[eregon]

@ylluminarious No worry, trying at the next release is fine.
I meant TruffleRuby should try to test this in CI or during release testing.

[ylluminarious]

@eregon I see that RC 4 was released today... but it looks like it's not on RVM master yet. I guess a PR still needs to be made to RVM.

[eregon]

@ylluminarious Actually RC4 is a in-between release only updating the functionality of Graal JavaScript, and as such doesn't contain any changes or fixes for Ruby.
So we need to wait for RC5, which is planned around the end of the month.

Longer term we are thinking to have daily builds or something similar to reduce time between fixes and testing.

[ylluminarious]

@eregon Oh, sorry, gotcha. Thanks for the info.

[eregon]

TruffleRuby 1.0.0-rc5 was released and added to RVM.
So now installing TruffleRuby using rvm and with MacPorts should work out of the box:

rvm get head
rvm install truffleruby

@ylluminarious Could you confirm it works as expected?

[ylluminarious]

@eregon It seems to work, but there were still some errors at compile-time:

Searching for binary rubies, this might take some time.
No binary rubies available for: osx/10.13/x86_64/truffleruby-1.0.0-rc5.
Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.
Checking requirements for osx_port.
Certificates bundle '/opt/local/etc/openssl/cert.pem' is already up to date.
Requirements installation successful.
truffleruby-1.0.0-rc5 - #downloading truffleruby-1.0.0-rc5-macos-amd64, this may take a while depending on your connection...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   630    0   630    0     0   2107      0 --:--:-- --:--:-- --:--:--  2107
100 41.6M  100 41.6M    0     0  8284k      0  0:00:05  0:00:05 --:--:-- 10.3M
truffleruby-1.0.0-rc5 - #extracting truffleruby-1.0.0-rc5-macos-amd64 to /Users/my_username/.rvm/src/truffleruby-1.0.0-rc5 - please wait
truffleruby-1.0.0-rc5 - #installing to /Users/my_username/.rvm/rubies/truffleruby-1.0.0-rc5 - please wait
truffleruby-1.0.0-rc5 - #compiling c-extensions - please wait
Error running './lib/truffle/post_install_hook.sh',
please read /Users/my_username/.rvm/log/1533405900_truffleruby-1.0.0-rc5/cext.log
truffleruby-1.0.0-rc5 - #making binaries executable - please wait
truffleruby-1.0.0-rc5 - #gemset created /Users/my_username/.rvm/gems/truffleruby-1.0.0-rc5@global
truffleruby-1.0.0-rc5 - #importing gemset /Users/my_username/.rvm/gemsets/global.gems - please wait
Error running 'command gem install /Users/my_username/.rvm/gem-cache/gem-wrappers-1.4.0.gem --local --no-ri --no-rdoc',
please read /Users/my_username/.rvm/log/1533405900_truffleruby-1.0.0-rc5/gem.install.gem-wrappers->=1.4.0.log
there was an error installing gem gem-wrappers
Error running 'command gem install rubygems-bundler --no-ri --no-rdoc',
please read /Users/my_username/.rvm/log/1533405900_truffleruby-1.0.0-rc5/gem.install.rubygems-bundler.log
there was an error installing gem rubygems-bundler
Error running 'command gem install /Users/my_username/.rvm/gem-cache/rvm-1.11.3.9.gem --local --no-ri --no-rdoc',
please read /Users/my_username/.rvm/log/1533405900_truffleruby-1.0.0-rc5/gem.install.rvm->=1.11.3.9.log
there was an error installing gem rvm
Error running 'command gem install bundler --no-ri --no-rdoc',
please read /Users/my_username/.rvm/log/1533405900_truffleruby-1.0.0-rc5/gem.install.bundler.log
there was an error installing gem bundler
truffleruby-1.0.0-rc5 - #generating global wrappers - please wait
Error running 'command gem install /Users/my_username/.rvm/gem-cache/gem-wrappers-1.4.0.gem --local --no-ri --no-rdoc',
please read /Users/my_username/.rvm/log/1533405900_truffleruby-1.0.0-rc5/gem.install.gem-wrappers->=1.4.0.log
Error running 'run_gem_wrappers regenerate',
please read /Users/my_username/.rvm/log/1533405900_truffleruby-1.0.0-rc5/gemset.wrappers.global.log
truffleruby-1.0.0-rc5 - #gemset created /Users/my_username/.rvm/gems/truffleruby-1.0.0-rc5
truffleruby-1.0.0-rc5 - #importing gemsetfile /Users/my_username/.rvm/gemsets/default.gems evaluated to empty gem list
truffleruby-1.0.0-rc5 - #generating default wrappers - please wait
Error running 'command gem install /Users/my_username/.rvm/gem-cache/gem-wrappers-1.4.0.gem --local --no-ri --no-rdoc',
please read /Users/my_username/.rvm/log/1533405900_truffleruby-1.0.0-rc5/gem.install.gem-wrappers->=1.4.0.log
Error running 'run_gem_wrappers regenerate',
please read /Users/my_username/.rvm/log/1533405900_truffleruby-1.0.0-rc5/gemset.wrappers.default.log

I can send you whatever logs you want from the above output. But still, despite these ominous errors, things seem to work:

$ rvm use truffleruby
Using /Users/my_username/.rvm/gems/truffleruby-1.0.0-rc5
$ type ruby
ruby is /Users/my_username/.rvm/rubies/truffleruby-1.0.0-rc5/bin/ruby
$ type irb
irb is /Users/my_username/.rvm/rubies/truffleruby-1.0.0-rc5/bin/irb
$ irb  
truffleruby-1.0.0-rc5 :001 > puts RUBY_ENGINE
truffleruby
 => nil 
truffleruby-1.0.0-rc5 :002 > puts RUBY_VERSION 
2.4.4
 => nil 
truffleruby-1.0.0-rc5 :003 > puts TruffleRuby.revision 
8827d247
 => nil 
truffleruby-1.0.0-rc5 :004 > 

[eregon]

@ylluminarious Could you post in a gist or send me the first two logs? /Users/my_username/.rvm/log/1533405900_truffleruby-1.0.0-rc5/cext.log and
/Users/my_username/.rvm/log/1533405900_truffleruby-1.0.0-rc5/gem.install.gem-wrappers->=1.4.0.log

Those basics will work, but does OpenSSL work?
Can you try: ruby -ropenssl -e 'p OpenSSL' ?

[ylluminarious]

@eregon

• Here is the requested cext.log
• Here is the requested gem.install.gem-wrappers->=1.4.0.log

Here is the result of ruby -ropenssl -e 'p OpenSSL':

/Users/my_username/.rvm/rubies/truffleruby-1.0.0-rc5/lib/mri/openssl.rb:13:in `gem_original_require': Library "/usr/local/opt/openssl/lib/libssl.dylib" does not exist. (RuntimeError)
    from /Users/my_username/.rvm/rubies/truffleruby-1.0.0-rc5/lib/mri/openssl.rb:13:in `require'
    from /Users/my_username/.rvm/rubies/truffleruby-1.0.0-rc5/lib/mri/openssl.rb:13:in `<top (required)>'
    from -e:1:in `gem_original_require'
    from -e:1:in `require'
    from -e:1:in `<main>'

[eregon]

@ylluminarious Thank you. Oddly, the linking step failed:

linking shared-object openssl.su
/Users/my_username/.rvm/rubies/truffleruby-1.0.0-rc5/lib/cext/linker.rb:54:in `link_bitcode': Linker failed (RuntimeError)
    from /Users/my_username/.rvm/rubies/truffleruby-1.0.0-rc5/lib/cext/linker.rb:48:in `main'
    from /Users/my_username/.rvm/rubies/truffleruby-1.0.0-rc5/lib/cext/linker.rb:108:in `<main>'

I'd guess this is because llvm-link is not in PATH and this is one place I forgot to do the search in MacPorts directories like it is done for clang and opt.

• Could you confirm llvm-link is not in PATH with which llvm-link?
• Could you temporarily (just in the current shell) add /opt/local/libexec/llvm-4.0/bin on PATH and retry?

rvm remove truffleruby
export PATH="/opt/local/libexec/llvm-4.0/bin:$PATH"
rvm install truffleruby

[ylluminarious]

@eregon Yes, llvm-link is not in PATH. which llvm-link yields nothing and after running find /opt/local -name "llvm-link" I found it at /opt/local/libexec/llvm-4.0/bin/llvm-link.

Your above series of commands works perfectly; Truffle installed with no problems at all:

$ rvm remove truffleruby           
truffleruby-1.0.0-rc5 - #removing src/truffleruby-1.0.0-rc5 - please wait
truffleruby-1.0.0-rc5 - #removing rubies/truffleruby-1.0.0-rc5 - please wait
truffleruby-1.0.0-rc5 - #removing gems - please wait
truffleruby-1.0.0-rc5 - #removing wrappers - please wait
truffleruby-1.0.0-rc5 - #removing environments - please wait
$ export PATH="/opt/local/libexec/llvm-4.0/bin:$PATH"
$ rvm install truffleruby
Warning! PATH is not properly set up, /Users/my_username/.rvm/gems/ruby-2.5.1/bin is not at first place.
         Usually this is caused by shell initialization files. Search for PATH=... entries.
         You can also re-add RVM to your profile by running: rvm get stable --auto-dotfiles
         To fix it temporarily in this shell session run: rvm use ruby-2.5.1
         To ignore this error add rvm_silence_path_mismatch_check_flag=1 to your ~/.rvmrc file.
Searching for binary rubies, this might take some time.
No binary rubies available for: osx/10.13/x86_64/truffleruby-1.0.0-rc5.
Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.
Checking requirements for osx_port.
Certificates bundle '/opt/local/etc/openssl/cert.pem' is already up to date.
Requirements installation successful.
truffleruby-1.0.0-rc5 - #downloading truffleruby-1.0.0-rc5-macos-amd64, this may take a while depending on your connection...
truffleruby-1.0.0-rc5 - #extracting truffleruby-1.0.0-rc5-macos-amd64 to /Users/my_username/.rvm/src/truffleruby-1.0.0-rc5 - please wait
truffleruby-1.0.0-rc5 - #installing to /Users/my_username/.rvm/rubies/truffleruby-1.0.0-rc5 - please wait
truffleruby-1.0.0-rc5 - #compiling c-extensions - please wait
truffleruby-1.0.0-rc5 - #making binaries executable - please wait
truffleruby-1.0.0-rc5 - #gemset created /Users/my_username/.rvm/gems/truffleruby-1.0.0-rc5@global
truffleruby-1.0.0-rc5 - #importing gemset /Users/my_username/.rvm/gemsets/global.gems - please wait
truffleruby-1.0.0-rc5 - #generating global wrappers - please wait
truffleruby-1.0.0-rc5 - #gemset created /Users/my_username/.rvm/gems/truffleruby-1.0.0-rc5
truffleruby-1.0.0-rc5 - #importing gemsetfile /Users/my_username/.rvm/gemsets/default.gems evaluated to empty gem list
truffleruby-1.0.0-rc5 - #generating default wrappers - please wait
$ ruby -ropenssl -e 'p OpenSSL'
OpenSSL 
$ type ruby # just a sanity check
ruby is /Users/my_username/.rvm/rubies/truffleruby-1.0.0-rc5/bin/ruby

[eregon]

@ylluminarious Success, finally :)

I will fix the search for llvm-link.

I'll try to add a test for llvm-link not in PATH in CI.
It's a bit tricky as Sulong expects LLVM binaries in PATH to be built but we try to not require that anymore. cc @rschatz

BTW, I think this issue affects Homebrew users which don't have /usr/local/opt/llvm@4/bin on PATH too.

[ylluminarious]

@eregon Yes, success indeed! :D

Thanks for your persistence in solving this issue. It is much appreciated. Your plan sounds good, and I think you're right; Homebrew users are probably also affected who have the PATH configuration that you mentioned.

Should we close this now or wait till you push the needed commits etc.?

[eregon]

@ylluminarious I propose to leave it open until the fix is in master and that we tested that it works without changing PATH. I will create a fix soon.

[eregon]

I fixed the search for llvm-link in 1a4983265b2000d38ca0ca6eab689b3b6c125913 and added a test in our CI to check tests work when there is no opt and llvm-link in PATH: https://github.com/oracle/truffleruby/commit/1a4983265b2000d38ca0ca6eab689b3b6c125913#diff-685dad263b0ff4155b484f5ef24373acR1064
So this should be fixed for the next release.

@ylluminarious Could you verify this is the only needed fix by changing your
/Users/my_username/.rvm/rubies/truffleruby-1.0.0-rc5/lib/cext/linker.rb
to https://gist.github.com/eregon/5e3d14d0f0cd07abe7960b177da19052 ?
(the only change is https://gist.github.com/eregon/5e3d14d0f0cd07abe7960b177da19052/revisions, which should fix this one llvm-link issue)
And then run:

/Users/my_username/.rvm/rubies/truffleruby-1.0.0-rc5/lib/truffle/post_install_hook.sh

It should recompile the OpenSSL C extension and exit successfully (exit code 0 from echo $?).

[ylluminarious]

@eregon It worked exactly as you outlined. It seems that nothing further is needed to fix this issue, then.

[eregon]

@ylluminarious Great, thank you for your help!
I'll close this issue then, the fix will be in the next release.

[eregon]

This should now be fixed in the TruffleRuby 1.0.0-rc6 release.
@ylluminarious Does rvm install truffleruby work out of the box for you now?

[ylluminarious]

@eregon Yes, everything seems to work perfectly. Thanks so much!

[eregon]

Great to hear :)