Truffleruby: Build failure 1.0.0-rc6 with macOS Mojave

[hsbt]

Hi, I couldn't install truffleruby-1.0.0-rc6 via ruby-build. See following log.

~/.r/c/truffleruby-1.0.0-rc6-macos-amd64 (master) > rbenv install truffleruby-1.0.0-rc6 
ruby-build: use openssl from homebrew
Installing truffleruby-1.0.0-rc6...
Compiling the OpenSSL C extension
/Users/hsbt/.rbenv/versions/truffleruby-1.0.0-rc6/lib/mri/mkmf.rb:468:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.

    from /Users/hsbt/.rbenv/versions/truffleruby-1.0.0-rc6/lib/mri/mkmf.rb:584:in `block in try_compile'
    from /Users/hsbt/.rbenv/versions/truffleruby-1.0.0-rc6/lib/mri/mkmf.rb:537:in `with_werror'
    from /Users/hsbt/.rbenv/versions/truffleruby-1.0.0-rc6/lib/mri/mkmf.rb:584:in `try_compile'
    from /Users/hsbt/.rbenv/versions/truffleruby-1.0.0-rc6/src/main/c/openssl/deprecation.rb:5:in `deprecated_warning_flag'
    from extconf.rb:25:in `<main>'

BUILD FAILED (OS X 10.14 using ruby-build 20180822-8-g336584c)

Inspect or clean up the working tree at /var/folders/jl/zgy9fk353nd6zn0xznlsdzvc6gxzs5/T/ruby-build.20180904205617.61332
Results logged to /var/folders/jl/zgy9fk353nd6zn0xznlsdzvc6gxzs5/T/ruby-build.20180904205617.61332.log

Last 10 log lines:
/var/folders/jl/zgy9fk353nd6zn0xznlsdzvc6gxzs5/T/ruby-build.20180904205617.61332 ~/.rbenv/cache/truffleruby-1.0.0-rc6-macos-amd64
/var/folders/jl/zgy9fk353nd6zn0xznlsdzvc6gxzs5/T/ruby-build.20180904205617.61332/truffleruby-1.0.0-rc6 /var/folders/jl/zgy9fk353nd6zn0xznlsdzvc6gxzs5/T/ruby-build.20180904205617.61332 ~/.rbenv/cache/truffleruby-1.0.0-rc6-macos-amd64

I try to reproduce with source package file. I can get same error without ruby-build. This is result of src/main/c/openssl/mkmf.log

=== OpenSSL for Ruby configurator ===
"/usr/local/opt/llvm@4/bin/clang -o conftest -I/Users/hsbt/.rbenv/cache/truffleruby-1.0.0-rc6-macos-amd64/lib/cext/include -I/Users/hsbt/.rbenv/cache/truffleruby-1.0.0-rc6-macos-amd64/lib/cext/include -I. -I/usr/local/opt/openssl/include  -g -Wimplicit-function-declaration -Wno-unknown-warning-option -Wno-int-conversion -Wno-int-to-pointer-cast -Wno-unused-value -Wno-incompatible-pointer-types -ferror-limit=500 /Users/hsbt/.rbenv/cache/truffleruby-1.0.0-rc6-macos-amd64/lib/cext/ruby.o /Users/hsbt/.rbenv/cache/truffleruby-1.0.0-rc6-macos-amd64/lib/cext/sulongmock.o conftest.c  -L. -L. -L/usr/local/opt/openssl/lib         "
In file included from conftest.c:1:
In file included from /Users/hsbt/.rbenv/cache/truffleruby-1.0.0-rc6-macos-amd64/lib/cext/include/ruby.h:33:
In file included from /Users/hsbt/.rbenv/cache/truffleruby-1.0.0-rc6-macos-amd64/lib/cext/include/ruby/ruby.h:28:
/Users/hsbt/.rbenv/cache/truffleruby-1.0.0-rc6-macos-amd64/lib/cext/include/sulong/truffle.h:39:10: fatal error: 'stdlib.h' file not found
#include <stdlib.h>
         ^~~~~~~~~~
1 error generated.
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: int main(int argc, char **argv)
4: {
5:   return 0;
6: }
/* end */

It seems caused by llvm or clang detection issues.

[eregon]

Thank you for the report!
That is a strange error indeed.

Maybe /usr/local/opt/llvm@4/bin/clang doesn't work correctly and fails to find system headers?
Could you try compiling:

#include <stdio.h>
int main() {
  puts("Hi");
  return 0;
}

with /usr/local/opt/llvm@4/bin/clang and see if it works correctly?

Maybe llvm@4 was installed before the update to Mojave and this could cause that clang to no longer be able to find system headers. If that's the case, reinstalling llvm@4 with Homebrew might help.

[hsbt]

I did invoke brew install --force-bottle llvm@4 before install truffleruby. but I got the same error 😇

~/Downloads > /usr/local/opt/llvm@4/bin/clang test.c 
test.c:1:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.

It seems packaging issue of llvm@4 with mojave.

[eregon]

Thanks, that confirms it's an issue between llvm@4 of Homebrew and macOS Mojave.

A workaround might be to use a more recent LLVM from Homebrew (llvm or llvm@5), prepend it on PATH and remove llvm@4. Then install TruffleRuby.
TruffleRuby will pick clang, opt and llvm-link from PATH if there is no /usr/local/opt/llvm@4/bin and no /opt/local/libexec/llvm-4.0/bin.

[eregon]

Maybe the --force-bottle is also a cause here, a quick search gives https://github.com/Homebrew/brew/issues/1781

[hsbt]

I see. But --force-bottle is not related to this issue. Because homebrew picked mojave binary with and without --force-bottle option.

~/Downloads > brew install llvm@4
==> Downloading https://homebrew.bintray.com/bottles/[email protected]
Already downloaded: /Users/hsbt/Library/Caches/Homebrew/downloads/b70515[e-mail]
==> Pouring [email protected]
==> Caveats
To use the bundled libc++ please add the following LDFLAGS:
  LDFLAGS="-L/usr/local/opt/llvm@4/lib -Wl,-rpath,/usr/local/opt/llvm@4/lib"

llvm@4 is keg-only, which means it was not symlinked into /usr/local,
because this is an alternate version of another formula.

If you need to have llvm@4 first in your PATH run:
  echo 'export PATH="/usr/local/opt/llvm@4/bin:$PATH"' >> ~/.bash_profile

For compilers to find llvm@4 you may need to set:
  export LDFLAGS="-L/usr/local/opt/llvm@4/lib"
  export CPPFLAGS="-I/usr/local/opt/llvm@4/include"

==> Summary
🍺  /usr/local/Cellar/llvm@4/4.0.1: 2,259 files, 1016.3MB
~/Downloads > /usr/local/opt/llvm@4/bin/clang test.c 
test.c:1:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.

[eregon]

@hsbt Could you try with llvm or llvm@5 as mentioned above?

[hsbt]

Absolutely yes. I try to compile rc6 with another compiler tomorrow.

[hsbt]

I got the same results with llvm@4.

~/Downloads > /usr/local/opt/llvm@5/bin/clang test.c 
test.c:1:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.
~/Downloads > /usr/local/opt/llvm/bin/clang test.c 
test.c:1:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.

[eregon]

I see, so neither LLVM 5 or 6 from Homebrew works yet on Mojave then.
Did you file an issue to Homebrew?
I think we cannot do much here until the issue is fixed in Homebrew.

The reason we cannot use the system LLVM is so far macOS only provides the clang executable, not opt and not llvm-link which we also need.
Could you check what which opt and which llvm-link gives in Mojave?

[hsbt]

I couldn't find opt and llvm-link commands in Mojave. I try to build truffleruby after final release of Mojave and Xcode 10.

[eregon]

I couldn't find opt and llvm-link commands in Mojave.

Right, so this is like previous macOS versions, macOS only ships clang.

I try to build truffleruby after final release of Mojave and Xcode 10.

OK.
@hsbt Could you report the issue to Homebrew that llvm@4, llvm@5 and llvm all fail to compile a simple C file on the Mojave preview due to not finding system headers?
I think they'll be interested to look at what goes wrong there.

[bumfo]

Have anyone tried running "/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg"

before on Mojave?

This solved brew installed clang problem on my machine.

source: https://stackoverflow.com/questions/50780183/cannot-install-brew-on-mojave-with-xcode10

[hsbt]

@bumfo Awesome. I can install truffleruby-1.0.0-rc6 with your instructions.

@eregon I'm closing this.

[ioquatix]

For me, installing llvm-7.0 with MacPorts and doing

export PATH=/opt/local/libexec/llvm-7.0/bin:$PATH

first, everything worked

rvm get head
rvm install truffleruby

[ioquatix]

I tried installing llvm/clang 4.0 but it won't install on Mojave. I suggest you bump the minimum version to 7.0 or at least add multiple search paths by default.

[chrisseaton]

I tried installing llvm/clang 4.0 but it won't install on Mojave.

With Homebrew or MacPorts? We don't test MacPorts so we don't know whether it works or not. Homebrew should work.

I suggest you bump the minimum version to 7.0

What's the problem with the minimum version being lower? You can use 7.0 (we don't test it). If we bumped it other people wouldn't be able to use 4 and how would it improve the situation for you?

or at least add multiple search paths by default

You can use any installed LLVM by setting PATH. We search for those we know about and test that will work correctly. We don't want to search for something we don't test because it may not work.

What's the underlying issue here? That you'd prefer to use MacPorts rather than Homebrew?

[ioquatix]

MacPorts reports that it's not compatible so the package won't install.

You can use any installed LLVM by setting PATH. We search for those we know about and test that will work correctly. We don't want to search for something we don't test because it may not work.

You already add llvm-4.0 to the path explicitly, no?

The underlying issue here is that with only llvm-7.0 installed, the build fails without extra steps, this isn't really an issue of MacPorts/Homebrew.

[chrisseaton]

Yes we add LLVM 4's Homebrew installation path to the system path, because we test with LLVM 4. We don't test with LLVM 7 installed via Homebrew, so we haven't added that to the path.

We can't reasonably test all LLVM versions - our test suite size will grow out of control.

Maybe we should relax and run with any LLVM we can find? It may cause more frustration if untested things fail.

[ioquatix]

Maybe we should relax and run with any LLVM we can find? It may cause more frustration if untested things fail.

If you are going to add explicit support for llvm-4.0, might I suggest you use some kind of configure/auto-detection. e.g. check llvm-x.0 and then add this as a configuration parameter. I can't imagine it would be good for a user to change $PATH and install some gems with different versions of LLVM.

[chrisseaton]

Ok we'll reconsider this and think about making it an option, rather than relying on $PATH.

[ioquatix]

Just FYI, here is the output of trying to install llvm/clang 4.0 using MacPorts:

^_^ > sudo port install llvm-4.0 clang-4.0
Password:
Waiting for lock on /opt/local/var/macports/registry/.registry.lock
--->  Computing dependencies for llvm-4.0
--->  Fetching archive for llvm-4.0
--->  Attempting to fetch llvm-4.0-4.0.1_2.darwin_18.x86_64.tbz2 from http://nou.nc.packages.macports.org/pub/macports/packages.macports.org/llvm-4.0
--->  Attempting to fetch llvm-4.0-4.0.1_2.darwin_18.x86_64.tbz2.rmd160 from http://nou.nc.packages.macports.org/pub/macports/packages.macports.org/llvm-4.0
--->  Installing llvm-4.0 @4.0.1_2
--->  Activating llvm-4.0 @4.0.1_2
--->  Cleaning llvm-4.0
--->  Computing dependencies for clang-4.0
--->  Fetching distfiles for clang-4.0
Error: clang-4.0 is not supported on macOS Mojave or newer.
Error: Failed to fetch clang-4.0: unsupported platform
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-4.0/clang-4.0/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets to report a bug.
Error: Processing of port clang-4.0 failed

[coorasse]

The solution provided by @bumfo worked for me while installing rubocop which depends on jaro_winkler (failing for missing stdlib.h)

[mhenrixon]

Probably beating a dead horse and sorry if I am spamming but @bumfo's solution worked for me too.

[eregon]

Installing the system headers is the current documented workaround on macOS Mojave.

[mhenrixon]

I didn't see that line obviously... sorry about that @eregon

Truffleruby is working fine though RSpec is insanely slow but that was expected. Just want to be able to provide my gem on more than MRI and jruby and so far it is working fantastically well.

[rafaelfranca]

As for Xcode 11 the system headers package will not be provided anymore. The correct fix on MacOS is to set the -isysroot flag when building or use xcrun --sdk macosx as CC wrapper. We can also tell people to set SDKROOT to the MacOS SDK like:

export SDKROOT=`xcrun --show-sdk-path`

@eregon do you know if we can change the CC wrapper only on MacOS? I tried to look on this file https://github.com/oracle/truffleruby/blob/700016057b446036cf99786fa723dd87acbbb2cd/lib/mri/mkmf.rb but I'm not familiar enough with how truffleruby is built to open a PR doing that. I'm totally open to change the documentation though, but it would be better if it just work on MacOS.

[eregon]

@rafaelfranca Thanks, we came to the same conclusion in #1733 and in discussions with the Sulong team, but did not apply changes yet.
cc @zapster

The actual code for C/C++ compilation is in https://github.com/oracle/truffleruby/blob/master/lib/truffle/rbconfig-for-mkmf.rb
I'll change that to wrap CC with xcrun --sdk macosx and check the impact on time to compile C files.

[eregon]

We fixed this by using xcrun --sdk macosx /path/to/toolchain/bin/clang in the Sulong toolchain and it will be in the next release, 19.3.0. So TruffleRuby should work out-of-the-box on macOS, without any configuration (no need to set SDKROOT).