Zig: Fail to build on macOS

[philix]

It looks like a LLVM issue, but maybe you have some insight about it

felipe@calvin:~/code/zig/build (readme *$)$ cmake .. -DCMAKE_BUILD_TYPE=Release  -DCMAKE_INSTALL_PREFIX=$(pwd)   -DZIG_LIBC_INCLUDE_DIR=/usr/include  -DZIG_LLVM_OLD_CXX_ABI=yes  -DCLANG_INCLUDE_DIRS=/usr/local/llvm-3.8/include/
Configuring zig version 0.0.0
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/felipe/code/zig/build
felipe@calvin:~/code/zig/build (readme *$)$ make
[  4%] Building CXX object CMakeFiles/run_tests.dir/src/buffer.cpp.o
[  8%] Building CXX object CMakeFiles/run_tests.dir/src/util.cpp.o
[ 12%] Building CXX object CMakeFiles/run_tests.dir/src/os.cpp.o
[ 16%] Building CXX object CMakeFiles/run_tests.dir/src/error.cpp.o
[ 20%] Building CXX object CMakeFiles/run_tests.dir/test/run_tests.cpp.o
[ 24%] Linking CXX executable run_tests
[ 24%] Built target run_tests
[ 28%] Building CXX object CMakeFiles/zig.dir/src/link.cpp.o
[ 32%] Building CXX object CMakeFiles/zig.dir/src/target.cpp.o
[ 36%] Building CXX object CMakeFiles/zig.dir/src/ast_render.cpp.o
[ 40%] Building CXX object CMakeFiles/zig.dir/src/bignum.cpp.o
[ 44%] Building CXX object CMakeFiles/zig.dir/src/tokenizer.cpp.o
[ 48%] Building CXX object CMakeFiles/zig.dir/src/c_tokenizer.cpp.o
[ 52%] Building CXX object CMakeFiles/zig.dir/src/parser.cpp.o
[ 56%] Building CXX object CMakeFiles/zig.dir/src/eval.cpp.o
[ 60%] Building CXX object CMakeFiles/zig.dir/src/analyze.cpp.o
[ 64%] Building CXX object CMakeFiles/zig.dir/src/codegen.cpp.o
[ 68%] Building CXX object CMakeFiles/zig.dir/src/buffer.cpp.o
[ 72%] Building CXX object CMakeFiles/zig.dir/src/error.cpp.o
[ 76%] Building CXX object CMakeFiles/zig.dir/src/main.cpp.o
[ 80%] Building CXX object CMakeFiles/zig.dir/src/os.cpp.o
[ 84%] Building CXX object CMakeFiles/zig.dir/src/util.cpp.o
[ 88%] Building CXX object CMakeFiles/zig.dir/src/errmsg.cpp.o
[ 92%] Building CXX object CMakeFiles/zig.dir/src/zig_llvm.cpp.o
[ 96%] Building CXX object CMakeFiles/zig.dir/src/parseh.cpp.o
In file included from /Users/felipe/code/zig/src/parseh.cpp:17:
In file included from /usr/local/Cellar/llvm38/3.8.0/lib/llvm-3.8/include/clang/Frontend/ASTUnit.h:18:
In file included from /usr/local/Cellar/llvm38/3.8.0/lib/llvm-3.8/include/clang/AST/ASTContext.h:18:
In file included from /usr/local/Cellar/llvm38/3.8.0/lib/llvm-3.8/include/clang/AST/ASTTypeTraits.h:19:
/usr/local/Cellar/llvm38/3.8.0/lib/llvm-3.8/include/clang/AST/ASTFwd.h:22:10: fatal error: 'clang/AST/DeclNodes.inc' file not found
#include "clang/AST/DeclNodes.inc"
         ^
1 error generated.
make[2]: *** [CMakeFiles/zig.dir/src/parseh.cpp.o] Error 1
make[1]: *** [CMakeFiles/zig.dir/all] Error 2
make: *** [all] Error 2

with VERBOSE=1

felipe@calvin:~/code/zig/build (readme *$)$ make VERBOSE=1
/usr/local/Cellar/cmake/3.6.1/bin/cmake -H/Users/felipe/code/zig -B/Users/felipe/code/zig/build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/local/Cellar/cmake/3.6.1/bin/cmake -E cmake_progress_start /Users/felipe/code/zig/build/CMakeFiles /Users/felipe/code/zig/build/CMakeFiles/progress.marks
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/Makefile2 all
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/run_tests.dir/build.make CMakeFiles/run_tests.dir/depend
cd /Users/felipe/code/zig/build && /usr/local/Cellar/cmake/3.6.1/bin/cmake -E cmake_depends "Unix Makefiles" /Users/felipe/code/zig /Users/felipe/code/zig /Users/felipe/code/zig/build /Users/felipe/code/zig/build /Users/felipe/code/zig/build/CMakeFiles/run_tests.dir/DependInfo.cmake --color=
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/run_tests.dir/build.make CMakeFiles/run_tests.dir/build
make[2]: Nothing to be done for `CMakeFiles/run_tests.dir/build'.
[ 24%] Built target run_tests
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/zig.dir/build.make CMakeFiles/zig.dir/depend
cd /Users/felipe/code/zig/build && /usr/local/Cellar/cmake/3.6.1/bin/cmake -E cmake_depends "Unix Makefiles" /Users/felipe/code/zig /Users/felipe/code/zig /Users/felipe/code/zig/build /Users/felipe/code/zig/build /Users/felipe/code/zig/build/CMakeFiles/zig.dir/DependInfo.cmake --color=
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/zig.dir/build.make CMakeFiles/zig.dir/build
[ 28%] Building CXX object CMakeFiles/zig.dir/src/parseh.cpp.o
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++    -I/usr/local/Cellar/llvm38/3.8.0/lib/llvm-3.8/include -I/usr/local/llvm-3.8/include -I/Users/felipe/code/zig -I/Users/felipe/code/zig/build -I/Users/felipe/code/zig/src  -O3 -DNDEBUG   -std=c++11 -fno-exceptions -fno-rtti -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -D__USE_MINGW_ANSI_STDIO -Werror=strict-prototypes -Werror=old-style-definition -o CMakeFiles/zig.dir/src/parseh.cpp.o -c /Users/felipe/code/zig/src/parseh.cpp
In file included from /Users/felipe/code/zig/src/parseh.cpp:17:
In file included from /usr/local/Cellar/llvm38/3.8.0/lib/llvm-3.8/include/clang/Frontend/ASTUnit.h:18:
In file included from /usr/local/Cellar/llvm38/3.8.0/lib/llvm-3.8/include/clang/AST/ASTContext.h:18:
In file included from /usr/local/Cellar/llvm38/3.8.0/lib/llvm-3.8/include/clang/AST/ASTTypeTraits.h:19:
/usr/local/Cellar/llvm38/3.8.0/lib/llvm-3.8/include/clang/AST/ASTFwd.h:22:10: fatal error: 'clang/AST/DeclNodes.inc' file not found
#include "clang/AST/DeclNodes.inc"
         ^
1 error generated.
make[2]: *** [CMakeFiles/zig.dir/src/parseh.cpp.o] Error 1
make[1]: *** [CMakeFiles/zig.dir/all] Error 2
make: *** [all] Error 2

[andrewrk]

We're about to upgrade to 3.9.0 which might make this issue go away. After upgrading to 3.9.0 I'll test the build on my Mac.

[philix]

Could you change the build strategy to build zig inside LLVM like clang does?

Right now, it's impossible for me to build it on my macOS. I am unable to understand the CMakeFiles, they have hardcoded paths, library names...

[philix]

I managed to get all include paths set, but now I'm getting linking errors because it seems like I didn't build libclang as static libraries...

include_directories(
  /Users/felipe/src/llvm39/build/include/
  /Users/felipe/src/llvm39/build/tools/cfe-3.9.0.src/include/)
include_directories(SYSTEM
  /Users/felipe/src/llvm39/llvm-3.9.0.src/include/
  /Users/felipe/src/llvm39/llvm-3.9.0.src/tools/cfe-3.9.0.src/include)
``

[andrewrk]

I'm about to get the build working on my Mac soon, I'll let you know how I do it.

Might be easier when this is merged: https://github.com/Homebrew/homebrew-core/pull/4579

[aindigo]

I was able to compile zig (100990b052be249c23743140d19ced37d463b51d) without any problem and compile an example zig program that calls the write macosx syscall ( El Capitan 10.11) .
you just have to download llvm+clang 3.9.0 from their site and point to it in the cmake execution with the flags CMAKE_PREFIX_PATH and CMAKE_CXX_COMPILER.

these are the step i followed :

https://gist.github.com/aindigo/0b3f76db7cbe053d478b4c731502ba40

[andrewrk]

I just tested this and it worked, although I had to install llvm and clang from source. Here are my steps:

tar xvf llvm-3.9.0.src.tar.xz
cd llvm-3.9.0.src
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=~/local -DCMAKE_PREFIX_PATH=~/local -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_OSX_DEPLOYMENT_TARGET=""
make
make install

tar xvf cfe-3.9.0.src.tar.xz
cd cfe-3.9.0.src
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=~/local -DCMAKE_PREFIX_PATH=~/local -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_OSX_DEPLOYMENT_TARGET=""
make
make install

git clone https://github.com/andrewrk/zig
cd zig
mkdir build
cd build
cmake .. -DCMAKE_PREFIX_PATH=~/local -DCMAKE_INSTALL_PREFIX=$(pwd) -DZIG_LIBC_LIB_DIR=$(dirname $(cc -print-file-name=crt1.o)) -DZIG_LIBC_INCLUDE_DIR=$(echo -n | cc -E -x c - -v 2>&1 | grep -B1 "End of search list." | head -n1 | cut -c 2-) -DZIG_LIBC_STATIC_LIB_DIR=$(dirname $(cc -print-file-name=crtbegin.o)) -DCMAKE_OSX_DEPLOYMENT_TARGET=""
make
make install
./run_tests

[stevetranby]

Your echo command in your third step to get the current framework folder doesn't work on my machine (El Capitan, XCode 8).

Mine prints out "/Library/Frameworks (framework directory)".

This seemed to work:

cmake .. -DCMAKE_PREFIX_PATH=~/local -DCMAKE_INSTALL_PREFIX=$(pwd) -DZIG_LIBC_LIB_DIR=$(dirname $(cc -print-file-name=crt1.o)) -DZIG_LIBC_INCLUDE_DIR=$(echo -n | cc -E -x c - -v 2>&1 | grep -B1 "End of search list." | head -n1 | cut -d' ' -f2) -DZIG_LIBC_STATIC_LIB_DIR=$(dirname $(cc -print-file-name=crtbegin.o)) -DCMAKE_OSX_DEPLOYMENT_TARGET=""

[stevetranby]

Well the cmake command worked. But after make and make install (appeared success) I ran the tests and got this error:

Test 1/168 hello world with libc...
Compile failed:
./zig build .tmp_source.zig --export exe --name test --output ./.tmp_exe --release --strip --color on --check-unused --library c
unable to infer -macosx-version-min or -mios-version-min

[stevetranby]

Oh, my bad! I failed to look at the gist above.
export MACOSX_DEPLOYMENT_TARGET=10.11 fixed my previous issues.

Now we have :\ ... will be fun to start playing with the language now that some tests work.

Test 42/168 multiple function definitions...OK
pipe failed
Test 43/168 bad directive...[1]    20650 abort      ./run_tests

Edit: the gist example syscall works fine

[andrewrk]

Hmm I'm not sure what's going on there.

[philix]

I managed to compile with @stevetranby instructions. Thanks!

$ ./run_tests
Test 1/169 hello world with libc...OK
Test 2/169 multiple files with private function...
Compile failed:
./zig build .tmp_source.zig --export exe --name test --output ./.tmp_exe --release --strip --color on --check-unused
unable to open '/Users/felipe/code/zig/build/lib/zig/std/bootstrap.zig': file not found

[andrewrk]

Did you do the make install step?

[philix]

@andrewrk I forgot that. The tests ran successfully when I ran it. Thanks!

[stevetranby]

This seems to not throw any errors.

#bogus1("")
extern fn b();
#bogus2("")
fn a() {}
pub fn main(args: [][]u8) -> %void {
}

Whereas this does. Prob the reason my run_tests failed ... ??

#bogus1("")
extern fn b();
#bogus2("")
//fn a() {}
pub fn main(args: [][]u8) -> %void {
}

This outputs on build

$ zig build --name test --export exe --output test test.zig
./test.zig:3:1: error: invalid directive: 'bogus2'
#bogus2("")
^

[andrewrk]

Hmm I'm clueless as to why this sort of thing would work for some people and not for others, since it has nothing to do with interaction with the operating system.

[stevetranby]

I'll try fresh on another machine. I'm not yet fully set up (or mostly not at a level to understand the core code yet) to debug this yet, but I'll try and take a brief look into the core.

[stevetranby]

@andrewrk after trying to narrow down which tests might be the culprit I finally noticed that the same test would fail or pass depending on how many I commented out, which led me to search for why exec fork would fail. It appears OSX has a low open file descriptor limit by default (256).

I'm guessing you could restructure your test runner to wait on a few tests, close the files, and then continue to the next set of tests so that it doesn't run up against the limit.

Check limit

ulimit -a 

Set new limit (I found 512 was still not enough) and all tests now pass.

ulimit -n 1024

https://superuser.com/questions/433746/is-there-a-fix-for-the-too-many-open-files-in-system-error-on-os-x-10-7-1

[stevetranby]

It might be smart to add this to a OSX script (or just document it) such that one might set a higher limit only for the tests and then revert back to what it was set to before.

My current build script for now (this is overkill, but I wanted to make sure the test ran fully compiled each time).

#!/usr/bin/env bash

# required for min/max build issues
export MACOSX_DEPLOYMENT_TARGET=10.11

# need more file descriptors for the test runner
OLD_LIMIT=$(ulimit -n)
echo $OLD_LIMIT
ulimit -n 2048

# fully build in case of changes or new pull from git (overkill)
cd $ZIG_ROOT/build && cmake .. -DCMAKE_PREFIX_PATH=~/local -DCMAKE_INSTALL_PREFIX=~/ -DZIG_LIBC_LIB_DIR=$(dirname $(cc -print-file-name=crt1.o)) -DZIG_LIBC_INCLUDE_DIR=$(echo -n | cc -E -x c - -v 2>&1 | grep -B1 "End of search list." | head -n1 | cut -d' ' -f2) -DZIG_LIBC_STATIC_LIB_DIR=$(dirname $(cc -print-file-name=crtbegin.o)) -DCMAKE_OSX_DEPLOYMENT_TARGET="" && make && make install && ./run_tests

# revert
ulimit -n $OLD_LIMIT

[andrewrk]

Aha! Thank you for figuring this out! Looks like perhaps zig leaks file descriptors in the implementation of exec. Let me look into this and maybe you won't need a script after all.

[aindigo]

i think the command run_test is creating source files compiling/executing test and then not closing them.

i have a ridiculously high ulimit cos mongodb cry about low ulimit, that's why i havent received that kind of errors.

[andrewrk]

Have another try after this commit that I just pushed.

[stevetranby]

Yep, tests all pass with only 256 file descriptor limit.