Zig: clarify approach to determine includes for libSystem on host macOS targeting macOS

[mikdusan]

Recently after upgrading to macOS 10.14 Mojave I finally encountered some issues others were having regarding finding macOS header files. This led me down the rabbit hole and I'd like to make certain zig is doing the desired thing.

So first thing I'd like to get clarification on the desired way to build C (and I assume we leave the door open to C++ for future) code on macOS for macOS. My understanding is:

The Way It Is

1. The zig toolchain (zig itself) builds against a specific version of LLVM+clang.
2. Zig products that are C-based are built with an embedded version of clang from step 1.
3. For maximum compatibility on building macOS applications the behaviour of LLVM+clang will mimic Xcode+clang, Apple's supported toolchain.
4. Currently Xcode+ld linker is used because LLVM+lld is inadequate for Mach-O.
5. Zig caches search paths related to header and library files used for libSystem in zig-cache/native_libc.txt.
6. Zig populates native_libc.txt by parsing information from generic compiler command line as documented in the same file.

Unclear

U1. In step 1 above, toolchain use is unclear to me. It's clear that LLVM+clang headers and libs are used. It's not clear which executable clang++ is used: LLVM's or Xcode's ? This question is only wrt building of zig0 and zig (stage1).

U2. Is it desirable that both LLVM's and Xcode's clang++ work?

note: I may add to this list as details are provided

[mattsta]

I spent last weekend fighting some of these issues too with the pass-through zig cc mode.

Long story short:

• you can see where zig thinks your headers are at with: ./bin/zig libc
  
  
  
  • which it retrieves here with path exploitation
  
  
• zig is expecting stdlib.h to reveal include_dir and sys/errno.h to reveal sys_include_dir, but on macOS both of those are in the same place and a second directory (with things like stdint.h) are never added to the search path. The current dirs detected by zig using libc_installation.cpp aren't enough to provide all headers to clang-in-zig on latest macOS.

One simple solution is always add the llvm-in-clang header directory to all builds by default and then include one -isysroot flag for the system sdk (see below).

Here's the rundown for macOS:

• mac has split /usr/include into two places:
  
  
  
  • System-specific directory: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
  
  
  • Standard C headers (from llvm, basically the same as zig's c_headers directory): /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1/include
    
    🤷‍♂️
  
  

You can programmatically query the "platform SDK" location with xcrun -sdk macosx --show-sdk-path:

% xcrun -sdk macosx --show-sdk-path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk

We can see other projects like llvm use xcrun to locate the system header directory then provide the header directory as the -isysroot argument to clang.

The problem is (and I don't quite understand why) the system root doesn't have all standard headers like stdint.h and stdbool.h, so for zig cc, I need to provide the llvm headers zig bundles with itself to get anything to compile (you could also provide the system's .../Toolchains/... directory, but they are both basically the same llvm headers at this point, so this simplifies the command line and, as a bonus, gives you the same standard C includes across all your different operating systems) :

./bin/zig cc -std=gnu11 -c -o matt.o matt.c -isystem lib/zig/include -isysroot $(xcrun -sdk macosx --show-sdk-path)

Notice all the directories the system Xcode-provided-clang yields:

% clang -E -Wp,-v -xc /dev/null
clang -cc1 version 10.0.1 (clang-1001.0.46.4) default target x86_64-apple-darwin18.2.0
...
#include <...> search starts here:
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks (framework directory)

The directory .../Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1/include has the common from-llvm headers like stdbool.h and stdint.h, but the system platform directories have sys/errno.h and stdlib.h: .../Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include.

Notice how the standard C headers are in Toolchains while the system (stdlib.h, sys/errno.h) are in Platforms.

but, the clang-inside-zig isn't populating _all_ those include directories. It's just picking the first one it finds with stdlib.h and sys/errno.h which doesn't provide the basic C headers on macOS anymore.

Another example: here's my hand-compiled llvm8 clang, which zig won't acknowledge as valid:

% CC=/home/matt/bin/llvm8/bin/clang ./bin/zig libc
unable to determine libc include path: stdlib.h not found in '/home/matt/bin/llvm8/bin/clang' search paths
unable to determine libc include path: sys/errno.h not found in '/home/matt/bin/llvm8/bin/clang' search paths

because my custom compiled llvm8 doesn't know anything about /usr/include being split and moved to inside of the Xcode.app bundle:

% /home/matt/bin/llvm8/bin/clang -E -Wp,-v -xc /dev/null
clang -cc1 version 8.0.0 based upon LLVM 8.0.0 default target x86_64-apple-darwin18.2.0
ignoring nonexistent directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /Users/matt/src/llvm8/lib/clang/8.0.0/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)

The overall solution other build systems like CMake follow: a compiler will know about its _own_ headers (as printed by -E -Wp,-v -xc), so as long as all the compiler-known headers are included, you just need to augment the compiler command with platform headers using -isysroot $(xcrun -sdk macosx --show-sdk-path)

How to get all of that done inside the current zig assumptions around stdlib.h and sys/errno.h when those detections don't work on macOS anymore? No idea. I'm just happy I got zig cc to finally compile something.

[andrewrk]

I'm just happy I got zig cc to finally compile something.

What's your use case for using zig cc directly? It's mainly intended to be used by the higher level --c-source CLI. More details: https://ziglang.org/download/0.4.0/release-notes.html#Zig-is-also-a-C-Compiler

[mattsta]

What's your use case for using zig cc directly?

Primarily just playing with demos from your livestream Q&A a week or two ago (thanks!)

I was toying with the idea of telling CMake to use CC="zig cc" too. Right now I custom compile new llvm versions across multiple different platforms in my C build farm, but since zig is statically bundling new llvm versions within a day of release, maybe copying around self-contained zig build platforms is easier than hand wrangling llvm compiles multiple times a year.

[mikdusan]

I spent last weekend fighting some of these issues too with the pass-through zig cc mode.

mattsta, I see you went down the rabbit hole too.

I've been looking very closely at Xcode-clang default search paths, along the lines of details which you listed as well. I'm going to use macOS 10.14 and Xcode 10.2.1 clang as reference:

sw_vers

ProductName:    Mac OS X
ProductVersion: 10.14.4
BuildVersion:   18E2034

clang --version

Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.5.0
Thread model: posix
InstalledDir: /Applications/Xcode_10.2.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

clang -E -Wp,-v -xc /dev/null

clang -cc1 version 10.0.1 (clang-1001.0.46.4) default target x86_64-apple-darwin18.5.0
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /Applications/Xcode_10.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/local/include
 /Applications/Xcode_10.2.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1/include
 /Applications/Xcode_10.2.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /Applications/Xcode_10.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include
 /Applications/Xcode_10.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks (framework directory)
 /Applications/Xcode_10.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/Library/Frameworks (framework directory)
End of search list.

The semantic breakdown of the search list is:

1. /usr/local/include, $SDK/local/include are locally installed headers that don't typically come with the OS and SDK, respectively.

2. $TOOLCHAIN/usr/lib/clang/10.0.1/include are compiler-specific headers. This is where new items that tracks with compiler versions are placed (like avx512 headers). It's also a place where the compiler can provide header files which are not part of classic system headers. Like stdbool.h. Another common one, stdint.h is basically let-the-SDK-provide-it and if it doesn't, then clang's kicks in. They use conditional #include_next to implement this. There's also opportunity for compiler to augment/patch SDK files as necessary.

3. $TOOLCHAIN/usr/include are toolchain headers. LLVM project libc++ will use c++/ and of course there's llvm/ and lld/ and more. The two main things in here for Xcode is c++/ and swift/.

4. $SDK/usr/include are system headers. As of macOS 10.14 Mojave, /usr/include is no longer populated. Even installing CommandLineTools will not add them. There is a manual way to do it but I am pretty sure SIP will interfere and csrutil will come into play so stay away if you can. The only way now and forward to get system headers will be to install SDKs via either CommandLineTools or Xcode, and locate their respective SDK locations (ie./Library/Developer or /Applications/Xcode.app).

5. $SDK/System/Library/Frameworks, $SDK/Library/Frameworks are frameworks. Here you'll mostly find system framework headers, and mostly facade dynamic library binaries which make linking possible.

Recommendations

1. search compiler-specific before zig-cache/native_libc.txt → sys_include_dir. This can be detected possibly by searching for exclusives such as stdbool.h could be the ticket. Alternatively clang -print-search-dirs might help and goes back to at least Xcode 8.0 ('2016). Unfortunately, clang -print-resource-dir is exactly suitable but only appeared as of Xcode 9.3 (early '2018).

2. Optional based on Zig project policy; search toolchain before zig-cache/native_libc.txt → sys_include_dir if Zig C++ interface is going to be a thing.

3. Optional based on Zig project poligy; search frameworks after zig-cache/native_libc.txt → sys_include_dir if Zig is going after the "behave like Xcode" mantra.

4. SDK headers vs. linking. There's a disconnect here. system headers are now SDK-only. Therefore, linking should be against the same SDK and not against system runtime /usr/lib/libSystem.

Consideration: LLVM-project clang

This would be a good time to decide if Zig is interested in supporting this. There is a logical path forward to use custom clang in a consistent manner, just need guidance if this is going to be a "not supported thing".

[mikdusan]

because my custom compiled llvm8 doesn't know anything about /usr/include being split and moved to inside of the Xcode.app bundle:

% /home/matt/bin/llvm8/bin/clang -E -Wp,-v -xc /dev/null
clang -cc1 version 8.0.0 based upon LLVM 8.0.0 default target x86_64-apple-darwin18.2.0
ignoring nonexistent directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /Users/matt/src/llvm8/lib/clang/8.0.0/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)

The overall solution other build systems like CMake follow: a compiler will know about its _own_ headers (as printed by -E -Wp,-v -xc), so as long as all the compiler-known headers are included, you just need to augment the compiler command with platform headers using -isysroot $(xcrun -sdk macosx --show-sdk-path)

Try running custom clang with xcrun. This is basically what makes all Xcode clang compilers aware of SDK locations (it's CommandLineTools 10.2.x for macOS 10.14 that is doing this. not sure if CommandLineTools 10.1.x for macOS 10.14 does it). Sadly, it doesn't help for toolkit location.

xcrun /opt/llvm-8.0.0/bin/clang -E -Wp,-v -xc /dev/null

and here is detail on xcrun env variables it sets before running clang. That's at least part, if not most of the magic. xcrun /usr/bin/env is an easy way to see the env. just compare with/without xcrun.

__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x52
SDKROOT=/Applications/Xcode_10.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
CPATH=/usr/local/include
LIBRARY_PATH=/usr/local/lib

[mattsta]

That's at least part, if not most of the magic. xcrun /usr/bin/env is an easy way to see the env. just compare with/without xcrun

Great point! $SDKROOT is what makes all the magic happen and we can see how clang reads SDKROOT for self-populating -isysroot internally.

Thanks for writing up a better formatted and more logically presented breakdown of all the header locations.

Basically on macOS it looks like there are now potentially up to 7 default header search paths, but worst case, they can all be managed as additional command line arguments.

[floooh]

I'm currently wrestling with the same problem (I'm on the latest macOS Mojave Beta).

My use-case is that I want to build UI applications on Mac with 3D rendering via GL or Metal, where the platform-specific parts are wrapped via C/ObjC headers (https://github.com/floooh/sokol).

Ideally I want to build a C implementation source which includes the above headers via "zig cc" (or in the build script via addCSourceFile), and also link via zig providing it with the required frameworks for linking.

Anyway, thanks for the detailed investigations @mikdusan and @mattsta! I think I'm going to mostly lurk here, but if anyone needs some testing capacity on macOS, I can help with that :)

[floooh]

FYI, this is how I could convince zig cc to compile my C code with the paths extracted from clang -E -Wp,-v -xc /dev/null just now:

zig cc -MD -MV -MF /Users/floh/projects/sokol-zig/zig-cache/tmp/p-9Fcn933TIU-sokol.o.d -nostdinc -fno-spell-checking -isystem /usr/local/Cellar/zig/HEAD-3eca5a4/lib/zig/include -isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -march=native -g -D_DEBUG -fstack-protector-strong --param ssp-buffer-size=4 -fno-omit-frame-pointer -fPIC -isystem /usr/local/include -isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1/include -isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include -isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -iframework /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks -isystem /Users/floh/projects/sokol-zig/src -o /Users/floh/projects/sokol-zig/zig-cache/tmp/p-9Fcn933TIU-sokol.o -c /Users/floh/projects/sokol-zig/src/sokol.c -ObjC -fobjc-arc

Note the following extra args extracted from clang -E...

-isystem /usr/local/include 
-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1/include 
-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include 
-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include 
-iframework /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks

...plus those Obj-C specifics:

-ObjC -fobjc-arc

I couldn't convince the build system's addCSourceFile() function to work though. First, adding the above paths via addNativeSystemIncludeDir doesn't seem to add the search paths for the C compiler invocation, and there would be a addNativeFrameworkIncludeDir needed for generating a -iframework command line arg.

And when adding the arguments to the addCSourceFile() function call, zig build-exe seems to be confused by the additional -isystem args during command line parsing:

Unexpected extra parameter: /Users/floh/projects/sokol-zig/src/sokol.c
See `/usr/local/Cellar/zig/HEAD-3eca5a4/bin/zig --help` for detailed usage information
The following command exited with error code 1:
/usr/local/Cellar/zig/HEAD-3eca5a4/bin/zig build-exe /Users/floh/projects/sokol-zig/src/main.zig --c-source -ObjC -fobj-arc -isystem  /usr/local/include -isystem  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1/include -isystem  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include -isystem  /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -isystem  /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks /Users/floh/projects/sokol-zig/src/sokol.c --verbose-cc --cache-dir /Users/floh/projects/sokol-zig/zig-cache --name bla -isystem /Users/floh/projects/sokol-zig/src -framework MetalKit -framework Foundation -framework Cocoa -framework Metal -framework Quartz --cache on 
exec failed

[gustavolsson]

Said this on twitter but it might be good to write here as well:

I couldn't convince the build system's addCSourceFile() function to work though. First, adding the above paths via addNativeSystemIncludeDir doesn't seem to add the search paths for the C compiler invocation, and there would be a addNativeFrameworkIncludeDir needed for generating a -iframework command line arg.

addNativeSystemIncludeDir() relies on a flag to be set as well (need_system_paths), which is set by exe.linkSystemLibrary("x") if x is not a libc library. I got zig to supply -isystem when calling both of these (the latter with a nonsense library). The next problem for me is to supply -framework.

And when adding the arguments to the addCSourceFile() function call, zig build-exe seems to be confused by the additional -isystem args during command line parsing:

It appears like addCSourceFile() only takes single arguments: https://github.com/ziglang/zig/blob/720ed74413f086a93f5ed66159300d8dd48e8034/src/main.cpp#L761-L778

[mikdusan]

This is probably relevant and much simpler. I found it pretty easy to get zig cc to compile an objective-c file on macOS 10.14.5 (Mojave) without the unofficial /usr/include tree installed.

zig cc \
  -fpic -fblocks -fobjc-arc \
  -resource-dir /Applications/Xcode_10.2.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1 \
  -isysroot /Applications/Xcode_10.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk \
  -c foo.m

The key args are:

• -resource-dir which is used to locate clang-specific tree. Here are includes and libs. Mostly only includes are important here. But there are some clang sanitizer and kext support libs too.

• -sysroot locates the SDK. includes, libs and frameworks searchpaths are all automatically populated from this root by (embedded) clang.

I am thinking zig-cache/native_libc.txt is the right place to:

• add key resource-dir
• add key sysroot
• deprecate or demote as secondary key sys_include_dir

To compute resource-dir:

• if clang version at least Apple LLVM version 9.1.0 (clang-902.0.39.1) → clang -print-resource-dir
• else heuristic match ^(.*/Toolchains/.*/clang/.*)/include$ from output of clang -E -Wp,-v -xc /dev/null
• else null

To compute sysroot:

• xcrun -show-sdk-path
• else /

[mikdusan]

closed: no actions to be taken

This is more or less a record of research into the details of Xcode/clang search paths and there are currently no action-items to take.

[andrewrk]

Here are the related action items:

• use case: depending on system headers and libraries #2041
• use case: ability to use zig as a drop-in replacement for a C compiler #3089