Zig: Zig on macOS Big Sur

[jedisct1]

Zig currently doesn't build, nor (for precompiled binaries) run on Big Sur.

llvm-config not found

Use -DCMAKE_PREFIX_PATH to have cmake look at /usr/local/opt/llvm:

cmake .. \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH=$(brew --prefix llvm)

cmake cannot find the compiler ID

Update: fixed in recent Zig versions.

Killed (signal 9)

Recent versions of Big Sur require libraries and executables to be signed. This includes code generated by compilers such as zig.

Unfortunately, adding a signature means having to use the codesign tool every time, or using the system ld linker instead of lld (LLVM's linker). The patch below (or the bigsur branch) does the later.

libSystem not found

__Workaround:__ presumably fixed in LLVM12, but until then, use the system linker hack and set the sysroot.

The system linker doesn't support '-error-limit, so don't add this option when using it.

The sysroot can be set by adding -syslibroot following by the path returned by the xcrun --show-sdk-path command.

Super ugly patch (or use that branch which is constantly kept in sync with the main branch):

diff --git a/src/link/MachO.zig b/src/link/MachO.zig
index 01830c156..ff24ded8b 100644
--- a/src/link/MachO.zig
+++ b/src/link/MachO.zig
@@ -397,6 +397,19 @@ pub fn flushModule(self: *MachO, comp: *Compilation) !void {
     assert(!self.libsystem_cmd_dirty);
 }

+fn getSDKPath(self: *MachO) ![]const u8 {
+    const argv = [_][]const u8{ "/usr/bin/xcrun", "--show-sdk-path" };
+    const result = try std.ChildProcess.exec(.{ .allocator = self.base.allocator, .argv = &argv });
+    if (result.stderr.len != 0) {
+        std.log.err("unexpected xcrun stderr: {}", .{result.stderr});
+    }
+    self.base.allocator.free(result.stderr);
+    if (result.term.Exited != 0) {
+        return error.XcrunReportedFailure;
+    }
+    return std.mem.trimRight(u8, result.stdout, "\r\n");
+}
+
 fn linkWithLLD(self: *MachO, comp: *Compilation) !void {
     const tracy = trace(@src());
     defer tracy.end();
@@ -529,14 +542,16 @@ fn linkWithLLD(self: *MachO, comp: *Compilation) !void {
             try fs.cwd().copyFile(the_object_path, fs.cwd(), full_out_path, .{});
         }
     } else {
-        // Create an LLD command line and invoke it.
+        // Create an LD command line and invoke it.
         var argv = std.ArrayList([]const u8).init(self.base.allocator);
         defer argv.deinit();
-        // Even though we're calling LLD as a library it thinks the first argument is its own exe name.
-        try argv.append("lld");

-        try argv.append("-error-limit");
-        try argv.append("0");
+        try argv.append("/usr/bin/ld");
+
+        const sdk_path = try self.getSDKPath();
+        defer self.base.allocator.free(sdk_path);
+        try argv.append("-syslibroot");
+        try argv.append(try self.getSDKPath());

         try argv.append("-demangle");

@@ -562,6 +577,7 @@ fn linkWithLLD(self: *MachO, comp: *Compilation) !void {
         }

         if (is_dyn_lib) {
+            try argv.append("-headerpad_max_install_names");
             try argv.append("-dylib");

             if (self.base.options.version) |ver| {
@@ -718,28 +734,19 @@ fn linkWithLLD(self: *MachO, comp: *Compilation) !void {
             .data = std.ArrayList(u8).init(self.base.allocator),
         };
         defer stdout_context.data.deinit();
-        const llvm = @import("../llvm.zig");
-        const ok = llvm.Link(
-            .MachO,
-            new_argv.ptr,
-            new_argv.len,
-            append_diagnostic,
-            @ptrToInt(&stdout_context),
-            @ptrToInt(&stderr_context),
-        );
-        if (stderr_context.oom or stdout_context.oom) return error.OutOfMemory;
-        if (stdout_context.data.items.len != 0) {
-            std.log.warn("unexpected LLD stdout: {}", .{stdout_context.data.items});
+
+        const result = try std.ChildProcess.exec(.{ .allocator = self.base.allocator, .argv = argv.items });
+        if (result.stdout.len != 0) {
+            std.log.warn("unexpected LD stdout: {}", .{result.stdout});
         }
-        if (!ok) {
-            // TODO parse this output and surface with the Compilation API rather than
-            // directly outputting to stderr here.
-            std.debug.print("{}", .{stderr_context.data.items});
+        if (result.stderr.len != 0) {
+            std.log.warn("unexpected LD stderr: {}", .{result.stderr});
+        }
+        self.base.allocator.free(result.stderr);
+        self.base.allocator.free(result.stdout);
+        if (result.term.Exited != 0) {
             return error.LLDReportedFailure;
         }
-        if (stderr_context.data.items.len != 0) {
-            std.log.warn("unexpected LLD stderr: {}", .{stderr_context.data.items});
-        }
     }

     if (!self.base.options.disable_lld_caching) {
diff --git a/src/main.zig b/src/main.zig
index 45fcb5ce6..5575c4abc 100644
--- a/src/main.zig
+++ b/src/main.zig
@@ -1707,7 +1707,7 @@ fn buildOutputType(
         }
     };

-    try updateModule(gpa, comp, zir_out_path, hook);
+    updateModule(gpa, comp, zir_out_path, hook) catch std.process.exit(1);

     if (build_options.is_stage1 and comp.stage1_lock != null and watch) {
         warn("--watch is not recommended with the stage1 backend; it leaks memory and is not capable of incremental compilation", .{});

Can't link libxml2.tbd

findLLVM wants to include libxml2 to linked libraries but does it by adding the -llibxml2.tbd option instead of -lxml2.

This is a bug in LLVM. When installed via Homebrew, LLVM 11 now includes a patch that fixes it.

---

Eventually, zig perfectly works on Big Sur.

Sorry for that enumeration of hacks rather sending a proper PR. Hope it can still help people who already upgraded to Big Sur.

[gaultier]

I fixed the issue Can't link libxml2.tbd happening (for me) on Catalina by simply symlinking: /usr/local/lib/libxml2.tbd -> /usr/local/lib/libxml2.dylib and that worked. Of course that is a workaround.

[jedisct1]

libxml2 is part of Xcode. You don't need to install your own version from Homebrew.

But adding a symlink inside the Xcode content would be really ugly.

[jedisct1]

Update: the libxml2.tbd issue appears to be solved in Homebrew:

https://github.com/Homebrew/homebrew-core/commit/02d57074267f4b5f31eaf0ee522e32d9f1e69b80

A new LLVM package with that fix is already available in Homebrew, and I just verified that it effectively fixes the libxml issue with Zig.

[maccam912]

@jedisct1 I have the latest llvm from homebrew, but still get the libxml2 issue. Is there some way I can check my version has that patch?

Just checked and I'm on version 10.0.1_1

[jedisct1]

Did you rebuild Zig from scratch, after removing the build directory?

[maccam912]

Yep, fresh build, even re-cloning the zig repo. What part of the process adds the "-llibxml2.tbd" flag? Thats still showing up for me.

[gaultier]

I am on Catalina with llvm 10.0.1_1 and the latest zig HEAD (as of this comment) and I got the same lbxml2.tbd error even after a complete fresh rebuild.
This fixed it for me:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index ceaecf555..28134f468 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -451,6 +451,7 @@ add_executable(zig0 "${ZIG_MAIN_SRC}" "${ZIG0_SHIM_SRC}")
 set_target_properties(zig0 PROPERTIES
     COMPILE_FLAGS ${EXE_CFLAGS}
     LINK_FLAGS ${EXE_LDFLAGS}
+    LINK_FLAGS "-L/usr/local/lib/"
 )
 target_link_libraries(zig0 zigcompiler)

@@ -505,6 +506,7 @@ add_executable(zig "${ZIG_MAIN_SRC}")
 set_target_properties(zig PROPERTIES
     COMPILE_FLAGS ${EXE_CFLAGS}
     LINK_FLAGS ${EXE_LDFLAGS}
+    LINK_FLAGS "-L/usr/local/lib/"
 )
 target_link_libraries(zig zigcompiler "${LIBSTAGE2}")
 if(MSVC)

That's because /usr/local/lib is not searched by default by the linker on macOS even though brew puts all the libraries (in form of symbolic links, at least) there. In /usr/local/lib/ I have created all the symbolic links I could think of so that it links:

 $ ls -la /usr/local/lib/*xml2*
lrwxr-xr-x  1 pgaultier  admin  40 Aug 31 18:08 /usr/local/lib/liblibxml2.tbd -> /usr/local/opt/libxml2/lib/libxml2.dylib
lrwxr-xr-x  1 pgaultier  admin  40 Aug 31 18:09 /usr/local/lib/liblibxml2.tbd.dylib -> /usr/local/opt/libxml2/lib/libxml2.dylib
lrwxr-xr-x  1 pgaultier  admin  40 Aug 31 18:06 /usr/local/lib/libxml2.dylib -> /usr/local/opt/libxml2/lib/libxml2.dylib
lrwxr-xr-x  1 pgaultier  admin  40 Aug 31 18:07 /usr/local/lib/libxml2.tbd -> /usr/local/opt/libxml2/lib/libxml2.dylib
lrwxr-xr-x  1 pgaultier  admin  28 Sep 13 19:31 /usr/local/lib/libxml2.tbd.dylib -> /usr/local/lib/libxml2.dylib

Of course if you do not have anywhere on your machine libxml2.dylib you need to install it: brew install libxml2.
Only one of those symbolic links is necessary but I don't have the heart right now to find out which one exactly, it is probably easy to find out by passing -v to the linker.

[maccam912]

Reading through the homebrew issue related to this it sounds like you can do a brew install --build-from-source llvm (do a brew uninstall llvm first if you have it). Building from source somehow gets the fix for the libxml2.tbd issue. I'm giving it a shot, will report back.

Update: No luck. It built fine but I still have llvm-config --libs --system-libs --ldflags ending with -llibxml2.tbd. I'm not sure how to make llvm-config not stick that on the end.

[maccam912]

I ended up just working around it by making a quick python executable that intercepts llvm-config, passes through all arguments, but finds any mentions of libxml2.tbd and replaces it with xml2. Worked for me, at least to compile zig!

[jedisct1]

Maybe the temporary hack that replaces libxml2.tbd can be added to Findllvm.cmake instead.

[jedisct1]

Update: -DZIG_WORKAROUND_6087=ON can now be used to work around the libxml2.tbd issue.

[kubkon]

This is so disappointing that whenever we bump either LLVM or macOS, things just stop working... 😞

[jedisct1]

Updated for the current master.

[andrewrk]

cmake cannot find the compiler ID.

The compiler ID is no longer computed at runtime with the merge of #6250 so I think this item can be crossed off.

libSystem not found

This LLD commit from master branch looks promising: https://github.com/llvm/llvm-project/commit/0ccda7c2326e1dc4e0d5d601dcc06c4b576acb0e

[jedisct1]

Thanks! Indeed, the compiler ID is not an issue any more.

And -syslibroot support in LLVM is great news!

[rudedogg]

I'm trying to get Zig going on macOS 11 beta 10.

I have LLVM 11 installed via homebrew, and am running brew install zig --HEAD. I get the following error:

Error: Failure while executing; `patch -g 0 -f -p1` exited with 1. Here's the output:
patching file CMakeLists.txt
Hunk #1 FAILED at 384.
1 out of 1 hunk FAILED -- saving rejects to file CMakeLists.txt.rej

Here are the contents of CMakeLists.txt.rej:

***************
*** 384,389 ****
      ${CLANG_LIBRARIES}
      ${LLD_LIBRARIES}
      ${LLVM_LIBRARIES}
  )


--- 384,392 ----
      ${CLANG_LIBRARIES}
      ${LLD_LIBRARIES}
      ${LLVM_LIBRARIES}
+     "-Wl,/usr/local/opt/llvm/lib/libPolly.a"
+     "-Wl,/usr/local/opt/llvm/lib/libPollyPPCG.a"
+     "-Wl,/usr/local/opt/llvm/lib/libPollyISL.a"
  )

Is there anything I can do to debug this? Or do I need to wait for new versions of LLVM/Zig?

[kubkon]

I'm trying to get Zig going on macOS 11 beta 10.

I have LLVM 11 installed via homebrew, and am running brew install zig --HEAD. I get the following error:

Error: Failure while executing; `patch -g 0 -f -p1` exited with 1. Here's the output:
patching file CMakeLists.txt
Hunk #1 FAILED at 384.
1 out of 1 hunk FAILED -- saving rejects to file CMakeLists.txt.rej

Here are the contents of CMakeLists.txt.rej:

***************
*** 384,389 ****
      ${CLANG_LIBRARIES}
      ${LLD_LIBRARIES}
      ${LLVM_LIBRARIES}
  )


--- 384,392 ----
      ${CLANG_LIBRARIES}
      ${LLD_LIBRARIES}
      ${LLVM_LIBRARIES}
+     "-Wl,/usr/local/opt/llvm/lib/libPolly.a"
+     "-Wl,/usr/local/opt/llvm/lib/libPollyPPCG.a"
+     "-Wl,/usr/local/opt/llvm/lib/libPollyISL.a"
  )

Is there anything I can do to debug this? Or do I need to wait for new versions of LLVM/Zig?

Funnily enough I was in the process of upgrading brew's Zig formula (see Homebrew/homebrew-core#63152) so that brew install zig --HEAD builds but I decided against it for the time being since this would require re-adding LLVM 10 into brew (Zig 0.6.0 is dependent on it). This doesn't seem to make much sense since the new release 0.7.0 of Zig is round the corner. Instead, might I suggest cloning the repo and building from source directly like so:

cd zig
mkdir build
cd build
cmake .. -DCMAKE_PREFIX_PATH=$(brew --prefix llvm)

This will put Zig installation in zig/build/bin (the binaries) and zig/build/lib (the libs) which you can either add to your path or copy into some other location that's already in your path.

[gaultier]

To have ‘make install’ work as expected, one can add to the cmake invocation:
-DCMAKE_INSTALL_PREFIX=/usr/local

[rudedogg]

Thanks so much @kubkon and @gaultier, I'm up and running!

Here's a beginner friendly guide for others running the Big Sur beta, to build from source:

1. Install LLVM 11 with Homebrew. This takes a long time since it's built from source, ~30 mins.

brew install llvm

2. Create and navigate to a zig directory, inside your home directory. For me:

mkdir /Users/rudedogg/zig
cd /Users/rudedogg/zig

3. Clone jedisct1s bigsur branch which fixes the libSystem error:

git clone --single-branch --branch bigsur https://github.com/jedisct1/zig.git zig-bigsur
cd zig-bigsur
mkdir build
cd build
cmake .. -DCMAKE_PREFIX_PATH=$(brew --prefix llvm) -DCMAKE_INSTALL_PREFIX=/usr/local

4. Copy the zig executable we just built from zig-bigsur/build into ~/zig:

cp zig ../../zig

5. Copy the zig-bigsur/lib directory into ~/zig, otherwise nothing will build:

cd ../
cp -r lib ../

6. Append the following to your ~/.zshrc file so you can simply run zig to test it:

# Zig
export PATH="/Users/rudedogg/zig:$PATH"

7. Allow the zig executable to run:

To do this I open finder, and open the context menu (right click) on the zig executable. Then hold the Option key when clicking the "Open" menu item. The label doesn't change, but it will open a dialog confirming you want to trust the executable and run it.

Now you should be ready for Hello World on https://ziglearn.org

[jedisct1]

Unfortunately, these issues are not solved yet :(

Big Sur now requires everything (libraries and executables) to be signed. Or else, they instantly receive a SIGKILL.

Zig doesn't build due to build not being signed. Which is simple enough to work around.

But way more annoying is the fact that zig is then unable to produce any usable library or executable. Again, due to the absence of a signature.

The system ld automatically signs everything it creates. However, lld currently doesn't.