zig build: implement C compiler driver on windows

[andrewrk]

Right now we have 2 tests disabled on windows due to not having MSVC++ compiler driver code in zig build system:

        cases.addBuildFile("example/shared_library/build.zig");
        cases.addBuildFile("example/mix_o_files/build.zig");

[andrewrk]

Labeling as bug because we have disabled tests.

[Sahnvour]

I'm working on this. Have the examples building with a temporary hack.

Usually, msvc compiler is not directly available from the environment as in the *nix world. For Zig's build system, should we assume that the user runs Zig from a developper command line (that sets up the dev environment) or find the appropriate compiler, linker and libraries paths ourselves ?
I believe the latter is better, and there's already code somewhere that does it, correct @andrewrk ?

[andrewrk]

I agree with you the latter is better, and yes there's already code for it. You can look for calls to zig_find_windows_sdk. Both stage1 and the self-hosted compiler use it. You might find it interesting to play with the self hosted compiler libc command. Here's an example of running it on linux:

$ stage2/bin/zig libc
# The directory that contains `stdlib.h`.
# On Linux, can be found with: `cc -E -Wp,-v -xc /dev/null`
include_dir=/nix/store/q2q1sg5sljia8sihhwcpbxir70yw33bw-glibc-2.27-dev/include

# The directory that contains `crt1.o`.
# On Linux, can be found with `cc -print-file-name=crt1.o`.
# Not needed when targeting MacOS.
lib_dir=/nix/store/fivq0nbggp4y8mhy3ixprqd7qyn1hy2j-glibc-2.27/lib

# The directory that contains `crtbegin.o`.
# On Linux, can be found with `cc -print-file-name=crtbegin.o`.
# Not needed when targeting MacOS or Windows.
static_lib_dir=/nix/store/4ga86h16l157r7bas9hcwxgl9d3r32s6-gcc-7.4.0/lib/gcc/x86_64-unknown-linux-gnu/7.4.0

# The directory that contains `vcruntime.lib`.
# Only needed when targeting Windows.
msvc_lib_dir=

# The directory that contains `kernel32.lib`.
# Only needed when targeting Windows.
kernel32_lib_dir=

# The full path to the dynamic linker, on the target system.
# Only needed when targeting Linux.
dynamic_linker_path=/nix/store/fivq0nbggp4y8mhy3ixprqd7qyn1hy2j-glibc-2.27/lib/ld-linux-x86-64.so.2

[andrewrk]

And here's the output for me on Windows:

c:\msys64\home\andy\dev\zig\build-release>bin\zig.exe build --build-file ..\build.zig --prefix .\stage2 install

c:\msys64\home\andy\dev\zig\build-release>stage2\bin\zig.exe libc
# The directory that contains `stdlib.h`.
# On Linux, can be found with: `cc -E -Wp,-v -xc /dev/null`
include_dir=C:\Program Files (x86)\Windows Kits\10\Include\10.0.17134.0\ucrt

# The directory that contains `crt1.o`.
# On Linux, can be found with `cc -print-file-name=crt1.o`.
# Not needed when targeting MacOS.
lib_dir=C:\Program Files (x86)\Windows Kits\10\Lib\10.0.17134.0\ucrt\x64

# The directory that contains `crtbegin.o`.
# On Linux, can be found with `cc -print-file-name=crtbegin.o`.
# Not needed when targeting MacOS or Windows.
static_lib_dir=

# The directory that contains `vcruntime.lib`.
# Only needed when targeting Windows.
msvc_lib_dir=C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.15.26726\lib\x64\

# The directory that contains `kernel32.lib`.
# Only needed when targeting Windows.
kernel32_lib_dir=C:\Program Files (x86)\Windows Kits\10\Lib\10.0.17134.0\um\x64\

# The full path to the dynamic linker, on the target system.
# Only needed when targeting Linux.
dynamic_linker_path=null

(side note - that null is a bug and should be empty instead)

[andrewrk]

I want to add a note - if we want to take advantage of this zig_find_windows_sdk code, we may have to port it to zig since zig build is entirely self hosted. Or... the stage1 compiler can expose the results in the same way the stage2 compiler does, with a libc command, and the build system can execute stage1 compiler as a child process and collect the output (which it already does for other stuff).

This will also help us resolve this problem:

https://github.com/ziglang/zig/blob/43df49cb35349e48b63356eef4a990c2233ec88f/src/analyze.cpp#L6630-L6632

Because the output above can be saved into zig-cache/libc.txt and zig can look there before doing libc detection. And the fact that we define zig to look for the libc in that directory, rather than re-doing its msvc detection on every invocation, makes the cache coherent, as well as probably speed up running the test suite.

@Sahnvour does this make sense? If you like I can quickly modify the Linux build system code to work this way so you can see what it would look like, and then I could pass the figurative baton back to you for the Windows side.

[Sahnvour]

I wasn't sure how to make use of zig_find_windows_sdk right now, so yeah I'd appreciate a kickstart.

However is depending on stage1 for libc really desirable if stage3 is to be the only shipped binary ? Full Zig implementation in the standard library should be the best solution ?
(FYI at the moment I have basic cl driver working, with hardcoded paths)

[andrewrk]

However is depending on stage1 for libc really desirable if stage3 is to be the only shipped binary ?

Note that stage3 will still depend on LLVM, libclang, LLD, and therefore libstdc++ and libc (see #853 for more details). And the Zig Build System will depend on stage3 (right now it depends on stage1). So depending on invoking the Zig compiler from the Zig Build System in order to rely on C++ code to find MSVC paths does not introduce a new dependency.

[Sahnvour]

Or... the stage1 compiler can expose the results in the same way the stage2 compiler does, with a libc command, and the build system can execute stage1 compiler as a child process and collect the output (which it already does for other stuff).

IIUC this adds a dependency on the stage1 binary if we're using its libc command (I didn't mean the actual libc) ?

[andrewrk]

The stage1 binary is already a dependency of zig build, so it does not add a new dependency. Here's the build runner having access to the path to the stage1 compiler executable: https://github.com/ziglang/zig/blob/43df49cb35349e48b63356eef4a990c2233ec88f/std/special/build_runner.zig#L30

[andrewrk]

@Sahnvour have a look at #1996 - I should be able to merge it into master by the end of the weekend.

So the build system would be able to execute zig libc, capture the output, and use src-self-hosted/libc_installation.zig parse function to parse it (we can move that logic to the standard library). Then you have the paths inside zig build, in userland.

[andrewrk]

Merged! Let me know if there are any other confusing issues or blockers than I can take care of.

[Sahnvour]

Thanks. Just looked at it quickly for now, but it seems moving libc_installation.zig to std would also pull a lot of other modules from self-hosted-src into it ?

[andrewrk]

The only function we need is the parse one, and one that operates only on a memory buffer. So we can extract that code out and put that in the std lib and leave libc_installation.zig in src-self-hosted, but using the code which was just moved to the std lib.

[andrewrk]

Hi @Sahnvour - I've been doing some research about this, and I have some exciting findings. I'm actually thinking that we can drop the use case of being a third party C compiler driver, and instead Zig can be its own C compiler driver!

I'm sorry that this is stepping on your toes - I know you just did a bunch of work on code which acts as a compiler driver to MSVC. But I didn't realize this new idea until just now.

So here's the deal:

• I just got zig cc working (#490)
• It works great. For all intents and purposes Zig now has all the power of Clang available without it counting as a system dependency.
• I'm about to push to master a commit that removes Zig as a C compiler driver for all targets. Instead, there is a new CLI parameter, --c-source [options] [file] which can be used to compile C code. This works on all targets.
• This issue can be closed, and is now considered Out Of Scope of the Zig project.

I'm really excited about this, and I hope you can see the beauty in it.

[Sahnvour]

Sure, that looks more elegant and practical. Especially when it avoids having to deal with (sometimes non-sensical) windows peculiarities. 😃
Since Zig allows to integrate pre-built libraries/objects, one can still use msvc if they really want to.

I didn't spend more than a few hours on this, don't feel bad about it.