Zig: Compilation of andrewrk/tetris takes a long time

[joachimschmidt557]

The compilation of https://github.com/andrewrk/tetris with zig build play takes a long time and the produces this error:

In file included from /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/x86_64/start.S:56:
In file included from /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/unix/sysv/linux/x86_64/sysdep.h:23:
In file included from /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/unix/x86_64/sysdep.h:18:
In file included from /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/unix/sysdep.h:20:
In file included from /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/unix/sysv/linux/sys/syscall.h:29:
/home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/unix/sysv/linux/include/bits/syscall.h:3:10: fatal error: 'misc/bits/syscall.h' file not found
#include <misc/bits/syscall.h>
         ^~~~~~~~~~~~~~~~~~~~~
1 error generated.

The following command failed:
/home/joachim/src/zig/build/bin/zig cc -MD -MV -MF /home/joachim/.cache/zig/stage1/tmp/1-3BVhBaqso9-start.o.d -nostdinc -fno-spell-checking -isystem /home/joachim/src/zig/build/lib/zig/include -march=native -g -fno-omit-frame-pointer -D_DEBUG -fno-stack-protector -fPIC -o /home/joachim/.cache/zig/stage1/tmp/1-3BVhBaqso9-start.o -c /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/x86_64/start.S -I /home/joachim/src/zig/build/lib/zig/libc/glibc/include -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/unix/sysv/linux/x86_64 -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/unix/sysv/linux/x86 -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/x86_64/nptl -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/x86/nptl -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/unix/sysv/linux/generic -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/unix/sysv/linux/include -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/unix/sysv/linux -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/nptl -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/pthread -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/unix/sysv -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/unix/x86_64 -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/unix/x86 -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/unix -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/x86_64 -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/x86 -I /home/joachim/src/zig/build/lib/zig/libc/glibc/sysdeps/generic -I /home/joachim/src/zig/build/lib/zig/libc/glibc -I /home/joachim/src/zig/build/lib/zig/libc/include/x86_64-linux-gnu -I /home/joachim/src/zig/build/lib/zig/libc/include/generic-glibc -I /home/joachim/src/zig/build/lib/zig/libc/include/x86_64-linux-any -I /home/joachim/src/zig/build/lib/zig/libc/include/any-linux-any -D_LIBC_REENTRANT -include /home/joachim/src/zig/build/lib/zig/libc/glibc/include/libc-modules.h -DMODULE_NAME=libc -Wno-nonportable-include-path -include /home/joachim/src/zig/build/lib/zig/libc/glibc/include/libc-symbols.h -DPIC -DSHARED -DTOP_NAMESPACE=glibc -DASSEMBLER -g -Wa,--noexecstack
tetris...The following command exited with error code 1:
/home/joachim/src/zig/build/bin/zig build-exe /home/joachim/src/tetris/src/main.zig --c-source -std=c99 /home/joachim/src/tetris/stb_image-2.22/stb_image_impl.c --library c --library glfw --library epoxy --cache-dir /home/joachim/src/tetris/zig-cache --name tetris -I /home/joachim/src/tetris/stb_image-2.22 --cache on 

Build failed. The following command failed:
/home/joachim/src/tetris/zig-cache/o/fJpu__w_zQBsn1tpniPCm6yiemgd6lVDdF0DVxjmGZz77zpgaTY5NToLm8nLhc99/build /home/joachim/src/zig/build/bin/zig /home/joachim/src/tetris /home/joachim/src/tetris/zig-cache play

[andrewrk]

For the glibc problem, ~I believe you are hitting #2044. Unfortunately, until #2044 is solved it means that there are some circumstances involving .h file inclusion order, which may require the workaround of wiping zig-cache.~ Nope once again it wasn't the cache's fault. See below

For the other problem, I believe it is a very recent regression in translate-c. Would you be willing to do a git bisect?

[joachimschmidt557]

It seems that wiping the zig-cache directory does not solve the problem. I'll try and do a git bisect.

[andrewrk]

Can you confirm which zig git version you are using?

[joachimschmidt557]

I'm using 0.5.0-1844-gd7dc7d7a5

[joachimschmidt557]

I'll try compiling with the newest commit now.

[andrewrk]

I don't recognize 0.5.0-1844-gd7dc7d7a51 neither does github

[dimenus]

I wiped my global cache to resolve this, not the local repo cache.

~/.cache/zig

[joachimschmidt557]

@andrewrk Sorry, I was on commit d7dc7d7a500541cf2ac0d3cfc28511b2fe7e88d4, I think.

[dimenus]

@joachimschmidt557 did you wipe your global cache or the repo cache? The file it's referring to is in your global user cache.

[joachimschmidt557]

Yes, I tried wiping the global and local caches, but I still get the same error, even with the newest commit.

[andrewrk]

Let's get the glibc bug figured out first. It looks like you've done a make install previously, and have not done a make install after updating to latest master. Alternatively you can delete your lib folder in the build directory, and zig will find the lib directory from the source repo.

[joachimschmidt557]

@andrewrk Yes, after wiping the build directory and building and installing zig from scratch, the error does not occur anymore.

[andrewrk]

I believe that was not necessary, and also deleting zig-cache was not necessary (I think I was wrong at first), the only problem was that you needed to run make install again (or delete the lib directory so that zig used lib from the source repo)

[joachimschmidt557]

Ok, thanks, however, the time taken for zig build play still seems very long.

[andrewrk]

Yes, now we are on to this part:

For the other problem, I believe it is a very recent regression in translate-c. Would you be willing to do a git bisect?

[joachimschmidt557]

Ok, I'll do the git bisect now.

[joachimschmidt557]

I have done the git bisecting and the first bad commit is 3b23929be565a66a94441d107448860079c2847d.

[andrewrk]

Thanks!

cc @Vexu

[andrewrk]

It's entirely possible that what's happening here is that commit made zig be able to translate a lot more macros, and so the translation takes longer because there are a ton of macros in the files that project is including. The next step will be adding some kind of introspection so that we can see where the time is being taken.

[joachimschmidt557]

I don't know if it helps, but I've created a callgrind output for the zig build command on the tetris repository: https://gist.github.com/joachimschmidt557/4e5a6eb7abdcfc1b2a855e789e81c004

[andrewrk]

@joachimschmidt557 if you provide the corresponding binary (which has debug info) that matches the above callgrind output, I'll take a look using kcachegrind.

[joachimschmidt557]

@andrewrk Here's the zig binary I used: https://send.firefox.com/download/2ed19fd51c0a7c56/#tzwyZVF0ZYoqpEhCrl7_aQ

[andrewrk]

Thanks- I was able to inspect the analysis but with a few minutes of digging I was unable to find any anomalies. I'll take a closer look again in the future.

[dbandstra]

I noticed that in the past month or two, building oxid started taking significantly longer too.

---

Zig commit: 4f2652d504c796bbba6d7ccf6a699dc01055e3e7 (the commit before the translate-c patch)
Oxid commit: 54bac25fb45d7218603e2585105ac94eb1bc5468 (Feb 17)

real    0m29.779s
user    0m25.221s
sys     0m2.084s

Build succeeded.

---

Zig commit: 3b23929be565a66a94441d107448860079c2847d (translate-c patch)
Oxid commit: 54bac25fb45d7218603e2585105ac94eb1bc5468 (same as above)

real    0m50.464s
user    0m47.252s
sys     0m1.154s

Build failed:

./zig-cache/o/3vwcxg5tOwQ3Z_zOtSMCRdt9Ejadmvpkqv7Cji_DEcE6I4uqlIYVlNym-QLl1hRA/cimport.zig:38410:61: error: invalid character: 'L'
pub const __LDBL_DENORM_MIN__ = 3.64519953188247460253e-4951L;
                                                            ^

---

Zig commit: abe7305e169be2047d65f96e6525d3828684f058 (master)
Oxid commit: 0785d0f38a1c74e633b112f97489ea5139458229 (master + minor workarounds)

real    3m35.067s
user    3m28.661s
sys     0m2.379s

Build succeeded.

I noticed that recently builds have been spending a long time (like 2 minutes?) at Semantic Analysis [42/76].

---

These are using zig built from source, deleting zig-cache before every run. I've started using downloaded binaries instead which are faster because I guess they're release builds. Using this, the last mentioned oxid commit builds in 45 seconds.

[joachimschmidt557]

@schmidh I just compiled with the newest commit 675f01f1768aa08c307640b53e8a5240fa190fab and the compilation still takes a long time at a Semantic Analysis [44/57] step. I'll change the issue title to accurately reflect the problem.

[Vexu]

Self hosted translate-c is taking a long time with libepoxy.

// zig translate-c a.h -lepoxy  22,35s
// C loc: 10207
#include <epoxy/gl.h>
// zig translate-c a.h -lglfw  5,39s
// C loc: 4547
#include <GLFW/glfw3.h>
// zig translate-c a.h -I stb_image-2.22 -lc  0,07s
#define STBI_ONLY_PNG 
#define STBI_NO_STDIO 
#include <stb_image.h>

[joachimschmidt557]

I've recently revisited this issue on the current master branch (tested with https://github.com/ziglang/zig/commit/237c5429b0e8a089c89d13031ae6004ebac55ba9) and the compilation speed (in particular the translation of C headers) does not seem to be an issue anymore.