Zig: Building the zig compiler fails due to memory allocation failure

[ask6155]

Hello,
Due to the recent changes in the internals of the zig compiler, I'm no longer able to build the compiler anymore.
The build fails at 99% where the self hosted component zig1.o is being built
Here's a small snippet

[ 95%] Building CXX object CMakeFiles/zigstage1.dir/src/stage1/util.cpp.o
[ 96%] Building CXX object CMakeFiles/zigstage1.dir/src/stage1/softfloat_ext.cpp.o
[ 97%] Linking CXX static library zigcpp/libzigstage1.a
[ 97%] Built target zigstage1
Scanning dependencies of target zig0
[ 97%] Building CXX object CMakeFiles/zig0.dir/src/stage1/zig0.cpp.o
[ 98%] Linking CXX executable zig0
[ 98%] Built target zig0
Scanning dependencies of target zig_build_zig1
[ 99%] Building self-hosted component /home/me/git/zig/build/zig1.o
allocation failed
make[2]: *** [CMakeFiles/zig_build_zig1.dir/build.make:77: CMakeFiles/zig_build_zig1] Aborted
make[1]: *** [CMakeFiles/Makefile2:107: CMakeFiles/zig_build_zig1.dir/all] Error 2
make: *** [Makefile:149: all] Error 2

I have 4GB of ram on my system and I'd assume it's enough ram to build the zig compiler as I was able to do that before the restructuring. After the commit 0.6.0+f8b3543ca I am unable to build the compiler due to enormous amount of ram that is required.

I'd like to be able to compile the build on my system because it's one the features of the compiler that I liked. Unlike other languages.

[tauoverpi]

Seems to use up to 5.4GiB on the current commit

[Snektron]

I've tested around a bit, and using jemalloc or mimalloc seems to improve memory usage quite a bit. You can use those simply by using LD_PRELOAD to override the usual (glibc or musl) allocator while building Zig:

... usual cmake setup
$ LD_PRELOAD=/usr/lib/libjemalloc.so.2 ninja

(substitute ninja with make if you use make instead)

These are the results i've obtained on my machine for building zig:
| Allocator | Peak memory usage | Wall time |
| ------------ | ------------: | ------------: |
jemalloc | 5 440 MB | 35.06s
mimalloc | 5 661 MB | 33.29s
glibc | 6 260 MB | 36.06s |
gc | 5 922 MB | 42.36

And running stdlib tests:
$ LD_PRELOAD=... /usr/bin/time --format="Peak RSS: %M KB, wall time: %e" ./zig test ../lib/std/std.zig

| Allocator | Peak memory usage | Wall time |
| ------------ | ------------: | ------------: |
jemalloc | 5 526 MB | 31.76s
mimalloc | 5 418 MB | 30.01s
glibc | 5 924 MB | 32.53s
gc | 5 317 MB | 41.34s

update: I've also tested boehm gc. This was not tested using LD_PRELOAD, but by linking with the library and replacing all malloc calls with the gc equivalent.

Also related: #6467

[Snektron]

~Update on the above: using the Boehm GC with stage 1 reduces the memory usage to 50%~
I'm not sure what happened, but i cannot reproduce the previous numbers i had for the other allocators. I've updated with the new values. Sorry to disappoint.

[marler8997]

@ask6155 did you try downloading more ram? Maybe we should add some logic in the build that checks how much ram the system has and automatically downloads more if there isn't enough?

[andrewrk]

Related: #6467

Things are going to get worse before they get better. The current plan is to (1) release 0.7.0 on October 26th and then (2) focus the main efforts of the next release cycle on finishing the self-hosted compiler, which has much better memory utilization. At this point the plan is #6378.

Hang on tight, it's gonna be a bumpy ride. But we'll get through to the other side.

[abthefish]

Andrew and all,
I tried to get around the 'allocation failed' error on a vbox Alpine 32b system by increasing swap with a swapfile. I added a 16GB /swapfile to the existing 4G swap partition.

free -m
Mem: 3542
Swap: 20479

I fail with the identical error output as the OP. I have not tried building/loading alternative *malloc libs; my interpretation of Snektron's update is this is (probably?) not an option.

Are there no work arounds to 'Hang on tight' at this point?

Thank you to all who are contributing to ziglang.

FWIW I followed this for the build; llvm v11 built without issue:
https://github.com/ziglang/zig#posix

[jedisct1]

I'd still recommend trying with jemalloc instead of the glibc malloc.

On my router that has a similar amount of memory, I can't build Zig with the default malloc either, but with jemalloc, compilation eventually completes.

[abthefish]

Thank your for that clarification. I'll try getting 'jemalloc' on my system.
Update: Nope : ) 'jemalloc' wasn't enough to coerce Zig 0.7 to compile on Alpine 32bit; good luck to the Zig team shrinking the memory footprint in 0.8. I'll 'hang on tight' and look forward to revisiting Zig down the road.