Crystal: Out of memory error compiling the compiler

[FixedTorres]

Hi, Crystal team!

Using /usr/bin/llvm-config-3.8 [version=3.8.0]
g++ -c -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc /usr/bin/llvm-config-3.8 --cxxflags
cc -fPIC -c -o src/ext/sigfault.o src/ext/sigfault.c
ar -rcs src/ext/libcrystal.a src/ext/sigfault.o
./bin/crystal build --release -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
fork: Cannot allocate memory
0x53f9be: ??? at ??
0x6ad6f5: ??? at ??
0x11f5881: ??? at ??
0x11f499c: ??? at ??
0x1147bab: ??? at ??
0x58f025: ??? at ??
0x5604bf: main at ??
0x7fe72ed3a830: __libc_start_main at ??
0x53b831: ??? at ??

Error: you've found a bug in the Crystal compiler. Please open an issue, including source code that will allow us to reproduce the bug: https://github.com/crystal-lang/crystal/issues

Any ideas? Why does this happen when I try to build Crystal from zero?

[refi64]

How much RAM do you have? Building Crystal can use up quite a bit...

[Sija]

See #3249 & #2183 :)

[lbguilherme]

We should probably catch those memory related errors and show something that is not "you've found a bug in the Crystal compiler"

[FixedTorres]

Hi guys.

I have. Ram: 4GB and CPU: Intel core i3 3110m.
How much RAM do I need?

I was watching at references #3249 & #2183 but I see that Crystal require a lot of RAM. ¿Why?

[refi64]

Ram: 4GB

I would think that would be enough...

[RX14]

Travis builds don't have more than 4gb of ram iirc, so it should be fine.

What os are you running on?

[FixedTorres]

Hi RX14.

I am using Linux Mint 18.1 64-bit with Desktop Mate.

[RX14]

Can you try building with -s? I see you're trying to compile crystal itself so that would be make stats=true crystal

[matiasgarciaisaia]

It took ~800MB in an Ubuntu 16.04.02 x64 VM to compile the compiler.

[spalladino]

Closing the issue. Please reopen with more info.

[anatol]

FYI I see the same issue when try to build Crystal Arch package. It started with Crystal 0.21.0 release.

==> Retrieving sources...
  -> Found crystal-0.21.0.tar.gz
  -> Found crystal-0.21.0-1-linux-x86_64.tar.gz
==> Validating source files with sha256sums...
    crystal-0.21.0.tar.gz ... Passed
==> Validating source_x86_64 files with sha256sums...
    crystal-0.21.0-1-linux-x86_64.tar.gz ... Passed
==> Extracting sources...
  -> Extracting crystal-0.21.0.tar.gz with bsdtar
  -> Extracting crystal-0.21.0-1-linux-x86_64.tar.gz with bsdtar
==> Starting prepare()...
==> Starting build()...
Using /usr/bin/llvm-config [version=3.9.1]
g++ -c -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc `/usr/bin/llvm-config --cxxflags`
cc -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fPIC  -D_FORTIFY_SOURCE=2  -c -o src/ext/sigfault.o src/ext/sigfault.c
ar -rcs src/ext/libcrystal.a src/ext/sigfault.o
./bin/crystal build --release  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
fork: Cannot allocate memory
0x54031e: ??? at /root/.cache/crystal/macro56261008.cr 10:28
0x10932aa: ??? at /var/cache/omnibus/src/crystal/src/process.cr 127:7
0x1476eb8: ??? at /var/cache/omnibus/src/crystal/src/process.cr 102:5
0x1475af1: ??? at /var/cache/omnibus/src/crystal/src/process.cr 145:14
0x13b7e05: ??? at /var/cache/omnibus/src/crystal/src/compiler/crystal/compiler.cr 134:7
0x1be6785: ??? at /var/cache/omnibus/src/crystal/src/compiler/crystal/compiler.cr 138:16
0x599d96: ??? at /var/cache/omnibus/src/crystal/src/compiler/crystal/command.cr 259:7
0x5685cd: main at /var/cache/omnibus/src/crystal/src/compiler/crystal/command.cr 49:5
0x7fa3bae9e291: __libc_start_main at ??
0x53b831: ??? at ??

Error: you've found a bug in the Crystal compiler. Please open an issue, including source code that will allow us to reproduce the bug: https://github.com/crystal-lang/crystal/issues
make: *** [Makefile:113: .build/crystal] Error 1
==> ERROR: A failure occurred in build().
    Aborting...
==> ERROR: Build failed, check /mnt/storage/archbuild/extra-x86_64/anatol/build

• @jhass

[RX14]

I can reproduce this also. Had a similar issue (compiles run until they OOM) trying to run crystal specs in release mode too.

In my opinion, Travis should have release mode builds in its build matrix.

[asterite]

@anatol Try compiling with --no-debug and see if the error goes away

[sifmelcara]

For your information, we also faced the same issue when packaging crystal on NixOS, NixOS/nixpkgs#23079 .
However I cannot reproduce the issue myself on my machines.

[zatherz]

16 GB of RAM and I can't build the compiler from master on Arch.

Parse:                             00:00:00.0001615 (   0.19MB)
Semantic (top level):              00:00:00.7481483 (  82.80MB)
Semantic (new):                    00:00:00.0044740 (  82.80MB)
Semantic (type declarations):      00:00:00.0677098 (  90.80MB)
Semantic (abstract def check):     00:00:00.0016340 (  90.80MB)
Semantic (ivars initializers):     00:00:00.1532023 (  98.80MB)
Semantic (cvars initializers):     00:00:00.0079806 (  98.80MB)
Semantic (main):                   00:00:33.3096178 ( 874.87MB)
Semantic (cleanup):                00:00:00.0012090 ( 874.87MB)
Semantic (recursive struct check): 00:00:00.0023756 ( 874.87MB)
Codegen (crystal):                 00:00:06.5269876 (1090.87MB)
fork: Cannot allocate memory
0x540b4e: ??? at ??
0x10fda55: ??? at ??
0x14316e4: ??? at ??
0x1cb6645: ??? at ??
0x5a23f9: ??? at ??
0x56e86d: main at ??
0x7ff988e1b43a: __libc_start_main at ??
0x53b871: ??? at ??

[RX14]

@zatherz What's your compiler's LLVM version: bin/crystal -v?

[zatherz]

bin/crystal is using LLVM 3.5.

Also, I freed up some memory. It's right now stuck on the bc+obj stage, and it's already using 5GB of RAM.

[RX14]

@zatherz LLVM 3.5 can't compile the latest compiler in release mode. It's been like this for ages. You need to compile the compiler with newer LLVM to be able to compile the compiler.

[zatherz]

Why is bin/crystal still built with LLVM 3.5?

[mverzilli]

See https://github.com/crystal-lang/omnibus-crystal/pull/17

[RX14]

@zatherz bin/crystal isn't a binary, its a shell script which uses your local make crystal built compiler or your system installed crystal (and sets CRYSTAL_PATH such that the compiler uses the checked-out stdlib). Your system crystal version probably uses llvm 3.5 because its the omnibus version of crystal, which should be used as a last resort instead of your system-packged crystal.

[zatherz]

Using /usr/bin/llvm-config [version=4.0.0]
./bin/crystal build --release --stats  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
Parse:                             00:00:00.0001500 (   0.19MB)
Semantic (top level):              00:00:00.9420306 (  82.80MB)
Semantic (new):                    00:00:00.0063139 (  82.80MB)
Semantic (type declarations):      00:00:00.0643344 (  90.80MB)
Semantic (abstract def check):     00:00:00.0014830 (  90.80MB)
Semantic (ivars initializers):     00:00:00.1514043 (  98.80MB)
Semantic (cvars initializers):     00:00:00.0057615 (  98.80MB)
Semantic (main):                   00:00:14.1093079 ( 842.80MB)
Semantic (cleanup):                00:00:00.0011992 ( 842.80MB)
Semantic (recursive struct check): 00:00:00.0018411 ( 842.80MB)
Codegen (crystal):                 00:00:08.8011780 (1018.80MB)
Codegen (bc+obj):                  01:19:13.9003315 (1018.80MB)
Codegen (linking):                fork: Cannot allocate memory
0x540b4e: ??? at ??
0x10fda55: ??? at ??
0x14fffc7: ??? at ??
0x14fef65: ??? at ??
0x1437a08: ??? at ??
0x1cb6645: ??? at ??
0x5a23f9: ??? at ??
0x56e86d: main at ??
0x7f9bd9ae343a: __libc_start_main at ??
0x53b871: ??? at ??

[RX14]

@zatherz run bin/crystal -v and you'll see it's still using llvm 3.5 (unless you changed something).

[zatherz]

The AUR package downloads the latest release to get the compiler, that's why it was using LLVM 3.5.

Using /usr/bin/llvm-config [version=4.0.0]
g++ -c -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc `/usr/bin/llvm-config --cxxflags`
cc -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fPIC  -D_FORTIFY_SOURCE=2  -c -o src/ext/sigfault.o src/ext/sigfault.c
ar -rcs src/ext/libcrystal.a src/ext/sigfault.o
./bin/crystal build --release --stats  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
Parse:                             00:00:00.0170711 (   0.19MB)
Semantic (top level):              00:00:00.8167270 (  75.10MB)
Semantic (new):                    00:00:00.0037930 (  83.10MB)
Semantic (type declarations):      00:00:00.0886403 (  83.10MB)
Semantic (abstract def check):     00:00:00.0029551 (  83.10MB)
Semantic (ivars initializers):     00:00:00.1836547 (  99.10MB)
Semantic (cvars initializers):     00:00:00.0071788 ( 107.10MB)
Semantic (main):                   00:00:26.1976139 ( 851.10MB)
Semantic (cleanup):                00:00:00.0016736 ( 851.10MB)
Semantic (recursive struct check): 00:00:00.0032535 ( 851.10MB)
Codegen (crystal):                 00:00:10.8515820 (1003.10MB)
Codegen (bc+obj):                  00:15:56.8540113 (1003.10MB)
Codegen (linking):                 00:00:00.9769090 (1003.10MB)

Macro runs:
 - /home/zatherz/.cache/pacaur/crystal-git/src/crystal/src/ecr/process.cr: 00:00:12.0600410

Codegen (bc+obj):
 - no previous .o files were reused

However,

Using /usr/bin/llvm-config [version=4.0.0]
./bin/crystal build  -o .build/all_spec spec/all_spec.cr
Using compiled compiler at `.build/crystal'
fork: Cannot allocate memory
0x409e80: ??? at ??
0x90ac26: fork_internal at /home/zatherz/.cache/pacaur/crystal-git/src/crystal/src/process.cr 127:7
0xa64356: codegen at /home/zatherz/.cache/pacaur/crystal-git/src/crystal/src/process.cr 102:5
0xd52e16: compile at /home/zatherz/.cache/pacaur/crystal-git/src/crystal/src/compiler/crystal/compiler.cr 138:16
0x4560ec: ??? at /home/zatherz/.cache/pacaur/crystal-git/src/crystal/src/compiler/crystal/command.cr 0:7
0x42f689: main at /home/zatherz/.cache/pacaur/crystal-git/src/crystal/src/compiler/crystal/command.cr 0:5
0x7ff70798b43a: __libc_start_main at ??
0x4098ca: _start at ??
0x0: ??? at ??

I'll try freeing a bit more space first.

[zatherz]

Specs finished, everything's okay. For people who come across this issue in the future:

• If you're on Arch and using the AUR package, first install crystal from the official repository, edit the PKGBUILD to remove the lines starting with source_i686, source_x86_64, sha256sums_i686 and sha256sums_x86_64, and only then build it
• If you can't get a build of Crystal against LLVM 4.0, build with --no-debug first and then rebuild from git with that build