Crystal: Use musl to build the Crystal binaries

[refi64]

Musl is a lightweigh, statically-linked, C99-compatible libc implementation. It's easy to install on Linux systems (in most repositories) and is easy to use instead of glibc (replace calls to gcc with musl-gcc). Using this to build the binaries would mean that they are not dependent on a particular version of the glibc shared object librar. An issue related to this was recently posted on the mailing list.

[jhass]

Does it work with clang too?

[ysbaddaden]

Alpine Linux is based on musl libc and it does provide a package for llvm/clang: http://pkgs.alpinelinux.org/package/main/x86_64/llvm

I'm configuring a container with alpine to try it out.

[ysbaddaden]

Using the pre-built Crystal 0.7.1 I get some missing symbols:

alpine:~/crystal-0.7.1-1# ldd embedded/bin/crystal 
        /lib64/ld-linux-x86-64.so.2 (0x7f5500006000)
        libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f5500006000)
        librt.so.1 => /lib64/ld-linux-x86-64.so.2 (0x7f5500006000)
        libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f5500006000)
        libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f5500006000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x7f54ffd13000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7f54ffb01000)
        libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f5500006000)
        ld-linux-x86-64.so.2 => /lib/ld-linux-x86-64.so.2 (0x7f54ff875000)
Error relocating embedded/bin/crystal: swapcontext: symbol not found
Error relocating embedded/bin/crystal: makecontext: symbol not found
Error relocating embedded/bin/crystal: mallinfo: symbol not found
Error relocating embedded/bin/crystal: getcontext: symbol not found
Error relocating embedded/bin/crystal: __register_atfork: symbol not found
Error relocating embedded/bin/crystal: __pthread_register_cancel: symbol not found
Error relocating embedded/bin/crystal: sysctl: symbol not found
Error relocating embedded/bin/crystal: __pthread_unregister_cancel: symbol not found
Error relocating embedded/bin/crystal: errno: symbol not found
Error relocating embedded/bin/crystal: __libc_stack_end: symbol not found

Maybe I'm just missing some packages (very likely).

[refi64]

I know that you can use musl with Clang, but you have to manually pass the arguments to not use glibc.

Besides, I'm suggesting that this be used for the prebuilt binaries.

[ysbaddaden]

I almost got Crystal to compile itself against musl libc (using a statically built crystal binary on alpine linux), and I'm down to a single missing symbol when linking: _errno_, which is defined differently.

int *__errno_location(void);
#define errno (*__errno_location());

I'm not sure how to translate this.

[jhass]

fun errno_location = __errno_location() : LibC::IntT*, errno = LibC.errno_location().value no?

[ysbaddaden]

Yes. Replacing calls to LibC.errno for LibC.errno_location().value (I tried to add some ifdef musl but I still got some undefined references to errno), fixing some missing dependencies, and I got version of Crystal built against musl libc!

alpine:~/crystal# ldd crystal 
        /lib/ld-musl-x86_64.so.1 (0x7fae5bb0c000)
        libz.so.1 => /lib/libz.so.1 (0x7fae5a10c000)
        libffi.so.6 => /usr/lib/libffi.so.6 (0x7fae59f07000)
        libedit.so.0 => /usr/lib/libedit.so.0 (0x7fae59cdb000)
        libncurses.so.5 => /usr/lib/libncurses.so.5 (0x7fae59a97000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x7fae597a4000)
        libevent-2.0.so.5 => /usr/lib/libevent-2.0.so.5 (0x7fae59565000)
        libpcl.so.1 => /usr/local/lib/libpcl.so.1 (0x7fae59361000)
        libpcre.so.1 => /usr/lib/libpcre.so.1 (0x7fae59108000)
        libgc.so.1 => /usr/lib/libgc.so.1 (0x7fae58ead000)
        libunwind.so.8 => /usr/local/lib/libunwind.so.8 (0x7fae58c93000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7fae58a81000)
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fae5bb0c000)

The problem now, is a segfault with gc when I try to compile something with it:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff68ffdac in GC_find_limit_with_bound () from /usr/lib/libgc.so.1
(gdb) bt
#0  0x00007ffff68ffdac in GC_find_limit_with_bound () from /usr/lib/libgc.so.1
#1  0x00007ffff68ffe30 in GC_init_linux_data_start () from /usr/lib/libgc.so.1
#2  0x00007ffff68fefa7 in GC_init () from /usr/lib/libgc.so.1
#3  0x0000555555641610 in *GC::init:Void ()
#4  0x000055555562fd25 in main ()

[bararchy]

@ysbaddaden , did you managed to make it work ?

[ysbaddaden]

No, it sadly reaches further than I'm capable.

I guess we should compare the generated LLVM bytecodes to understand if the error comes from the generated crystal code, or maybe this has something to do with the way GC is initialized (it's patched for musl libc).

I also tried against Alpine 3.2 which is in beta and ships LLVM 3.6. I should retry with the stable Alpine 3.1 and LLVM 3.5.

[katlogic]

I'm using statically linked crystal against glibc - the resulting (stripped, not recommended as you lose exception backtraces) binary is 21MB. The benefit of musl is questionable, as largest chunk of the static blob is llvm, ncurses and other dependencies, removing glibc/pthread and other gnu runtime will shed off 3-5MB at most.

[ysbaddaden]

I didn't experiment against musl-libc to distribute a smaller binary, but to be able to use crystal on musl based systems like Alpine; ie. make it more portable, without having to build a static binary on a glibc system and eventually have to use the glibc when I could use musl instead.

[katlogic]

@ysbaddaden: Fair point about Alpine. Without proper -ggdb backtrace, one could only speculate it's related to https://github.com/ivmai/bdwgc/blob/master/dyn_load.c#L55 - a general boehmgc issue with musl (dyn_load.c is awfuly platform specific regarding dynamic library linkmaps, and musl is not exactly svr4-compliant in this regard).

If that turns out to be true, either you can recompile libgc with dynamic library support disabled, or use TinyGC which does not implement it at all (it's just simple allocator wrappers) and is generally simpler to troubleshoot on new platform.

EDIT: Forgot to add, GC_find_limit_with_bound segfaults are normal part of operation. It's how libgc probes for memory areas and uses SEGV handler to recover and it makes gdb misleading. You have to continue over these until it crashes for good.

[ysbaddaden]

Wow, thanks a lot for detailing how to conduct further research!

[ozra]

So, for this issue in itself - statically linking with glibc will suffice? Considering the other statics included.
I like the idea of totally static, dependency free binaries - downloading and trying a neat language should just work, be effort free and a nice first impression ;-)

[ysbaddaden]

The issue will be closed when we can use a musl built crystal to compile apps.

Statically linking against glibc and other dependencies was already operational. Building against musl using this static build is also possible, with a small change around errno, but compiling with this compiler generates segfaulting binaries, which I didn't investigate, yet.

[ozra]

Ah, of course, I got the issue wrong. I got a Nim app running with musl - made it insanely small, so it's cool if it works for possible future embedded device hacking, whatnot :-)