Crystal: Building Crystal 0.24.1 fails on Alpine 3.7

[j8r]

Compiling with make returns a build error when running the crystal build command (3rd line).

 make
Using /usr/bin/llvm-config [version=5.0.0]
CRYSTAL_CONFIG_PATH=`pwd`/src ./bin/crystal build  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
_main.o: In function `*Errno::new<String>:Errno':
main_module:(.text+0x2f4b): undefined reference to `errno'
_main.o: In function `*String::Builder@IO#write_utf8<Slice(UInt8)>:Nil':
main_module:(.text+0x331e): undefined reference to `errno'
_main.o: In function `*Iconv#initialize<String, String, (Symbol | Nil)>:Nil':
main_module:(.text+0x35b5): undefined reference to `errno'
_main.o: In function `*Crystal::System::Random::sys_getrandom<Slice(UInt8)>:Int64':
main_module:(.text+0x63d4): undefined reference to `errno'
_main.o: In function `*IO::FileDescriptor+@IO::FileDescriptor#unbuffered_write<Slice(UInt8)>:Nil':
main_module:(.text+0x7014): undefined reference to `errno'
_main.o:main_module:(.text+0x77ea): more undefined references to `errno' follow
collect2: error: ld returned 1 exit status
Error: execution of command failed with code: 1: `cc "${@}" -o '/root/.cache/crystal/srv-crystal-0.24.1-src-ecr-process.cr/macro_run'  -rdynamic  -lpcre -lgc -lpthread /srv/crystal-0.24.1/src/ext/libcrystal.a -levent -lrt -ldl -L/usr/lib -L/usr/local/lib`
make: *** [Makefile:118: .build/crystal] Error 1

Here is a gist that lists the commands I've executed in my docker container.

[ysbaddaden]

errno is actually __errno_location on musl-libc. The target seems wrong, and it's possibly caused by macro runs.

Anyway, don't bother cross building:

$ docker pull ysbaddaden/crystal-alpine:0.24.1

It's automatically built a few days after a crystal release.

[j8r]

@ysbaddaden Your work like crystal-alpine has helped me, thanks, but I haven't found your build commands. Anyway this issue has to be fixed.

Furthermore, I'm working on https://github.com/crystal-lang/crystal/issues/5467 - the ultimate goal is to have a working statically-linked crystal compiler built on Alpine with musl for x86_64, armhf and aarch64. For this I need to generate a crystal.o and link on exact same envionments, and this error is still occuring when running ./bin/crystal build src/compiler/crystal.cr -o crystal.cr --cross-compile --target x86_64-linux-musl.

[jirutka]

I think that the problem is in compiler/crystal/semantic/flags.cr:11:

def flags
  @flags ||= parse_flags(target_machine.triple.split('-'))
end

It seems that flags are initialized with components of the target triplet (which includes needed flags linux and musl on Alpine) only when no other flags are passed to the compiler. That’s not the case when compiling crystal compiler, because make crystal adds flags without_openssl and without_zlib (see Makefile:120).

[ysbaddaden]

Unrelated. CLI flag definitions are added to the compiler flags (initially populated from the target triple, yes) but they have nothing to do with the error at hand.

Anyway I can't reproduce (ubuntu trusty host, fresh master checkout):

$ bin/crystal --version
Crystal 0.24.1 (2018-02-05)
LLVM: 5.0.1
Default target: x86_64-unknown-linux-gnu

``sh $ bin/crystal build src/compiler/crystal.cr -o crystal --cross-compile --target x86_64-linux-musl -Dwithout_openssl -Dwithout_zlib cc 'crystal.o' -o 'crystal' -rdynamic src/llvm/ext/llvm_ext.o/usr/bin/llvm-config-5.0 --libs --system-libs --ldflags 2> /dev/null` -lstdc++ -lpcre -lm -lgc -lpthread /home/github/crystal/src/ext/libcrystal.a -levent -lrt -L/usr/lib -L/usr/local/lib

```sh
$ ls -al crystal.o
-rw-rw-r-- 1 julien julien 24M mars  24 11:34 crystal.o

[jirutka]

Anyway I can't reproduce (ubuntu trusty host, fresh master checkout):

This issue is about error when compiling crystal on Alpine Linux using upstream’s prebuilt crystal binary, not cross compiling on Ubuntu… Try to reproduce it using e.g. alpine-chroot-install.

I’m sure that the problem is with flags not being propagated. I’ve tried to modify all places with {% if flag?(:musl) %} to hard-code musl variant and it compiles without any problem on Alpine. However, I’m not sure if it’s really caused by compiler/crystal/semantic/flags.cr:11 yet, I’m verifying it now. (EDIT: confirmed)

[jirutka]

Unrelated. CLI flag definitions are added to the compiler flags (initially populated from the target triple, yes) but they have nothing to do with the error at hand.

No, they have everything with the error at hand. ;)

First I built crystal using prebuilt binary from Releases with fix:

--- a/src/compiler/crystal/semantic/flags.cr
+++ b/src/compiler/crystal/semantic/flags.cr
@@ -8,7 +8,7 @@
   #
   # See `Compiler#flags`.
   def flags
-    @flags ||= parse_flags(target_machine.triple.split('-'))
+    @flags = (@flags || Set(String).new) | parse_flags(target_machine.triple.split('-'))
   end

   # Overrides the default flags with the given ones.

and hack hard-coding musl variants instead of {% if flag?(:musl) %}:

--- a/src/errno.cr
+++ b/src/errno.cr
@@ -2,18 +2,7 @@
 require "c/string"

 lib LibC
-  {% if flag?(:linux) %}
-    {% if flag?(:musl) %}
-      fun __errno_location : Int*
-    {% else %}
-      @[ThreadLocal]
-      $errno : Int
-    {% end %}
-  {% elsif flag?(:darwin) || flag?(:freebsd) %}
-    fun __error : Int*
-  {% elsif flag?(:openbsd) %}
-    fun __error = __errno : Int*
-  {% end %}
+  fun __errno_location : Int*
 end

 # Errno wraps and gives access to libc's errno. This is mostly useful when
@@ -219,27 +208,11 @@

   # Returns the value of libc's errno.
   def self.value : LibC::Int
-    {% if flag?(:linux) %}
-      {% if flag?(:musl) %}
-        LibC.__errno_location.value
-      {% else %}
-        LibC.errno
-      {% end %}
-    {% elsif flag?(:darwin) || flag?(:freebsd) || flag?(:openbsd) %}
-      LibC.__error.value
-    {% end %}
+    LibC.__errno_location.value
   end

   # Sets the value of libc's errno.
   def self.value=(value)
-    {% if flag?(:linux) %}
-      {% if flag?(:musl) %}
-        LibC.__errno_location.value = value
-      {% else %}
-        LibC.errno = value
-      {% end %}
-    {% elsif flag?(:darwin) || flag?(:freebsd) || flag?(:openbsd) %}
-      LibC.__error.value = value
-    {% end %}
+    LibC.__errno_location.value = value
   end
 end
--- a/src/iconv.cr
+++ b/src/iconv.cr
@@ -8,12 +8,6 @@
     original_from, original_to = from, to

     @skip_invalid = invalid == :skip
-    {% unless flag?(:freebsd) || flag?(:musl) %}
-    if @skip_invalid
-      from = "#{from}//IGNORE"
-      to = "#{to}//IGNORE"
-    end
-    {% end %}

     @iconv = LibC.iconv_open(to, from)

After that I removed the last hack and rebuilt crystal using binary from the previous build as bootstrap crystal. And it works!

[RX14]

@jirutka It works for every single other target, why would musl be special? Please show me crystal -v on the compiler that got you the failed build. If you're using the prebuilt compiler from releases that's a compiler for x86_64-linux-gnu, not for musl. You need to override the target with --target if you're going to use that compiler on musl, or target_machine.triple will just be x86_64-linux-gnu.

The compiler's target triple is not determined by the currently running system, it's determined statically at compile time. There's no bug here. Just add --target x86_64-linux-musl (no need for --cross-compile).

[jirutka]

I've specified --target, of course. Please look into the code, reason about it, and try to reproduce it on Alpine (or any other musl native) system. I can prepare you a script to reproduce this issue if you really don't believe me...

[j8r]

No @RX14 , see the gist. I've also tried with./bin/crystal build src/compiler/crystal.cr -o crystal.cr --cross-compile --target x86_64-linux-musl.

Please for those that have doubts, try the steps posted in this gist on a Alpine Linux host.

[j8r]

Thanks @jirutka for your work! It would be great if you submit a PR with this changes!
I'm close to be able to continue the work on #5467 :)

[jirutka]

I'll send PR in few hours, I'm not at computer now.

[RX14]

Your diagnosis makes no sense, we always rely on the x86_64, and linux flags, even when setting -D options. For every build. If this was broken every build in the last 3 years would have failed.

And as expected, the code says this should work:

program.flags << "release" if release?
program.flags << "debug" unless debug.none?
program.flags << "static" if static?
program.flags.concat @flags

where program.flags is the set initialized from parse_flags and @flags are the flags from the commandline. There's no overriding going on. Your PR simply makes your compiler only ever work on musl.

[RX14]

The key thing we've been missing from the error: /root/.cache/crystal/crystal-0.24.1-src-ecr-process.cr/macro_run. The error is occuring when compiling a macro {{run()}}. --target arguments are not passed to the child compiler used for compiling macro runs, because it is assumed that the bin/crystal -v default target is correct for the system the compiler is running on. This assumption is not always correct. However, we cannot simply pass in the --target option for macro runs, as this would break cross-compilation to ARM: it would try to compile the macro runs (that run on the x86 host) for ARM which would obviously fail.

Looks like we need the --build, --host and --target options from ./configure, except --host is already --target and --target is already CRYSTAL_CONFIG_TARGET.

[jirutka]

Okay, let's take it from different side - why the heck do you add flags from target triplet _only_ when no extra flags are set? This doesn't make damn sense.

If my empirically verified diagnosis doesn't make sense to you, then troubleshoot and fix the error, I'm really interested. ;) BTW, I haven't fixed it just by accident, I spent half a day looking into the code and finding where and how are flags initialized.

[RX14]

why the heck do you add flags from target triplet only when no extra flags are set? This doesn't make damn sense

we don't, you've misunderstood the code.

[jirutka]

Aha, your previous comment finally makes sense! I'll try something...

[RX14]

def flags
  @flags ||= parse_flags(target_machine.triple.split('-'))
end

means to initialize the flags once to the flags from the target triplet. We then take the resulting set and add new flags to it using the code that I posted above.

[RX14]

Just to make it clearer, the problem i'm talking about is this source here: https://github.com/crystal-lang/crystal/blob/dedc726bcafd48f017865020441cfce6c45b741c/src/compiler/crystal/macros/macros.cr#L106

we create a whole new compiler instance with the default flags, so no flags, not even the triple, get passed though.

[jirutka]

Only if it's called _before_ this:

def flags=(flags : String)
    @flags = parse_flags(flags.split)
end

[RX14]

@jirutka I don't think that method is ever called. Certainly not in the snippet that adds the flags from the commandline that I copied above.

[jirutka]

@RX14 I’m sorry, you were right. I thought that I’ve tried to compile crystal with the mentioned hack and _without_ flags fix first, but apparently I didn’t, because after I’ve removed it and recompiled again now, it still works. So that fix has no effect and only the hack I’ve used in first round has effect.

However, I still thinks that compiler/crystal/semantic/flags.cr:10-17 is at least quite confusing and error prone.

Anyway, important is that I’m able to build Crystal on Alpine (using hack) and you’ve already identified the real cause of the problem (in https://github.com/crystal-lang/crystal/issues/5689#issuecomment-375895582), so it can be fixed.

[jirutka]

@j8r I’ve created Alpine package for Crystal (source) and pushed it to the Alpine’s testing repository (it’s building now). The next step is to build it for other architectures…

I’ve also successfully built completely static linked crystal binary: crystal-0.24.2-x86_64-alpine-linux-musl.tar.gz.

[RX14]

@jirutka glad you've managed to bootstrap using the hack, looking forward to having crystal be a native alpine package!

Cross-compiling for other architectures should be fairly standard now you have a working compiler for musl. The issues here are just because of the funky compiler build we have.

[jirutka]

Cross-compiling for other architectures should be fairly standard now you have a working compiler for musl.

Probably a noob question (I don’t have much experience with cross-compilation), but do I need cross-compiled LLVM libs to cross-compile Crystal? Currently I’m getting quite a weird error and not 100% sure if it’s really caused by missing system deps:

./bin/crystal build --release --progress --threads 12  --no-debug --target aarch64-alpine-linux-musl --cross-compile  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
Error in src/compiler/crystal.cr:1: while requiring "prelude"

# This is the file that is compiled to generate the
^

in src/prelude.cr:15: while requiring "exception"

require "exception"
^

in src/exception.cr:1: while requiring "callstack"

require "callstack"
^

in src/callstack.cr:1: while requiring "c/dlfcn": can't find file 'c/dlfcn' relative to '/home/jirutjak/aports/main/crystal/src/cryst
al-0.24.2/src'

require "c/dlfcn"
^

make: *** [Makefile:118: .build/crystal] Error 1

[RX14]

@jirutka no, sorry I forgot we only support musl on x86_64 and i386 for now. You can see the targets we support here: https://github.com/crystal-lang/crystal/tree/master/src/lib_c

Adding an arm musl target shouldn't be too hard though: #5467.

[jirutka]

@j8r Statically linked crystal for aarch64: https://dev.alpinelinux.org/archive/crystal/crystal-0.24.2-aarch64-alpine-linux-musl.tar.gz. It may be buggy (see #5861)!

[j8r]

@jirutka nice work, love it! I had started to work on this but had hard times - you managed to do it!

@RX14 when the musl targets for armhf and aarch64 will be reviewed to work properly and merged, could we plan to move on Alpine Linux for all crystal compiler builds?
The only anoying issue is like you said on gitter - no ARM CIs, or we need to hack around.
Hacks (with qemu) or not, ARM CIs are likely to be slower that others.

[j8r]

Now solved, similar to https://github.com/crystal-lang/crystal/issues/4480

[jirutka]

@j8r Statically linked crystal 0.25.0:

• crystal-0.25.0-x86_64-alpine-linux-musl.tar.gz
• crystal-0.25.0-aarch64-alpine-linux-musl.tar.gz

[j8r]

@jirutka awesome! There is a chance to have Crystal 0.25.0 for Alpine 3.8 :tada:

[jirutka]

@j8r It's not a chance, it's certain. ;)

[mjago]

@jirutka you rock! Just started looking at this stuff today and came across this thread ~ good work 👍