Crystal: Partial static compile

[bararchy]

I'm trying to static compile OpenSSL into my project.

Makefile is at: https://github.com/bararchy/sslscanner/blob/master/Makefile

It crashes with:

make[1]: Entering directory '/home/unshadow/Desktop/git-projects/sslscanner'
crystal build bin/scan.cr --release --link-flags "-static -L/home/unshadow/Desktop/git-projects/sslscanner/openssl/ -I/home/unshadow/Desktop/git-projects/sslscanner/openssl/include/ -I/home/unshadow/Desktop/git-projects/sslscanner/openssl/  -DVERSION=\"a52ddef-wip-static\" -lssl -lcrypto -lz -ldl -I/usr/lib/ -lpcre -levent"
/home/unshadow/Desktop/git-projects/sslscanner/openssl//libcrypto.a(dso_dlfcn.o): In function `dlfcn_globallookup':
dso_dlfcn.c:(.text+0x20): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: cannot find -lpcre
/usr/bin/ld: cannot find -levent
/usr/bin/ld: cannot find -lpcre
/usr/bin/ld: cannot find -levent
_main.o: In function `~procProc(Nil)@src/sslscanner.cr:23':
main_module:(.text+0x1bd79): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
collect2: error: ld returned 1 exit status

I'm not really sure where to go from here :\
Any help would be appreciated :)

[refi64]

Looks like your system doesn't have the static libraries for PCRE and libevent, so you'll probably have to build them yourself.

Static linking on Linux is basically a can of decaying worms. You might have better luck trying something like AppImage.

[bararchy]

Can't I use only what I have statically for the static linking and libevent and libpcre as dynamic linking ?

[ysbaddaden]

You must specify a full path to the static libraries to include at link time. Passing them with --link-flags won't work, because the -lcrypto and -lssl are still specified, and the linker will search for shared libraries by default, and only use the specified static libraries if shared libraries can't be found.

[bararchy]

@ysbaddaden If I can't use --link-flags to link them , where and how should I specify full path ?

[bararchy]

So, I compiled all the needed libs staticlly and moved the .a files to the needed path

make sslscanner STATIC_BUILD=TRUE
make[1]: Entering directory '/home/unshadow/Desktop/git-projects/sslscanner'
crystal build bin/scan.cr --release --link-flags "-static -L/home/unshadow/Desktop/git-projects/sslscanner/openssl/ -I/home/unshadow/Desktop/git-projects/sslscanner/openssl/include/ -I/home/unshadow/Desktop/git-projects/sslscanner/openssl/ -I/home/unshadow/Desktop/git-projects/sslscanner/openssl/pcre/  -DVERSION=\"bff04da-wip-static\" -lssl -lcrypto -lz -lpcre -levent -ldl -I/usr/lib/"
/home/unshadow/Desktop/git-projects/sslscanner/openssl//libcrypto.a(dso_dlfcn.o): In function `dlfcn_globallookup':
dso_dlfcn.c:(.text+0x20): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
_main.o: In function `~procProc(Nil)@src/sslscanner.cr:23':
main_module:(.text+0x1bd79): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/home/unshadow/Desktop/git-projects/sslscanner/openssl//libevent.a(evutil.c.o): In function `evutil_getaddrinfo_common_':
evutil.c:(.text+0x2521): warning: Using 'getprotobynumber' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
evutil.c:(.text+0x2350): warning: Using 'getservbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

The good thing is, it seems that it worked :) , when using ldd on the file it says "not a dynamic executable" and the file size is big, 4mb big.

The bad thing is, the program it self now crashes with:

Usage: ./scan [host] [port]
Invalid memory access (signal 11) at address 0x0
[4338694] ???
[4269518] ???
[6298832] ???
[0] ???

[refi64]

See, I told you static linking on Linux was a can of worms!! ;)

What if you try using the nightly version of Crystal with -g?

[bararchy]

@kirbyfan64 lol you did say that XD

What does -g does ? I'll pull Crystal head now.

[bararchy]

@kirbyfan64 Git HEAD installed, not seeing a -g option anywhere

[refi64]

Oops, I'm an idiot. I meant -d. It puts debugging symbols in the program so it'll tell you where it crashed.

[bararchy]

@kirbyfan64 No worries :)

With -d the build crashes without even completing the compile process :

Invalid memory access (signal 11) at address 0x1c
[94918406654550] ???
[94918397372782] __crystal_sigfault_handler +30
[139702843957376] ???
[139702865736583] _ZNK4llvm3DIE13getUnitOrNullEv +23
[139702865843452] _ZN4llvm10DwarfDebug25finishVariableDefinitionsEv +108
[139702865843720] _ZN4llvm10DwarfDebug18finalizeModuleInfoEv +72
[139702865881014] _ZN4llvm10DwarfDebug9endModuleEv +38
[139702865682656] _ZN4llvm10AsmPrinter14doFinalizationERNS_6ModuleE +512
[139702860814617] _ZN4llvm13FPPassManager14doFinalizationERNS_6ModuleE +73
[139702860865237] _ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE +869
[139702874707399] ???
[139702874708052] LLVMTargetMachineEmitToFile +228
[94918403628025] ???
[94918403173871] ???
[94918397505666] ???
[94918397370116] main +11348
[139702821446289] __libc_start_main +241
[94918397271258] _start +42
[0] ???

with stats

Parse:                             00:00:00.0000801 (   0.19MB)
Semantic (top level):              00:00:00.0972156 (  29.51MB)
Semantic (new):                    00:00:00.0009355 (  29.51MB)
Semantic (type declarations):      00:00:00.0105900 (  29.51MB)
Semantic (abstract def check):     00:00:00.0006722 (  29.51MB)
Semantic (cvars initializers):     00:00:00.0058404 (  37.51MB)
Semantic (ivars initializers):     00:00:00.0006442 (  37.51MB)
Semantic (main):                   00:00:00.1040167 (  61.57MB)
Semantic (cleanup):                00:00:00.0003782 (  61.57MB)
Semantic (recursive struct check): 00:00:00.0004097 (  61.57MB)
Codegen (crystal):                 00:00:00.2114705 (  69.70MB)
Codegen (bc+obj):                 Invalid memory access (signal 11) at address 0x1c
[94707899542102] ???
[94707890260334] __crystal_sigfault_handler +30
[139874723971200] ???
[139874745750407] _ZNK4llvm3DIE13getUnitOrNullEv +23
[139874745857276] _ZN4llvm10DwarfDebug25finishVariableDefinitionsEv +108
[139874745857544] _ZN4llvm10DwarfDebug18finalizeModuleInfoEv +72
[139874745894838] _ZN4llvm10DwarfDebug9endModuleEv +38
[139874745696480] _ZN4llvm10AsmPrinter14doFinalizationERNS_6ModuleE +512
[139874740828441] _ZN4llvm13FPPassManager14doFinalizationERNS_6ModuleE +73
[139874740879061] _ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE +869
[139874754721223] ???
[139874754721876] LLVMTargetMachineEmitToFile +228
[94707896515577] ???
[94707896061635] ???
[94707890393218] ???
[94707890257668] main +11348
[139874701460113] __libc_start_main +241
[94707890158810] _start +42
[0] ???

[refi64]

Well, looks like you accidentally found a bug in Crystal... :/

What do you get when you run ls -al $(which ld)?

[bararchy]

@kirbyfan64

-rwxr-xr-x 4 root root 553 Aug  2  2015 /usr/lib/hardening-wrapper/bin/ld

Btw, I found out that the issue when static compiling (not using --debug) is the resolve function (getaddressinfo) , when using IP for the sockets I can make the program work.

[refi64]

@bararchy Ah, ok. I had thought you might be using the gold linker, which can have bugs with static linking, but you aren't...

You must have fallen victim to this glibc. Unfortunately, there is no known fix for the bug, which has been open for over three years.

Only other idea I have is to try to build a copy of musl (a glibc alternative) and use that instead, since it tends to be tamer when static linking...but you'll have to also rebuild libevent, libgc, and all the rest of Crystal's dependencies in order for that to work properly.

I'm dead serious when I say that you really should try AppImage...

[rg-3]

@kirbyfan64 with AppImage, are the shared libraries bundled within the app?

[refi64]

@r-obert Yes.

[rg-3]

@kirbyfan64 wow. that's good for commercial software on linux besides just being generally awesome for distributing OSX-style "apps" on linux. great link, thank you.

[bararchy]

Closing as it seems to be an issue with general static compiling and not Crystal specific.