Crystal: Why are executables so big?
I just noticed that the executables created by Crystal are rather big – around 500 kb even for a hello world programm. If I code that in C, the executable is 8 kb and with nasm it is only 1 kb. This is of course impossible to compete with and also not what Crystal focuses on, but I do wonder where all these additional kb come from, as they have no functional benefit here.
And would it be possible to optimize that and reduce the file size of the executable (I know there are also modern languages that produce much smaller executables – Nim's hello world is 180 kb)?
Thyra
All 21 comments
I'm pretty sure that Crystal statically links its standard library.
SplittyDev
on 21 Dec 2016
Quick question: what do you mean with "If I code that in C". What is "that"? Is it a program with GC, non-blocking IO, a runtime with an event loop and scheduler, built-in regex, etc.? Crystal is more a high-level language rather than a low-level language. I also think (can't remember now) that libevent, libgc and libpcre are statically linked.
We compile to an executable mostly because it's faster than a VM (in most cases), fast boot time and easier distribution, but small executables was never a goal (or at least it isn't one of my goals, others at the team might differ) There was also some discussion on this on Google Groups.
asterite
on 21 Dec 2016
Crystal's standard library is distributed as source so the parts of the standard library which are used have to be compiled in every executable, not shared. Consider that even puts "hello World" uses evented IO, that means you need libevent bindings, the fiber scheduler, and all the code related to that in the executable too. The signal handling code is going to add some code too.
RX14
on 21 Dec 2016
@asterite Pretty sure libevent and libpcre and the like are dynamically linked. I think the only static library in your typical executable is libcrystal from the src/ext sources.
RX14
on 21 Dec 2016
@asterite The Google Groups discussion is very interesting! I never intended to use Crystal for embedded systems (even though I wish I could), but are there any plans to make some of these features optional (I don't need GC or regex for my hello world programm)?
Thyra
on 21 Dec 2016
I think Regex could be excluded, but it would be a bit weird (the standard library does use Regex in some places, so you won't be able to use that functionality). Removing the GC... I don't think that's going to happen. For a non-GC language I can recommend Rust.
What are you trying to build?
asterite
on 21 Dec 2016
I don't have anything specific in mind, I was just surprised that this small program produced such a big executable and now I am wondering whether it would make sense to try and make Crystal more versatile in this respect to make it suitable for more scenarios...
Anyways, I guess there are other things more important, thanks for clarifying :-)!
Thyra
on 21 Dec 2016
@asterite PCRE could be avoided if not needed, if lib definitions had an initializer (see the Android pull request).
ysbaddaden
on 21 Dec 2016
@ysbaddaden Yes, something like that would be nice. The main issue with PCRE is that we are doing this at the start of the program:
LibPCRE.pcre_malloc = ->GC.malloc(LibC::SizeT)
LibPCRE.pcre_free = ->GC.free(Void*)
So PCRE is considered to be used.
In fact, the compiler already excludes libraries that are not used. For example, if we add a Regex.init method that we invoke in Regex#initialize, and then call these lib methods only once (behind a bool), and then compile an empty program, libpcre is not linked anymore.
But maybe a macro initialized section could be added to lib somehow, I'll try to implement it later.
asterite
on 21 Dec 2016
$ cat linking.cr
puts "Hello, world"
$ crystal build --release linking.cr
$ otool -L linking
linking:
/usr/lib/libpcre.0.dylib (compatibility version 1.0.0, current version 1.1.0)
/usr/local/opt/bdw-gc/lib/libgc.1.dylib (compatibility version 2.0.0, current version 2.3.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)
/usr/local/opt/libevent/lib/libevent-2.0.5.dylib (compatibility version 7.0.0, current version 7.9.0)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
They are, in fact, dynamically linked.
matiasgarciaisaia
on 22 Dec 2016
@Thyra Also, at least on mac, I just tried a simple puts "Hello world!" program, and doing crystal build program.cr it results in a ~400KB file. However, if I turn on optimizations, that is crystal build program.cr --release I get a ~100KB program. I also tried it with nim, both with --opt:speed and --opt:size and I'm getting ~190KB and ~145KB programs... so I think our executables aren't that big in the end.
Did you check the executable size for the normal build or the optimized build?
asterite
on 22 Dec 2016
@asterite That's strange! I turned on optimizations as well and still got ~450KB:
$ cat hello_world.cr
puts "Hello, World!"
$ crystal build --release hello_world.cr
$ stat --printf="%s\n" hello_world
453002
$ crystal -v
Crystal 0.20.1 [18e7617] (2016-12-05)
$ uname -r
3.13.0-106-generic
$ cat /etc/issue
Ubuntu 14.04.5 LTS \n \l
I can confirm the generated executable in Ubuntu is around 400kb, while it's only 100kb in OSX.
It still isn't one of the Crystal goals, but... ¯\_(ツ)_/¯
matiasgarciaisaia
on 22 Dec 2016
What do you get if you run ldd on the executable in Ubuntu?
refi64
on 23 Dec 2016
Ubuntu 16.04.01 LTS Linux 4.4.0-31-generic x86_64:
$ ldd linking
linux-vdso.so.1 => (0x00007fff6057f000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f66ba0a6000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f66b9e9e000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f66b9c99000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f66b9a83000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f66b96ba000)
/lib64/ld-linux-x86-64.so.2 (0x000055be00b41000)
$ objdump -d linking
# see https://transfer.sh/Ayrdy/linking.objdump for the output (3MB)
macOS Sierra 10.12.2:
$ otool -L linking
linking:
/usr/lib/libpcre.0.dylib (compatibility version 1.0.0, current version 1.1.0)
/usr/local/opt/bdw-gc/lib/libgc.1.dylib (compatibility version 2.0.0, current version 2.3.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)
/usr/local/opt/libevent/lib/libevent-2.0.5.dylib (compatibility version 7.0.0, current version 7.9.0)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
$ objdump -d linking > linking.objdump
# see https://transfer.sh/T7Tc8/linking.objdump for the output (1.5MB)
Yeah, so it seems everything is statically-linked on Linux and dynamically-linked on OSX. Maybe this should be documented somewhere?
refi64
on 23 Dec 2016
However, on archlinux (compiling puts "Hello World!"):
$ ldd foo
linux-vdso.so.1 (0x00007ffd43dce000)
libpcre.so.1 => /usr/lib/libpcre.so.1 (0x00007f63fe52f000)
libgc.so.1 => /usr/lib/libgc.so.1 (0x00007f63fe2c5000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f63fe0a8000)
libevent-2.0.so.5 => /usr/lib/libevent-2.0.so.5 (0x00007f63fde5e000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f63fdc56000)
libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f63fda52000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f63fd83b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f63fd49d000)
/lib64/ld-linux-x86-64.so.2 (0x00007f63fe7a2000)
libatomic_ops.so.1 => /usr/lib/libatomic_ops.so.1 (0x00007f63fd29a000)
Clearly dynamically linked. foo is only 124KiB.
On linux it depends on your distribution. There's nothing to document because crystal doesn't decide this. Crystal says "I require these libraries", and the linker decides whether they are dynamically or statically linked based on which are available on the system at the time.
RX14
on 23 Dec 2016
@RX14 On Manjaro Linux when I try to compile with --link-flags="-static" the result is a big binary.
@matiasgarciaisaia The ldd output for this binary is not a dynamic executable
$ cat hello.cr
puts "hello!"
$ crystal build hello.cr --link-flags="-static"
$ du -h hello*
1.5M hello
4.0K hello.cr
$ ldd hello
not a dynamic executable
$ ./hello
hello!
faustinoaq
on 24 Dec 2016
This also doesn't increase linearly, right?
We are really just talking about the base size of a binary, not an issue with all binaries being too big.
andyfleming
on 29 Dec 2016
I would worry more about startup time than executable size.
Which are excellent BTW, at least here on OS X.
$ cat go.cr
puts "from cr"
$ crystal build --release go.cr
$ time ./go
from cr
real 0m0.006s
rdp
on 6 Jan 2017
Relevant gist:
https://gist.github.com/teaearlgraycold/c7b181f7bc543ee9c37cfd45df5f8856#gistcomment-2154719
faustinoaq
on 12 Dec 2017
Related issues
Sija
·
3Comments
TechMagister
·
3Comments
costajob
·
3Comments
lgphp
·
3Comments
pbrusco
·
3Comments
Most helpful comment
Quick question: what do you mean with "If I code that in C". What is "that"? Is it a program with GC, non-blocking IO, a runtime with an event loop and scheduler, built-in regex, etc.? Crystal is more a high-level language rather than a low-level language. I also think (can't remember now) that libevent, libgc and libpcre are statically linked.
We compile to an executable mostly because it's faster than a VM (in most cases), fast boot time and easier distribution, but small executables was never a goal (or at least it isn't one of my goals, others at the team might differ) There was also some discussion on this on Google Groups.