Deno: target x86_64-unknown-linux-musl

[hayd]

Previously I was able to use --target x86_64-unknown-linux-musl on rusty_v8 and deno (excluding the plugin). https://github.com/denoland/rusty_v8/issues/49

It's unclear to me if it's best to _only_ target x86_64-unknown-linux-musl or to additionally support it. e.g. sccache seems to _only_ provide binaries for that target.

The one concern is that glibc is required for plugins (but I don't know if that means glibc cannot be used on a binary created with musl 🤷‍♂ ).

This is useful for platforms without glibc, examples alpine linux and amazon linux (which is occasionally a pain building, so would be helpful if a compatible binaries was in deno's CI).

xlink: #3243 #1658 #1495 #3356

[hayd]

Some progress using rust-musl-builder, but run into GLIBC issue (maybe?):

https://gist.github.com/hayd/ae5cbe81117863fff98c2f0c877f2b34

/usr/bin/ld: /home/rust/src/deno/target/x86_64-unknown-linux-musl/release/deps/deno-9173e3687c7172b6: hidden symbol `__dso_handle' isn't defined
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status

Note: In this context it seems like rusty_v8 builds:

cargo build --package rusty_v8
   Compiling ... #snipped 
   Compiling rusty_v8 v0.1.0
    Finished dev [unoptimized + debuginfo] target(s) in 33m 20s

Not sure what that means...

But it seems like it's the deno(bin) that fails (rather than rusty_v8):

Building [=====================================================> ] 256/257: deno(bin)
# same failure

[bnoordhuis]

Not sure if it's helpful but __dso_handle is normally provided by libgcc's (not glibc's) crtbegin.o. It sounds like that's not getting linked in on your system?

On my system:

$ dpkg -S `cc -print-file-name=crtbegin.o`
libgcc-9-dev:amd64: /usr/lib/gcc/x86_64-linux-gnu/9/crtbegin.o

$ nm -o `cc -print-file-name=crtbegin.o` | grep __dso_handle
/usr/lib/gcc/x86_64-linux-gnu/9/crtbegin.o:0000000000000000 D __dso_handle

[hayd]

🤷‍♂ I think the objective is for musl to statically link, specifically not to link them to that specific system... this might already be possible with some flags?

(The environment is in the Dockerfile.)

cc @chrmoritz ?

[bnoordhuis]

I think the objective is for musl to statically link

crtbegin.o is linked in statically, not dynamically.

[hayd]

It is in the docker image too, perhaps an env issue? (e.g. RUST_FLAGS or CFLAGS? 😬 )

rust@c3b60eef5da1:~/src/deno/cli$ dpkg -S `cc -print-file-name=crtbegin.o`
libgcc-7-dev:amd64: /usr/lib/gcc/x86_64-linux-gnu/7/crtbegin.o
rust@c3b60eef5da1:~/src/deno/cli$ nm -o `cc -print-file-name=crtbegin.o` | grep __dso_handle
/usr/lib/gcc/x86_64-linux-gnu/7/crtbegin.o:0000000000000000 D __dso_handle

rust@c3b60eef5da1:~/src/deno/cli$ env
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
LESSCLOSE=/usr/bin/lesspipe %s %s
GN_ARGS=clang_use_chrome_plugins=false treat_warnings_as_errors=false use_sysroot=false use_glib=false use_gold=true clang_base_path="/tmp/clang"
HOSTNAME=c3b60eef5da1
RUST_BACKTRACE=full
GN_VERSION=latest
PKG_CONFIG_ALL_STATIC=true
DENO_VERSION=0.31.0
OPENSSL_DIR=/usr/local/musl/
LIBZ_SYS_STATIC=1
OPENSSL_LIB_DIR=/usr/local/musl/lib/
NINJA=/bin/ninja
PWD=/home/rust/src/deno/cli
PKG_CONFIG_ALLOW_CROSS=true
HOME=/home/rust
PG_CONFIG_X86_64_UNKNOWN_LINUX_GNU=/usr/bin/pg_config
CLANG_BASE_PATH=/tmp/clang
DEP_OPENSSL_INCLUDE=/usr/local/musl/include/
TERM=xterm
SHLVL=1
DENO_BUILD_MODE=release
PQ_LIB_STATIC_X86_64_UNKNOWN_LINUX_MUSL=1
TARGET=musl
GN=/bin/gn
PATH=/home/rust/.cargo/bin:/usr/local/musl/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
NINJA_VERSION=1.8.2
OPENSSL_INCLUDE_DIR=/usr/local/musl/include/
OPENSSL_STATIC=1
LESSOPEN=| /usr/bin/lesspipe %s
_=/usr/bin/env

[hayd]

@jeromegn Do you have any tips/suggestions here, noting your comment:
https://github.com/emk/rust-musl-builder/issues/65#issuecomment-469049466

Did the musl binaries you created for fly work on other linux systems?

[jeromegn]

I'm not sure I have much to add. I mostly used the linked Dockerfile and it worked. Inspired (read: ripped off) from: https://hub.docker.com/r/alexmasterov/alpine-libv8

That said, there are alpine images with glibc compiled in there, that might help. It's a lot of trial and error to get it to compile. And then the performance can vary a bit between musl and glibc.

[kesor]

If anyone is interested, I have compiled DENO with V8 in Alpine Linux Docker container - without glibc, but completely using the Alpine muslc library.

https://gist.github.com/kesor/68df53a5d76784a235ca6b0e7efed4d9

Enjoy!

[hayd]

@kesor This is great, good work! Do these (or some of these) options make sense to be incorporated as gn args?
That patch looks scary to maintain. :)

[kesor]

@kesor This is great, good work! Do these (or some of these) options make sense to be incorporated as gn args?
That patch looks scary to maintain. :)

I spent hours trying to find GN args to pass via env var or some other way. Couldn’t find how. My knowledge of GN is limited, probably someone with more experience can find the way. Didn’t mean to provide that patch as “the solution” only as a proof of concept.

PS: Alpine has other llvm versions as well, maybe with llvm8 the binary will be smaller?

Also note that the eventual binary still has dependency on libgcc at the least, maybe other dependencies I didn’t quite catch, it does run with the provided docker image resulting from the Dockerfile build.

[hayd]

In llvm10 there is no_inline_line_tables which shrinks the binary.
I think the solution is that BUILD.gn is modified to allow arguments that support what your diff does :)

Are both these required? Do you have any idea of the reasons?

-      ldflags += [ "-Wl,--color-diagnostics" ]
+      ldflags += [  ]

-  libs = []
+  libs = ["execinfo"]

[kesor]

In llvm10 there is no_inline_line_tables which shrinks the binary.
I think the solution is that BUILD.gn is modified to allow arguments that support what your diff does :)

Are both these required? Do you have any idea of the reasons?

-      ldflags += [ "-Wl,--color-diagnostics" ]
+      ldflags += [  ]

-  libs = []
+  libs = ["execinfo"]

Color diagnostics was reported as an invalid option for the linker. Wouldn’t work with it. I also think that there was a different linker used for parts of the build for some reason. Maybe that can be the root cause and resolving it can solve the arguments issue.

libexecinfo was also required because of failing linking. I don’t really know the reason why.

The most annoying thing was the custom location of the header files which was not automatically detected, and I couldn’t find a way to provide an include path via GN args. I would also assume that this path will change for different llvm/alpine versions.

[hayd]

I would also assume that this path will change for different llvm/alpine versions.

Yes, I think we'd want to pass this (as a GN_ARGS)...?

@ry do you have any pointers in adding these/or suggest whether this a good/bad way forward?

https://gist.github.com/kesor/68df53a5d76784a235ca6b0e7efed4d9#file-rusty-changed-build-diff

[nathanblair]

Also note that the eventual binary still has dependency on libgcc at the least

Is this referring to the libgcc alpine package? I wanted to avoid this dependency by compiling with musl and the linux musl headers in order for deno to be a fully static binary. This can be achieved with node but I'm not sure how far down the rust toolchain requires glibc.

[kesor]

Also note that the eventual binary still has dependency on libgcc at the least

Is this referring to the libgcc alpine package? I wanted to avoid this dependency by compiling with musl and the linux musl headers in order for deno to be a fully static binary. This can be achieved with node but I'm not sure how far down the rust toolchain requires glibc.

@nathanblair Note that libgcc is not the same as glibc, whereas musl is an alternative glibc, there is no "special" alternative for libgcc on Alpine.

[nizox]

Are both these required? Do you have any idea of the reasons?

-  libs = []
+  libs = ["execinfo"]

I had a similar problem with execinfo and was able to workaround it with:
https://github.com/nizox/v8/commit/f66447ba9d7e8c901f40552c5409a3b9457e8be3

Also, you should be able to replace the libgcc dependency with libunwind: https://github.com/nizox/chromium_build/commit/39a009c6d3898c64c0d3abc65feb9f82f304dec8

[lygstate]

Can deno static link to c library to resolve issue like golang does?