Ring: Failure building release targeting x86_64-unknown-linux-musl

[bruceadams]

I've only been able to produce the failure when building a binary. Libraries build just fine. _Using_ a library in a binary is when the problem appears. The key message from the failure appears to be this:

  = note: /Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libring-863fd845bbadbb87.rlib(bn.o): In function `memcpy':
/usr/include/x86_64-linux-gnu/bits/string3.h:53: undefined reference to `__memcpy_chk'

I tried four combinations; one fails. All of my attempts were using Ubuntu 16.04 in a Docker container.

C compiler | debug | release
-----------|---------|--------
gcc | success | _failure_
clang | success | success

I'm more than happy to try other combinations. Ideally, we would find a way to see this issue in a build and/or test run within ring. I do not (yet?) understand ring's build and test combinations well enough.

Demonstration of the problem

Building the tlsclient example in rustls using two, published, Docker images for the two C build chains: gcc and clang. I use a function, rmusl, in bash for the complex docker command for running builds inside a Docker container with the current Rust project context.

function rmusl() {
    tag=$1
    shift
    docker run --interactive \
               --rm \
               --tty \
               --user "$(id -u):$(id -g)" \
               --volume $HOME/.cargo/git:/home/rust/.cargo/git \
               --volume $HOME/.cargo/registry:/home/rust/.cargo/registry \
               --volume $PWD:$PWD \
               --workdir $PWD \
               bruceadams/rustup-musl:$tag \
               "$@"
}

gcc debug build

cargo clean
rmusl gcc cargo build --target x86_64-unknown-linux-musl --example tlsclient
   Compiling strsim v0.5.2
   Compiling regex-syntax v0.3.9
   Compiling base64 v0.2.1
   Compiling log v0.3.6
   Compiling libc v0.2.19
   Compiling winapi-build v0.1.1
   Compiling ring v0.6.1
   Compiling rustc-serialize v0.3.22
   Compiling utf8-ranges v0.1.3
   Compiling slab v0.1.3
   Compiling untrusted v0.3.2
   Compiling lazy_static v0.2.2
   Compiling kernel32-sys v0.2.2
   Compiling bytes v0.3.0
   Compiling bitflags v0.4.0
   Compiling nix v0.5.1
   Compiling cfg-if v0.1.0
   Compiling net2 v0.2.26
   Compiling ws2_32-sys v0.2.1
   Compiling memchr v0.1.11
   Compiling aho-corasick v0.5.3
   Compiling winapi v0.2.8
   Compiling thread-id v2.0.0
   Compiling thread_local v0.2.7
   Compiling regex v0.1.80
   Compiling time v0.1.35
   Compiling miow v0.1.5
   Compiling mio v0.5.1
   Compiling webpki v0.8.0
   Compiling webpki-roots v0.6.0
   Compiling rustls v0.5.3 (file:///Users/ba/rustls)
   Compiling env_logger v0.3.5
   Compiling docopt v0.6.86
    Finished debug [unoptimized + debuginfo] target(s) in 90.55 secs

gcc release build

cargo clean
rmusl gcc cargo build --target x86_64-unknown-linux-musl --example tlsclient --release
   Compiling utf8-ranges v0.1.3
   Compiling bitflags v0.4.0
   Compiling ring v0.6.1
   Compiling rustc-serialize v0.3.22
   Compiling libc v0.2.19
   Compiling memchr v0.1.11
   Compiling nix v0.5.1
   Compiling cfg-if v0.1.0
   Compiling net2 v0.2.26
   Compiling regex-syntax v0.3.9
   Compiling log v0.3.6
   Compiling bytes v0.3.0
   Compiling aho-corasick v0.5.3
   Compiling untrusted v0.3.2
   Compiling winapi v0.2.8
   Compiling slab v0.1.3
   Compiling strsim v0.5.2
   Compiling base64 v0.2.1
   Compiling lazy_static v0.2.2
   Compiling winapi-build v0.1.1
   Compiling ws2_32-sys v0.2.1
   Compiling kernel32-sys v0.2.2
   Compiling time v0.1.35
   Compiling thread-id v2.0.0
   Compiling thread_local v0.2.7
   Compiling miow v0.1.5
   Compiling webpki v0.8.0
   Compiling mio v0.5.1
   Compiling rustls v0.5.3 (file:///Users/ba/rustls)
   Compiling webpki-roots v0.6.0
   Compiling regex v0.1.80
   Compiling docopt v0.6.86
   Compiling env_logger v0.3.5
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-nostdlib" "-static" "-Wl,--eh-frame-hdr" "-Wl,-(" "-m64" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crt1.o" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crti.o" "-L" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/examples/tlsclient.0.o" "-o" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/examples/tlsclient" "-Wl,--gc-sections" "-Wl,-O1" "-nodefaultlibs" "-L" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps" "-L" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/build/ring-b04a0c9d4dc7829a/out/lib" "-L" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/librustls.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libbase64-abdbb66a514e8505.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libenv_logger-c716af707f2027e1.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libwebpki_roots-bb72cc6baf4af80d.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libwebpki-09bb2e338e7b2cf6.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libmio-4cd51b3d244c94b3.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/liblog-bf16bb9a4912b11d.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libmiow-4eaf42dcaf735bda.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libwinapi-0889532d327ff4e2.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libtime-750bfdd52feafcb7.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libslab-4f8b9e9b6e35857c.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libdocopt-664aa2a6d9ccf7ca.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/librustc_serialize-6b938435173797f7.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libnet2-755d3e4f87237d0e.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libregex-36c8e259ac5ba542.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libutf8_ranges-5c6a6dacba3be7ce.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libthread_local-a3c0092e9fb6507d.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libthread_id-bcd46c79a620a618.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libring-863fd845bbadbb87.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/liblazy_static-7f1b96a3a3eb529d.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libbytes-f9fbfc75eb2416e0.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libaho_corasick-d1dfd931d7cac82f.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libmemchr-c555f740a543880f.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libnix-e3c0cb7ca8e41f17.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/liblibc-e1db4c5f3a4f3c2f.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libregex_syntax-6602c4e3d91326a4.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libbitflags-75746cc7f0e9d928.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libstrsim-5346333e54880be8.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libcfg_if-72c1f992b13d5087.rlib" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-f5a209a9.rlib" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-f5a209a9.rlib" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-f5a209a9.rlib" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/librand-f5a209a9.rlib" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcollections-f5a209a9.rlib" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-f5a209a9.rlib" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc_jemalloc-f5a209a9.rlib" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-f5a209a9.rlib" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_unicode-f5a209a9.rlib" "/Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libuntrusted-6d5be7309ea48309.rlib" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-f5a209a9.rlib" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-f5a209a9.rlib" "-l" "stdc++" "/home/rust/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crtn.o" "-Wl,-)"
  = note: /Users/ba/rustls/target/x86_64-unknown-linux-musl/release/deps/libring-863fd845bbadbb87.rlib(bn.o): In function `memcpy':
/usr/include/x86_64-linux-gnu/bits/string3.h:53: undefined reference to `__memcpy_chk'
/usr/include/x86_64-linux-gnu/bits/string3.h:53: undefined reference to `__memcpy_chk'
collect2: error: ld returned 1 exit status


error: aborting due to previous error

error: Could not compile `rustls`.

To learn more, run the command again with --verbose.

clang release build

cargo clean
rmusl clang cargo build --target x86_64-unknown-linux-musl --example tlsclient --release
   Compiling bitflags v0.4.0
   Compiling ring v0.6.1
   Compiling libc v0.2.19
   Compiling winapi-build v0.1.1
   Compiling lazy_static v0.2.2
   Compiling rustc-serialize v0.3.22
   Compiling log v0.3.6
   Compiling untrusted v0.3.2
   Compiling utf8-ranges v0.1.3
   Compiling regex-syntax v0.3.9
   Compiling winapi v0.2.8
   Compiling strsim v0.5.2
   Compiling kernel32-sys v0.2.2
   Compiling bytes v0.3.0
   Compiling cfg-if v0.1.0
   Compiling base64 v0.2.1
   Compiling memchr v0.1.11
   Compiling aho-corasick v0.5.3
   Compiling slab v0.1.3
   Compiling net2 v0.2.26
   Compiling nix v0.5.1
   Compiling ws2_32-sys v0.2.1
   Compiling time v0.1.35
   Compiling thread-id v2.0.0
   Compiling miow v0.1.5
   Compiling thread_local v0.2.7
   Compiling regex v0.1.80
   Compiling mio v0.5.1
   Compiling webpki v0.8.0
   Compiling rustls v0.5.3 (file:///Users/ba/rustls)
   Compiling webpki-roots v0.6.0
   Compiling env_logger v0.3.5
   Compiling docopt v0.6.86
    Finished release [optimized] target(s) in 120.13 secs

[briansmith]

undefined reference to `__memcpy_chk'

Is it possible that your GCC is configured to use _FORTIFY_SOURCE automatically, for some reason only for non-debug builds? And/or, is it possible that your INCLUDE path doesn't include the directory containing the _FORTIFY_SOURCE header-only library?

The reference to __memcpy_chk seems to be a result of the use of _FORTIFY_SOURCE. It seems musl C decided they don't want to implement the GCC _FORTIFY_SOURCE ABI. Instead, they expect the implementation of _FORTIFY_SOURCE to be done in a header-only library.

See:

• http://wiki.musl-libc.org/wiki/Future_Ideas#Fortify
• http://www.openwall.com/lists/musl/2015/02/04/3
• http://www.openwall.com/lists/musl/2015/06/17/1

[bruceadams]

Thank you so much for the pointer to _FORTIFY_SOURCE. I doubt I would've ever figured that out on my own.

It looks like Ubuntu's defaults for GCC include setting _FORTIFY_SOURCE, see: https://wiki.ubuntu.com/ToolChain/CompilerFlags#fortify-source Also, _FORTIFY_SOURCE is "[o]nly activated when compiled with -O1 or higher."

It looks like Red Hat does not set _FORTIFY_SOURCE by default. Building a release targeting x86_64-unknown-linux-musl with vanilla GCC on CentOS works fine.

I see this issue running builds on Travis-CI, which is Ubuntu based. An easy workaround is to specify:

language: cpp
compiler: clang

For example: https://github.com/bruceadams/wdscli/blob/master/.travis.yml

I can wish that Ring could avoid this build issue on Ubuntu, but I don't know of a safe and effective way to make this happen.

[briansmith]

First, I recommend poking the musl project about the status of the stuff I mentioned in https://github.com/briansmith/ring/issues/409#issuecomment-273652676 and see if they have a suggestion for how to make it work.

And/or, when the ABI is musl and the target OS is linux, we could add -U_FORTIFY_SOURCE to CPPFLAGS in ring's build system. If that works for you, I'd accept a PR that does that as long as the PR includes along the "XXX TODO comment" that explains why this hack is implemented.

If contributing a PR is problematic (because IBM), please email me at [email protected].

[briansmith]

@bruceadams Any luck with this?

[bruceadams]

I did not manage to ask a question into the Musl community. I see some more recent chatter about _FORTIFY_SOURCE, http://www.openwall.com/lists/musl/2016/09/20/1, but no real sign of progress.

[briansmith]

@bruceadams. Thanks for the patch. We just rewrote the whole build system so that it doesn't use Makefiles any more. I took the liberty of writing a commit using the same idea from your commit. Please see fb830c4de07bc91247d1aa925d8f3323c0fbb01b. Does that fix the problem?

[briansmith]

@bruceadams ping?

[bruceadams]

Yes! Looks great! Thanks!

[briansmith]

https://github.com/briansmith/ring/issues/713#issuecomment-717654176 describes the current state of musl support. Marking this as a duplicate of #713.