Libelektra: musl build fails with gcc 6(+?) on debian stretch

[ingwinlu]

When I try to replicate the elektra-gcc-configure-debian-musl locally I get the following errors during build:

jenkins@5391be09a3a0:~/ws/build$ ../scripts/configure-debian-musl -DINSTALL_SYSTEM_FILES=OFF -DCMAKE_INSTALL_PREFIX=$WORKSPACE/local -DKDB_DB_SYSTEM=$WORKSPACE/system -DBUILD_DOCUMENTATION=OFF  ..
<snip>
jenkins@5391be09a3a0:~/ws/build$ make -j 10
<snip>
[ 10%] Linking C shared library ../../../lib/libelektra-core.so
/usr/bin/ld: CMakeFiles/elektra-core.dir/internal.c.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: CMakeFiles/elektra-core.dir/keyhelpers.c.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: CMakeFiles/elektra-core.dir/keyname.c.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: CMakeFiles/elektra-core.dir/keyset.c.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: CMakeFiles/elektra-core.dir/keytest.c.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: CMakeFiles/elektra-core.dir/keyvalue.c.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: CMakeFiles/elektra-core.dir/nolog.c.o: relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: CMakeFiles/elektra-core.dir/opmphm.c.o: relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: CMakeFiles/elektra-core.dir/owner.c.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: CMakeFiles/elektra-core.dir/__/loader/dl.c.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output

Any idea why it works on the (old) build server but not for me?

logs

https://build.libelektra.org/jenkins/blue/organizations/jenkins/libelektra/detail/PR-2063/48/pipeline
https://build.libelektra.org/jenkins/job/elektra-gcc-configure-debian-musl/lastSuccessfulBuild/consoleText

update

musl and gcc 6 is not the problem as it builds fine on alpine or via openwrt toolchain. Seems like debian gcc does something it should not.

[ingwinlu]

Running with VERBOSE=1

[  5%] Building C object src/libs/elektra/CMakeFiles/elektra-core.dir/keytest.c.o
cd /home/jenkins/ws/build/src/libs/elektra && /usr/bin/musl-gcc  -DHAVE_KDBCONFIG_H -Delektra_core_EXPORTS -I/home/jenkins/ws/build/src/include -I/home/jenkins/ws/src/include -I/home/jenkins/ws/build/src/libs/elektra  -std=gnu99  -Wno-deprecated-declarations  -Wstrict-prototypes  -Wno-long-long -pedantic -Wno-variadic-macros -Wall -Wextra -Wno-overlength-strings -Wsign-compare -Wfloat-equal -Wformat -Wformat-security -Wshadow -Wcomments -Wtrigraphs -Wundef -Wuninitialized -Winit-self -Wmaybe-uninitialized -Wsign-compare -Wfloat-equal -O2 -g -DNDEBUG -fPIC   -o CMakeFiles/elektra-core.dir/keytest.c.o   -c /home/jenkins/ws/src/libs/elektra/keytest.c

/usr/bin/ld: CMakeFiles/elektra-core.dir/keytest.c.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC

so -fPIC is set....

[ingwinlu]

the only difference I can find is the obvious gcc version missmatch (6.3 locally vs 4.9 on the jessie machine). Can anybody build musl locally with 6.3?

[markus2330]

I get plenty of warnings but I cannot reproduce your error. Afaik "relocation" are only warnings. So the error might be somewhere else. A link to the build job is generally useful.

[ 15%] Linking C shared library ../../../bin/libelektra-core.dll

[ingwinlu]

@markus2330 I added links to the op a few minutes ago.

[ingwinlu]

it builds fine in our jessie container so the issue is somewhere with gcc 6 :/

[markus2330]

Thank you for adding the link!

I also could not find anything useful in the output. "Nonrepresentable section on output" might be caused by wrong visibility and wrong use of PIC. But we do not use any symbol visibility and you already checked PIC.

Locally I have installed (on stretch):

binutils-mingw-w64-x86-64                       install
g++-mingw-w64-x86-64                            install
gcc-mingw-w64-base                              install
gcc-mingw-w64-x86-64                            install
mingw-w64-common                                install
mingw-w64-x86-64-dev                            install

and

~e/scripts/configure-mingw-w64 ~e && make -j 5

works just fine. Compiler version:

x86_64-w64-mingw32-g++-posix --version
#> x86_64-w64-mingw32-g++-posix (GCC) 6.3.0 20170516

[ingwinlu]

why mingw? i am trying to build with musl?

The changelog for gcc 6 includes the following:
Support for the musl C library was added for the AArch64, ARM, MicroBlaze, MIPS, MIPS64, PowerPC, PowerPC64, SH, i386, x32 and x86_64 targets. It can be selected using the new -mmusl option in case musl is not the default libc. GCC defaults to musl libc if it is built with a target triplet matching the -linux-musl pattern.
https://gcc.gnu.org/gcc-6/changes.html

However adding that setting seems to not resolve any issues.

[markus2330]

:smile: sorry my bad. The auto completion chose the wrong configure script. And once it was compiling...

I can reproduce the problem now.

First I thought it is caused with mixing g++ (non-musl) and gcc-musl. But this does not seem the case. Even after removing everything which uses g++, the same error occurs:

diff --git a/scripts/configure-debian-musl b/scripts/configure-debian-musl
index 0123f4467..64b2707e7 100755
--- a/scripts/configure-debian-musl
+++ b/scripts/configure-debian-musl
@@ -4,12 +4,13 @@ SCRIPTS_DIR=$(dirname "$0")
 . "${SCRIPTS_DIR}/configure-common"

 cd $BUILD
-# CFLAGS="-fPIC" LDFLAGS="-fPIC" CXXFLAGS="-fPIC"
-CC=musl-gcc CXX=g++ cmake \
+ CFLAGS="-fPIC" LDFLAGS="-fPIC" CXXFLAGS="-fPIC" CC=musl-gcc CXX=g++ cmake \
        -DINSTALL_BUILD_TOOLS=ON \
        -DBUILD_TESTING=OFF -DBUILD_FULL=OFF -DBUILD_STATIC=OFF \
-       -DPLUGINS="ALL;-yajl;-journald;-CRYPTO;-dbus;-dbusrecv;-xmltool;-curlget;-multifile" \
-       -DBINDINGS="cpp" \
-       -KDB_DEFAULT_STORAGE=ini \
+       -DPLUGINS="noresolver;mini" \
+       -DENABLE_OPTIMIZATIONS=OFF \
+       -DBINDINGS="" \
+       -DKDB_DEFAULT_STORAGE=mini\
+       -DKDB_DEFAULT_RESOLVER=noresolver \
        -DTARGET_PLUGIN_FOLDER="" \
        $*

(Btw. there is an error in scripts/configure-debian-musl it should be -DKDB_DEFAULT_STORAGE=ini, the D is missing. Can you fix this?)

I think it is easier to use an OpenWRT compiler. Seems like the Debian musl compiler is too buggy. What about https://github.com/hnw/docker-openwrt-sdk

[ingwinlu]

The easiest way is to just keep the musl build on gcc 4.9.2 for now.
I would prefer to fix the issues with gcc 6 but that might take some real effort...

[markus2330]

I am afraid musl in Debian does not receive much testing. OpenWRT is the way forward. It also better tests what we are interested in. (If Elektra works with OpenWRT.)

Would be great if you can take a short look at that Docker image.

[ingwinlu]

The Docker image itself is nearly useless for our use case.
We would have to provide a package building environment similar to how our debian packaging works.

We could use the Makefile from downstream, but it would still be some work to make it happen.

[markus2330]

You are right, it is not so easy:

docker run -i -t -u openwrt -w /home/openwrt/sdk -v $PWD:/work yhnw/openwrt-sdk:17.01-ar71xx /bin/bash
git clone https://github.com/ElektraInitiative/libelektra.git
mkdir build-native && cd build-native
cmake -DINSTALL_BUILD_TOOLS=ON -DINSTALL_SYSTEM_FILES=OFF -DCMAKE_INSTALL_PREFIX=/home/openwrt/sdk/staging_dir/host/ ../libelektra/
make -j 5 install

cd ..
mkdir build && cd build
CC=`pwd`/../staging_dir/toolchain-mips_24kc_gcc-5.4.0_musl-1.1.16/bin/mips-openwrt-linux-gcc CXX=`pwd`/../staging_dir/toolchain-mips_24kc_gcc-5.4.0_musl-1.1.16/bin/mips-openwrt-linux-g++ cmake -DCMAKE_CROSSCOMPILING=ON ../libelektra/
make -j 5

Unfortunately the second part does not work because /home/openwrt/sdk/staging_dir/host/bin/elektra-export-errors: /home/openwrt/sdk/staging_dir/host/lib/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/openwrt/sdk/staging_dir/host/bin/elektra-export-errors)

[ingwinlu]

I am following a different approach and try building with the official instructions available at https://wiki.openwrt.org/doc/howto/buildroot.exigence.

So far I have not found out how to configure and build a minimal set of requirements to install elektra (not even the included release).
https://wiki.openwrt.org/doc/howtobuild/single.package

[markus2330]

Did you create a Dockerfile while trying to build with the official instructions?

Compiling the toolchain takes forever, I hoped we could avoid to do that. The ./scripts/feeds are only useful if we package Elektra for OpenWRT in a way so that it downloads from master. Would be quite an effort to do that. Rebuilding what the maintainer already packaged is not useful.

In my test above, the problem is already in the native build of Elektra. It builds but you cannot run the executables. The host libstdc++ seems to be incompatible with the host g++.

Btw. the cross-compilation worked up to the point where Elektra needs the cross compilation tools of Elektra. So the toolchain in the docker image might be okay.

[ingwinlu]

Did you create a Dockerfile while trying to build with the official instructions?

No but I run everything in a container and plan to add one once compilation works.

Compiling the toolchain takes forever, I hoped we could avoid to do that.

We would only need to do it once per upstream release.

downloads from master

Yes, or from the PR.

Rebuilding what the maintainer already packaged is not useful.

I only try to do that now to have a baseline of what works and what does not.

[ingwinlu]

honestly I am surprised you even got so far with your approach. how did you get Elektras deps compiled with just those commands?

[ingwinlu]

@markus2330 I will probably add an alpine image and test it that way. I feel like it is the best option for the following reasons:

• build whole system against musl (so we check musl compat)
• use different shell (ash) so we test that as well
• check all our definition scripts and fix stuff like #2075 in the process

Sounds good?

[ingwinlu]

Additionally we get errors for all the 'wrong' glibc assumptions that automatically assumes _GNU_SOURCE to be set

[markus2330]

Actually, as said the whole problem is that the wrong stdlibc++ is picked.

So following works (so simply not installing and using the binaries of the other build directory):

docker run -i -t -u openwrt -w /home/openwrt/sdk -v $PWD:/work yhnw/openwrt-sdk:17.01-ar71xx /bin/bash
git clone https://github.com/ElektraInitiative/libelektra.git
mkdir build-native && cd build-native
cmake -DINSTALL_BUILD_TOOLS=ON -DINSTALL_SYSTEM_FILES=OFF -DCMAKE_INSTALL_PREFIX=/home/openwrt/sdk/staging_dir/host/ ../libelektra/

cd ..
mkdir build && cd build
export PATH="`pwd`/../build-native/bin:$PATH" 

But you are right, for the second part a cmake toolchain file needs to be written. Otherwise the build tools are confused. (Simply setting CMAKE_CROSSCOMPILING does not work.)

But let us give Alpine Linux a try first.

[ingwinlu]

Resolved as we have a working musl build now.