Runtime: [Alpine.38.Arm64] gCurrentThreadInfo: symbol not found

[echesakovMSFT]

Checked coreclr fails to run on Alpine 3.8 (Arm64)

+ /root/helix/work/correlation/corerun /root/helix/work/correlation/xunit.console.dll baseservices.compilerservices.XUnitWrapper.dll -parallel collections -nocolor -noshadow -xml testResults.xml -notrait 'category=outerloop' -notrait 'category=failing'
dlopen failed to open the libcoreclr.so with error Error relocating /root/helix/work/correlation/libcoreclr.so: gCurrentThreadInfo: symbol not found

[RussKeldorph]

@janvorli @jashook Is this the thing where we have to build a custom loader to support these relocations?

[janvorli]

This looks like an issue with build of coreclr. The gCurrentThreadInfo is a thread local variable defined in threads.cpp, so coreclr should not have it listed as an unresolved symbol.

[echesakovMSFT]

This seems to be an issue with other TLS symbols as well
running under Alpine 3.8 on arm64 (mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.8-helix-arm64v8-46e69dd-20190327215724)

bash-4.4$ ldd libcoreclr.so
        ldd (0xaaaaab186000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0xffff87626000)
        libintl.so.8 => /usr/lib/libintl.so.8 (0xffff87609000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xffff87499000)
        libc.musl-aarch64.so.1 => ldd (0xaaaaab186000)
Error relocating libcoreclr.so: gCurrentThreadInfo: symbol not found
Error relocating libcoreclr.so: _ZN21ThreadEventBufferList29gCurrentThreadEventBufferListE: symbol not found

bash-4.4$ ldd libclrjit.so
        ldd (0xaaaabc204000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0xffffa6aa2000)
        libintl.so.8 => /usr/lib/libintl.so.8 (0xffffa6a85000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xffffa6915000)
        libc.musl-aarch64.so.1 => ldd (0xaaaabc204000)
Error relocating libclrjit.so: gJitTls: symbol not found

[janvorli]

I've been looking into it today. When I build coreclr directly on the arm64 Alpine 3.9 on RPI3, the issue is not present. When using crossbuild, the symbols are missing. I've used the same clang version 5.0.2 in both cases. My guess is that it is caused by older binutils. On Alpine there is version 2.31, for crossbuild we use version 2.27 (that we build ourselves from sources during docker image creation).
I am currently trying to build version 2.31 to see if it fixes the problem.

[echesakovMSFT]

@janvorli Can you please point me at where we are building binutils during Docker image creation? https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/master/src/alpine/3.6/WithNode/amd64/Dockerfile -- wrong Dockerfile

Found right one https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/master/src/ubuntu/16.04/cross/arm64-alpine/Dockerfile

[jashook]

https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/master/src/ubuntu/16.04/cross/arm64-alpine/Dockerfile#L13

[echesakovMSFT]

I see that we are building binutils.2.27 but I wonder why ld.gold then reports 2.26.1

root@9330d1a8ec5b:/opt/code# ld.gold --version
GNU gold (GNU Binutils for Ubuntu 2.26.1) 1.11
Copyright (C) 2015 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

I am using this image ubuntu-16.04-cross-arm64-alpine10fcdcf-20190208200917

[janvorli]

This is not the ld.gold that's being used. The /usr/bin/aarch64-alpine-linux-musl-ld.gold is the one.

[echesakovMSFT]

@janvorli I see - thanks

[janvorli]

I can confirm that the cross build with the updated binutils works. There are no missing symbols.

I've noticed an additional problem with the way we build the binutils. We were building them without plugin support, so the LTO was turned off. It is just a matter of adding the --enable-plugins=yes option to the configure command line when building binutils.

[janvorli]

@echesakovMSFT are you going to update the Dockerfile or would you like me to do that?

[echesakovMSFT]

@janvorli Can you please update the Dockerfile and as soon the image is pushed to registry I will update the yaml files and run testing in Helix using the new image

[janvorli]

Sure, I will do it right away.

[echesakovMSFT]

@janvorli Thank you for finding the root cause

[echesakovMSFT]

Do you know technical details of what was wrong with the older linker?

[janvorli]

I don't. Looking at the objdump -t output, I could see that most of the symbols were marked with ".hidden" and there were much less symbols than in the build I've built directly on Alpine ARM64. I have guessed that the linker is most likely the culprit based on these differences. I am not sure what the marking really means though.

[janvorli]

Unfortunately, I have found that it works fine for me only because I've forgotten commented out line in the cross/toolchain.cmake that adds linker option to use the gold linker. When I enabled it again, I got the same missing symbols as before.
I now think the reason may be that we are building the binutils standalone. When I was originally adding arm64 musl support, I was building the https://github.com/richfelker/musl-cross-make repo and using the linker from that. But it takes ages to build and so when @jashook was adding support for the arm64 musl to the docker images, we've decided to build the binutils alone which was much faster. It is possible though that our build misses some config options.
I'll look into it in the morging.

[janvorli]

I've spent quite some time today investigating the issue. I've tried:

• use the binutils built from the https://github.com/richfelker/musl-cross-make repo
• use the binutils 2.31.1 built with additional options that I've found being used during binutils-gold package build in the Alpine repo
• add "-z now -z relro" options to clang linker options (as the clang in Alpine repo is built with those set by default and they may influence how symbols are linked)
• tried using clang-7
• revert change that we've made recently that set the default symbol visibility to hidden

Nothing worked, the issue persists.

The only thing that fixes the problem is to not to use the ld.gold and use the regular ld instead. In fact, we should not be using the ld.gold as we don't have PGO data available for Linux musl arm64 and we turn on the ld.gold only when we have the PGO data available. The reason we use it with PGO is that only the ld.gold can consume these data.
The cross build, as opposed to the native build, uses ld.gold by default due to a bug in cross/toolchain.cmake that unnecessarily sets ld.gold as the linker to use.
I've tried to force the usage of ld.gold when building natively on arm64 device - and the missing symbols appeared.
So the reason why the native build worked was that it is using ld instead of ld.gold and the issue is not caused by the cross building.

Finally, I've tried to use clang-7 with the new llvm linker called lld. That linker should also be capable of using the PGO data, so it could replace the ld.gold if we moved to clang-7 and using the lld helped. And that worked too - no missing symbols.

My suggestion to fix the issue for the preview 5 is to remove the incorrect unconditional setting of ld.gold as a linker from cross/toolchain.cmake.
Then at some point after the preview 5 release, we should consider moving to clang-7 and the LLVM lld linker so that we can use PGO.

[echesakovMSFT]

Fixed in https://github.com/dotnet/coreclr/pull/23721