Runtime: [ARM32/Linux] stable default clang version for cross compilation

[hqueue]

From last week, Arm32 (armhf) packages of Coreclr for Ubuntu 14.04 and Ubuntu 16.04 are published on myget. https://github.com/dotnet/coreclr/issues/8549#issuecomment-275252583.

However they can not be ran correcly when built with -O3 optimization, since default clang version used to build ARM32 binaries is 3.6 and it has problem with -O3 optimization as described at https://github.com/dotnet/coreclr/blob/master/Documentation/building/linux-instructions.md#how-to-enable--o3-optimization-level-for-armlinux. As a workaround, we are now using -O1 optimization for Linux arm32 Release instead of -O3 optimization.
Related issues: https://github.com/dotnet/core-setup/issues/790#issuecomment-275326153 https://github.com/dotnet/core-setup/issues/725#issuecomment-275324269

To address this clang problem,

• It is necessary to change default clang version when compiling Linux arm32 targets, i.e. using recent version of clang with fix
• Same clang should be used for all core-setup, coreclr and corefx.
• Test with packaging all core-setup, coreclr and corefx

This issue will track all issues regarding clang version for core-setup, coreclr and corefx.

One of possible candiadates is clang-3.9 and clang-3.9 will be tested for a while.

[hqueue]

/cc: @gkhanna79 @janvorli @jyoungyun @hseok-oh @chunseoklee

FYI. Previous discussion on clang version was made at Core-Setup(https://github.com/dotnet/core-setup/issues/790#issuecomment-275326153) and CoreCLR(https://github.com/dotnet/coreclr/pull/9211)

[gkhanna79]

Thank you @hqueue. Can you please link this issue at https://github.com/dotnet/core-setup/issues/725 as well?

[hqueue]

@gkhanna79 I've already add this item in a "common" section :)

[hqueue]

We have tested with clang 3.9 with -O3 optimization.

• For coreclr and corefx, no problem is found when cross building for Linux/ARM32 with clang 3.9 after enabling -O3 optimization.
• For coreclr, no regression is found in unit test. All failing tests are also failed with current infra, i.e. clang-3.6 with -O1 option. Test is performed on rpi3 with ubuntu.16.04-arm.
• For core-setup, no problem is found when cross building for Linux/ARM32 with clang 3.9.

Threfore fortunately, we can simply choose clang 3.9 for cross compiling for Linux/ARM and don't have to investigate other version if there is no new requirement.

@janvorli @gkhanna79 What do you think of it?
If you agree I will make PRs to make clang-3.9 as a default compiler for cross compiling for Linux/ARM and also enable -O3 optimization for Release build for Linux/ARM.

[gkhanna79]

Thanks for validating these scenarios @hqueue. To drive for simplicity, it will be good to have a single clang version we could use - not just for Arm but also for x64. Do you think 3.9 is good for x64 as well?

Aside from that, is 3.9 publicly released (and not under development)?

[hqueue]

@gkhanna79 I also agree that using clang 3.9 for all dotnet platform will be ideal and I expect no problem using 3.9. However we need more verfication from developers of x64 dotnet for using 3.9, since I don't have much experience in x64 dotnet yet.

Aside from that, is 3.9 publicly released (and not under development)?

It seems that llvm 3.9.1 is officially released on Dec. 23, 2016 as in http://releases.llvm.org/.
I've also installed stable 3.9.1 on host machine (Ubuntu 14.04) as described in http://apt.llvm.org/ as below.

# /etc/apt/sources.list
# 3.9 
deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9 main
deb-src http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9 main

$ wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key|sudo apt-key add -
$ apt-get install clang-3.9

BTW We have to install clang-3.9 on CI machine before we merge PRs using clang 3.9 as a default, since CI also use default compilers.

[janvorli]

@gkhanna79 I have recently changed CoreRT to use clang 3.9 by default too and the CI machines that CoreRT use were already updated to contain clang 3.9. So moving CoreCLR and other related projects to that version as well would be nice.
We would need to update more VMs in the CI though, since for CoreRT, only Ubuntu 14.04 is used and so only those VMs were updated, as far as I know.

[gkhanna79]

@hqueue Thanks for the details. Looks like http://apt.llvm.org/ also has links for 16.04 and 16.10. Is there a reason you used the above approach to install as opposed to the instructions in Install (stable branch)section?

@janvorli Thanks for the confirmation.

[hqueue]

@gkhanna79 I think I followed the instrtuction in Install (stable branch) and Ubuntu section. We have to update /etc/apt/source.list to locate and install clang-3.9 with apt-get install clang-3.9. If not, apt-get can't locate clang-3.9 package.

[hqueue]

@gkhanna79 As you may already know, arm32 binary is still built using clang 3.6 with -O1 optimization, not -O3. And I think it's same for the official pipeline build for arm32. (right?)
Is is ok to use this approach for the official pipeline build which may be used for 2.0.0 release ? Or should we move to clang 3.9 for arm32 cross build and enable -O3 optimiozation to prepare 2.0.0 ? For other architectures, I'm not sure about whether it's stable for other architectures.

[janvorli]

I think it would be the best to switch our build to clang 3.9 now all over the board.

[Petermarcu]

If we're going to do this, we need to do it soon.

[gkhanna79]

And I think it's same for the official pipeline build for arm32. (right?)

Yes, this is correct since official images do not carry clang 3.9. Before switching whole sale to 3.9, can you please help confirm that CoreCLR tests are passing with it for x64 as well? Once you can help confirm that, we can look into transitioning to 3.9.

[gkhanna79]

For other architectures, I'm not sure about whether it's stable for other architectures.

@janvorli Have you validated that this works as expected for x64 on Linux (since on OSX we endup using whatever clang comes with XCode)?

[janvorli]

@gkhanna79 I have not done a full validation yet. But I can do an easy way of validation - create a PR that makes clang 3.9 the default and let the lab run the tests. We already have clang3.9 installed in the lab for Ubuntu to support CoreRT builds.

[gkhanna79]

@janvorli If you can give it a shot, that will be great and will help us get answers soon.

[hseok-oh]

CoreCLR cross build (linux arm) depends on docker images: microsoft/dotnet-buildtools-prereqs:ubuntu1404_cross_prereqs_v3, microsoft/dotnet-buildtools-prereqs:ubuntu1604_cross_prereqs_v3
But these docker images look like don't have clang-3.9 yet.

[janvorli]

@gkhanna79 my clang3.9 test build in the CI ran ok (with help from @mmitche to ensure that it is built on a VM that has clang3.9 installed), all the CoreCLR tests passed (except for the spanbench that's failing on all jobs now).

[gkhanna79]

Thanks @janvorli. how about we bring this up in stages:

1) Stage 1 - Update the Arm build definitions to carry clang 3.9 and transition the builds to it.
2) Stage 2 - once (1) is stable, we update X64 docker images to use clang 3.9 as well.

With this, we could even stop after stage (1) and yet use -O3 across the board without switching x64 to 3.9, if required.

What do you think?

[janvorli]

@gkhanna79 that sounds good to me. We would just need to make the clang3.9 the default optionally for arm.

[gkhanna79]

We would just need to make the clang3.9 the default optionally for arm.

I agree.

[gkhanna79]

@hqueue Per https://github.com/dotnet/core-setup/issues/1411#issuecomment-277866406, what changes need to be made to sources file to pull down clang 3.9 (for Arm) so that the Docker image can be updated?

We will need to update Core-Setup's cross build scripts for this since that is what is used for building the Docker images.

[jyoungyun]

I added below codes to create new docker image including clang 3.9.

RUN apt-get update \                                                            
    && echo "deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9 main" >> /etc/apt/sources.list.d/llvm.list \
    && echo "deb-src http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9 main" >> /etc/apt/sources.list.d/llvm.list \
    && wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key|apt-key add - \         
    && apt-get update \                                                         
    && apt-get install -y \                                                     
        llvm-3.9 \                                                              
        clang-3.9 \                                                             
        lldb-3.9 \                                                              
    && rm -rf /var/lib/apt/lists/*

I uploaded t2wish/dotnetcore:ubuntu1404_cross_prereqs_v4 docker image which contains clang 3.9 for tizen build. And let me know if you ready to use new docker image for clang 3.9, then I will update core-setup build script code.

[hqueue]

@gkhanna79 I think @jyoungyun addressed how to pull clang 3.9 down for Docker image.

There is one concern with arm cross CI regarding upgrading clang to 3.9.
Are you going to prepare Docker image with both clang 3.6 and clang 3.9 ?
If we include only clang 3.9, then arm cross CI may break until build scripts is merged.

Therefore I suggest follow steps,

1. Prepare new Docker image for ARM CI (arm/armel) which contain clang 3.6 and clang 3.9
2. Update CI script to use new Docker image
3. Update build script in core-setup, coreclr and corefx to use clang 3.9 as default for arm cross build.

What do you think of it ?

[gkhanna79]

Sounds good to me @hqueue. Thanks for sharing the steps @jyoungyun. Can you please get a PR going for updating Core-Setup cross build scripts to pull clang 3.9?

@MichaelSimons Will it be possible for you to use the steps above to create v4 version of crossbuild docker images that contain clang 3.9 as well?

[hqueue]

Can you please get a PR going for updating Core-Setup cross build scripts to pull clang 3.9?

@gkhanna79 Are you talking about updating build script to use clang 3.9 for arm cross build ?

[jyoungyun]

@gkhanna79 I'm sorry but I don't understand what Core-Setup cross build scripts does mean. If the new docker image which contains clang 3.9 is available, I will update the scripts/arm32_ci_script.sh script to use new docker image and update the src/corehost/build.sh script to change default clang version to clang 3.9 in cross build case. However, both of these fixes require the new docker image updated. Do you want to put our PR first before updating new docker image? Or do you think other script should be fixed first?

[MichaelSimons]

@gkhanna79, yes this is possible. What is the time frame this needs to be done in?

[gkhanna79]

@MichaelSimons when do you think you will be able to get it done? Can it be done by Monday, if not today? I would like this to have a bake time and once this is in place, @hqueue and @jyoungyun would need to get the scripts updated.

[MichaelSimons]

@gkhanna79, I can work on this now. I assume you want this done for both 14.04 and 16.04, correct? If so, can someone specify the changes I should make for 16.04?

[hqueue]

I assume you want this done for both 14.04 and 16.04, correct?

@MichaelSimons Yes, we need both 14.04 and 16.04 image for ARM CI.

If so, can someone specify the changes I should make for 16.04?

I think @jyoungyun shared Docker file for Trusty (Ubuntu 14.04) in comment above (https://github.com/dotnet/core-setup/issues/1411#issuecomment-292426308) which contains followings

deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9 main
deb-src http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9 main

And I think you can use followings for Xenial (Ubuntu 16.04)

deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-3.9 main
deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-3.9 main

[MichaelSimons]

@hqueue @jyoungyun - I could not get Xenial to build.

FROM ubuntu:16.04

RUN apt-get update \
    && apt-get install -y wget \
    && echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-3.9 main" >> /etc/apt/sources.list.d/llvm.list \
    && echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-3.9 main" >> /etc/apt/sources.list.d/llvm.list \
    && wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \
    && apt-get update \
    && apt-get install -y \
        llvm-3.9 \
        clang-3.9 \
        lldb-3.9 \
    && rm -rf /var/lib/apt/lists/*

E: Unable to locate package llvm-3.9
E: Couldn't find any package by glob 'llvm-3.9'
E: Couldn't find any package by regex 'llvm-3.9'
E: Unable to locate package clang-3.9
E: Couldn't find any package by glob 'clang-3.9'
E: Couldn't find any package by regex 'clang-3.9'
E: Unable to locate package lldb-3.9
E: Couldn't find any package by glob 'lldb-3.9'
E: Couldn't find any package by regex 'lldb-3.9'

If you can specify how the Dockerfile should be defined, I will get the microsoft/dotnet-buildtools-prereqs image updated.

[hqueue]

@MichaelSimons Can you try with below file ? I've just tested and successfully added clang-3.9 by extending microsoft/dotnet-buildtools-prereqs:ubuntu1604_cross_prereqs_v3 and add a command for llvm 3.9.

https://github.com/hqueue/docker/blob/master/dotnet/ubuntu1604_cross_prereqs/Dockerfile

FROM microsoft/dotnet-buildtools-prereqs:ubuntu1604_cross_prereqs_v3

RUN apt-get update \
    && echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-3.9 main" >> /etc/apt/sources.list.d/llvm.list \
    && echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-3.9 main" >> /etc/apt/sources.list.d/llvm.list \
    && wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key|apt-key add - \
    && apt-get update \
    && apt-get install -y \
        llvm-3.9 \
        clang-3.9 \
        lldb-3.9 \
        python-lldb-3.9 \
    && rm -rf /var/lib/apt/lists/*

[MichaelSimons]

@hqueue - I built the new cross images requested.

microsoft/dotnet-buildtools-prereqs:ubuntu-14.04-cross-0cd4667-20172211042239
microsoft/dotnet-buildtools-prereqs:ubuntu-16.04-cross-ef0ac75-20175511035548

We have introduced a new naming scheme for the images. This is in support of our forkable engineering effort. The name indicates the commit sha of the Dockerfile the image was generated from and the timestamp of when it was built. We have plans to move these to the open so you will be able to make the necessary Dockerfile changes yourself and they will automatically get built upon checkin.

[hqueue]

The name indicates the commit sha of the Dockerfile the image was generated from and the timestamp of when it was built. We have plans to move these to the open so you will be able to make the necessary Dockerfile changes yourself and they will automatically get built upon checkin.

@MichaelSimons Looks good and nice to hear the news :)

[jyoungyun]

@MichaelSimons Thanks for sharing. I will update the ci script for using new docker image soon. And it is good news to add a new repository(?) to access the Dockerfile. :)

[gkhanna79]

@wtgodbe @MichaelSimons updated the crossbuild images to support clang 3.9 for cross builds (see https://github.com/dotnet/core-setup/issues/1411#issuecomment-293150266). Can you please update CoreCLR build definitions to consume these Docker images for cross build?

Likewise, can you create PRs for the same for CoreFX and Core-Setup too?

[gkhanna79]

@jyoungyun Once the official build pipeline is updated to use the above images, will you create the PRs for CoreCLR/CoreFX/Core-Setup to specify -O3 for Arm/armel builds?

[wtgodbe]

Here are the PR's to update the images in the 3 repos:

https://github.com/dotnet/core-setup/pull/2057
https://github.com/dotnet/corefx/pull/18349
https://github.com/dotnet/coreclr/pull/10953

[gkhanna79]

@jyoungyun @hqueue We have updated the build pipeline to use the new Docker images @MichaelSimons produced that contain clang 3.9. Will you be updating the repos to use the -O3 switch for the build?

Also, can I merge https://github.com/dotnet/core-setup/pull/2038/files?

[hqueue]

Also, can I merge https://github.com/dotnet/core-setup/pull/2038/files?

@gkhanna79 Yes, please :)

[qmfrederik]

I'm a bit late to this thread, but since we're making clang 3.9 the default for arm and armel, would it make sense to use clang3.9 by default for arm64 (aarch64) as well?

We've seen some issues with older versions of clang, so this seems like a good default to me.

[gkhanna79]

Closing as this work complete.