Runtime: Optimizing RootFS generation

[gkhanna79]

@hqueue @hseok-oh @jyoungyun

I wanted to open a seperate issue to discuss this topic. Essentially, when we generate RootFS today for Ubuntu 14.04/16.04 Armhf, it takes about 17 mins on our official build machines, something that I would like to reduce without impacting correctness of cross-compilation.

One approach to this would be for us to create our Docker images with these RootFS already generated. However, it relies on the fact that the generated RootFS changes very infrequently - what are your thoughts on the topic? Does the cross toolset (persisted in RootFS) change very often?

If not, I can see us generating a Docker image that contains RootFS for Armhf, armel, x86, etc.

Let me know what you think.

CC @janvorli

[hqueue]

One approach to this would be for us to create our Docker images with these RootFS already generated. However, it relies on the fact that the generated RootFS changes very infrequently - what are your thoughts on the topic? Does the cross toolset (persisted in RootFS) change very often?

@gkhanna79 I agree with your opinion on creating Docker image with RootFS for Ubuntu 14.04/16.04 Armhf, because I don't think cross toolset changes often since they have been already released and I don't experience critical failure due to old cross toolset in Ubuntu 16.04 armhf.

However for armel, esp. Tizen, I'm not sure about this. I'm afraid we have to construct RootFS on demand with the latest version, because it is also being developed in open source community with non-minor changes. Furthermore RootFS for armel(Tizen) can be constructed within a minuite.
You can compare time required to consturct RootFS for Ubuntu.16.04-arm and Tizen.4.0.0-armel when you you look into corefx CI log at https://github.com/dotnet/corefx/pull/15900. Therefore if this issue is about optimizing RootFS generation, I think it would be desirable to embed RootFS for arm(Ubuntu 14.04/16.04) and make RootFS for armel(Tizen 4.0.0) everytime for the time being.

• For armel(Tizen 4.0.0), it took only about 12 secs to construct RootFS, (https://ci.dot.net/job/dotnet_corefx/job/master/job/linuxarmemulator_softfp_cross_release_prtest/20/consoleFull)

00:01:08.937 + sudo ./cross/build-rootfs.sh armel tizen --skipunmount
...
00:01:19.724 + docker run -i --rm -v /mnt/j/workspace/dotnet_corefx/master/linuxarmemulator_softfp_cross_release_prtest:/opt/corefx -w /opt/corefx t2wish/dotnetcore:ubuntu1404_cross_prereqs_v2 /opt/corefx/build-native.sh -buildArch=armel -release -- cross verbose
...
00:09:40.795 Finished: SUCCESS

• For arm(Ubuntu 16.04), it took about 10 mins.(https://ci.dot.net/job/dotnet_corefx/job/master/job/linuxarmemulator_hardfp_cross_release_prtest/18/consoleFull)

00:01:23.867 + sudo ./cross/build-rootfs.sh arm xenial --skipunmount
...
00:11:40.467 + docker run -i --rm -v /mnt/j/workspace/dotnet_corefx/master/linuxarmemulator_hardfp_cross_release_prtest:/opt/corefx -w /opt/corefx chcosta/dotnetcore:ubuntu1404_cross_prereqs_v1 /opt/corefx/build-native.sh -buildArch=arm -release -- cross verbose
...
00:20:15.096 Finished: SUCCESS

/cc: @lemmaa @jyoungyun @hseok-oh @chunseoklee
What do you think for Tizen ? Do we need the latest RootFS for Tizen? or can we use a pre-constructed RootFS for a while and update RootFS sometimes when needed ?

[hseok-oh]

I agree with your opinion that uses docker image including RootFS already generated for Ubuntu. And I think it's better to generate RootFS for Tizen for each build request because it has small overhead to generate RootFS for Tizen.

[hqueue]

@gkhanna79 BTW if we are going to use same Docker image with RoofFS for core-setup, coreclr and corefx, we have to construct a single RootFS for all three repos and this is being acvhieved by using build-rootfs.sh in https://github.com/dotnet/core-setup/issues/1432 (not merged yet), https://github.com/dotnet/coreclr/pull/9411 (not merged yet), dotnet/corefx#15755 (merged).

[hseok-oh]

We need to prepare to contribute rootfs generation. For example, I want to add patch process for ubuntu 14.04 to build libcoreclrtraceptprovider.so (#8684), but I'll get failure in CI without docker image update, and ARM test may fail if this change merged.

[gkhanna79]

I think it would be desirable to embed RootFS for arm(Ubuntu 14.04/16.04) and make RootFS for armel(Tizen 4.0.0) everytime for the time being

Excellent - for now, I am most interested in optimizing the time it takes for the armhf rootfs for 14.04/16.04 and given that Tizen one gets constructed soon and is under development, it sounds reasonable to construct it on the fly.

@hqueue Can you please get the Core-Setup and dotnet/coreclr#9411 issues merged so that I can look into Docker image update for this?

@hseok-oh I agree that you should be able to contribute to RootFS generation. The way I envision this to happen is by having the respective DockerFiles being made available in a public repo that you can contribute to, followed by some automatic validation of the updated image (though, in the short term, it maybe manual validation).

@MichaelSimons @dleeapho Can you please share your thoughts on how we can make this happen for 2.0?

[hqueue]

Can you please get the Core-Setup and dotnet/coreclr#9411 issues merged so that I can look into Docker image update for this?

@gkhanna79 Oh, I've made a mistake. PR for Core-Setup and CoreFX has been alreay merged.
And I'm waiting PR in CoreCLR (#9411) to be reviewed and merged, since I don't have merge permission.

We can use build-rootfs.sh in Core-Setup (dotnet/core-setup#1493) to construct a RootFS for all core-setup, coreclr and corefx, because build-rootfs.sh in Core-Setup and CoreCLR will have exactly same structure and the only difference is supported architecture, i.e. Core-Setup does not support arm64 yet.

[hqueue]

@gkhanna79 If we are going to embed RootFS in docker image, are we going to maintain seperate docker images for each rootfs ?

[gkhanna79]

@hqueue I would that we will have all the RootFS in a single Docker image in a well defined layout, something like:

\rootfs\<arch>\<platform>

where

<arch> can be arm, armel (when it is stabilized), x86.
<platform> can be Ubuntu 14.04, Ubuntu 16.04, Tizen

And when invoking the build, we will set ROOTFS_DIR environment variable to point to the right one that build-rootfs.sh can pickup from https://github.com/dotnet/coreclr/blob/master/cross/build-rootfs.sh#L123.

[seanshpark]

@gkhanna79 , sounds good :)

[seanshpark]

CC @parjong , @wateret

[hqueue]

@gkhanna79 Due to Jenkins was not working yesterday, I've triggered CI for https://github.com/dotnet/corefx/pull/15900) now and it worked.

\rootfs\<arch>\<platform>

I also think the suggested layout will work with three scripts, i.e. build-rootfs.sh, CI script and build script.

BTW currently we are using different docker image for ubuntu.14.04-arm, ubuntu.16.04-arm and tizen.4.0.0-armel. Are you going to merge them too ? or just let each docker image have all three rootfs for arm?

[gkhanna79]

Thanks for the confirmation @seanshpark @hqueue.

We will continue to maintain Ubuntu 14.04 and Ubuntu 16.04 Docker images. For Tizen, I am not clear yet - what would you suggest? Will either 14.04 or 16.04 images work for Tizen build once we generate Tizen rootfs on them?

[jyoungyun]

You can generate Tizen rootfs in either 14.04 and 16.04 images if the libxml2-utils is installed in the docker image by https://github.com/dotnet/core-setup/blob/master/cross/armel/tizen-fetch.sh. However, in the case of Tizen, we will use the 14.04 image by default.

[hqueue]

@gkhanna79 As @jyoungyun said, there is no problem to build dotnet for Tizen in 14.04 and 16.04 if required packages are installed to construct rootfs. And I also prefer 14.04 over 16.04 if we have to choose one docker image.

[hqueue]

@gkhanna79 While updating arm32 CI script, I've added a option --skipRootFS to skip rootfs generation in arm CI. (https://github.com/dotnet/corefx/pull/16378, https://github.com/dotnet/coreclr/pull/9445)

[gkhanna79]

Based upon the recent discussions, here is my take away:

1) Update Arm Rootfs toolset in Docker images so that the patch to enable libcoreclrtraceptprovider.so is available.

2) Add RootFS for Linux/x86 for Xenial (16.04) only.

@hqueue @hseok-oh Can you confirm?

CC @MichaelSimons

[hseok-oh]

@gkhanna79 To build Linux/x86 using cross build, additional packages need: gcc-multilib, g++-multilib

[seanshpark]

I'm not sure about installing gcc-multilib, g++-multilib if you use rootfs.

[hseok-oh]

@seanshpark I confirmed that gcc-multilib and g++-multilib are needless. Thank you.

[hqueue]

@gkhanna79 Agreed :)

[MichaelSimons]

I rebuilt the docker images to include the libcoreclrtraceptprovider.so fix. The new image tags are ubuntu1404_cross_prereqs_v3 and ubuntu1604_cross_prereqs_v3

[gkhanna79]

Excellent - thank you @MichaelSimons!

[hqueue]

@MichaelSimons Thanks!

@hseok-oh Should we update arm CI to use this new docker image?

[hseok-oh]

@hqueue I has not make new commit to make libcoreclrtraceptprovider.so yet. So there will be no build fail in CI now if you use old docker image. But I'll make new PR soon to build libcoreclrtraceptprovider.so, so we need new docker image.

[hqueue]

@hseok-oh I see. ARM CI using docker (#9445) is not merged yet either and it will take some time berfore merged. After dotnet/coreclr#9853 is merged and dotnet/coreclr#9445 (arm CI using Docker) can be tested and merged.

BTW does existing arm CI have same problem ?

[MichaelSimons]

Regarding https://github.com/dotnet/coreclr/issues/9903 and the creation of an x86 Ubuntu Docker image with the RootFS. To my knowledge there isn't an official i386 Ubuntu Docker image. I asked what i386 Ubuntu base image we should using. The response was that developers are using https://hub.docker.com/r/ioft/i386-ubuntu/ and https://hub.docker.com/r/i686/ubuntu/. If we are going to be using these images for official builds, I think we need to know exactly how the images are built and what they contain. Looking at https://hub.docker.com/r/i686/ubuntu/, you can see the image isn't using autobuild nor are the Dockerfiles it is built with available. This means we have no guarantee of what the image contains. This is a risk IMO. https://hub.docker.com/r/ioft/i386-ubuntu/ is a little better in that you can trace the source down to the https://hub.docker.com/r/ioft/i386-ubuntu_core/ base image but you still have no guarantee what that image contains. Aside from the unknown contents, it doesn't look like the any of these images are getting the latest patches applied to them, the images are 9 months to over a year old.

I can think of 5 options for us to proceed with.

1. Get an official i386 Ubuntu image. We can ask Docker/community to create this or contribute ourselves. Others have already been asking for this.
2. Ask https://hub.docker.com/r/ioft/i386-ubuntu/ and https://hub.docker.com/r/i686/ubuntu/ to make their build process transparent so that we can have certainty in what the base images contain.
3. Create our own i386 Ubuntu image.
4. Use an existing i386 base image and accept it as is.
5. Don't use Docker for this scenario.

cc @gkhanna79

[gkhanna79]

@hqueue What are your thoughts on this?

[hseok-oh]

@MichaelSimons To avoid risk that you commented, I prefer option 3: create our own i386 ubuntu image. As you say, description in https://hub.docker.com/r/ioft/i386-ubuntu_core/ offer how to generate x86 core image, and https://hub.docker.com/r/ioft/i386-ubuntu/builds/b65pltnr7hyrsuhktrom2vr/ shows dockerfile to generate final image.

[hqueue]

@MichaelSimons @gkhanna79 I expect all of us may choose option 1 and use an official image if available. Unless Ubuntu provide an official i386 image soon, I think option 3 is the only feasiable way we can choose right now and we may contribute it to Docker community later. However it seems that an official Linux image is usually maintained by official Linux distributors though.

[lemmaa]

@MichaelSimons @gkhanna79 I suggest option 3 for the first. In parallel we can try option 1 and move to there when they prepared the official images.

[BruceForstall]

This issue appears obsolete, so I'm closing it. (FWIW, we use Docker in the CI and Official Builds for Ubuntu/arm32 )