Source-build: Building 3.0 on Arch Linux

[alucryd]

Reopening an issue since for some reason issue #1242 was completely nuked along with all the valuable information it contained.

The latest head of the release/3.0 branch still doesn't build. Did you guys make progress on this?

[omajid]

issue #1242 was completely nuked

Yikes. I can't seem to access it either. @dleeapho Any chance we could get that issue back?

Did you guys make progress on this?

I tried installing Arch on a machine, but I didn't have much luck. Tried it in a container and I ran into the dreaded "too many files open" issue :(

Can you git clone https://github.com/dotnet/source-build --recrusive and make this change:

diff --git a/tools-local/arcade-services/eng/Versions.props b/tools-local/arcade-services/eng/Versions.props
index c69a2ff..6350ff1 100644
--- a/tools-local/arcade-services/eng/Versions.props
+++ b/tools-local/arcade-services/eng/Versions.props
@@ -12,7 +12,7 @@
     <CommandLineParserVersion>2.2.1</CommandLineParserVersion>
     <CredentialManagementVersion>1.0.2</CredentialManagementVersion>
     <HandlebarsNetVersion>1.9.5</HandlebarsNetVersion>
-    <LibGit2SharpVersion>0.26.0</LibGit2SharpVersion>
+    <LibGit2SharpVersion>0.27.0-preview-0020</LibGit2SharpVersion>
     <log4netVersion>2.0.8</log4netVersion>
     <SystemNetHttpVersion>4.3.3</SystemNetHttpVersion>
     <MicrosoftAzureKeyVaultVersion>3.0.3</MicrosoftAzureKeyVaultVersion>

And try a ./build.sh?

I also opened https://github.com/dotnet/corefx/pull/41750 and https://github.com/dotnet/cli/pull/12880

[alucryd]

@omajid Thanks for the PRs!

What's this newer libgit2 for? The current one works fine with openssl 1.0 installed, is 0.27.0 built with openssl 1.1?

[robinchew]

Here's a Google cache of #1242 https://webcache.googleusercontent.com/search?q=cache:CIeT58k-DkgJ:https://github.com/dotnet/source-build/issues/1242 and I've also attached a screenshot.
https://user-images.githubusercontent.com/7122710/66716574-d7897a80-ee01-11e9-8ea8-79f5b8a10c6d.png

[omajid]

is 0.27.0 built with openssl 1.1

Kind of; 0.27 is more flexible and supports both.

[alucryd]

Kind of; 0.27 is more flexible and supports both.

Works like a charm with just openssl 1.1 in the chroot, thanks!

[omajid]

Works like a charm with just openssl 1.1 in the chroot, thanks!

Does that mean you can build .NET Core 3.0 in the chroot? Or that it gets you past libgit2 erorrs?

[alucryd]

Just the libgit2 errors, those were previously fixed by having openssl 1.0 in the chroot, but now it works with 1.1, hopefully we can start retiring 1.0 from our repos soon.

[adaggarwal]

Reopening an issue since for some reason issue #1242 was completely nuked along with all the valuable information it contained.

The latest head of the release/3.0 branch still doesn't build. Did you guys make progress on this?

I think it didn't transfer cleanly. Here's the original issue - https://github.com/dotnet/core-eng/issues/7996

Here is the coreFX reference to this issue.

cc: @alucryd @omajid

[omajid]

I think it didn't transfer cleanly. Here's the original issue - dotnet/core-eng#7996

Unfortunately, dotnet/core-eng is, as far as I know, restricted to folks inside Microsoft. As an outsider-to-Microsoft, the optic of this change look like kicking the community out from a community-based issue. I know that's probably not intentional on anyone's part. Is there any chance we could get the issue back in a place everyone can see it?

I can see the corefx issue, but it just contains a pointer to the can't-be-seen-core-eng-issue.

[adaggarwal]

I think it didn't transfer cleanly. Here's the original issue - dotnet/core-eng#7996

Unfortunately, dotnet/core-eng is, as far as I know, restricted to folks inside Microsoft. As an outsider-to-Microsoft, this change looks like someone just completely restricted/hid the issue :(

I can see the corefx issue, but it just contains a pointer to the can't-be-seen-core-eng-issue.

Ah. I understand now. I tried to get the issue back. But it wasn't very consistent. Using zenhub, I could get all the comments and timestamps. It just lacks the reactions. I hope we can reference #1312 for history.

[dagood]

FYI, I found this in the help doc for GitHub transfers:

You can't transfer an issue from a private repository to a public repository.

I guess it makes sense from a "make it a little harder to expose secrets" sense, but unfortunate for this case. Would have been nice to be able to use the "native" move to bring it back as-was.

[omajid]

I have now locally seen both arch-x64 and fedora.32-x64 builds fail with failures similar to:

    chmod +x /builddir/build/BUILD/dotnet-v3.0.0/src/corefx.4ac4c0367003fe3973a3648eb0715ddb0e3bbcea/artifacts/bin/testhost/netcoreapp-Linux-Release-x64/                                                                        
    chmod: cannot access '/builddir/build/BUILD/dotnet-v3.0.0/src/corefx.4ac4c0367003fe3973a3648eb0715ddb0e3bbcea/artifacts/bin/testhost/netcoreapp-Linux-Release-x64/': No such file or directory                               
8>/builddir/build/BUILD/dotnet-v3.0.0/src/corefx.4ac4c0367003fe3973a3648eb0715ddb0e3bbcea/external/runtime/runtime.depproj(63,5): error MSB3073: The command "chmod +x /builddir/build/BUILD/dotnet-v3.0.0/src/corefx.4ac4c0367003fe3973a3648eb0715ddb0e3bbcea/artifacts/bin/testhost/netcoreapp-Linux-Release-x64/" exited with code 1.     

Need to debug this some more.

Edit: I tried building on Fedora 30 (where this is known to work) with /etc/os-release modified to say 32. The build failed there with this exact same error. I doubt it's an issue in the system libraries or tools; otherwise it would have built on this hacked-Fedora-30 system too. This error seems more of an RID issue to me now.

[tmds]

This error seems more of an RID issue to me now.

It think like the issue is with:

<RuntimeIdentifier>$(PackageRID)</RuntimeIdentifier>

in runtime.depproj. I think it fails to restore some things when the rid is not known, causing them to be missing when they are used later (with the chmod).

[SuddenGunter]

@adaggarwal any info when it can be fixed?

[adaggarwal]

@SuddenGunter is coreFX standalone still failing on Arch Linux?

[Saancreed]

@adaggarwal Actually it seems like the build itself isn't failing anymore, at least the current release/3.0@c1778515a3bee34cc09c757b5563d0af0c8b1e99 with pre-rebased RID patch applied on top of it. Some tests in System.Net.Ping.Functional.Tests are of course failing, but it is somewhat expected.

Unfortunately, I was running into this issue because I'm in GMT+1 timezone and I had to hack a solution with find ~ -name _._ -exec touch {} + before running the build.

Would you like me to try and bisect which commit fixed it or can we simply wait for 3.0.1 (assuming it comes before 3.1 releases)?

[adaggarwal]

Unfortunately, I was running into this issue because I'm in GMT+1 timezone and I had to hack a solution with find ~ -name _._ -exec touch {} + before running the build.

Interesting. It should have gone as we are ahead of the commit mentioned in this comment.

Would you like me to try and bisect which commit fixed it or can we simply wait for 3.0.1 (assuming it comes before 3.1 releases)?

3.1 lacks some updates from 3.0 and will be updated with the merge PR. 3.0.1 update is pretty close to get in. Won't be a bad idea to try that one out. Even 3.1 would be a good baseline to try; as the merge PR goes in.

[Saancreed]

Unfortunately, I was running into this issue because I'm in GMT+1 timezone and I had to hack a solution with find ~ -name _._ -exec touch {} + before running the build.

Interesting. It should have gone as we are ahead of the commit mentioned in this comment.

From what I see, only the master branch of corefx has that commit for now, and current master builds fine without any hacks or additional patches required.

Would you like me to try and bisect which commit fixed it or can we simply wait for 3.0.1 (assuming it comes before 3.1 releases)?

3.1 lacks some updates from 3.0 and will be updated with the merge PR. 3.0.1 update is pretty close to get in. Won't be a bad idea to try that one out. Even 3.1 would be a good baseline to try; as the merge PR goes in.

Then I'm going to try building the release/3.1 of corefx and whatever commits are referenced by this repo's eng/Version.Details.xml for both release/3.0 and release/3.1 branches; I assume that if all of these build fine then 3.0.1 should be fine as well.

[adaggarwal]

Then I'm going to try building the release/3.1 of corefx and whatever commits are referenced by this repo's eng/Version.Details.xml for both release/3.0 and release/3.1 branches; I assume that if all of these build fine then 3.0.1 should be fine as well.

Yes. I think we could build once we have that. We can target specific problems we face, if any. And then try to investigate the delta with master that would get us to a successful build. Although, I hope the latter won't be needed.

[Saancreed]

Looks like all mentioned commits can be successfully built. What remains now is the zip timestamp issue, but maybe we can work around that by exporting TZ=Etc/UTC before restoring/building.

Small update: Exporting TZ=Etc/UTC before calling build.sh seems to be enough to fix the timestamp issue, assuming that nuking the user's home directory is enough to clear all possible package/config caches. And with extra /usr/bin/ping6 -> ping symlink, all tests that weren't skipped pass successfully as well (on commit 54d63c3e283e0c3937a35ab4e99f391c84ff119e at least).

[Saancreed]

Well, looks like my little celebration was a bit premature: even though I can build directly the corefx commit referenced by source-build, building the whole thing still fails at corefx step. The error message is now:

    chmod: cannot access '/build/dotnet/src/source-build/bin/src/corefx.54d63c3e283e0c3937a35ab4e99f391c84ff119e/artifacts/bin/testhost/netcoreapp-Linux-Release-x64/': No such file or directory
  /build/dotnet/src/source-build/bin/src/corefx.54d63c3e283e0c3937a35ab4e99f391c84ff119e/external/runtime/runtime.depproj(65,5): error MSB3073: The command "chmod +x /build/dotnet/src/source-build/bin/src/corefx.54d63c3e283e0c3937a35ab4e99f391c84ff119e/artifacts/bin/testhost/netcoreapp-Linux-Release-x64/" exited with code 1.

And /build/dotnet/src/source-build/bin/src/corefx.54d63c3e283e0c3937a35ab4e99f391c84ff119e/artifacts/bin folder has only the following content (all folders themselves): binplacePackages external.runtime netfx netstandard ref

So, it looks like the problem is somewhere in source-build patches for corefx or arguments/envvars passed by source-build to build.sh. Any ideas how to find the culprit?

Also it appears that passing --capability=CAP_IPC_LOCK to systemd-nspawn is again required, this time probably because of Darc running on pre-3.0 runtime.

[tmds]

@Saancreed see https://github.com/dotnet/source-build/issues/1310#issuecomment-545249791. Patching RuntimeIdentifier (e.g. use linux-x64) in runtime.depproj may help.

[Saancreed]

@tmds With <RuntimeIdentifier>linux-x64</RuntimeIdentifier> building corefx now succeeds, but then I have another failure at core-setup with a nice exception:

  /build/dotnet/src/source-build/packages/restored/microsoft.dotnet.build.tasks.packaging/1.0.0-beta.19463.3/build/Packaging.targets(1258,5): error : Error when creating nuget lib package from /build/dotnet/src/source-build/bin/src/core-setup.0bdbe6b3a7caf32a4f594196effe56a941980d7b/artifacts/packages/Release/specs/runtime.arch-x64.Microsoft.NETCore.DotNetAppHost.nuspec. System.IO.DirectoryNotFoundException: Could not find a part of the path '/build/dotnet/src/source-build/bin/src/core-setup.0bdbe6b3a7caf32a4f594196effe56a941980d7b/artifacts/bin/arch-x64.Release/corehost'. [/build/dotnet/src/source-build/bin/src/core-setup.0bdbe6b3a7caf32a4f594196effe56a941980d7b/src/pkg/projects/Microsoft.NETCore.DotNetAppHost/Microsoft.NETCore.DotNetAppHost.pkgproj]

arch-x64.Release contains only two subfolders: crossgen and symbols. Interestingly, there also exists arch.-x64.Release folder in the same location, and that one _does_ have corehost and corehost_test. Looks like another RID computation gone wrong for a rolling release distro.

Update: Yep, this line outputs Computed RID for native build is arch.-x64 because VERSION_ID being unset is not handled correctly here.

[Saancreed]

After changing line mentioned above to __rid_plat="$ID${VERSION_ID:+.$VERSION_ID}" building core-setup now succeeds and the next failure is at AspNetCore with error NETSDK1083: The specified RuntimeIdentifier 'arch-x64' is not recognized.

[Saancreed]

Okay, looks like I was able to hack together a solution by injecting arch and arch-x64 nodes into RID graphs of bootstrapped SDK/runtime. With this final change, I was able to build everything from current release/3.0 with only 113 warnings. Yay?

Patches and PKGBUILD I used are here. If applying patch to RuntimeIdentifierGraph.json fails, make sure that lines to be inserted have CRLF line endings.

[adaggarwal]

Related comment

[alucryd]

@Saancreed Thanks for the PKGBUILD, I was able to build the head of release/3.1 using only the core-setup patch, I pushed 3.1.0 to [community-staging] and added you to the contributor list: https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/dotnet-core

[huupoke12]

@alucryd Shouldn't we keep the version 2.1 instead of 2.2 in the repository? Since 2.1 is LTS but 2.2 is not and its support will end at 2019-12-23.

[alucryd]

The 2.2 package is only temporary while emby switches to 3.1 (which is due next release, beta targeting 3.1 has started). Due to the amount of backporting I already need to do to get 2.2 to build, I'm not really keen on trying to package 2.1. This will have to do.

[ptrxyz]

@Saancreed Thanks for the PKGBUILD, I was able to build the head of release/3.1 using only the core-setup patch, I pushed 3.1.0 to [community-staging] and added you to the contributor list: https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/dotnet-core

@SanderSade , @alucryd , It seems this does not work anymore. At least if I install dotnet-sdk from the Arch repo in a clean environment and do a basic testing:

The template "Console Application" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /home/ptrxyz/dev/test/test.csproj...
/usr/share/dotnet/sdk/3.1.100/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(88,5): error NETSDK1083: The specified RuntimeIdentifier 'arch-x64' is not recognized. [/home/ptrxyz/dev/test/test.csproj]
/usr/share/dotnet/sdk/3.1.100/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(88,5): error NETSDK1083: The specified RuntimeIdentifier 'arch-x64' is not recognized. [/home/ptrxyz/dev/test/test.csproj]
/usr/share/dotnet/sdk/3.1.100/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(88,5): error NETSDK1083: The specified RuntimeIdentifier 'arch-x64' is not recognized. [/home/ptrxyz/dev/test/test.csproj]

Restore failed.
Post action failed.
Description: Restore NuGet packages required by this project.
Manual instructions: Run 'dotnet restore'

Am I missing something? This is based on version 3.1.0.sdk100-1.

[fmauNeko]

@ptrxyz I opened https://bugs.archlinux.org/task/65128 about this.
This is probably due to dotnet/corefx#41750 being pushed back to milestone 5.0.

[ptrxyz]

@ptrxyz I opened https://bugs.archlinux.org/task/65128 about this.
This is probably due to dotnet/corefx#41750 being pushed back to milestone 5.0.

Ah right, just saw that. Thanks!

[alucryd]

That pull request is now backported into our package, thanks guys.

[Saancreed]

It might be a good time to finally close this issue I think — not only .NET Core 3.0 reached end of support a week ago, but we also have succeeded in building 3.0 on Arch and replaced it with 3.1 in our repos some time ago.