Source-build: Add support for banana linux

[tmds]

Banana Linux is a linux-x64 compatible flavor of Linux for which there is no rid assigned.

Perhaps one of the CI build's /etc/os-release ID can be changed to 'banana', to ensure source-build is buildable on an unknown linux-x64 distro?

@dotnet/source-build-contrib

[tmds]

@crummel @dagood we are often running into issues with newer versions of Fedora due to the rid being unknown.
The suggestion here is to change /etc/os-release as part of one of the ci builds to ensure source-build works on an unknown glibc linux-x64.
Can we do this?

cc @omajid

[adaggarwal]

@crummel Chris Rummel FTE @dagood Davis Goodin FTE we are often running into issues with newer versions of Fedora due to the rid being unknown.
The suggestion here is to change /etc/os-release as part of one of the ci builds to ensure source-build works on an unknown glibc linux-x64.
Can we do this?

I think that's how dotnet-install.sh determines the os. Incase we replace it, we might need a file/placeholder that would contain the RID?

[omajid]

I think that's how dotnet-install.sh determines the os. Incase we replace it, we might need a file/placeholder that would contain the RID?

I think dotnet-install.sh falls back to linux-x64 rid correctly.

For example, arch-x64 RID doesn't exist in the RID graph, but we get past dotnet-install.sh on Arch Linux just fine: https://github.com/dotnet/source-build/issues/1310.

[dagood]

I think that's how dotnet-install.sh determines the os. Incase we replace it, we might need a file/placeholder that would contain the RID?

It seems better to me to look at it from a dev's point of view: what do I expect source-build to do when it can't recognize my OS? My understanding is that the point of adding banana (which I think is a really good idea) is to find situations like that.

For the prebuilt SDK, I imagine I'll normally need linux-x64 (glibc). Then I'd probably want a non-portable build with my new RID in the right place in the RID graph that allows me to restore Microsoft assets if necessary for cross-RID publish, etc. Specifying those each time I want to build seems reasonable, because my OS is unknown (not mentioned in the sources). (Edit: Sounds like I may only need to specify the RID I want, or not even that.)

[crummel]

I like this idea, it'll keep us on top of new versions and unknown distros. It does seem like we'll probably have to soft-patch the RID graph though, unless something in the runtime changes to allow arbitrary unknown RIDs. We can try this out and see what all breaks.

[dagood]

we'll probably have to soft-patch the RID graph

I thought we were already doing that with:

https://github.com/dotnet/source-build/blob/be813dfe3b144fb6e389e07d8de3a7dbc92d0acb/repos/corefx.proj#L97-L107

[crummel]

Oh, I think IsBootstrapping went away when we switched from BuildTools to Arcade. That should be an easy fix though.

[tmds]

(Edit: Sounds like I may only need to specify the RID I want, or not even that.)

Yes. Source-build should figure out the target rid from /etc/os-release and assume a parent of linux-{arch} (on glibc Linux).
As optional parameters, the user may want to name the target rid (e.g. rhel.8 instead of rhel.8.1)
And specify a list of parent rids.

For this issue, I think we can focus on patching /etc/os-release as a pre-build CI step. And verify we get an sdk for the patched os-release rid.

[tmds]

I think we can focus on patching /etc/os-release as a pre-build CI step. And verify we get an sdk for the patched os-release rid.

@dagood @crummel can we do this?

[dagood]

The approach sounds good to me, if that's the question. I don't know where it sits in the backlog... seems weird that @leecow is assigned and it's in a closed ZenHub epic. @crummel do you know the background, should it be unassigned to retriage or something like that?

[crummel]

Yeah, we'll assign this to someone in triage. Thanks for the discussion around how to do this everyone!

[tmds]

@crummel @dagood having this CI build will catch rid issues that are now hitting us post-release.

[tmds]

And considering https://github.com/dotnet/source-build/issues/1428, we should build the source-tarball on _banana_ Linux, and verify it builds on _pineapple_ Linux.

[dagood]

Issues have come up with versions--stuff like banana-x64/banana.2-x64 vs. banana.2.9-x64 failing on this regex:

https://github.com/dotnet/source-build/blob/1eba1a673a160ae4635270fdcf823da4bfe66d3d/tools-local/tasks/Microsoft.DotNet.SourceBuild.Tasks.XPlat/PublishCoreSetupBinaries.cs#L26-L31

Concretely, @lupin-de-mid hit this issue with astra.1.6-x64. ~Also, the SDK seemed to parse os-release as astra.16-x64 at runtime, not sure if this is related (or a problem).~(edit: see below clarification) Context starting here: https://gitter.im/dotnet/source-build?at=5f21801125cae90e6d8afb48

I think this problem will repro on Ubuntu too (e.g. ubuntu.20.04-x64).

[lupin-de-mid]

Concretely, @lupin-de-mid hit this issue with astra.1.6-x64. Also, the SDK seemed to parse os-release as astra.16-x64 at runtime,

I workaround problem by changing /etc/os-release from VERSION=1.6 to VERSION=16. So RID in dotnet --info output is defined at compile time

[dagood]

I think this problem will repro on Ubuntu too (e.g. ubuntu.20.04-x64).

We actually run Ubuntu 18.04 CI, which doesn't hit this, so I'm wrong here. Will need to look deeper.

[dagood]

Something to consider for this issue is patching up the RID graph in the prebuilt/tooling SDK that we download in the tarball production build. It isn't necessary in all cases (the catch-all linux-x64 may work) but it's probably necessary for cases like alpine-x64 / linux-musl-x64. See https://github.com/dotnet/source-build/issues/1868. If we don't do this as part of resolving this issue, we should probably open a new issue to track support for that.

[tmds]

RHEL 9 is Banana Linux: we can't build .NET Core on it until the rids have rippled through :/

[omajid]

Also starting to see issues on the recently-forked Fedora 35 :(

[tmds]

Now that https://github.com/dotnet/runtime/pull/50818 is in.

Can we add a CI job to source-build that verifies we can build a source-tarball on _banana_ Linux, and use it to build .NET on _pineapple_ Linux?

The CI job can write to /etc/os-release to change the distro id.

@MichaelSimons @crummel @dseefeld can you take a look into this?

[tmds]

cc @ericstj

[tmds]

@MichaelSimons @crummel @dseefeld friendly reminder.

It would be great if we can close the circle by adding a CI job to verify source-build is no longer blocked by unknown rids.

[crummel]

I'm looking into this now, I'll hopefully have some x64 RHEL machines soon so I'm planning on making CI changes for that anyway.