source-build 'dotnet restore' fails for platform rid

[tmds]

repro:

$ dotnet new console --no-restore
$ dotnet restore -r fedora.30-x64

Using Microsoft 3.0.100-preview8-013656:

  Restore completed in 16.81 ms for /tmp/console2/console2.csproj.

Using source-build 3.0.100-preview8-013656 sdk on Fedora 30:

error NU1101: Unable to find package Microsoft.NETCore.App.Runtime.fedora.30-x64. No packages exist with this id in source(s): nuget.org

The source-build sdk doesn't fall back to the linux-x64 rid.

project.assets.json diff:

         "downloadDependencies": [
           {
             "name": "Microsoft.AspNetCore.App.Runtime.linux-x64",
             "version": "[3.0.0-preview8.19405.7, 3.0.0-preview8.19405.7]"
           },
           {
-            "name": "Microsoft.NETCore.App.Runtime.linux-x64",
+            "name": "Microsoft.NETCore.App.Runtime.fedora.30-x64",
             "version": "[3.0.0-preview8-28405-07, 3.0.0-preview8-28405-07]"
           }
         ],
-        "frameworkReferences": {
-          "Microsoft.NETCore.App": {
-            "privateAssets": "all"
-          }
-        },
-        "runtimeIdentifierGraphPath": "/home/tmds/Downloads/dotnet-3.0-p8/sdk/3.0.100-preview8-013656/RuntimeIdentifierGraph.json"
+        "frameworkReferences": [
+          "Microsoft.NETCore.App"
+        ]
       }
     },
     "runtimes": {

CC @omajid @crummel @dagood

[tmds]

If I edit ./sdk/3.0.100-preview8-013656/Microsoft.NETCoreSdk.BundledVersions.props, and remove fedora.30-x64 from:

    <KnownFrameworkReference Include="Microsoft.NETCore.App"
                              TargetFramework="netcoreapp3.0"
                              RuntimeFrameworkName="Microsoft.NETCore.App"
                              DefaultRuntimeFrameworkVersion="3.0.0-preview8-28405-07"
                              LatestRuntimeFrameworkVersion="3.0.0-preview8-28405-07"
                              TargetingPackName="Microsoft.NETCore.App.Ref"
                              TargetingPackVersion="3.0.0-preview8-28405-07"
                              RuntimePackNamePatterns="Microsoft.NETCore.App.Runtime.**RID**"
                              RuntimePackRuntimeIdentifiers="fedora.30-x64;linux-arm;linux-arm64;linux-musl-arm64;linux-musl-x64;linux-x64;osx-x64;rhel.6-x64;tizen.4.0.0-armel;tizen.5.0.0-armel;win-arm;win-arm64;win-x64;win-x86"
                              IsTrimmable="true"
                              />

it works.

[dagood]

I would guess this is a portable vs. non-portable issue. I wouldn't expect fallback for the framework packs because they aren't restored using PackageReferences.

The KnownFrameworkReference expects a fedora.30-x64 runtime pack based on Microsoft.NETCore.App.Internal's runtime.json (saw this for the first time now, interesting dependency...), but one apparently isn't available.

Maybe there's some non-portable build magic that still needs to be removed?

[crummel]

Yeah, this looks like we might have missed somewhere we removed non-portableness. I'm looking into this now.

[dagood]

I wouldn't expect fallback for the framework packs because they aren't restored using PackageReferences.

I phrased this poorly, just to clarify, I mean that I wouldn't expect fallback once the SDK decides what it wants and tells NuGet to download it. The SDK performing fallback when the target RID isn't specified in RuntimePackRuntimeIdentifiers sounds right.

[tmds]

For my education, is the Microsoft.NETCore.App.Runtime.xxx package what is needed to build self-contained applications?
Probably we could - at some point, not 3.0 - build this from source-build?

[tmds]

I got a build from @omajid where this is now working because of the switch to portable.

This means the using linux-x64 in some places. linux-x64 means: works on a lot of Linuxes. This may no longer be the case. For example, the packs/Microsoft.NETCore.App.Host.linux-x64 may not be really linux-x64.

@omajid mentioned build is switching to portable because of ASP.NET Core integration. I don't see how ASP.NET Core and portable are linked, because afaik ASP.NET Core is all managed code?

[dagood]

For my education, is the Microsoft.NETCore.App.Runtime.xxx package what is needed to build self-contained applications?
Probably we could - at some point, not 3.0 - build this from source-build?

Yep, design is over here: https://github.com/dotnet/designs/pull/50.

Source-build does produce this package, along with the other packs. Here's the list of what Core-Setup produced in a recent CI run:

  New NuGet package(s) after building core-setup:
    -> Microsoft.DotNet.PlatformAbstractions 3.0.0-preview8-28405-07
    -> Microsoft.Extensions.DependencyModel 3.0.0-preview8-28405-07
    -> Microsoft.NET.HostModel 3.0.0-preview8-28405-07
    -> Microsoft.NETCore.App 3.0.0-preview8-28405-07
    -> Microsoft.NETCore.App.Host.linux-x64 3.0.0-preview8-28405-07
    -> Microsoft.NETCore.App.Internal 3.0.0-preview8-28405-07
    -> Microsoft.NETCore.App.Ref 3.0.0-preview8-28405-07
    -> Microsoft.NETCore.App.Runtime.linux-x64 3.0.0-preview8-28405-07
    -> Microsoft.NETCore.DotNetAppHost 3.0.0-preview8-28405-07
    -> Microsoft.NETCore.DotNetHost 3.0.0-preview8-28405-07
    -> Microsoft.NETCore.DotNetHostPolicy 3.0.0-preview8-28405-07
    -> Microsoft.NETCore.DotNetHostResolver 3.0.0-preview8-28405-07
    -> NETStandard.Library.Ref 2.1.0-preview8-28405-07
    -> runtime.linux-x64.Microsoft.NETCore.App 3.0.0-preview8-28405-07
    -> runtime.linux-x64.Microsoft.NETCore.DotNetAppHost 3.0.0-preview8-28405-07
    -> runtime.linux-x64.Microsoft.NETCore.DotNetHost 3.0.0-preview8-28405-07
    -> runtime.linux-x64.Microsoft.NETCore.DotNetHostPolicy 3.0.0-preview8-28405-07
    -> runtime.linux-x64.Microsoft.NETCore.DotNetHostResolver 3.0.0-preview8-28405-07

[crummel]

I don't see how ASP.NET Core and portable are linked, because afaik ASP.NET Core is all managed code?

You can see the original discussion here, but the short version is that the core runtime and the ASP.NET runtime are required to have the same RID. ASP.NET doesn't have a non-portable build, because it is all managed code, and this would essentially be adding a new feature from them that nobody except source-build would use. This change actually brings us more in line with the official build, which uses a portable runtime even with the non-portable SDK.

[tmds]

Source-build does produce this package, along with the other packs. Here's the list of what Core-Setup produced in a recent CI run:

Nice. Is it the plan to include this in the dotnet installation? Then source-build sdk could do self-contained without nuget.org involvement.

PS: this is definitely a package that shouldn't be source-built under the linux-x64 rid.

You can see the original discussion here, but the short version is that the core runtime and the ASP.NET runtime are required to have the same RID.

Can you give an example of something (some code/project file) that is expecting this?

[crummel]

Sure, core-sdk expects them to be the same here. I tried overriding this in a couple different ways and did manage to get things building, but I still had failures at runtime due to ASP.NET expecting to live in same folder as the core runtime here. That was when I looked into how the official build was doing this and discovered it was using the portable runtime all the time.

[dagood]

Nice. Is it the plan to include this in the dotnet installation? Then source-build sdk could do self-contained without nuget.org involvement.

Unfortunately this isn't supported in the current design: https://github.com/dotnet/source-build/issues/1046#issuecomment-496356312.

linux-x64 means: works on a lot of Linuxes. This may no longer be the case. For example, the packs/Microsoft.NETCore.App.Host.linux-x64 may not be really linux-x64.

PS: this is definitely a package that shouldn't be source-built under the linux-x64 rid.

I don't understand these comments... is there a reason that the Microsoft portable build would support more Linuxes than a source-build portable build? Why shouldn't we build a portable linux-x64 runtime pack during source-build?

[omajid]

I don't understand these comments... is there a reason that the Microsoft portable build would support more Linuxes than a source-build portable build? Why shouldn't we build a portable linux-x64 runtime pack during source-build?

I think this is about implicit guarantees of what linux-x64 means, and not about source-build vs Microsoft's builds as such.

Correct me if I am wrong, but Microsoft builds the portable build on a RHEL 7 machine. That uses a certain version of glibc to build .NET Core. When running, .NET Core can use any feature from that glibc version. glibc is backwards compatible: all Linux distributions that have a newer version of glibc (than RHEL 7) will be able to run this build of .NET Core. So we can run a linux-x64 SDK on a newer distribution.

Building on a more recent platform - say Fedora 30, or Debian 10 - means that .NET Core may (will?) use glibc features only present in a newer version of glibc. In other words an SDK built on these newer platforms require the new versions of glibc to run. So if we build .NET Core on Debian 10, it will claim to be linux-x64 but not actually work on older linux-x64 distributions such as RHEL 7 and Debian 8.

As I understand it, @tmds 's concern is that we can build source-build on two different platforms, both will produce a "portable" linux-x64 SDK but because of the glibc versions differences neither build may really support linux-x64.

For self-contained applications, will this work at all? If someone builds "portable" linux-x64 SDK on Debian 10, then uses that for building self-contained linux-x64 applications, will they work on RHEL 7?

[tmds]

Sure, core-sdk expects them to be the same here. I tried overriding this in a couple different ways and did manage to get things building, but I still had failures at runtime due to ASP.NET expecting to live in same folder as the core runtime here. That was when I looked into how the official build was doing this and discovered it was using the portable runtime all the time.

Thanks for those references. Is it about rid-named folders mostly?

Why shouldn't we build a portable linux-x64 runtime pack during source-build?

If you build binaries you link with what you have on the system unless that has been taken care of specifically (e.g. openssl 1.0 vs 1.1). glibc can be especially tricky because it has different versions for symbols. afaik Microsoft has been building linux-x64 on rhel.7 because that has the oldest version of glibc. If you build on a newer os, you may require some symbol version which aren't present on an os with an older glibc version.

[crummel]

I honestly stopped looking into it after my first couple workarounds failed and I found out how the official build was doing it. We didn't think this would cause any issues, now that we know it's a bit more problematic, I can go back and try a few more things after I finish up the prebuilt stuff I'm currently doing.

[dagood]

Thanks, wasn't aware of this dependency, very interesting. Building non-portable builds/RIDs certainly seems like the clearest thing to do. I agree that it can't be put anywhere accessible in the end product as linux-x64.

(They're built on CentOS 7, by the way: https://github.com/dotnet/core-setup/blob/c947763d1fa6e935086a35d81bd1e7f5ef750b93/azure-pipelines.yml#L100, https://github.com/dotnet/coreclr/blob/0bbc89c7279a2655e8cfe8466eb69d85899033a8/eng/platform-matrix.yml#L143.)

[wfurt]

There are two aspects. binary compatibility with given distribution. glibc, curl and openssl are some example where compatibility may be problem. But there is also the RID graph and package selection where build may or may not be portable at .NET layer.
The portable flag impacts how packages are built and what we see as compatible.

[tmds]

I can go back and try a few more things after I finish up the prebuilt stuff I'm currently doing.

@crummel have you had the chance to look into this?

[crummel]

I can confirm the issue: built on Fedora 31, running on RHEL7.7 I get:

../dotnet: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ../dotnet)
../dotnet: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by ../dotnet)
../dotnet: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ../dotnet)

So two issues:

• Can we patch this for now?
  
  
  
  • This will take patches across the ASP.NET repos, core-sdk, and possibly core-setup. For now I've gone back to the ASP.NET PR. I haven't made a lot of progress on the differing RIDs (and I'm not sure if it will cause other problems) so I'm attempting a non-portable ASP.NET runtime right now.
  
  
• What's the correct long-term fix for this?
  
  
  
  • cc @nguerrera, @JunTaoLuo, @ericstj, @dagood
  
  
  • ~Can we just require building on a lowest-common-denominator OS?~
  
  
  • Is there any reason a portable ASP.NET runtime can't or shouldn't work with a non-portable core runtime?
  
  
  • It's not great to build an ASP.NET non-portable runtime that nobody else uses, but is that the easiest way out?
  
  

[omajid]

Can we just require building on a lowest-common-denominator OS?

I am not sure I understand this question. If we are using source-build to build a package that should become (say) a part of Fedora 31, how do we ask people to build on a different OS?

[crummel]

I guess that's true, we can't target the non-portable SDK to the correct RID while we're building on a different OS. Scratching that one.

[dagood]

It's not great to build an ASP.NET non-portable runtime that nobody else uses, but is that the easiest way out?

Based on what I've learned in this thread, this seems to me to be the right thing to do, that it's literally impossible to make a linux-x64 that is as compatible as the Microsoft linux-x64 because the old version of libraries aren't available to compile against when source-building for a new distro.

It seems like if you want to publish something that will run across, say, Fedora 31, 32, and 33, you have to get access to a 31 distro (machine, VM, Docker container...) then use its source-built .NET Core SDK to publish a fedora.31-x64 app that runs on all later binary-compatible versions. Or you download a runtime pack that has the native bits pre-compiled by Microsoft or otherwise.

[dagood]

I think I crossed some wires... the above is my understanding why .NET Core needs to be built for a specific RID, but it seems possible that since ASP.NET Core is all managed code, it could still be built as linux-x64.

[tmds]

Based on what I've learned in this thread, this seems to me to be the right thing to do, that it's literally impossible to make a linux-x64 that is as compatible as the Microsoft linux-x64 because the old version of libraries aren't available to compile against when source-building for a new distro.
It seems like if you want to publish something that will run across, say, Fedora 31, 32, and 33, you have to get access to a 31 distro (machine, VM, Docker container...) then use its source-built .NET Core SDK to publish a fedora.31-x64 app that runs on all later binary-compatible versions. Or you download a runtime pack that has the native bits pre-compiled by Microsoft or otherwise.

Exactly!
You can split this in two main use-cases:

• target linux-x64 (Microsoft built) to work across a range of distros
• target the platform rid, which is why https://github.com/dotnet/cli/issues/12325 makes sense

say, Fedora 31, 32, and 33, you have to get access to a 31 distro

I wouldn't consider this a real use-case.

but it seems possible that since ASP.NET Core is all managed code, it could still be built as linux-x64.

Yes, afaik ASP.NET Core is even the any rid, that is: no rid at all.

[dagood]

ASP.NET Core is all managed code

Oh right, this is actually not completely true as far as runtime packs and shared framework go. The build runs crossgen to create native images in the DLLs--I'm not sure if those have the same compatibility requirements as the base runtime.

If they do, I can imagine someone wanting to get the Microsoft-built ASP.NET Core runtime pack, and they might be unable to (by normal means) when source-build gives them a linux-x64 runtime pack that can't be distinguished from the one on nuget.org.

say, Fedora 31, 32, and 33, you have to get access to a 31 distro

I wouldn't consider this a real use-case.

Makes sense to me, does seem against the point of building from source in the first place.

[nguerrera]

Catching up here as I've been away. Expect more replies.

If I edit ./sdk/3.0.100-preview8-013656/Microsoft.NETCoreSdk.BundledVersions.props, and remove fedora.30-x64 from:

That line indicates which runtime packs will be available from nuget feeds (at this time runtime packs always come from nuget), so we should not add the source build RID to it unless there is some work done to make it available.

[nguerrera]

The build runs crossgen to create native images in the DLLs--I'm not sure if those have the same compatibility requirements as the base runtime.

@fadimounir Does crossgen emit anything differently if built for a specific distro? I don't think it does, but I want to be sure.

[tmds]

The build runs crossgen to create native images in the DLLs--I'm not sure if those have the same compatibility requirements as the base runtime.

Ah, yes, crossgen will make this linux-x64.

does seem against the point of building from source in the first place.

source-build makes things compatible at the source code level. backwards binary compatibility is a much harder problem.

[fadimounir]

@nguerrera crossgen should emit the same image for different distros. Running crossgen however requires the correct crossgen/distro compatibility. For instance, crossgen that runs on linux-musl can't run on linux. But the output R2R image should be the same

[crummel]

I've opened https://github.com/dotnet/source-build/pull/1221 to revert to building the non-portable core runtime. This still uses the portable ASP.NET runtime.

[crummel]

https://github.com/dotnet/source-build/pull/1221 has been merged, we're back to a non-portable core runtime by default.

[omajid]

This might be a silly question, but dotnet restore -r rhel.8.1-x64 fails on my end. Is that expected?

[tmds]

No. It should fall back to linux-x64.

[omajid]

Okay, then this sounds like a bug:

$ dotnet new console --no-restore
The template "Console Application" was created successfully.
$ dotnet restore -r rhel.8.1-x64
/dotnet-regular-tests/restore-with-rid/restore-with-rid.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Runtime.rhel.8-x64. No packages exist with this id in source(s): 0, nuget.org
  Restore failed in 686.31 ms for /dotnet-regular-tests/restore-with-rid/restore-with-rid.csproj.

[tmds]

Can you check if rhel-8 is in the RuntimePackRuntimeIdentifiers (see https://github.com/dotnet/source-build/issues/1202#issuecomment-525735256)?
If it's there, that's the cause of the bug.

[omajid]

It's there:

    <KnownFrameworkReference Include="Microsoft.NETCore.App"                                                         
                              TargetFramework="netcoreapp3.0"                                                        
                              RuntimeFrameworkName="Microsoft.NETCore.App"                                           
                              DefaultRuntimeFrameworkVersion="3.0.0-preview9-19423-09"                               
                              LatestRuntimeFrameworkVersion="3.0.0-preview9-19423-09"                                
                              TargetingPackName="Microsoft.NETCore.App.Ref"                                          
                              TargetingPackVersion="3.0.0-preview9-19423-09"                                         
                              RuntimePackNamePatterns="Microsoft.NETCore.App.Runtime.**RID**"                        
                              RuntimePackRuntimeIdentifiers="linux-arm;linux-arm64;linux-musl-arm64;linux-musl-x64;linux-x64;osx-x64;rhel.6-x64;rhel.8-x64;tizen.4.0.0-armel;tizen.5.0.0-armel;win-arm;win-arm64;win-x64;win-x86"         
                              IsTrimmable="true"                                                                     
                              />          

Surprisingly, rhel.7-x64 isn't there, but both rhel.6-x64 and rhel.8-x64 are.

[tmds]

source-build adds it.
Portable build masked the issue, but the root cause wasn't fixed.
rhel-7 isn't there because that isn't the rid that gets built.
rhel-6 is there because it has its own package, which is different from linux-x64.