Sdk: Regression: self-contained publish fails on Linux ARM in 3.0 Preview 7

[mthalman]

Steps to reproduce

1. dotnet new web --framework netcoreapp3.0
2. dotnet publish -r linux-arm -c Release -o out

Expected behavior

Publish succeeds

Actual behavior

An exception occurs: System.ArgumentException: Can not add property libclrjit.so to Newtonsoft.Json.Linq.JObject. Property with the same name already exists on object.

 Microsoft (R) Build Engine version 16.3.0-preview-19329-01+d31fdbf01 for .NET Core
 Copyright (C) Microsoft Corporation. All rights reserved.

   Restore completed in 68.02 ms for /app/app.csproj.
   You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018: The "GenerateDepsFile" task failed unexpectedly. [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018: System.ArgumentException: Can not add property libclrjit.so to Newtonsoft.Json.Linq.JObject. Property with the same name already exists on object. [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Newtonsoft.Json.Linq.JObject.ValidateToken(JToken o, JToken existing) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Newtonsoft.Json.Linq.JContainer.InsertItem(Int32 index, JToken item, Boolean skipParentCheck) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Newtonsoft.Json.Linq.JObject.InsertItem(Int32 index, JToken item, Boolean skipParentCheck) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Newtonsoft.Json.Linq.JContainer.AddInternal(Int32 index, Object content, Boolean skipParentCheck) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Newtonsoft.Json.Linq.JContainer.Add(Object content) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Microsoft.Extensions.DependencyModel.DependencyContextWriter.WriteAssetList(IEnumerable`1 runtimeFiles) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Microsoft.Extensions.DependencyModel.DependencyContextWriter.AddAssets(JObject libraryObject, String key, RuntimeAssetGroup group) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Microsoft.Extensions.DependencyModel.DependencyContextWriter.WriteTargetLibrary(Library library) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Microsoft.Extensions.DependencyModel.DependencyContextWriter.<WriteTarget>b__7_0(Library library) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at System.Linq.Enumerable.SelectArrayIterator`2.MoveNext() [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Newtonsoft.Json.Linq.JContainer.AddInternal(Int32 index, Object content, Boolean skipParentCheck) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Newtonsoft.Json.Linq.JContainer.Add(Object content) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Newtonsoft.Json.Linq.JObject..ctor(Object content) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Microsoft.Extensions.DependencyModel.DependencyContextWriter.WriteTarget(IReadOnlyList`1 libraries) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Microsoft.Extensions.DependencyModel.DependencyContextWriter.WriteTargets(DependencyContext context) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Microsoft.Extensions.DependencyModel.DependencyContextWriter.Write(DependencyContext context) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Microsoft.Extensions.DependencyModel.DependencyContextWriter.Write(DependencyContext context, Stream stream) [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Microsoft.NET.Build.Tasks.GenerateDepsFile.WriteDepsFile(String depsFilePath) in /_/src/Tasks/Microsoft.NET.Build.Tasks/GenerateDepsFile.cs:line 173 [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Microsoft.NET.Build.Tasks.GenerateDepsFile.ExecuteCore() in /_/src/Tasks/Microsoft.NET.Build.Tasks/GenerateDepsFile.cs:line 190 [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute() in /_/src/Tasks/Common/TaskBase.cs:line 38 [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Microsoft.NET.Build.Tasks.GenerateDepsFile.Execute() in /_/src/Tasks/Microsoft.NET.Build.Tasks/GenerateDepsFile.cs:line 182 [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/app/app.csproj]
 /usr/share/dotnet/sdk/3.0.100-preview7-012769/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(168,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/app/app.csproj]
 The command '/bin/sh -c dotnet publish -r $rid -c Release -o out' returned a non-zero code: 1

Environment data

This issue was discovered in our CI validation tests for .NET Docker when running the test scenario on Linux ARM hardware.

This is a regression and was introduced between the following two SDK versions: 3.0.100-preview7-012743 and 3.0.100-preview7-012769.

[livarcocc]

And @peterhuene another one. Can you please give some priority to this one, as this might be a blocking issue?

[peterhuene]

I don't have an ARM device to repro with, so I will need a binlog to investigate. I suspect this might be related to recent changes @dsplaisted made.

@mthalman would you be able to provide a binlog from the dotnet publish command by passing the /bl option and emailing me the resulting MSBuild.binlog file? Thanks!

[peterhuene]

Never mind, I was able to reproduce just fine with a cross-build. I'll investigate.

[peterhuene]

There are two entries in the RuntimeList.xml for libclrjit.so:

<File Type="Native" Path="runtimes/linux-arm/native/libclrjit.so" FileVersion="0.0.0.0" />
<File Type="Native" Path="runtimes/x64_arm/native/libclrjit.so" FileVersion="0.0.0.0" />

@dagood is this a known issue with the arm runtime manifests? Note the weird "RID" of x64_arm.

Obviously there's a bug here in the SDK to be more robust against duplicates in the manifest, but this should be corrected in the manifest first to unblock the scenario.

[peterhuene]

I can confirm that removing the duplicate moves past the failure to generate the deps file.

[dagood]

My expectation would be for linux-arm to be used. x64_arm is for running crossgen cross-target (https://github.com/dotnet/core-setup/issues/5969). That doesn't mean it has to be in the runtime list in particular, but that's why it's in the package with that path. To unblock Preview 7 I'll prep a workaround in Core-Setup to remove it from the list.

/cc @wtgodbe

[peterhuene]

Thanks! I'd like to keep this bug assigned to me in the CLI so that we can make this task more robust, but I don't think that needs to be addressed for preview7.

Would you like me to file a core-setup bug?

[dagood]

Thanks! I'd like to keep this bug assigned to me in the CLI so that we can make this task more robust, but I don't think that needs to be addressed for preview7.

Sounds good. It might actually end up being right to keep it out of the list since it's tooling, not something that ships with self-contained apps--not sure how focused that file is intended to be. Or some extra attribute to avoid putting the logic in the SDK to filter out stuff that doesn't match the RID.

Would you like me to file a core-setup bug?

Yes please. 😄

[dsplaisted]

The runtime list is supposed to be the list of files that will be included for self-contained apps. So it should not include the x64_arm file in this case.

I think the SDK should fail more gracefully here, but I don't think it should try to choose the best file to use in a case like this.

[peterhuene]

Opened dotnet/core-setup#7039.

[dagood]

I merged https://github.com/dotnet/core-setup/pull/7042 into Core-Setup release/3.0 to fix this for preview 7. (FYI for those on this thread.)

[peterhuene]

Both the fix to the RuntimeList.xml and for better handling in the .NET Core SDK when there is a missing or invalid RuntimeList.xml have been merged.