Roslyn: CS8603 shows up non-deterministically, leading to broken cloud builds

[AArnott]

Version Used:
VS 29229.7.master
SDK 3.0.100-preview8-013437

Steps to Reproduce:

git clone https://[email protected]/devdiv/DevDiv/_git/VS.RPC.Contracts
cd VS.RPC.Contracts\src
git checkout b0cde86
dotnet build

Expected Behavior:

See a build warning:

Utilities\DiscriminatedTypeJsonConverter`1.cs(191,20): error CS8603: Possible null reference return. [E:\A_work\177\s\src\Microsoft.VisualStudio.RpcContracts\Microsoft.VisualStudio.RpcContracts.csproj]

We want to see this consistently in local builds and in Azure Pipelines.

Actual Behavior:

No warning in local builds.

We only see the expected warning/error on Azure Pipelines.

Note we constrain the SDK version with a global.json file so we really should see consistent behavior in both places.

[AArnott]

CC: @CharlesWillis3

[gafter]

I wonder if this could be due to a different compiler being used in dotnet build and in Visual Studio.

[AArnott]

Maybe, I don't know. But it doesn't repro locally whether I'm in VS, or use dotnet build or msbuild at the command line. It seems to only work properly in AZP.

[AArnott]

This is showing up again in this build. It doesn't show up locally for me, but the Azure Pipelines build produces the build warning-as-error.

What's particularly interesting about this one is its relation to https://github.com/dotnet/coreclr/pull/26521 (and more particularly #36591), so supposedly this should work because object.Equals(object) returns true before the rest of this expression is evaluated so the arg should be considered non-null. So it seems the compiler's special handling may be non-deterministic or otherwise faulty.

[jaredpar]

@AArnott can you determine the version of the compiler in both scenarios? Knowing the version of the SDK is sufficient (we can track it to a compiler).

Given that the failure is deterministic in both places it's likely just a behavior difference between the builds.

[AArnott]

We pin the .NET Core SDK version to be 3.0.100-preview8-013437. So it should theoretically be the same in AZP and locally.

[jaredpar]

@AArnott

We pin the .NET Core SDK version to be 3.0.100-preview8-013437. So it should theoretically be the same in AZP and locally.

That doesn't appear to be the case though. In the AZP build you're using desktop msbuild, not the dotnet CLI.

C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\msbuild.exe /nologo /bl:E:\A_work\292\a/build_logs/msbuild.binlog ...

That means on the AZP machine you're getting the csc.exe from the Visual Studio installation, not the the .NET SDK.

[AArnott]

That doesn't appear to be the case though. In the AZP build you're using desktop msbuild, not the dotnet CLI.

I knew that using the installed msbuild.exe meant I was using some of the installation, but I thought it at least got some tools and import files from the SDK I prescribe. Is that not the case?

Anyway, I don't have a choice on using msbuild.exe because I need to sign the files and the MicroBuild signing msbuild task doesn't work on dotnet build yet.

Do you have any suggestion on how to resolve that? I've already made the ask to the MicroBuild team that they support MSBuild Core. I'll go ping them on that again in the meantime.

[AArnott]

Also: why does it work when we build locally in VS then? Wouldn't the local VS use the same msbuild.exe as the Azure Pipelines msbuild.exe used?

[jaredpar]

I knew that using the installed msbuild.exe meant I was using some of the installation, but I thought it at least got some tools and import files from the SDK I prescribe. Is that not the case?

@nguerrera can answer the specifics here better than I can for tasks / targets.

As for the compiler though when you build with the desktop msbuild you get the desktop csc and when you build with the CLI you get the csc that ships with the selected .NET SDK.

[AArnott]

OK, it may be that it doesn't repro locally because we have IntPreview installed (with C# 3.4) while the VSEng-MicroBuildVS2019 hosts have C# 3.2 installed. So ours have the same fixes as the SDK we've pinned, and we thus only see the errors when running on msbuild.exe from the "older" agents.