Fsharp: Can't build using MSBuild.exe from VS 16.6.0

[csmager]

After I upgraded my VS 2019 Build Tools to 16.6.0, my projects no longer build. I'm using net461, but I've tried net48 and netcoreapp3.1 with the same result. They do build using dotnet build, but using MSBuild.exe fails - even with a new, empty project.

I created a new project, the .fsproj file looks like this:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net48</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="Library.fs" />
  </ItemGroup>
</Project>

And then build using C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\Bin\MSBuild.exe.

Expected behavior

Expected to build successfully.

Actual behavior

I get 3 identical errors: Could not load file or assembly 'System.Buffers, Version=4.0.3.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified..

The relevant portion of the verbose log looks like this:

0>Task "Fsc"
0>C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\FSharp\fsc.exe -o:obj\Debug\net48\BuildTest.dll
-g
--debug:portable
--noframework
--define:TRACE
--define:DEBUG
--define:NETFRAMEWORK
--define:NET48
--optimize-
--tailcalls-
-r:C:\Users\cmager\.nuget\packages\fsharp.core\4.7.1\lib\net45\FSharp.Core.dll
-r:C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.8\mscorlib.dll
-r:C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.8\System.Core.dll
-r:C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.8\System.Data.dll
-r:C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.8\System.dll
-r:C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.8\System.Drawing.dll
-r:C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.8\System.IO.Compression.FileSystem.dll
-r:C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.8\System.Numerics.dll
-r:C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.8\System.Runtime.Serialization.dll
-r:C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.8\System.Xml.dll
-r:C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.8\System.Xml.Linq.dll
--target:library
--warn:3
--warnaserror:3239,76
--fullpaths
--flaterrors
--subsystemversion:6.00
--highentropyva+
--targetprofile:mscorlib
--nocopyfsharpcore
--deterministic+
--simpleresolution
obj\Debug\net48\.NETFramework,Version=v4.8.AssemblyAttributes.fs
obj\Debug\net48\BuildTest.AssemblyInfo.fs
Library.fs

0>Microsoft (R) F# Compiler version 10.9.1.0 for F# 4.7
0>Copyright (c) Microsoft Corporation. All Rights Reserved.
0>
0>C:\dev\build-test\BuildTest\unknown(0,1): Error FS0193 : Could not load file or assembly 'System.Buffers, Version=4.0.3.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.
0>
0>C:\dev\build-test\BuildTest\unknown(0,1): Error FS0193 : Could not load file or assembly 'System.Buffers, Version=4.0.3.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.
0>
0>C:\dev\build-test\BuildTest\unknown(0,1): Error FS0193 : Could not load file or assembly 'System.Buffers, Version=4.0.3.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.
0>The command exited with code 1.
0>Done executing task "Fsc" -- FAILED.

Known workarounds

dotnet build works, though I can't use this as I have a mixed solution and my WPF projects won't build with this. Alternative is to downgrade to 16.5.x.

[abelbraaksma]

The system cannot find the file specified

If you check that file by hand, is it available? And perhaps it's an implicit dependency with a different version than another implicit dependency. I've had something similar with System.Memory (iirc, System.Collections.Generic.Immutable and System.Json linked different memory assemblies, since MSBuild doesn't use app.config, the workaround was to change the fsproj by hand to use SpecificVersion, and to use the fully qualified name, Nuget doesn't set this correctly).

[csmager]

I've actually reduced this even further. I just created a script file with printfn "Hello world" as its only content. If I run this:

C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\FSharp\fsc.exe test.fsx

Then I get this output:

yielding source C:\dev\build-test\test.fsx (failed parse)
Microsoft (R) F# Compiler version 10.9.1.0 for F# 4.7
Copyright (c) Microsoft Corporation. All Rights Reserved.

unknown(0,1): error FS0193: Could not load file or assembly 'System.Buffers, Version=4.0.3.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.

So it seems its the compiler that can't find this dependency rather than my project. I've checked and there's no System.Buffers.dll in C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\FSharp\.

[abelbraaksma]

The "failed parse" is very odd...

[Treit]

I am seeing this as well and as you noted it reproduces with a trivially simple 'Hello world' F# project. I notice that it builds fine in Visual Studio but not using msbuild.exe from the command line.

[abelbraaksma]

I've checked and there's no System.Buffers.dll in

As a workaround, did you try adding the file by hand (and possibly other missing refs)? What might also work is adding the dll to the GAC. This is system wide and could resolve such errors (but there are drawbacks to this approach).

If, after adding the file to the dir, you still get these errors, you may try to use an assemblyredirect in app.config.

Another cause for this error could be that fsc expects the dll on the search path for assemblies, and somehow this search path is different for MSBuild and VS. I can't investigate myself right now, but somehow, fsc needs to be able to find its references (at least until there's a fix).

cc @cartermp, @KevinRansom.

[cartermp]

The behavior about msbuild.exe vs dotnet build/dotnet msbuild isn't unexpected here - SDK-style projects use dotnet-style commands to build. This is also the case when you build from VS. This is because they are .NET SDK-style projects. @csmager did you recently convert your projects to the .NET SDK?

[KevinRansom]

Okay, System.Buffers was missed off from the BuildTools sku.

[KevinRansom]

These changes are missing from dev16.6 branch:
file source="$(BinariesFolder)\FSharp.Compiler.Private\$(Configuration)\$(TargetFramework)\System.Buffers.dll" file source="$(BinariesFolder)\FSharp.Compiler.Private\$(Configuration)\$(TargetFramework)\System.Memory.dll" file source="$(BinariesFolder)\FSharp.Compiler.Private\$(Configuration)\$(TargetFramework)\System.Numerics.Vectors.dll" file source="$(BinariesFolder)\FSharp.Compiler.Private\$(Configuration)\$(TargetFramework)\System.Reflection.Metadata.dll" file source="$(BinariesFolder)\FSharp.Compiler.Private\$(Configuration)\$(TargetFramework)\System.Resources.Extensions.dll" file source="$(BinariesFolder)\FSharp.Compiler.Private\$(Configuration)\$(TargetFramework)\System.Runtime.CompilerServices.Unsafe.dll" file source="$(BinariesFolder)\FSharp.Compiler.Private\$(Configuration)\$(TargetFramework)\System.Threading.Tasks.Dataflow.dll"

I believe this is the missing commit: https://github.com/dotnet/fsharp/pull/8939

[KevinRansom]

So this only impacts the Build Tools for Visual Studio 2019 for Visual Studio. The other sku's should be fine.

[csmager]

Thanks @KevinRansom. Guess it's a good thing it only affects Build Tools. For now I've manually copied the missing binaries into the Microsoft\FSharp folder and all is well. Glad this will be fixed in a later release.

[abelbraaksma]

Is it possible to add a unit test to capture such cases? MSBuild itself is not open source I believe,but maybe something as simple as art least trying to build a default project could catch the majority of such cases?

A lot of CI tools out there are still configured with MSBuild.

[NatElkins]

@abelbraaksma I think it is open source, assuming this is what gets shipped with VS https://github.com/Microsoft/msbuild

[Treit]

For anyone else hitting this, just to make the workaround unambiguous, this should do the trick:

copy "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\PublicAssemblies\*.dll" "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\FSharp"

[KevinRansom]

@abelbraaksma , it's really an integration issue, we will add a final integration test. When I said let's add a ci-leg to test it, I learned just how hard that stuff is, in no - uncertain terms.

[abelbraaksma]

@KevinRansom, wow, I didn't expect that. In my simple view of things, I'd imagine pulling latest master from MSBuild, and run it against a handful of sln and prj files. Check exit code and output dir.

That may not catch every possible bug, but it should catch extreme cases like this one. Ideally you'd do some integration test using MSBuild for all fsc tests you have out there, but I can imagine that'd be far more work.

But i must admit, I know little to nothing about how the CI is set up, let alone adding such scenarios. I assume the complexity has something to do with creating a clean environment for each run (tearup/teardown?), and pairing the versions between separate gits.

[KevinRansom]

@abelbraaksma , if you look at the fix it's over there in setup land. That's how the files get into visual studio. For the IDE we build a vsix over in vsintegration, that we can install into a visual studio, however, the bug here was the package we gave to visual studio to install for the build server didn't include some new dependencies the compiler took. There are two issues that are inherently tough to get right, creating and causing to be signed a new file, and adding dependencies. In this case we caught the issue in the IDE relatively early but the Build Server Sku, we missed, because we don't really dogfood it. Anyway, we can add something that will catch this if we mess up again, but it's not really the sort of thing we can catch here, without simulating the work of the setup team.

[abelbraaksma]

Ok, if I understand you correctly, that means that feeding back the setup is a no-go, and the setup team should add themselves at least a minimal test that shows that MSBuild works with a handful of project types.

Another thought is to add something along the lines of the service area tests: if a dependency gets added, a test will fail, requiring it to be added to those other locations as well. Even if such test is only informational, it'll work as a reminder.

[abelbraaksma]

FYI, now also reported on SO. Once a fix is out, we should update the answer there too: https://stackoverflow.com/questions/62085300/f-broken-in-vs-build-tools-2019-16-6-0/62085478

[Gh0stWalk3r]

For anyone else hitting this, just to make the workaround unambiguous, this should do the trick:

copy "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\PublicAssemblies\*.dll" "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\FSharp"

Anyone stumbling across this and not finding the PublicAssemblies folder -> use the Visual Studio IDE path instead to copy missing dlls from.
e.g. C:Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDEPublicAssemblies

[loop-evgeny]

Just wasted an hour on this. How has an issue like this, which totally breaks the product for anyone using F#, as I understand, been open for over a month and not fixed in the next hotfix?!

[abelbraaksma]

Is this not fixed in 16.7? I think it ought to be. @cartermp, I thought this was something for the setup team, being out of our hands kinda, do they have any follow up for this?

(I actually haven't updated my build servers, that rely on MSBuild, as I figured it would come with the next update)

[cartermp]

Note that VS 16.7 is not yet released. You can confirm the relevant files are there in that branch: https://github.com/dotnet/fsharp/blob/release/dev16.7/setup/Swix/Microsoft.FSharp.Compiler.MSBuild/Microsoft.FSharp.Compiler.MSBuild.csproj#L93

@loop-evgeny This doesn't break things for most F# developers. Only those using msbuild.exe and the F# tools from the VS Build Tools SKU. It will be fixed in the 16.7 update.

[ctollerud]

This is so disheartening. I'm not sure what's worse; the bug or the seeming indifference about it. It's these kinds of things that's going to prevent fsharp from ever becoming a serious contender.

[cartermp]

Closing as fixed