Home: dotnet restore no longer works for VS solutions that contain a mix of SDK and non-SDK projects

[livarcocc]

_From @DaveSlinn on March 12, 2018 23:46_

Original issue was #7294 but prior to sdk version 2.1.100 that came out last week the VS 2017 15.6, dotnet restore would only display errors for projects it didn't understand. Now it quits altogether and as such, has broken our automated builds.

I ask again - why can't it just ignore non-SDK projects it finds in the VS .sln files?

Here is an image of the step in our build def that now stops the build:

And here is an image of the build working on a build PC without the 15.6 update (and thus, using dotnet sdk v 2.1.4):

_Copied from original issue: dotnet/cli#8774_

[livarcocc]

I am moving this to NuGet for them to take a look first.

As for why we can't ignore projects we don't know... It is because we don't know which projects we actually don't know about. For instance, natively, the CLI does not support UAP projects, however, with the addition of a PackageReference, everything works.

We don't whitelist supported projects because there is no way for that list to be authoritative.

[livarcocc]

Is this a change in NuGet behavior, where the error code is now being returned as non-zero?

[nkolev92]

Looking at this here, the difference between the 2 paths is that we don't do continue on error in the RestoreUseSkipNonexistentTargets= 'true' case.

Now I'd assume that's because SkipNonexistentTargets="true" should handle it.

It seems like we're missing a SkipNonexistentProjects in there (like in all other places we use SkipNonexistentTargets), but that didn't fix the problem.

Not sure if I understand this, correctly, but it's suggesting it's the expected behavior?

/// When this flag is present, the top level target(s) in the build request will be skipped if those targets
/// are not defined in the Project to build. This only applies to this build request (if another target calls
/// the "missing target" at any other point this will still result in an error).

//cc
@jeffkl

Might be related to https://github.com/NuGet/Home/issues/6651

The thing is that neither of this repros with pure msbuild, presumably because all imports work, MSB4019 is just that complaining about the imports.

[nkolev92]

Some more investigation:

:\Users\nkolev\Documents\Code\NuGet\NuGet.Client [dev-nkolev92-initialauthpluginwork ≡ +1 ~0 -0 !]> dotnet msbuild /t:_FilterRestoreGraphProjectInputItems C:\Users\nkolev\Source\Repos\ConsoleApp13\ConsoleApp13.sln /p:RestoreUseSkipNonexis
tentTargets="true" /bl
Microsoft (R) Build Engine version 15.6.82.30579 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

C:\Program Files\dotnet\sdk\2.1.100\MSBuild.dll /bl /Logger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,C:\Program Files\dotnet\sdk\2.1.100\dotnet.dll /m /p:RestoreUseSkipNonexistentTargets=true /t:_FilterRestoreGraphProjectInputItems /v:m C:\Users\nkolev\Source\Repos\ConsoleApp13\ConsoleApp13.sln
C:\Users\nkolev\Source\Repos\ConsoleApp13\Database1\Database1.sqlproj(57,3): error MSB4019: The imported project "C:\Program Files\dotnet\sdk\2.1.100\Microsoft\VisualStudio\v11.0\SSDT\Microsoft.Data.Tools.Schema.SqlTasks.targets" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.

Fails if I'm calling a different top level target from the failing one.

Works just fine if I call the failing target directly.

C:\Users\nkolev\Documents\Code\NuGet\NuGet.Client [dev-nkolev92-initialauthpluginwork ≡ +1 ~0 -0 !]> dotnet msbuild /t:_IsProjectRestoreSupported C:\Users\nkolev\Source\Repos\ConsoleApp13\ConsoleApp13.sln /p:RestoreUseSkipNonexistentTarget
s="true" /bl
Microsoft (R) Build Engine version 15.6.82.30579 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

C:\Program Files\dotnet\sdk\2.1.100\MSBuild.dll /bl /Logger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,C:\Program Files\dotnet\sdk\2.1.100\dotnet.dll /m /p:RestoreUseSkipNonexistentTargets=true /t:_IsProjectRestoreSupported /v:m C:\Users\nkolev\Source\Repos\ConsoleApp13\ConsoleApp13.sln

Basically the issue comes down to the fact that SkipNonExistentTargets ends up erroring when all imports cannot be resolved.

Is this the expected behavior?
@jeffkl @andygerlicher?
Anything we can do to mediate this?

[nkolev92]

We need to do further investigation to understand all the impact here.

We can potentailly revert to not using SkipNonexistentTargets, but we need to do some quick performance analysis, with and without the optimization.

/cc
@rrelyea
@mikeharder

[mikeharder]

@nkolev92: I should be able to run build perf tests pretty quickly, just let me know what you want me to measure.

[nkolev92]

@mikeharder Sounds great.

The idea here is that the "quick" fix for CLI would be not use the SkipNonexistentTargets optimization, and use the old ContinueOnError approach of discovering compatible projects.
That one is more fault tolerant, which is appropriate only for the CLI scenario.

The ask here is:

• Restore with RestoreUseSkipNonexistentTargets="true", which is the default currently,
  and
• Restore with RestoreUseSkipNonexistentTargets="false".

In my quick tests, I did not notice a perf degradation, but it'd be nice to have a more comprehensive run.

[nkolev92]

A duplicate of this was reported to mseng

[nkolev92]

We have analyzed this issue in depth. Here are our findings:

Old Behavior
When restoring a solution with nuget.exe 4.5 (or dotnet.exe from 2.1.4 sdk) restore used to ignore projects whose imports cannot be found, and print a warning such as:

C:\App1\App1.csproj : warning NU1503: Skipping restore for project 'C:\App1\App1.csproj'. The project file may be invalid or missing targets required for restore. [C:\App1\App1.sln]

For these particular projects, dotnet build would not work, due to the same missing imports. So restore's behavior was inconsistent compared to build in the way it treated "faulty" projects.

Effectively, dotnet.exe was never able to build correctly here. Whether it fails in restore or build is not particularly relevant. It's an unsupported scenario.

New Behavior
In nuget.exe 4.6 (or dotnet.exe 2.1.100), we made major perf strides (300% or more for large solutions of 40+ projects), by refactoring a healthy portion of the restore targets.
This also "regressed" the way NuGet was handling faulty projects, by now treating them the same way as build.

The impact of this regression is that users that did dotnet restore on a full solution, and ran build with a custom scripts now need to change their behavior.

Summary
The importance of the performance improvements outweighs that of the above mentioned scenario, so we have decided not to act on this immediately.

Note that this will happen only in dotnet.exe restore scenarios and not in msbuild (full framework), nuget.exe, or Visual Studio restore.
Variants where meta projects are used could be affected as well. (docker, which has since been addressed with a docker focus fix)

Workaround
Users can still revert to the old, more forgiving, behavior by setting RestoreUseSkipNonexistentTargets to "false" to dotnet restore. For example:

dotnet.exe restore /p:RestoreUseSkipNonexistentTargets="false"

//cc @rrelyea @NuGet/nuget-client

[nkolev92]

Closing this as per the above comment.

[lopezdp]

None of this is working... How can I restore after this error:

error MSB4019: The imported project "C:\Program Files\dotnet\sdk\2.1.602\Microsoft\VisualStudio\v16.0\WebApplications\Microsoft.WebApplication.targets" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.

[nkolev92]

@lopezdp What kind of a project is that?

msbuild /t:restore or nuget.exe restore is the approach for projects not supported by the sdk.

[lopezdp]

VS2019 neither of those commands worked either. Not sure what info to provide...

I can build from IDE but ultimately I am trying to dotnet publish so I can deploy minor changes to an existing application.

Any idea or resources you suggest I look at?

Thanks!

@lopezdp What kind of a project is that?

msbuild /t:restore or nuget.exe restore is the approach for projects not supported by the sdk.

[nkolev92]

What's the type of the project you created?
Does it have PackageReference?

How does it fail when you run msbuild /t:restore?
Do you have a repro project?

[lopezdp]

What's the type of the project you created?
Does it have PackageReference?

How does it fail when you run msbuild /t:restore?
Do you have a repro project?

msbuild : The term 'msbuild' is not recognized as the name of a cmdlet, function, script file, or operable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

I will add you as a collaborator on the project now to take a look. This is a new project that I was assigned and just trying to figure out how to run it to deploy a small change for now.

Here is the repo: https://github.com/lopezdp/BetweenSessions

[nkolev92]

Do you have VS installed on the machine? It's likely msbuild.exe. Try that too.

If you do have VS, open a developer command prompt and run the command there.

[lopezdp]

Do you have VS installed on the machine? It's likely msbuild.exe. Try that too.

If you do have VS, open a developer command prompt and run the command there.

Yes VS2019 is installed.

I tried with msbuild.exe and same error result observed.

Really want to run from PowerShell if possible.

Do you have any resources that you may recommend?

[nkolev92]

@lopezdp
VSWhere should help.
You can find the location of msbuild with it.

https://github.com/Microsoft/vswhere

[lopezdp]

Do you have VS installed on the machine? It's likely msbuild.exe. Try that too.

If you do have VS, open a developer command prompt and run the command there.

Did not work from Cmd prompt either inside of my VS2019 install. Any help would be great.

[lopezdp]

lol.

[PinkhamSoftware]

Currently experiencing this issue.

[christophedemey]

Also this issue for me.

[Condor2708]

Same for me

[empz]

Same here....

[Bosch-Eli-Black]

For me, doing dotnet restore on the entire folder, instead of the solution file, fixed the issue.

That is, this command fails:

dotnet restore "my-project\my-project.sln"

This command works:

dotnet restore "my-project\"

[mrmartan]

@nkolev92 @livarcocc

Well, this is a mess. MS keeps closing issues on this while the community keeps opening them. This one is like the fourth in a chain I went through.

AFAICT in many cases this boils down to projects you want included in your Solution but you are pretty fine they don't get built by CI. You want them in the solution because of the convenience it gives you when developing on your box. Some of such projects are non-SDK or rather non-cross-platform (like SSDT sqlproj) and simply running dotnet build in a folder containing your solution. This is a problem for multiple use-cases, CI is among them. While in CI you can workaround it by building specific projects (instead of the solution) it is not possible always.
As an example take SonarQube scanner. You have to run it in conjunction with building the solution as it uses the solution to get project IDs. I had to resort to providing a solution configuration that excludes incompatible projects from build.
Now imagine your team uses a mix of platforms. Some use Windows, some Linux, some Mac. While I have not tested this in Visual Studio for Mac, I know for a fact this breaks simple (VS Code) usage with dotnet build *.sln on Linux.

I can imagine reasons why you would want not to skip incompatible projects but I would like the option to tell the .NET SDK to treat them as warnings instead. Maybe a switch in global.json