Msbuild: IncrementalClean deletes transitively-acquired content in Visual Studio

[rainersigwald]

[This is based on feedback from connect.microsoft.com.]

If you have:

• A project that copies some files to the output directory (call it ClassLibraryIndirect)
• A project that depends on ClassLibraryIndirect (call it ClassLibraryDirect)
• A project that depends on ClassLibraryDirect (call it IncrementalClean)

And you build using Visual Studio, some very surprising behavior can occur.

Demo projects: https://github.com/rainersigwald/IncrementalCleanDemo

On the first (full) build of the main project, the output file is present in ClassLibraryIndirect's bin directory, and in IncrementalClean's--as expected. Content files should be transitively passed along. Selected state.

If you make a change to the main project (to get VS's fast-up-to-date check to cause an actual build) and build just it again, the content file gets deleted from the main project's bin directory. changes

Looking through a diag-level log of the second build, the problem is that the IncrementalClean target deletes the file, because it incorrectly thinks it's no longer an output.

This does not happen when building from the command line using no special properties.

But it does happen when building from the command line setting the property BuildProjectReferences=false, which simulates Visual Studio's mechanism of building individual projects.

Whether or not a project is built in isolation should not affect the outputs it produces.

[rainersigwald]

The proximate cause of this is that the transitive GetCopyToOutputDirectoryItems target doesn't include the indirect content file in the latter build, while it does in the initial, successful build as well as the successful full command-line build.

There's a very confusing comment for GetCopyToOutputDirectoryItems that indicates that it isn't transitive--but clearly sometimes it is.

[kzu]

We're hitting this too, trying to improve the way we perform the Pdb>Mdb in-place, and not getting them copied transitively :(

[abbottbd]

+1 this has been a thorn in our side at work for ages!

[aswinraaj]

We are getting the same problem when using sonarqube for performing static code analysis which creates temporary files . Incremental clean deletes it before the entire solution is even completed building

[jcouv]

+1 I'm hitting this daily in my inner development loop.

[angularsen]

We worked around this by moving all content files from each individual library, where we feel they naturally belong, to dumping all of them in our main app project. It's not pretty, but it works. We have spent way too much time investigating this and very frustrated to see the age of this problem and no update on fixing this.

[KirillOsenkov]

We've been hitting this as well and the easy workaround we found is to add:

<Target Name="IncrementalClean" />

to a .targets file that's included in all projects. Disabling incremental clean is mostly harmless since we regularly do git clean -xdff anyway. A decent workaround.

[rainersigwald]

Another workaround is to flatten your dependency tree: ensure that there are direct ProjectReferences instead of relying on transitive copying behavior.

[gulbanana]

I can confirm that I was extremely surprised by this behaviour. The symptom is a failure to pack, since the PDB files which have been deleted aren't present as an input to nuget, which the pack targets require even if you aren't building a symbol package..

[gulbanana]

Actually I'm not totally sure it's the same issue, but it seems very similar. My problem is the same one that's described in this StackOverflow question: https://stackoverflow.com/questions/44175371/pdb-file-is-being-removed-by-target-incrementalclean

[herrgruebner]

2018 and still going on. Was a fun one to track down.

[herrgruebner]

In our case we were using msbuild as part of a powershell script with the 'clean' parameter. We put a bit at the start of the script to blow away the output directory and got rid of the clean, which isn't great, but it works.

[akoessler]

Is there any plan to fix this issue in the near future?

[cclauson]

Just encountered this, is bugging some people at my organization, would love it if it were fixed! :)

[cclauson]

Hello,

I spent some time debugging this in structured log viewer, and eventually reached the same conclusion that is described in this comment mentioned by @rainersigwald :
https://github.com/Microsoft/msbuild/blob/e7019886b3fd520b547a7353d78d106d72a4b828/src/XMakeTasks/Microsoft.Common.CurrentVersion.targets#L4091-L4100
(should have read this at the beginning...)

Succinctly, the target _SplitProjectReferencesByFileExistence consumes @(ProjectReferenceWithConfiguration), which is computed in the target AssignProjectConfiguration. The bug occurs when _SplitProjectReferencesByFileExistence is run without first running AssignProjectConfiguration.

As a workaround, I'm currently putting the following code in a file which is imported by all my projects:
<Target Name="ForceAssignProjectConfigurationBeforeSplitProjectReferencesByFileExistence_KLUDGE" BeforeTargets="_SplitProjectReferencesByFileExistence" DependsOnTargets="AssignProjectConfiguration" />

Am not noticing any problems or performance degradation. Unless I'm missing something, maybe the fix is to add DependsOnTargets="AssignProjectConfiguration" to the _SplitProjectReferencesByFileExistence target in Microsoft.Common.CurrentVersion.targets?

Also, I tried @rainersigwald repro project here:
https://github.com/rainersigwald/IncrementalCleanDemo
Have confirmed that this approach fixes it.

[Drachenkaetzchen]

I'm having a similar issue. Upon a clean build, everything works fine. Once I rebuild, I see that all references are visible in explorer for a second, then are deleted. In fact, setting MSBuild output to verbose showed me that IncrementalClean was executed, leading me to this issue.

I did not have any files in my referred projects, so this was extremely brutal to track down.

In my case, the culprit was FodyWeavers. I used to run Fody 3.2.9. After upgrading to Fody 3.2.13, the problem started.

After lots of investigation, I found out that the issue started with Fody 3.2.12 where as with Fody 3.2.11 it works fine. Symptom is that with Fody 3.2.12, the affected referred project always gets rebuilt, causing the issue. That's how I actually found out about Fody in the first place; I went through all projects in my solution, clicking "rebuild" and seeing if VS thinks they're up-to-date or are rebuilt.

In fact, it seems that this only happens if you're using Packages.config. Fody 3.2.14 fixes this.

I understand that this comment is not strictly related to the original issue, but as it was extremely hard to track down and during my research I also stumbled over this issue like many others having similar effects, I thought it's valid to publish my findings.

[pasha-top]

The proximate cause of this is that the transitive GetCopyToOutputDirectoryItems target doesn't include the indirect content file in the latter build, while it does in the initial, successful build as well as the successful full command-line build.

There's a very confusing comment for GetCopyToOutputDirectoryItems that indicates that it isn't transitive--but clearly sometimes it is.

Thanks, it is worked!)
Unexpected behavior msbuild break my brain)
For transit copying files is necessary, than in transit project have been at least one file with mark "Copy if never/Copy always"

[superware]

Any plans to fix this critical issue anytime soon? @rainersigwald?

[jeremysimmons]

Thanks @cclauson

I used Directory.Build.props to apply this to all my csproj files at once, without having to edit them.

    <Project>
      <Target 
        Name="ForceAssignProjectConfigurationBeforeSplitProjectReferencesByFileExistence_KLUDGE" 
        BeforeTargets="_SplitProjectReferencesByFileExistence" 
        DependsOnTargets="AssignProjectConfiguration" />
    </Project>

[InKahootz]

Even after trying to apply the target in my top most Directory.Build.props it will still clean out my transitively acquired files from time to time.

Edit: My situation is only different in that I'm using nuget packages that contain my indirect files. Here's my project structure.

IndirectLibrary - Using EFCore Sqlite nuget 2.2.3 (outputs x64/x86 e_sqlite3.dlls in bin folder)
DirectLibrary - Using Indirect
Gui - Depends on DirectLibrary

Incremental clean even after the targets kludge doesn't stop IncrementalClean from deleting x64/e_sqlite3.dll and x86/e_sqlite3.dll

[pratthinds]

I am experiencing essentially the same issue as InKahootz.

The issue is happening when an incremental build occurs on a project that does not contain the Nuget dependencies directly. The unmanaged DLLs for some reason are incorrectly removed by the IncrementalClean step -- they should be left alone.

[livarcocc]

Assigning to @rainersigwald and 16.5 P1 because it seems it is being fixed by #4865. If that's not the case, we can move this back to backlog and un-assign.

[Chiel92]

How and where do I use this fix? I'm still suffering from this problem and I can't find information on the newly introduced MSBuildCopyContentTransitively.

I'm currently working around this issue by adding a dummy content file set to CopyAlways to the project that first introduces the content that needs to be copied transitively up the dependency tree. This then causes Visual Studio to always trigger all dependent projects to execute their post build steps etcetera, which prevents this bug from happening.

[Chiel92]

I'm still experiencing this issue in Visual Studio Version 16.5.4. This issue is closed implying that it should be fixed. @rainersigwald Do I require to make changes to my project structure in order to have this fixed? Or should it work out of the box (which it doesn't, so that would imply I should make a new issue?)