Sdk: Performance impact by searching excluded directories

[MovGP0]

Steps to reproduce

• Create a web project
• have a large folder with thousands of files in it (ie. node_modules)
• exclude the folder in the .csproj file for performance reasons

  <ItemGroup>
    <Compile Remove="node_modules\**" />
    <Content Remove="node_modules\**" />
    <EmbeddedResource Remove="node_modules\**" />
    <None Remove="node_modules\**" />
  </ItemGroup>

Expected behavior

• the folder should be ignored by the dotnet command
• project should load really fast in Visual Studio and JetBrains Raider

Actual behavior

• the folder is searched recursively
  
  
  
  • (issue was determined using the Sysinternals Process Monitor)
  
  
• project takes way too long to load
• creating/renaming/refactoring files and folders in Visual Studio takes way longer
• more RAM consumption

Environment data

dotnet --info output:

.NET Command Line Tools (2.0.0)

Product Information:
 Version:            2.0.0
 Commit SHA-1 hash:  cdcd1928c9

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.15063
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.0.0\

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.0
  Build    : e8b8861ac7faf042c87a5c2f9f2d04c98b69f28d

[MovGP0]

see also: https://developercommunity.visualstudio.com/content/problem/101727/visual-studio-2017-seaches-excluded-folders.html

[dasMulli]

The problem here is that the items are added beforehand so to exclude the folder being searched is to add it to the DefaultItemExcludes property like the web SDK already does for ASP.NET Core project.

To do it in non ASP.NET Core projects, you'd have to add a similar property redefinition to your csproj:

<PropertyGroup>
  <DefaultItemExcludes>node_modules/**;$(DefaultItemExcludes)</DefaultItemExcludes>
</PropertyGroup>

Note that there is a performance issue in glob expansion with patterns like **/node_modules/** at the moment so you'll have to add the folders explicitly (best at the start of the DefaultItemExcludes as shown above)

[MovGP0]

thank you! that's a working solution.

[Piedone]

What do you mean by "the items are added beforehand" @dasMulli? If the first and only reference to node_modules in the csproj is <None Remove="node_modules\**" /> is it still somehow added before this by some other process? Shouldn't this fully exclude the folder from the build process?

Did you find a solution @MovGP0 that doesn't necessitate excluding items twice?

[dasMulli]

@Piedone "it's complicated", basically having the Sdk="..." in the csproj magically adds stuff (like C-like #include) both before and after the content you define in the project file, and then there are multiple passes over the text - (so one for properties, then items and so on (omitting some other stuff now)).
So the evaluation for Properties will set DefaultItemExcludes first from the SDK and then from our definition.
A further pass will execute - among other things - some <Content Include="..." Exclude="$(DefaultItemExcludes)..." /> definitions from the SDK before processing any <None> / <Content> includes / removes from our main project definition (any property group to be exact). So if we want to save performance, we need to modify DefaultItemExcludes to make the SDK skip them. If we only do a <SomeItem Remove="..."> we don't save the time to scan the files, we only remove files from the project that were already found.

[Piedone]

Got it, thank you for the quick reply and the thorough explanation. So if I understand correctly (and my short tests seem to confirm this) if you care about performance, and you want Visual Studio and MSBuild to basically not know about certain files then the thing you want to use, and you don't need anything else, is DefaultItemExcludes, right? So no need for (or any benefit to) also use e.g. the following for git file? <EmbeddedResource Remove=".git" /> and <None Remove=".git" />. I'd use something like this:

  <PropertyGroup>
    <DefaultItemExcludes>$(DefaultItemExcludes);node_modules\**</DefaultItemExcludes>
  </PropertyGroup>

I wish it wasn't necessary to have a PhD in MSBuild to use Node but working on my thesis now :).