Sdk: Build is significantly slower with a large node_modules folder

[nil4]

Steps to reproduce

In an empty folder, run dotnet new. Change the generated project.json file to:

{
  "version": "1.0.0-*",
  "buildOptions": {  "emitEntryPoint": true  },
  "dependencies": {  },
  "frameworks": {  "net461": {}  }
}

Run dotnet restore; then run dotnet build and observe that the build banner (first two lines below) are displayed instantly, and the build finishes in less than 2 seconds:

Project Test (.NETFramework,Version=v4.6.1) will be compiled because expected outputs are missing
Compiling Test for .NETFramework,Version=v4.6.1

Compilation succeeded.
    0 Warning(s)
    0 Error(s)

Time elapsed 00:00:01.8254959

In the folder containing project.json, create an NPM package.json file with the following content:

{
  "dependencies": {
    "babel-core": "^6.8.0",
    "babel-loader": "^6.2.4",
    "babel-plugin-transform-runtime": "^6.8.0",
    "babel-preset-es2015": "^6.6.0",
    "babel-preset-react": "^6.5.0",
    "babel-preset-stage-0": "^6.5.0",
    "bootstrap": "^4.0.0-alpha.2",
    "classnames": "^2.2.5",
    "css-loader": "^0.23.1",
    "es6-promise": "^3.1.2",
    "exports-loader": "^0.6.3",
    "extract-text-webpack-plugin": "^1.0.1",
    "file-loader": "^0.8.5",
    "font-awesome": "^4.6.1",
    "imports-loader": "^0.6.5",
    "node-sass": "^3.7.0",
    "query-string": "^4.1.0",
    "react": "^15.0.2",
    "react-dom": "^15.0.2",
    "react-router": "^2.4.0",
    "react-select": "1.0.0-beta13",
    "react-spinkit": "^1.1.7",
    "sass-loader": "^3.2.0",
    "source-map-loader": "^0.1.5",
    "style-loader": "^0.13.1",
    "ts-loader": "^0.8.2",
    "typescript": "^1.8.10",
    "typings": "^0.8.1",
    "url-loader": "^0.5.7",
    "webpack": "^2.1.0-beta.6"
  }
}

Run npm install. With NPM 3.8.8 and Node 6.0, this creates 2095 sub-folders containing more than 20000 files under the node_modules folder. This is an example web application development scenario, but the issue reproduces whenever a large number of files/folders are present in the project folder.

Run dotnet build and note that there is a significant delay before the any output is displayed (on my machine, around 5-6 seconds with CPU usage above 20%). The build takes a lot longer (3x):

Project Test (.NETFramework,Version=v4.6.1) will be compiled because expected outputs are missing
Compiling Test for .NETFramework,Version=v4.6.1

Compilation succeeded.
    0 Warning(s)
    0 Error(s)

Time elapsed 00:00:07.1625644

After the build output is displayed, there is another large pause (>5 seconds) before the command prompt returns.

Update project.json to exclude the node_modules folder from the compilation:

  "buildOptions": {
    "emitEntryPoint": true,
    "compile": {
        "exclude": ["node_modules"]
    }
  },

Running dotnet build hits dotnet/cli#2841 and the build fails silently. Update project.json again to include .cs files in the compilation:

  "buildOptions": {
    "emitEntryPoint": true,
    "compile": {
        "include": ["**/*.cs"],
        "exclude": ["node_modules"]
    }
  },

There is still a large pause before and after the command output, but the build finished somewhat faster (2x slowdown). A unexpected warning is now displayed about the obj\assembly.cs file:

Project Test (.NETFramework,Version=v4.6.1) will be compiled because Input items added from last build
Compiling Test for .NETFramework,Version=v4.6.1
D:\projects\Test\warning CS2002: Source file 'obj\Debug\net461\dotnet-compile.assemblyinfo.cs' specified multiple times

Compilation succeeded.
    1 Warning(s)
    0 Error(s)

Time elapsed 00:00:04.6567803

Update project.json to exclude the obj folder:

  "buildOptions": {
    "emitEntryPoint": true,
    "compile": {
        "include": ["**/*.cs"],
        "exclude": ["node_modules", "obj"]
    }
  },

The pauses before/after are the same, the build completes in 4.5 seconds, but the warning is gone.

Expected behavior

The build slowdown caused by a large node_modules folder can be avoided by adding node_modules to compile/exclude.

Actual behavior

Build remains significantly slower when node_modules is added to compile/exclude.

Environment data

dotnet --info output:

.NET Command Line Tools (1.0.0-rc2-002611)

Product Information:
 Version:     1.0.0-rc2-002611
 Commit Sha:  bf8f0edd89

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.10586
 OS Platform: Windows
 RID:         win10-x64

[GunterGlad]

I have just found the same issue. Our node_modules is pretty big. It is supposed to be excluded:
"buildOptions": {
"compile": {
"exclude": [
"node_modules"
]
}
},
My compile time was 2m30. I deleted the node_modules folder and my compile time wend down to 30s.

[jusbuc2k]

I'm having the same issue across all 4 four dev workstations in my current project. Compile time (dotnet build) is at least 2 times longer when the node_modules folder exists with about 580 folders (which is from about 20 packages in devDependencies). I also have node_modules, jspm_modules, and some other folders excluded in buildOptions/compile/exclude.

[GunterGlad]

A tip for anyone. It may not work for you. So it depends on how you deploy your front end and how you have your projects setup. But to solve the problem for us I create a new project (dot not matter the type as long as you can compile you UI in it). Make sure no other project has any references to it. Then for debugging we started 2x web servers. 1 for .net core. And 1 for UI (webpack dev server). And added another html file to re-direct to the dev server. Then when finished editing UI we had a compile UI that would simply generate out bundle.js and then copy if over to the appropriate location. This fixed the performance for me. My compile time went from 2.5 minutes to 20 seconds. So you just need to move the npm folder out of your project folder if possible.

[TheRealPiotrP]

This issue was created with the project.json infrastructure and doesn't appear to repro in recent rc3 bits. If someone sees this repro atop rc3 or another build that supports csproj please open a new issue... ideally with a repro.

[rhyek]

Related: dotnet/cli#5656