Nx: @nrwl/linter: heap error when linting 2800+ files

[Hotell]

Expected Behavior

lint should work consistently no matter if executed via linter builder or directly via eslint cli

Current Behavior

with following workspace configuration

{
"pb-legacy": {
      "root": "src",
      "sourceRoot": "src/js",
      "projectType": "library",
      "schematics": {},
      "architect": {
        "lint-broken": {
          "builder": "@nrwl/linter:lint",
          "options": {
            "linter": "eslint",
            "config": ".eslintrc-pb-legacy",
            "tsConfig": ["tsconfig.eslint.pb-legacy.json"],
            "exclude": ["**/node_modules/**", "!src/js/**"]
          }
        },
        "lint": {
          "builder": "@nrwl/workspace:run-commands",
          "options": {
            "commands": [
              {
                "command": "eslint --ext .js,.ts,.tsx --config .eslintrc-pb-legacy src/js"
              }
            ]
          }
        }
    }
}

1. 🚨 yarn nx run pb-legacy:lint-broken throw error because of out of memory

<--- Last few GCs --->

[41332:0x103b12000]    79269 ms: Scavenge 1362.1 (1423.2) -> 1361.2 (1423.7) MB, 2.5 / 0.0 ms  (average mu = 0.341, current mu = 0.292) allocation failure 
[41332:0x103b12000]    79276 ms: Scavenge 1362.1 (1423.7) -> 1361.2 (1424.2) MB, 2.7 / 0.0 ms  (average mu = 0.341, current mu = 0.292) allocation failure 
[41332:0x103b12000]    79283 ms: Scavenge 1362.4 (1424.2) -> 1361.8 (1425.2) MB, 2.8 / 0.0 ms  (average mu = 0.341, current mu = 0.292) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x101e4d4dbe3d]
    1: StubFrame [pc: 0x101e4e52d6cd]
Security context: 0x31dba791e6e9 <JSObject>
    2: /* anonymous */(aka /* anonymous */) [0x31db3fcce2d1] [/Users/hotell/Projects/devel/productboard/pb-frontend/node_modules/eslint/lib/linter/linter.js:1] [bytecode=0x31dbccfcb971 offset=0](this=0x31db56a026f1 <undefined>,ruleId=0x31dbcd459771 <String[17]: constructor-super>)
    3: arguments adaptor frame: 3->1
   ...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10096d6b6 node::Abort() (.cold.1) [/usr/local/Cellar/node@10/10.18.1/bin/node]
 2: 0x10003b9e6 node_module_register [/usr/local/Cellar/node@10/10.18.1/bin/node]
 3: 0x10003bba7 node::OnFatalError(char const*, char const*) [/usr/local/Cellar/node@10/10.18.1/bin/node]

few debug logs from builder:

 nx run pb-legacy:lint-broken

builder options:
   { 
    linter: 'eslint',
     config: '.eslintrc-pb-legacy',
     tsConfig: [ 'tsconfig.eslint.pb-legacy.json' ],
     exclude: [ '**/node_modules/**', '!src/js/**' ],
     format: 'stylish',
     files: [],
     force: false,
     silent: false,
     fix: false,
     cache: false,
     cacheLocation: undefined,
     outputFile: undefined 
}

Linting "pb-legacy"...

- eslintConfigPath: '/Users/hotell/Projects/devel/my-project/.eslintrc-pb-legacy',
- tsConfigs: [ 'tsconfig.eslint.pb-legacy.json' ]
- filesCount: 2839

1. ✅ yarn nx run pb-legacy:lint properly lints codebase

Context

@nrwl/angular : Not Found
  @nrwl/cli : 9.1.2
  @nrwl/cypress : 9.1.2
  @nrwl/eslint-plugin-nx : 9.1.2
  @nrwl/express : Not Found
  @nrwl/jest : 9.1.2
  @nrwl/linter : 9.1.2
  @nrwl/nest : Not Found
  @nrwl/next : Not Found
  @nrwl/node : 9.1.2
  @nrwl/react : 9.1.2
  @nrwl/schematics : Not Found
  @nrwl/tao : 9.1.2
  @nrwl/web : 9.1.2
  @nrwl/workspace : 9.1.2
  typescript : 3.8.3

[hakalb]

I have also experienced this error and finally came up with a solution. The problem origin from how TypeScript and ESLint works together, providing the files to lint.

When creating a new workspace with Nx everything works, since there are limited number of files to lint. As the code base grows this will eventually become a problem.

Suggested fix

To support enterprise workspaces I did the following:

Created a new file tsconfig.eslint.json containing a reference to tsconfig.base.json and the paths that should support linting. For a default Nx workspace, paths would be apps and libs.

{
  "extends": "./tsconfig.base.json",
  "include": [
    "./apps/**/*",
    "./libs/**/*"
  ]
}

Then change parserOptions in .eslintrc to point to the new file.

{
  ...
  "parserOptions": {
    "ecmaVersion": 2020,
    "sourceType": "module",
    "project": "./tsconfig.eslint.json"
  }
  ...
}

The lint block in angular.json should also look something like this.

{
  ...
  "lint": {
    "builder": "@nrwl/linter:lint",
    "options": {
      "linter": "eslint",
      "tsConfig": [
        "libs/shared/ui/loading-message/tsconfig.lib.json",
        "libs/shared/ui/loading-message/tsconfig.spec.json"
      ],
      "exclude": [
        "**/node_modules/**",
        "!libs/shared/ui/loading-message/**/*"
      ]
    }
  ...
}

Now everything should work as expected. Hopefully it will help someone else struggling getting it to work.

Context

Nx version 10.0.12.

Angular CLI: 10.0.6
Node: 12.17.0
OS: darwin x64

Angular: 10.0.9
... animations, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, platform-server
... router, service-worker
Ivy Workspace: Yes

Package                            Version
------------------------------------------------------------
@angular-devkit/architect          0.1000.5
@angular-devkit/build-angular      0.1000.6
@angular-devkit/build-optimizer    0.1000.6
@angular-devkit/build-webpack      0.1000.6
@angular-devkit/core               10.0.6
@angular-devkit/schematics         10.0.6
@angular/cdk                       10.1.3
@angular/cli                       10.0.6
@angular/flex-layout               10.0.0-beta.32
@angular/material                  10.1.3
@angular/material-moment-adapter   10.1.3
@ngtools/webpack                   10.0.6
@schematics/angular                10.0.6
@schematics/update                 0.1000.6
rxjs                               6.6.2
typescript                         3.9.7
webpack                            4.43.0

Finally

Kudos to this issue that got me on track by explaining the problem in more detail.

[JamesHenry]

This situation should be greatly helped by the new builder coming in https://github.com/nrwl/nx/pull/3763

[Hotell]

FYI this OOM error is mainly caused by nx error boundaries rule rather than current linter builder implementation.

[kamiljano]

that didn't help much. I upgraded to @nrwl/*: 11.0.2 and I'm still experiencing the same issue, although my project is significantly smaller than the project the OP is describing. It only has ~350 files in total, but the linting always crashes on my e2e module with ~40 TS files in it and like 5 JSON files.

[Hotell]

As I mentioned, what is slowing things down by huge amount is implementation of nx module boundaries eslint plugin. Try to turn that rule off @kamiljano ✌️. There were no changes in the implementation in v11.

Regarding new eslint builder, while the implementation is better than the old builder, there are no real perf improvements ( we migrated 180 projects nx workspace ).