Hi, in the error stack trace, it seems that you are using a different configuration from "production", can you kindly share that configuration?

alan-agius4 on 21 Feb 2019

Sorry for mismatch here is configuration params that you need

"config": {
              "optimization": true,
              "outputHashing": "all",
              "sourceMap": false,
              "extractCss": true,
              "namedChunks": false,
              "aot": true,
              "extractLicenses": true,
              "vendorChunk": false,
              "buildOptimizer": true,
              "fileReplacements": [
                {
                  "replace": "src/environments/environment.ts",
                  "with": "src/environments/environment.config.ts"
                }
              ]
            }

arayik-yervandyan on 21 Feb 2019

I have the same error:

screen shot 2019-02-26 at 11 43 52 am

My env:

screen shot 2019-02-26 at 11 44 23 am

nicobytes on 26 Feb 2019

Guys here's described the "workaround" for this problem: https://github.com/angular/angular-cli/issues/5618

Bloodcast69 on 27 Feb 2019

👍2

[ ] new
[ ] build
[X] serve prod
[ ] test
[ ] e2e

I am facing same issue while ng serve --prod

Reference Image

mumairofficial on 18 Mar 2019

👍13

I to got the same error. NodeJs process uses around 1.5GB of RAM, and after few minutes of attempt to clean/allocate memory, it fails with error.

themanojshukla on 25 Mar 2019

UPDATE :-

I've seen this issue only on WINDOWS machine particularly. Anyone else, noticed that? Or have got same issue in other machines?

Note: I tried 2 Windows machine, all tests gave same result. Both machine have same config almost.
i.e. 8GB RAM/Admin Access For Bash/CMD/PowerShell/ i3 & i5 Processor (quad core, both).

And, I admit that the project that I'm working on is a MESS in terms of code quality.
I tried one clean code, full fledged application, ON WINDOWS, it was slower in build process, but worked fine.

I tried MESSY project's code, it never compile on default node heap allocation (around 1.2 GB, not sure.). It works if I allocate more heap memory to node process, say 5 GB. But, in that case too, it took more than 30 mins to build. I tried the 5 GB node heap option with and without --aot AND/OR --prod .

Same code when tried on CentOS could instance of 1GB RAM/ Virtual Xenon Processor (1 Core), generates build within 90 Seconds. with and without --aot AND/OR --prod

I'm not sure, but, looks like problem is with Node Service not with Angular CLI itself, coz I've seen some performance lag in new NodeJS releases after last update (locally on WINDOWS machine only).

Some observations below.:-

Even though allocating 5 GB of heap, it never used more than 1.9 GB (max) and allocation process was very slow.. it was increasing RAM consumption 1-2 MB per 3-5 seconds. And, was not using Disk Resource as it used to be in older version. (Disk access was very high previously on WINDOWS machine, but build process was faster without extra RAM allocation. This time, same machine almost same code with some bug fixes, but node was upgraded, disk consumption was lowered but slow build generation.)

Hope somebody can confirm this.

themanojshukla on 26 Mar 2019

👎35 👍3

Yes, I am also facing the same issue

tibinthomas on 27 Mar 2019

👍2

Same issue here running on a Mac book pro. macOs Mojoave. Any environment with production: true

sswayney on 27 Mar 2019

👍18

same issue on ubuntu server 18.04, but this solved my problem:
node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod

nokhodian on 28 Mar 2019

👍169 ❤47 🚀38 🎉36 😄26 👀24 👎19 😕9

I've tried uninstalling Angular CLI 7, and installed 6.0.8. verified the downgraded version by ng version, still it breaks with the same error.

Maybe this could be because of nodeJs too. Coz, I'm building same code in CentOS 7, 1.75GB RAM (with 512MB swap memory), single core processor, and with angular CLI 7, it works there. It is breaking in my 2 windows machines.

For those, who can't afford allocating more RAM (due to less memory available) can create a SWAP memory from the space available from Hard Drive of system. It'll be little slower, but should work without System Upgrade. I've done the same. 👍

themanojshukla on 28 Mar 2019

In my case the problem only appears, when building --prod --source-map. The normal --prod build runs just fine.

Using --max_old_space_size=8192 as suggested by @nokhodian it works with memory consumption peaking at ~3.8GiB during source map generation.

Build environment:

Angular CLI: 7.3.8
Node: 10.15.2
OS: linux x64
Angular: 7.2.12
... animations, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, router, service-worker

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.10.4
@angular-devkit/build-angular     0.10.4
@angular-devkit/build-optimizer   0.10.4
@angular-devkit/build-webpack     0.10.4
@angular-devkit/core              7.0.4
@angular-devkit/schematics        7.1.0
@angular/cdk                      7.3.7
@angular/cli                      7.3.8
@angular/flex-layout              7.0.0-beta.19
@angular/material                 7.3.7
@angular/pwa                      0.11.0
@ngtools/webpack                  7.0.4
@schematics/angular               7.1.0
@schematics/update                0.13.8
rxjs                              6.4.0
typescript                        3.2.4
webpack                           4.19.1

j2L4e on 5 Apr 2019

👍8

In my angular.json file I have setted "sourceMap": false, but it doesn't helped me. Same issue for prod build

arayik-yervandyan on 10 Apr 2019

just chiming in here with same error message when running ng build when the build is working on a particular scss file

ormico on 14 Apr 2019

I also have this issue but can resolve it with "NODE_OPTIONS=--max-old-space-size=8192". Ram usage during build gets up about 14GB.

geo242 on 17 Apr 2019

😕7

Same here. I found that after further searching. I've sent out instructions to our team to set this environment variable and set it on our build servers. Seems to work and build seems to be faster than before it stopped working.

ormico on 18 Apr 2019

I started running into the same issue with Azure Pipelines: #https://github.com/Microsoft/azure-pipelines-image-generation/issues/854 I assumed it was a hosting problem, since don't have the issue on premise.

Calidus on 18 Apr 2019

We are seeing this issue as well. With ng serve being the worse, but even normal ng build having issues now. Could this be sass related? Something seems off.

      "old_space": {
        "memorySize": 1280512000,
        "committedMemory": 1278567272,
        "capacity": 1155253856,
        "used": 1152333192,
        "available": 2920664
      },

Above is my old_space head info from the generated report. What could be making these numbers get so large?

j3gb3rt on 18 Apr 2019

Has anyone experienced this in a project that doesn't use sass?

j3gb3rt on 18 Apr 2019

👍12

We are experiencing this problem when we use
ng build --prod
Sometimes needed because we use different environment settings.
node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng serve --prod solves the problem but the build time is exceeded to 3 minutes.
I don't think this issue is solved

hvdscheer on 25 Apr 2019

🎉5

This issue is the same as https://github.com/angular/angular-cli/issues/5618#issuecomment-450151214, as far as I can tell.

I don't see a lot of things I can follow up and investigate though. @themanojshukla mentioned Windows was taking a lot longer. @j3gb3rt is asking about sass, and I have seen reports in the past about sass making things slower.

Does anyone have a reproduction that we can look at and try to debug?

filipesilva on 26 Apr 2019

👍1

I have run into this issue on a project that does not use sass. I did some environment testing because I originally thought it was an issue relating to Azure Dev Ops.

Using Azure Dev Ops Pipelines:

Hosted Linux agent with Node 10: Build succeeds
Hosted Windows vs2017 Agent with Node 10: Build fails
Hosted Windows vs2017 Agent with Node 6: Build fails
Hosted Windows vs2019 Agent with Node 10: Build fails
Hosted Windows vs2019 Agent with Node 6: Build fails
Hosted Windows Container Agent with Node 10: Build Failed 5/10 attempts
Install Build Agent on my dev machine(win10) with Node 10: Build Succeeds
Install Build Agent on my dev machine(win10) with Node 6: Build Succeeds
Dev machine(win10) using cmd with Node 10: Build Succeeds
Dev machine(win10) using cmd with Node 6: Build Succeeds

Calidus on 26 Apr 2019

👍3

Yes @filipesilva and @Calidus my projects also don't use sass.

Let me give some more observations...

The pet project (following best practices as much as can be) I created in around a year ago was working very fine that time ( I don't remember the CLI version I used). Now when I'm building it using --aot --prod it takes bit more time than before but never breaks. No sass used.
New project (of course production code base & I admit, really really messy codes), it breaks on --aot --prod. No Sass used. Breaks on windows always. Sometimes works on ubuntu without allocating extra heap on node process, but success rate on ubuntu is 10% say.
Another mid-level project (messy codes), works without extra heap allocation but takes a bit more time on both Windows & Ubuntu.
Another pet project (following best practices as much as can be) works fine so far.. but very less files to compile, so can't be sure on processing loads.

When running ng build --aot --prod for the first time after machine start/restart. it takes 57seconds. afterwards, 22-25seconds only.

First Run Screenshot..

Second Run Screenshot..

Both are for same project.. and are of while writing this comment.

Hope it helps.

themanojshukla on 26 Apr 2019

I'm facing the same issue. Every time I build I get "JavaScript heap out of memory". I don't want to use node --max_old_space_size=8192 since this doesn't sounds best practice. Any other way to fix this?

noopur-dabhi on 2 May 2019

@noopur-dabhi I don't think it is fair to say that using node --max_old_space_size=8192 is not best practice.

It is nothing but allowing NODE process _(which is going to build your app now)_ to use more than DEFAULT heap memory (from your RAM or SWAP area).

And this type of allocation is very very common in most of the app development (especially deployment) process. Example, in Java, if we want to restrict our java app heap usage, we add additional flag like --Xmx64m this means maximum heap memory allowed is 64MB.

So if 'Best Practice' is the only concern, I think it's not a problem using node --max_old_space=8192, this means you are allowing NODE process to use maximum of 8192MB from memory. nothing to do with Best Practice Concern.

themanojshukla on 3 May 2019

👍13 👎5

This is still crashing for me> any suggestions?
I tried "ng build --aot --prod"
and "node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod"

Angular Live Development Server is listening on localhost:4200, open your browser on http://localhost:4200/ **
92% chunk asset optimization TerserPlugin
<--- Last few GCs --->

[9568:000002A08DC547B0] 183499 ms: Mark-sweep 1350.2 (1423.6) -> 1350.2 (1424.1) MB, 952.0 / 0.0 ms (average mu = 0.111, current mu = 0.000) allocation failure GC in old space requested
[9568:000002A08DC547B0] 184477 ms: Mark-sweep 1350.2 (1424.1) -> 1350.2 (1424.1) MB, 977.5 / 0.0 ms (average mu = 0.058, current mu = 0.000) last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 000001D7B1850481]

Security context: 0x008d8961d9b1
1: /* anonymous */ [0000006BEE99B319] [C:CODEV2UXnode_moduleswebpack-sourcesnode_modulessource-maplibsource-node.js:~342] pc=000001D7B3E70B48
2: SourceNode_walk [000001779ECCDF31] [C:CODEV2UXnode_moduleswebpack-so...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20190506.161503.9568.001.json
Node.js report completed
1: 00007FF775E1896A public: __cdecl v8::internal::GCIdleTimeHandler::GCIdleTimeHandler(void) __ptr64+4554
2: 00007FF775DC8956 uv_loop_fork+85542
3: 00007FF775DC941D uv_loop_fork+88301
4: 00007FF7761EE72E void __cdecl v8::internal::FatalProcessOutOfMemory(class v8::internal::Isolate * __ptr64,char const * __ptr64)+798
5: 00007FF7761EE667 void __cdecl v8::internal::FatalProcessOutOfMemory(class v8::internal::Isolate * __ptr64,char const * __ptr64)+599
6: 00007FF7762A2144 public: static bool __cdecl v8::internal::Heap::RootIsImmortalImmovable(int)+14068
7: 00007FF776297F52 public: bool __cdecl v8::internal::Heap::CollectGarbage(enum v8::internal::AllocationSpace,enum v8::internal::GarbageCollectionReason,enum v8::GCCallbackFlags) __ptr64+7234
8: 00007FF776296768 public: bool __cdecl v8::internal::Heap::CollectGarbage(enum v8::internal::AllocationSpace,enum v8::internal::GarbageCollectionReason,enum v8::GCCallbackFlags) __ptr64+1112
9: 00007FF776295C25 public: void __cdecl v8::internal::Heap::CollectAllGarbage(int,enum v8::internal::GarbageCollectionReason,enum v8::GCCallbackFlags) __ptr64+949
10: 00007FF7762A0174 public: static bool __cdecl v8::internal::Heap::RootIsImmortalImmovable(int)+5924
11: 00007FF7763D7E4D private: class v8::internal::HeapObject * __ptr64 __cdecl v8::internal::Factory::AllocateRawArray(int,enum v8::internal::PretenureFlag) __ptr64+61
12: 00007FF7763D87E2 private: class v8::internal::Handle __cdecl v8::internal::Factory::NewFixedArrayWithFiller(enum v8::internal::Heap::RootListIndex,int,class v8::internal::Object * __ptr64,enum v8::internal::PretenureFlag) __ptr64+66
13: 00007FF77641B0BA public: static class v8::internal::wasm::AsmType * __ptr64 __cdecl v8::internal::wasm::AsmType::Void(void)+69050
14: 00007FF776407A06 class v8::internal::MaybeHandle __cdecl v8::internal::BigIntLiteral(class v8::internal::Isolate * __ptr64,char const * __ptr64)+127094
15: 00007FF77649AFA0 public: static int __cdecl v8::internal::StoreBuffer::StoreBufferOverflow(class v8::internal::Isolate * __ptr64)+64256
16: 000001D7B1850481
npm ERR! code ELIFECYCLE
npm ERR! errno 134
npm ERR! Exit status 134
npm ERR!
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

lcpautotester on 7 May 2019

Unfortunately I am facing with the same issue. Before upgrade to 7+ I was able to builid Angular App on DevOps azure envinorment. Now it's not possible. I'm getting following error:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 00007FF7B388F04A v8::internal::GCIdleTimeHandler::GCIdleTimeHandler+5114
2: 00007FF7B386A0C6 node::MakeCallback+4518
3: 00007FF7B386AA30 node_module_register+2032
4: 00007FF7B3AF20EE v8::internal::FatalProcessOutOfMemory+846
5: 00007FF7B3AF201F v8::internal::FatalProcessOutOfMemory+639
6: 00007FF7B4012BC4 v8::internal::Heap::MaxHeapGrowingFactor+9556
7: 00007FF7B4009C46 v8::internal::ScavengeJob::operator=+24310
8: 00007FF7B400829C v8::internal::ScavengeJob::operator=+17740
9: 00007FF7B4010F87 v8::internal::Heap::MaxHeapGrowingFactor+2327
10: 00007FF7B4011006 v8::internal::Heap::MaxHeapGrowingFactor+2454
11: 00007FF7B3BCCDB7 v8::internal::Factory::NewFillerObject+55
12: 00007FF7B3C62CC6 v8::internal::WasmJs::Install+29414
13: 0000035C75D5C5C1
npm ERR! code ELIFECYCLE
npm ERR! errno 134

vindemi on 7 May 2019

@vindemi you are going to need to edit your Build Pipeline. Remove the npm build task and replace it with a command line task with the following script node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod make sure to set your working directory to the folder with your package.json.

Calidus on 7 May 2019

👍1

@vindemi you are going to need to edit your Build Pipeline. Remove the npm build task and replace it with a command line task with the following script node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod make sure to set your working directory to the folder with your package.json.

its working thanks

safeer-khan123 on 10 May 2019

Of note as well is that Node.js 12 automatically configures its memory options based on the available system memory.

clydin on 10 May 2019

👍9 ❤1

Did this solution work for anybody? I still get the same error on VS2017 hosted process on Azure. Here is my YAML for the step. What is the root cause of this? Running this even locally on a beefy I7 I get 100% CPU hit and nearly 5GB of RAM usage. Is there something I need to refactor? I have a bunch of standard components with LESS files.

`steps:

script: |
node --max_old_space_size=8192
$(Build.SourcesDirectory)node_modules.binng.cmd build --prod
displayName: 'Command Line Script'
`

Error:

`##[section]Starting: Command Line Script

Task : Command Line
Description : Run a command line script using cmd.exe on Windows and bash on macOS and Linux.
Version : 2.148.0
Author : Microsoft Corporation

Help : More Information

Generating script.
========================== Starting Command Output ===========================

[command]"C:windowssystem32cmd.exe" /D /E:ON /V:OFF /S /C "CALL "d:a_temp3e7b4265-9fa3-47c6-b6b7-b690af8bcac6.cmd""

Browserslist: caniuse-lite is outdated. Please run next command yarn upgrade caniuse-lite browserslist
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 00007FF747A7F04A v8::internal::GCIdleTimeHandler::GCIdleTimeHandler+5114
2: 00007FF747A5A0C6 node::MakeCallback+4518
3: 00007FF747A5AA30 node_module_register+2032
4: 00007FF747CE20EE v8::internal::FatalProcessOutOfMemory+846
5: 00007FF747CE201F v8::internal::FatalProcessOutOfMemory+639
6: 00007FF748202BC4 v8::internal::Heap::MaxHeapGrowingFactor+9556
7: 00007FF7481F9C46 v8::internal::ScavengeJob::operator=+24310
8: 00007FF7481F829C v8::internal::ScavengeJob::operator=+17740
9: 00007FF748200F87 v8::internal::Heap::MaxHeapGrowingFactor+2327
10: 00007FF748201006 v8::internal::Heap::MaxHeapGrowingFactor+2454
11: 00007FF747DBCBE8 v8::internal::Factory::AllocateRawArray+56
12: 00007FF747DC2CFA v8::internal::Factory::NewTransitionArray+58
13: 00007FF7482A6684 v8::internal::CodeStubAssembler::InitializeFunctionContext+26932
14: 00007FF747FDCBFE v8::internal::JSReceiver::GetOwnPropertyDescriptor+17550
15: 00007FF747FDCD69 v8::internal::JSReceiver::GetOwnPropertyDescriptor+17913
16: 00007FF747FDEB9B v8::internal::JSReceiver::GetOwnPropertyDescriptor+25643
17: 00007FF747FCDD94 v8::internal::JSReceiver::class_name+4132
18: 00007FF747FDE2D9 v8::internal::JSReceiver::GetOwnPropertyDescriptor+23401
19: 00007FF747EAACFE v8::internal::LookupIterator::PrepareTransitionToDataProperty+478
20: 00007FF747FD1E11 v8::internal::JSReceiver::class_name+20641
21: 00007FF747DACFE9 v8::internal::wasm::WasmCodeManager::LookupCode+17993
22: 00007FF747DB00A4 v8::internal::wasm::WasmCodeManager::LookupCode+30468
23: 0000023D7E05C5C1

[error]Cmd.exe exited with code '134'.

[section]Finishing: Command Line Script`

janosfekete on 15 May 2019

no it didn't work for me either. Ended up doing a completely different work around.

lcpautotester on 15 May 2019

You may be using a class inside itself of the same name.

garrettmac on 15 May 2019

Also happens with angular 8 (rc4). When I turn off sourcemaps for production it works again.

johannesjo on 21 May 2019

With Angular 8, I am unable to build on Azure. I am on the highest tier on Azure Web App with 7GB of RAM (S3). I have tried the --max_old_space_size=8192 trick, but I get the following error.

I have turned of AOT for production. Making my main.js file 3MB instead of 2.5MB, but it does build on Azure.
On Azure, Node 10.15.2 is the highest node installed on the machine at the moment.

#

# Fatal process OOM in insufficient memory to create an Isolate
#

<--- Last few GCs --->


<--- JS stacktrace --->

Failed exitCode=-1073741819, command="D:\Program Files (x86)\nodejs\10.15.2\node.exe" --max_old_space_size=8192 node_modules\@angular\cli\bin\ng build myApp --prod --configuration staging --delete-output-path=false --progress=false --outputPath D:\local\Temp\8d6e507f4dfaa02\wwwroot

jsgoupil on 30 May 2019

@johannesjo @jsgoupil I think the issue with 8 has to do with a memory leak on prod builds with differential loading. Can you try disabling it? See https://angular.io/guide/deployment#opting-out-of-differential-loading

cc @alan-agius4

filipesilva on 30 May 2019

👍2

@filipesilva Nope. Doesn't fix anything:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

johannesjo on 30 May 2019

I upgraded to Angular 8 this morning and started experiencing this issue. I tried @filipesilva suggestion and it now compiles.

https://angular.io/guide/deployment#opting-out-of-differential-loading

browserslist file

not dead
not IE 9-11

Changed to

dead
IE 9-11

tsconfig.json file

  "compilerOptions": {
    "target": "es2015"
    ...

Changed to

  "compilerOptions": {
    "target": "es5"
    ...

gak10100 on 30 May 2019

@johannesjo @jsgoupil are you using sass/scss?

filipesilva on 30 May 2019

@filipesilva yes.
@gak10100 did you set sourceMap to true in the angular.json?

johannesjo on 30 May 2019

@johannesjo my sourceMapis set to false. I tried it with sourceMap set to true and it still builds

gak10100 on 30 May 2019

@gak10100 and are you using scss/sass?

johannesjo on 30 May 2019

I think the increased memory requirement might be related to sass processing then. We moved from node-sass to sass in 8.

Can you follow the instructions in @angular-devkit/build-angular: use sass instead of node-sass (ce15899) to force usage of node-sass instead?

filipesilva on 30 May 2019

❤1

@johannesjo Yes
AngularBuildError
AngularBuildSuccess

gak10100 on 30 May 2019

👍1

You can also try using Node 12, as it should increase the memory automatically. If there's enough memory, builds should succeed.

filipesilva on 30 May 2019

Just FYI I'm on node v12.3.1

gak10100 on 30 May 2019

@gak10100 thank you.

I tried all combinations of the following:
node: 10.15.3/12.3.1
angular: 7.4, 8.0
os: win10/ubuntu18.04

All result in the same error.

I have quite the powerful gaming laptop so memory and cpu power shouldn't normally be an issue.
The code of project is open source and can be looked at here: https://github.com/johannesjo/super-productivity

johannesjo on 30 May 2019

Upgrading from node 10 to node 12 resolved my compiling memory issue when using Angular 8.

However, when I try to run the compiled code from ng build --prod, it just displays a blank screen :/ Works fine with ng serve though

Troyd-Destin on 30 May 2019

👍1

@johannesjo I'm very, very grateful for the repro! It's so hard to find repros for these problems.

filipesilva on 30 May 2019

@filipesilva I tried a lot of things here (took me quite a while to gather this). I use SASS.

With --max_old_space_size:
ES2015 AOT ON => fail with Fatal process OOM in insufficient memory to create an Isolate
ES2015 AOT OFF => success
ES5 AOT ON => fail with Fatal process OOM in insufficient memory to create an Isolate

Without --max_old_space_size
ES2015 AOT ON => fail with Ineffective mark-compacts near heap limit Allocation failed
ES2015 AOT OFF => success
ES5 AOT ON => fail with Ineffective mark-compacts near heap limit Allocation failed
ES5 AOT OFF => success

This browserslist, I am not sure where it is picking it from, I upgraded a large application so ng update or anything about schematic did not do its job properly, so I updated everything manually.
I tried to drop it in the root folder. It does not change anything.

As mentioned, Azure does not have Node 12. But only Node 10. I want to re-iterate, this is on Azure, an S3 machine.

I have never used the --aot flag. I use the --prod flag. I used angular.json with "aot" and "buildOptimizer" set to true. I moved my app from Angular 6.0.4 to 8.0.0.

jsgoupil on 30 May 2019

@johannesjo I tried your repo, here are my results.

First I updated manually to CLI and build-angular latest stable. The update is in https://github.com/filipesilva/super-productivity/tree/benchmark-8.

This is what I see for ng version:

kamik@RED-X1C6 MINGW64 /d/sandbox/oom/v8 (benchmark-8)
$ ng version

     _                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/


Angular CLI: 8.0.1
Node: 10.10.0
OS: win32 x64
Angular: 8.0.0-rc.4
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... platform-server, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.800.1
@angular-devkit/build-angular     0.800.1
@angular-devkit/build-optimizer   0.800.1
@angular-devkit/build-webpack     0.800.1
@angular-devkit/core              8.0.1
@angular-devkit/schematics        8.0.1
@angular/cdk                      8.0.0-rc.0
@angular/cli                      8.0.1
@angular/http                     7.2.14
@angular/material                 8.0.0-rc.1
@angular/pwa                      0.10.7
@angular/service-worker           7.2.14
@ngtools/webpack                  8.0.1
@schematics/angular               7.0.7
@schematics/update                0.800.1
rxjs                              6.5.1
typescript                        3.4.5
webpack                           4.30.0

Then I ran a custom benchmark tool we have in this repository to measure resource usage for commands.

kamik@RED-X1C6 MINGW64 /d/sandbox/oom/v8 (benchmark-8)
$ ../../../work/cli/bin/benchmark -- ng build --prod
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\oom\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 155162.80 ms (157246.00, 143200.00, 165534.00, 155899.00, 153935.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 17.40 % (16.74, 17.06, 18.40, 17.31, 17.52)
[benchmark]   Peak CPU usage: 190.04 % (165.60, 236.10, 189.10, 187.50, 171.90)
[benchmark]   Average Memory usage: 930.58 MB (928.38, 935.07, 917.83, 950.89, 920.71)
[benchmark]   Peak Memory usage: 1640.18 MB (1583.62, 1619.55, 1624.59, 1669.88, 1703.28)

So I am seeing peak memory usage juuuuust below the 1.7g default limit. I don't quite understand how you are reaching the memory limit of your machine. Is there anything else about your environment you can think of? Maybe symlinks somewhere.

@jsgoupil I am not on Azure, but I did my tests on a Windows machine. It is expected that AOT takes more memory mostly because it's a more complex build. More is happening there.

But I find your results interesting. You say that without --max_old_space_size+AOT off the builds succeed. So they succeed with less than 1.7g memory (the default for now). But you say that with --max_old_space_size=8192+AOT on, it always fails.

So the AOT build is taking at least 5x more memory. Which I don't think is very reasonable. I mean up to 2x I could imagine for an insanely complex TS setup, but 5x is just too much. I think it is more likely that there's some memory leak in the compiler, or something silly going on.

Do you have any repo we could look at? Even if only privately shared.

filipesilva on 31 May 2019

Hello @filipesilva , thanks for being willing to look into this. I can share my repo, it is indeed private and on Bitbucket. I am able to build AOT on my windows machine with Node 12. It does take quite a while but it works. I'm not used into taking benchmark and check all that stuff.
Can you go on my profile and send me a quick email, I can get you setup.

jsgoupil on 31 May 2019

@filipesilva thanks for checking this.

I'm not sure if understand you correctly.

First: Am I assuming correctly that on your machine the build is running fine? Or is it just, that it technically _shouldn't_ fail according to the benchmark?

Second: Do you mean a repository of another angular app? Maybe the travis build logs are helpful, too? If I'm not mistaken they should be public anyway:
https://travis-ci.org/johannesjo/super-productivity/builds/539736050
Results seem to be the same, even though the mac build is taking its sweet time again :)

Please let me know, if I can supply anything else of assistance.

johannesjo on 31 May 2019

I think the increased memory requirement might be related to sass processing then. We moved from node-sass to sass in 8.

Can you follow the instructions in @angular-devkit/build-angular: use sass instead of node-sass (ce15899) to force usage of node-sass instead?

I installed node-sass as a dev dependency but still experienced insanely high memory consumption. Our builds are using more than 12GB.

stevenoctopus on 4 Jun 2019

Not sure if this will help but here is a graph of our memory consumption.

The first spike is a default build with differential loading. Second spike is differential loading using node-sass.

The final spike is a build with differential loading turned off.

Both builds with differential loading failed after they reached the memory limit of the server. The last build was successful.

memoryconsumption

We have about 20 different projects defined in our angular.json. It would normally fail when building the second or third project.

stevenoctopus on 4 Jun 2019

👍3

Below are two more issue related to the slowdown in version 8. The below have logs and a reproduction as well.

https://github.com/angular/angular-cli/issues/14666#issue-452183942
https://github.com/angular/angular-cli/issues/14604#issuecomment-498425823

alan-agius4 on 5 Jun 2019

@Troyd-Destin same here as you, did you solve your problem?

Svetomechc on 6 Jun 2019

Hey all,

Just want to give you an update on this issue. I've been looking into it to see if I can find something. Below are my findings so far.

filipesilva on 7 Jun 2019

Debugging the memory situation in CLI 8

Environment:

kamik@RED-X1C6 MINGW64 /d/sandbox/memory-debug
$ ng version

Angular CLI: 8.0.1
Node: 12.3.1
OS: win32 x64
Angular:
...

Package                      Version
------------------------------------------------------
@angular-devkit/architect    0.800.1
@angular-devkit/core         8.0.1
@angular-devkit/schematics   8.0.1
@schematics/angular          8.0.1
@schematics/update           0.800.1
rxjs                         6.4.0

kamik@RED-X1C6 MINGW64 /d/sandbox/memory-debug
$ npm -v
6.9.0

kamik@RED-X1C6 MINGW64 /d/sandbox/memory-debug
$ yarn -v
1.12.3

Measuring memory usage

We have a private benchmarking package for the CLI https://github.com/angular/angular-cli/tree/master/packages/angular_devkit/benchmark.

To install it, clone the CLI repo, then run follow these commands:

yarn
yarn build
npm pack dist/@angular-devkit/benchmark
npm install -g angular-devkit-benchmark-0.800.0-beta.18.tgz

The filename on the last command might change, depending on the version of the CLI you clone.

To use it, run benchmark -- command. The command will depend on the project you use. To benchmark a prod build, do benchmark -- ng build --prod.

You can also do benchmark --iterations 20 -- ng build --prod to get a larger sample size (20 instead of 5).

Methodology

I have gathered a bunch of projects of various sizes, and have setup a v7 and v8 folder for each.

To get the different versions I first update a project to version 8 following https://update.angular.io/. Thats the v8 variant.

Then I make the v7 variant by replacing:

@angular-devkit/build-angular version from ~0.800.0 to 0.13.8
@angular/cli version from ~8.0.1 to 7.3.9
target in ./tsconfig.json from es2015 to es5.

To change between node versions I used NVM. To disable Differential Loading (DL) I changed the target in ./tsconfig.json from es2015 to es5.

On each I do benchmark -- ng build --prod, then compare the results.

Benchmark matrix:

v8, DL, Node 10.10.0
v8, DL, Node 12.3.1
v8, no DL, Node 10.10.0
v8, no DL, Node 12.3.1
v7, Node 10.10.0
v7, Node 12.3.1

Results

new-project

From ng new new-project

Generated using Angular CLI 8.0.1

v8, DL, Node 10.10.0

[benchmark] Benchmarking process over 20 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\new-project\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 46922.00 ms (39800.00, 45773.00, 45097.00, 45295.00, 46447.00, 47061.00, 49343.00, 48787.00, 58921.00, 50950.00, 45493.00, 45809.00, 47836.00, 45142.00, 45632.00, 45060.00, 45956.00, 45688.00, 45292.00, 49058.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 14.31 % (10.82, 12.30, 14.18, 13.79, 13.90, 13.75, 16.09, 15.20, 17.63, 16.20, 14.68, 14.51, 15.48, 14.43, 13.26, 14.56, 13.38, 13.03, 12.87, 16.10)
[benchmark]   Peak CPU usage: 109.84 % (92.30, 89.10, 137.60, 126.60, 103.10, 92.20, 128.20, 103.00, 142.20, 114.00, 90.60, 104.60, 107.80, 96.90, 93.70, 128.20, 123.40, 95.30, 92.20, 135.90)        
[benchmark]   Average Memory usage: 412.94 MB (410.61, 414.42, 412.02, 414.26, 414.95, 413.02, 411.16, 408.59, 408.12, 411.88, 417.27, 416.50, 410.62, 413.21, 416.36, 411.94, 411.58, 414.87, 412.33, 
415.01)
[benchmark]   Peak Memory usage: 972.38 MB (940.09, 1007.36, 939.21, 1014.48, 1012.67, 953.59, 934.79, 944.62, 860.78, 1008.56, 1019.51, 1005.97, 902.56, 1008.87, 999.36, 1005.65, 973.90, 1012.61, 974.84, 928.23)

v8, DL, Node 12.3.1

[benchmark] Benchmarking process over 20 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\new-project\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 43714.45 ms (42475.00, 42905.00, 42806.00, 45229.00, 44435.00, 42946.00, 43730.00, 44409.00, 43320.00, 43331.00, 45213.00, 46699.00, 44467.00, 42552.00, 42833.00, 42773.00, 42621.00, 45640.00, 43103.00, 42802.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 15.08 % (13.71, 15.04, 14.23, 15.94, 15.58, 14.19, 14.84, 15.49, 15.26, 14.15, 15.76, 15.63, 16.43, 14.73, 15.51, 15.28, 15.06, 16.25, 14.92, 13.62)
[benchmark]   Peak CPU usage: 114.45 % (103.10, 153.10, 114.10, 98.50, 122.00, 98.40, 121.90, 109.40, 126.60, 109.40, 87.40, 109.30, 112.50, 114.00, 142.10, 120.30, 104.60, 150.00, 109.40, 82.80)    
[benchmark]   Average Memory usage: 443.99 MB (399.76, 469.47, 461.83, 416.02, 425.19, 433.08, 395.28, 459.70, 462.49, 419.98, 462.35, 476.16, 459.91, 463.51, 461.39, 423.32, 461.10, 424.20, 465.06, 
439.96)
[benchmark]   Peak Memory usage: 1041.65 MB (1056.51, 1166.15, 1030.32, 975.46, 1107.44, 920.04, 1028.92, 1062.10, 1067.24, 1075.11, 1040.00, 1087.46, 1005.02, 1073.69, 1040.01, 1034.72, 1059.59, 1003.81, 1081.23, 918.20)

v8, no DL, Node 10.10.0

[benchmark] Benchmarking process over 20 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\new-project\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 39715.85 ms (41254.00, 38917.00, 38954.00, 39452.00, 39908.00, 39124.00, 41544.00, 39056.00, 39008.00, 38737.00, 39794.00, 41248.00, 41226.00, 42391.00, 38638.00, 41751.00, 36910.00, 41124.00, 39960.00, 35321.00)
[benchmark]   Average Process usage: 1.02 process(es) (1.37, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.15 process(es) (4.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 19.27 % (20.15, 18.37, 17.48, 19.86, 20.59, 19.83, 20.60, 19.03, 18.46, 18.66, 19.95, 18.84, 20.64, 21.30, 18.69, 20.99, 17.54, 19.45, 18.83, 16.05)
[benchmark]   Peak CPU usage: 118.53 % (170.40, 120.30, 89.00, 109.40, 140.70, 120.30, 132.90, 98.40, 107.80, 86.00, 93.70, 101.50, 132.80, 128.20, 112.60, 154.70, 90.60, 145.40, 134.40, 101.60)     
[benchmark]   Average Memory usage: 313.02 MB (360.81, 308.63, 312.78, 311.23, 310.24, 310.67, 312.26, 309.85, 310.46, 313.62, 310.08, 313.48, 308.39, 306.88, 309.72, 311.82, 311.05, 309.78, 310.82, 
307.85)
[benchmark]   Peak Memory usage: 800.08 MB (774.48, 774.38, 763.67, 807.96, 786.14, 815.06, 835.76, 762.01, 767.66, 861.41, 825.32, 855.64, 794.63, 767.91, 798.90, 805.75, 824.83, 823.76, 791.04, 765.35)

v8, no DL, Node 12.3.1

[benchmark] Benchmarking process over 20 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\new-project\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 29947.20 ms (28681.00, 30854.00, 28829.00, 28346.00, 28471.00, 29073.00, 28534.00, 28585.00, 27104.00, 27578.00, 27421.00, 28399.00, 29042.00, 27301.00, 27480.00, 33175.00, 37174.00, 34747.00, 34038.00, 34112.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 15.59 % (14.65, 17.22, 15.86, 14.82, 15.21, 15.75, 15.03, 15.15, 12.67, 14.51, 13.17, 14.72, 14.46, 12.78, 13.69, 17.00, 19.96, 18.69, 17.80, 18.63)
[benchmark]   Peak CPU usage: 97.90 % (90.70, 106.20, 90.70, 90.70, 117.30, 89.00, 96.80, 103.10, 92.20, 75.00, 92.20, 93.80, 96.90, 92.20, 81.20, 134.40, 98.40, 106.20, 95.30, 115.60)
[benchmark]   Average Memory usage: 303.56 MB (294.81, 314.64, 298.23, 310.19, 303.41, 295.35, 293.39, 294.60, 310.87, 311.56, 316.21, 296.54, 305.72, 315.03, 315.93, 313.65, 311.67, 291.89, 283.43, 
294.12)
[benchmark]   Peak Memory usage: 847.21 MB (822.65, 948.96, 819.03, 852.73, 859.57, 764.29, 749.08, 811.35, 887.66, 905.15, 938.75, 799.78, 878.00, 912.18, 929.51, 819.72, 897.77, 769.11, 734.79, 844.19)

v7, Node 10.10.0

[benchmark] Benchmarking process over 20 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\new-project\v7)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 34065.15 ms (48589.00, 30736.00, 29772.00, 32748.00, 35670.00, 37796.00, 35251.00, 32773.00, 33065.00, 32493.00, 32957.00, 32942.00, 33630.00, 34067.00, 33241.00, 32454.00, 32997.00, 33366.00, 32509.00, 34247.00)
[benchmark]   Average Process usage: 1.06 process(es) (2.20, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.20 process(es) (5.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 15.93 % (17.74, 15.14, 14.45, 15.67, 16.94, 18.67, 17.13, 14.81, 15.50, 15.68, 15.22, 15.17, 17.00, 15.39, 15.49, 15.34, 15.15, 15.99, 15.94, 16.11)
[benchmark]   Peak CPU usage: 99.83 % (170.30, 98.50, 79.60, 117.20, 117.20, 96.90, 90.70, 82.70, 118.80, 106.20, 79.70, 82.90, 107.80, 93.70, 87.50, 84.30, 84.30, 89.00, 117.20, 92.10)
[benchmark]   Average Memory usage: 307.49 MB (455.09, 299.47, 299.79, 305.62, 300.43, 298.91, 298.40, 301.08, 300.22, 299.42, 298.72, 299.10, 299.30, 301.11, 299.16, 298.94, 300.26, 297.80, 298.66, 
298.40)
[benchmark]   Peak Memory usage: 805.72 MB (847.09, 820.75, 771.20, 851.10, 817.67, 831.61, 782.52, 815.22, 832.90, 798.27, 786.08, 753.38, 779.23, 821.07, 804.25, 824.09, 852.30, 765.94, 754.26, 805.57)

v7, Node 12.3.1

[benchmark] Benchmarking process over 20 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\new-project\v7)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 33382.50 ms (33115.00, 33274.00, 31484.00, 30663.00, 30814.00, 30642.00, 31120.00, 31219.00, 33208.00, 39141.00, 36040.00, 36106.00, 35269.00, 34795.00, 33526.00, 33014.00, 34918.00, 33872.00, 32882.00, 32548.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 17.92 % (17.56, 17.77, 16.47, 16.61, 15.86, 16.67, 16.90, 15.90, 18.98, 23.05, 18.36, 19.39, 20.03, 18.58, 17.52, 16.43, 19.60, 17.50, 17.58, 17.72)
[benchmark]   Peak CPU usage: 98.91 % (106.30, 99.90, 90.60, 98.40, 86.00, 87.50, 104.70, 93.70, 115.60, 128.10, 117.20, 103.10, 95.40, 85.90, 96.90, 81.30, 100.00, 85.90, 89.10, 112.60)
[benchmark]   Average Memory usage: 292.19 MB (293.59, 295.74, 293.63, 290.62, 293.48, 291.67, 292.69, 293.37, 293.36, 293.54, 293.94, 292.08, 291.17, 291.83, 292.20, 291.82, 283.89, 292.66, 291.06, 
291.52)
[benchmark]   Peak Memory usage: 846.53 MB (828.64, 886.21, 835.02, 820.26, 837.29, 883.92, 873.06, 882.54, 825.58, 889.52, 880.10, 831.73, 827.94, 868.43, 820.85, 871.18, 763.72, 800.90, 900.42, 803.35)

Super-productivity

From https://github.com/aviabird/angularspree

Provided by @johannesjo in https://github.com/angular/angular-cli/issues/13734#issuecomment-497393904

v8, DL, Node 10.10.0

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\super-productivity\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 269523.80 ms (257464.00, 255811.00, 285836.00, 279557.00, 268951.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 13.37 % (12.17, 12.49, 13.95, 15.03, 13.22)
[benchmark]   Peak CPU usage: 193.76 % (175.00, 179.80, 207.80, 223.40, 182.80)
[benchmark]   Average Memory usage: 1139.50 MB (1125.28, 1128.75, 1165.98, 1146.50, 1131.01)
[benchmark]   Peak Memory usage: 1753.78 MB (1712.68, 1726.49, 1809.78, 1796.38, 1723.58)

v8, DL, Node 12.3.1

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\super-productivity\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 302133.40 ms (379732.00, 271669.00, 295372.00, 289442.00, 274452.00)
[benchmark]   Average Process usage: 1.19 process(es) (1.97, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.80 process(es) (5.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 12.91 % (13.05, 12.47, 13.46, 13.09, 12.45)
[benchmark]   Peak CPU usage: 199.36 % (217.20, 198.50, 203.00, 178.10, 200.00)
[benchmark]   Average Memory usage: 1506.77 MB (1742.04, 1453.37, 1455.98, 1438.85, 1443.59)
[benchmark]   Peak Memory usage: 2335.48 MB (3000.09, 2137.67, 2118.84, 2190.41, 2230.37)

v8, no DL, Node 10.10.0

$ benchmark -- ng build --prod
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\super-productivity\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 137125.80 ms (134088.00, 140877.00, 137668.00, 139447.00, 133549.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 10.91 % (10.27, 11.17, 11.13, 11.15, 10.82)
[benchmark]   Peak CPU usage: 155.30 % (143.80, 154.70, 164.00, 153.10, 160.90)
[benchmark]   Average Memory usage: 944.34 MB (948.63, 948.80, 960.01, 950.97, 913.27)
[benchmark]   Peak Memory usage: 1664.48 MB (1723.53, 1670.94, 1670.71, 1684.96, 1572.24)

v8, no DL, Node 12.3.1

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\super-productivity\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 138378.80 ms (136618.00, 140146.00, 136281.00, 139331.00, 139518.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 10.63 % (10.72, 10.78, 10.38, 10.12, 11.17)
[benchmark]   Peak CPU usage: 154.38 % (162.50, 148.40, 167.20, 145.30, 148.50)
[benchmark]   Average Memory usage: 1173.09 MB (1182.91, 1179.12, 1162.80, 1162.17, 1178.46)
[benchmark]   Peak Memory usage: 2064.48 MB (2104.92, 2113.08, 2071.26, 2007.69, 2025.45)

v7, Node 10.10.0

$ benchmark -- ng build --prod
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\super-productivity\v7)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 126331.00 ms (115769.00, 114542.00, 127353.00, 144113.00, 129878.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 12.75 % (11.16, 10.43, 12.28, 15.91, 13.96)
[benchmark]   Peak CPU usage: 160.92 % (175.00, 139.10, 140.60, 185.90, 164.00)
[benchmark]   Average Memory usage: 850.62 MB (851.26, 851.57, 859.25, 837.64, 853.40)
[benchmark]   Peak Memory usage: 1456.63 MB (1458.19, 1463.49, 1470.42, 1447.42, 1443.61)

v7, Node 12.3.1

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\super-productivity\v7)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 128415.80 ms (121753.00, 121720.00, 124237.00, 143535.00, 130834.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 12.02 % (11.37, 11.24, 12.02, 13.66, 11.79)
[benchmark]   Peak CPU usage: 155.96 % (160.90, 175.10, 176.60, 186.00, 81.20)
[benchmark]   Average Memory usage: 1126.36 MB (1138.10, 1105.06, 1135.59, 1126.35, 1126.68)
[benchmark]   Peak Memory usage: 1892.42 MB (1864.26, 1773.99, 2055.74, 1777.96, 1990.18)

filipesilva on 7 Jun 2019

👍3

My conclusions are this point are:

I see increased memory usage just from using Node 12:
- Super-productivity showed peak 1753 MB on v8, DL, Node 10.10.0 vs 2335 MB v8, DL, Node 12.3.1
I see about 15% extra memory usage between v7 and v8 for the same (non DL) case:
- Super-productivity showed peak 1664 MB on v8, no DL, Node 10.10.0 vs 1456 MB v7, Node 10.10.0
- Super-productivity showed peak 2064 MB on v8, no DL, Node 12.3.1 vs 1892 MB v7, Node 12.3.1
On Windows, the peak process usage is always 1. Our minifier plugin (Terser) is the only thing that uses extra processes. I think it's not using them on windows, I don't know why. Maybe I'll investigate it later. But it's not super relevant for memory usage because parallelizing something will always use more memory, even if it's not a lot. Might be related though.

I have gathered a few more example repositories, but after doing all these benchmarks manually I think I have to automate this. I really should have automated it from the start really. So I'll set up a repository that does this using CI.

filipesilva on 7 Jun 2019

👍3

well actually we have a project where setting max_old_space_size=8192 worked, but failed with max_old_space_size=4096, however with 8192 the memory usage peaked at around 4-6g btw. we used max_old_space_size=2048 with angular v7.x.

I'm pretty sure there is a leak. Maybe it only happens with build-optimizer?

schmitch on 7 Jun 2019

Any news? workarounds? Do you guys need more benchmarks?

hugo-marello on 7 Jun 2019

This is also a problem we are facing with https://github.com/vmware/clarity on the master branch right now. We are unable to build and deploy our website because we run out of memory and other SSR issues. To see memory issues, run npm run website:prerender to trigger the website build.

gnomeontherun on 7 Jun 2019

Heya, I've put up results and methodology in https://github.com/filipesilva/angular-cli-perf-benchmark. Will be adding more projects there next week.

The only extra detail that I have is that in the awesome-angular-workshop project build times almost doubled from CLI 7 to 8. But they didn't on the other projects I tested.

If someone wants to follow the methodology there using the scripts, it should be easy enough. If you find something interesting let me know. If you have a large project that you feel could yield interesting benchmarks, let me know (or add a PR) and I'll add it there.

filipesilva on 7 Jun 2019

👍1

same issue here on
Darwin Miguels-MacBook-Pro.local 17.7.0 Darwin Kernel Version 17.7.0: Thu Dec 20 21:47:19 PST 2018; root:xnu-4570.71.22~1/RELEASE_X86_64 x86_64

angular cli 5.2.11

MVCpp on 8 Jun 2019

Build Problem Solved. Suggestion is to change build logic. Instead of angular ng build use node build

arayik-yervandyan on 10 Jun 2019

👎15 😕2

@arayik-yervandyan I'd like to keep this open, as there's more to this issue than just increasing the memory limit. Some projects are exhibiting disproportionate increases in memory usage, and we also seem to have memory that is not freed on differential loading builds.

filipesilva on 10 Jun 2019

👍3

@filipesilva sure no problem. Just for the information I can say that in our version we increased memory limit into 4096 or 8192 I didn't remember correctly but the problem still exists and when build was changes from ng build into node build then build process passed successfully.

arayik-yervandyan on 10 Jun 2019

When you say node build, what exactly do you mean?

filipesilva on 10 Jun 2019

I think he meant something in the lines of node --max_old_space_size=4096 ./node_modules/@angular/cli/bin/ng build --prod, so the argument would be passed to node environment. But there are other issues linked to this one, like the huge time and memory spent while compiling.

hugo-marello on 10 Jun 2019

👍1

I agree with @hugo-marello , my use case is indeed this one. During a simple ng serve + args, I went through out of memory errors every time it tried to recompiled. The CPU went crazy at 100% for me. I tried the workaround by setting node --max_old_space_size=4096 ./node_modules/@angular/cli/bin/ng + args and it worked well to me.

I do see bigger times to recompile nevertheless, but it doesn' crash anymore. So, i feel that something is still off by going to the angular 8 (+cli). By the way, i already use node 12.

johanchouquet on 11 Jun 2019

I finished benchmarking the projects I had in https://github.com/filipesilva/angular-cli-perf-benchmark.

When looking at those results, keep this note in mind:

It's important to note that the first build of the CLI version 8 with differential loading and CLI version 7 variants will always use more cores and take more memory.
This happens because terser-webpack-plugin uses both parallel processes and a cache that is cleared on reinstalling the node modules. Cache hits don't spawn extra processes nor perform any work.

This is a bit unfortunate and makes the results harder to read. If I repeat this in the future I'll try to find a way of always busting/ignoring the Terser cache. In the results below I am mostly looking at the numbers with cache, but add a single comparison without cache as well.

Below are the results. Outliers are in bold. N10 and N12 means Node 10.16.0 and Node 12.4.0 respectively. CLI7, CLI8 and CLI8DL mean Angular CLI version 7, version 8 and version 8 with differential loading.

Results

cli-eight-project

In N12 going from CLI7 to CLI8 adds ~20mb (5%) to memory usage and ~60mb (7%) to peak memory.

CLI8DL adds ~190mb to both memory and peak memory, and extra 60% build time.

N10 shows similar numbers.

N12 seems to use about 6% extra memory over N10.

Builds without cache in N12 go from 1063mb peak memory usage in CLI 7 to 1191mb in CLI8DL.

super-productivity

In N12 going from CLI7 to CLI8 adds ~60mb (5%) to memory usage but peak memory stays the same.

CLI8DL adds ~200mb to both memory and peak memory, and extra 90% build time.

N10 shows similar numbers.

N12 seems to use about 30% extra memory over N10.

Builds without cache in N12 go from 2886mb peak memory usage in CLI 7 to 3131mb in CLI8DL.

awesome-angular-workshop

In N12 going from CLI7 to CLI8 adds ~410mb (45%) to memory usage and ~300mb (30%) to peak memory**.

CLI8DL adds ~200mb to both memory and peak memory, and extra 150% build time.

N10 shows similar numbers.

N12 seems to use about 25% extra memory over N10.

Builds without cache in N12 go from 2515mb peak memory usage in CLI 7 to 2554mb in CLI8DL.

angular.io

In N12 going from CLI7 to CLI8 adds ~50mb (6%) to memory usage and ~350mb (25%) to peak memory.

CLI8DL adds ~270mb to memory, 100mb to peak memory, and extra 90% build time.

N10 shows similar numbers.

N12 seems to use about 17% extra memory over N10.

Builds without cache in N12 go from 2898mb peak memory usage in CLI 7 to 2243mb in CLI8DL.

spartacus

In N12 going from CLI7 to CLI8 adds ~290mb (32%) to memory usage and ~420mb (25%) to peak memory.

N10 shows 40% extra memory usage and peak memory.

N12 seems to use about 27% extra memory over N10.

Builds without cache in N12 go from 2011mb peak memory usage in CLI 7 to 2326mb in CLI8.

clarity

In N12 going from CLI7 to CLI8 adds ~50mb (2%) to memory usage but peak memory stays the same.

N10 shows similar numbers.

N12 seems to use about 70% extra memory over N10.

Builds without cache in N12 go from 6000mb peak memory usage in CLI 7 to 5000mb in CLI8.

Conclusions

The first build always takes longer and uses more memory due to caching of some build results. On realistic scenarios most builds will invalidate some of the cache and take longer too.

Using differential loading seems to add a ~200mb to memory usage. This value does not change much, which seems to indicate that something that doesn't depend much on project size is being retained in memory between builds.

Node 12 is using between 6% and 70% extra memory than Node 10. Most projects tested showed a ~30% increase.

awesome-angular-workshop suffers a disproportionate increase in memory usage (45%) and build time going (150%) from CLI7 to CLI8DL. This indicates a localized regression.

spartacus shows a higher memory usage difference between CLI7 and CLI8 in Node 10 when compared to Node 12. This indicates Node 12 is managing memory better in some cases.

clarity uses 70% extra memory in Node 12 when compared to Node 10. This indicates that Node 12 can use much more memory in some cases.

filipesilva on 11 Jun 2019

If you are on Node 12 and hitting the memory limit, you should try Node 10 instead for now.

We'll look into the retained memory during Differential Loading builds and the outliers like awesome-angular-workshop.

filipesilva on 11 Jun 2019

👍1

actually im pretty sure that even with "target": "es5" that the memory usage was gone trough the roof. I would not just blame differential loading for that. I would actually blame TerserPlugin which can be highly ineffiecent and actually is since angular 8.

schmitch on 11 Jun 2019

@schmitch as far as I can tell DL accounts for an almost static amount of extra memory (~200ish). So it's not great but also not a big deal in most cases.

Regarding TerserPlugin, you can disable it with --optimization=false. You can set this option on the prod config in angular.json too. But cases where you turn it off are cases where you'd just use non prod mode instead really.

filipesilva on 11 Jun 2019

While Node 12 will attempt to automatically configure the --max_old_space_size option, the V8 method used to do so will cap the value at 2048MB.

clydin on 11 Jun 2019

Reported the new 2048MB default maximum in https://github.com/nodejs/node/issues/28202.

filipesilva on 13 Jun 2019

Reported Node 12 using more memory in https://github.com/nodejs/node/issues/28205.

filipesilva on 13 Jun 2019

Ran into this today when trying to enable source maps for my production build.

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0x8dc510 node::Abort() [ng build --prod]

Senneseph on 13 Jun 2019

I had this error; turns out it was because I had left a backup node_modules folder in my project directory. Deleting it fixed the issue.

simonewebdesign on 17 Jun 2019

My Angular builds are taking a lot longer today and some are even failing with a FATAL ERROR.

saxenarajat on 19 Jun 2019

ng serve --prod


** Angular Live Development Server is listening on localhost:4200, open your browser on http://localhost:4200/ **
 92% chunk asset optimization TerserPlugin                                       
<--- Last few GCs --->

[1730:0x103800000]   217578 ms: Scavenge 1350.0 (1422.7) -> 1349.4 (1423.2) MB, 7.1 / 0.0 ms  (average mu = 0.212, current mu = 0.135) allocation failure 
[1730:0x103800000]   217589 ms: Scavenge 1350.2 (1423.2) -> 1349.6 (1423.7) MB, 7.4 / 0.0 ms  (average mu = 0.212, current mu = 0.135) allocation failure 
[1730:0x103800000]   217606 ms: Scavenge 1350.4 (1423.7) -> 1350.1 (1424.7) MB, 12.6 / 0.0 ms  (average mu = 0.212, current mu = 0.135) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x2d9cf905be3d]
    1: StubFrame [pc: 0x2d9cf90134b0]
Security context: 0x3a5b0e11e6e1 <JSObject>
    2: new SourceNode [0x3a5b0f4ff279] [/Users/rac/Desktop/git/requisite-variety/node_modules/webpack-sources/node_modules/source-map/lib/source-node.js:~35] [pc=0x2d9cfb660621](this=0x3a5b97cf6d31 <SourceNode map = 0x3a5be9ed4219>,aLine=289,aColumn=31,aSource=0x3a5b7b5a01f1 <String[122]: /Users/rac/Desktop/git/requisite...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10003ae75 node::Abort() [/usr/local/bin/node]
 2: 0x10003b07f node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0x1001a7ae5 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0x100572ef2 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 5: 0x1005759c5 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/usr/local/bin/node]
 6: 0x10057186f v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 7: 0x10056fa44 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
 8: 0x10057c2dc v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 9: 0x10057c35f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
10: 0x10054bca4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/usr/local/bin/node]
11: 0x1007d3b54 v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
12: 0x2d9cf905be3d 
Abort trap: 6

npm v 6.4.1
node v10.13.0
"@angular/core": "~7.2.0"
"@angular/cli": "~7.3.7",

bhumin3i on 20 Jun 2019

@filipesilva , I gave it a shot with the benchmark tool you gave.

On my computer, it passes, it's really on Azure that it is not. On my computer, with AOT, I get about 2GB for the build.
I gave a shot with AOT on/off, because DL is not of a concern for me.
I have updated Angular to 8.0.2.

On Azure, without AOT, it works, but I get a Aw Snap out of memory after 15 minutes of inactivity in Chrome. I can't point the finger to AOT just yet because I did move from Angular 6 to Angular 8. Also, on my machine, I don't get this Aw snap (the code is not minified either and I use ng serve).

This whole Angular 8 is giving me quite the headache to run on production. But we wanted to get the benefit of a higher version of TypeScript...

AOT=on

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build 3cconnect --prod --configuration staging (at D:\git\clients\hbti\Portal\hbti-web-master\HBTI.ServiceTech.Web2.Client)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 189010.00 ms (345029.00, 148765.00, 153791.00, 152923.00, 144542.00)
[benchmark]   Average Process usage: 1.33 process(es) (2.64, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 2.40 process(es) (8.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 13.13 % (10.44, 13.75, 14.26, 13.56, 13.65)
[benchmark]   Peak CPU usage: 184.38 % (171.90, 175.00, 196.90, 187.50, 190.60)
[benchmark]   Average Memory usage: 1024.72 MB (1143.24, 1001.46, 1041.20, 953.13, 984.56)
[benchmark]   Peak Memory usage: 1860.07 MB (3000.74, 1577.36, 1616.97, 1516.06, 1589.24)

AOT=off

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build 3cconnect --prod --configuration staging (at D:\git\clients\hbti\Portal\hbti-web-master\HBTI.ServiceTech.Web2.Client)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 233620.00 ms (233375.00, 231213.00, 226946.00, 249170.00, 227396.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 15.51 % (16.71, 15.90, 15.03, 14.74, 15.16)
[benchmark]   Peak CPU usage: 232.84 % (240.70, 306.30, 206.20, 236.00, 175.00)
[benchmark]   Average Memory usage: 1474.39 MB (1467.56, 1476.16, 1427.16, 1526.26, 1474.80)
[benchmark]   Peak Memory usage: 2213.60 MB (2217.81, 2224.59, 2184.01, 2224.69, 2216.90)

jsgoupil on 20 Jun 2019

same issue on ubuntu server 18.04, but this solved my problem:
node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod

Just dropping this in here but if anyone is using AWS codebuild make sure to update your build environment to be configured with more than the minimum 3GB memory, I followed the steps above but had to re-configure my environment with more memory.

gpazo on 21 Jun 2019

👍1

Heya all,

First of all sorry for the delay. I AFK for a good chunk of last week.

I'm trying to give as much detail about what I did to investigate this sort of problem because in the past I have been asked about it (looking at you @wardbell, @johnpapa, and @DanWahlin :wave:). I also find it useful for myself to come back to these write-ups in the future because I forget the details of how to do these things, or want to share with colleagues so they can do the same. A lot of work described here is done by fiddling directly in node_modules and changing logic in there to write files that I later debug, or log information, or use different packages, stuff like that. I find that fiddling with the JS sources really is the best way of investigating things because I don't know what I'm looking for most of the time, so it helps to explore.

At this point we've already ascertained that Node 12 has a default 2048MB memory limit (https://github.com/nodejs/node/issues/28202) and is using more memory than Node 10 (https://github.com/nodejs/node/issues/28205). This is unfortunate but a reality at the moment, so we should focus on Node 10 for now while the node folks address memory issues. I'm using Node 10.16.0 myself for testing.

I've been working on profiling where memory is used in the problematic builds. The first step to do that is decide on what project to get profiling data from. I chose @johnpapa's awesome-angular-workshop because in https://github.com/angular/angular-cli/issues/13734#issuecomment-500849058 I came to the conclusion it was the project where we could see the biggest impact:

awesome-angular-workshop suffers a disproportionate increase in memory usage (45%) and build time going (150%) from CLI7 to CLI8DL. This indicates a localized regression.

I made a fork of it at https://github.com/filipesilva/awesome-angular-workshop, commit SHA 9076a3d, with local modifications to get reproducible benchmarks easily. That's the code I used in https://github.com/filipesilva/angular-cli-perf-benchmark. The build command I use is ng build 5-ngrx-end --prod, since this workspace has several projects and I just want to use this specific one.

My initial idea was to use the Chrome Node DevTools, gather a bunch of profiling data, then work away at trying to explore it to find actionable information.

To open the Chrome Node DevTools, open in Chrome and click "Open dedicated DevTools for Node". When you have a node process started with the --inspect-brk flag, DevTools should automatically attach. An easy way to run Angular CLI like this is to make a npm script: "ng-inspect": "node --inspect-brk node_modules/@angular/cli/bin/ng",. Then if you run npm run ng-inspect -- build --prod and have DevTools open, it will automatically connect. At this point you can set breakpoints and take profiles.

The memory tab has a couple of different profiling tools and descriptions of what they do. When you use these, you will have to go to sources and click the resume script execution (top right) before things happen. I spent some time trialling the different types of profiles on simpler node processes to figure out which one would be better. I was already used to reading the CPU profiles from when I debug build time, so the Allocation sampling (aka heap profile) sounded interesting because it should show how much memory each function is using at a given time. Heap snapshots show whats in memory at a given time and what is retaining those objects, it would also work but isn't as easy to read and get the big picture of what's happening.

To be honest capturing profiles with DevTools has been a lot harder than I expected. It actually does not seem to deal well with capturing profiles from large running Node processes. I saw it crashing while taking profiles, or at the end when processing them, for most of the profiles I tried to take.

I tried to explore alternatives. One of them was to profile smaller parts at a time so that DevTools didn't crash, but it looked like it would always crash on some types of profiles anyway, even on new CLI projects, so that didn't seem to matter much. I tried ndb but it seemed to suffer from the same problems. automated-chrome-profiling seemed promising byt hadn't been updated in a while, and only took CPU profiles. @manekinekko suggested but as far as I could tell it wouldn't give me very granular information on memory usage. node-oom-heapdump looked good for heap snapshots but required native bindings, which I didn't like much because it would make it harder to later integrate directly into the CLI for user profiling. sampling-heap-profiler looked great for heap profiles, but also needed native bindings. @ofrobots suggested https://github.com/google/pprof-nodejs which I tried and seemed to give me really nice visualizations, but it also needed native bindings and needed a go tool to read it, which would make it harder to integrate.

Ultimately I opted to make some custom profiling code based on the Node experimental profiler support described here. I drew inspiration from code in sampling-heap-profiler and the profiler API to figure out what to use. From this I prototyped some automated profiling code that looks like https://github.com/angular/angular-cli/pull/14936. At the time of writing I manually control flags inside the file but maybe in the future we can expose it somehow. It can take CPU profiles of the whole process, or heap snapshot/profiles at set intervals. This seemed like a good way of doing it because I might want to ask people with build speed or memory problems in the future to take these profiles and send to us.

With this setup I was able to take reproducible profiles with minimal fiddling around. Reducing the fiddling is really important when it takes a long time to gather your information, because the more fiddling you need to do, the higher the likelyhood of you botching a given sample. If you botch a sample and only find it hours later, then you might have to throw away most of your conclusions for that period since your starting points were bad and so conclusions and comparisons weren't accurate.

Armed with this need tool I managed to capture a few heap profiles of CLI7 and CLI8 (without differential loading) builds to compare them. I opened them on DevTools and tried to figure out what parts of the build pipeline were involved in each section of the profile.

The CLI7 one looked like this:
cli7-annotated

The CLI8 one looked like this:
cli8-annotated

CLI8 uses around 180mb more memory, for a total of 760mb, versus the 580mb used by CLI7. What stood out the most to me was the difference in CSS processing. It looks like it takes a lot longer in CLI8. In CLI8 we started using dart-sass (published in npm as just sass) by default instead of node-sass (see https://github.com/angular/angular-cli/pull/11791, https://github.com/angular/angular-cli/pull/13950) so it's a great candidate for investigation.

We expected dart-sass to not be as performant as node-sass so we let you go back to node-sass by installing it in your project (npm install node-sass --save-dev). As long as it's installed, we'll pick it up automatically. You can also use dart-sass with the fibers package for better performance by installing it in the same way as node-sass. Both node-sass and fibers use native bindings and might not work on all machines, which is why we don't install them by default.

Then I compared profiles for CLI8, CLI8 with node-sass, CLI8 with fibers, and CLI 7 side by side to see how they changed.
sass-comparisons

So using node-sass seems to return memory usage CLI7-levels, at least on this project. Using just fibers only reduces memory by some 20mb.

As @clydin pointed out, this might not be the full picture since the heap profiles are only for memory managed by V8. node-sass might be using more memory that doesn't show up in the profile. So I benchmarked the processes themselves:

CLI 8

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build 5-ngrx-end --prod (at D:\sandbox\memory-debug\awesome-angular-workshop)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 157486.80 ms (157292.00, 159517.00, 156464.00, 159606.00, 154555.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 18.31 % (18.68, 18.63, 18.04, 18.76, 17.46)
[benchmark]   Peak CPU usage: 205.60 % (237.50, 204.60, 198.40, 196.90, 190.60)
[benchmark]   Average Memory usage: 1008.24 MB (1048.68, 975.86, 1017.15, 1020.31, 979.21)
[benchmark]   Peak Memory usage: 1560.48 MB (1588.38, 1489.02, 1583.70, 1574.51, 1566.81)

CLI 8 with node-sass

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build 5-ngrx-end --prod (at D:\sandbox\memory-debug\awesome-angular-workshop)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 103204.00 ms (98623.00, 103302.00, 105260.00, 104674.00, 104161.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 16.70 % (15.46, 15.94, 17.36, 17.48, 17.26)
[benchmark]   Peak CPU usage: 152.50 % (160.90, 89.10, 190.60, 128.10, 193.80)
[benchmark]   Average Memory usage: 676.14 MB (681.11, 674.49, 676.65, 675.08, 673.38)
[benchmark]   Peak Memory usage: 1443.15 MB (1444.21, 1439.97, 1435.09, 1449.96, 1446.53)

CLI 8 with fibers

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build 5-ngrx-end --prod (at D:\sandbox\memory-debug\awesome-angular-workshop)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 126098.00 ms (136145.00, 125063.00, 123654.00, 123849.00, 121779.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 14.55 % (16.18, 14.48, 14.25, 14.05, 13.79)
[benchmark]   Peak CPU usage: 190.30 % (215.50, 173.40, 184.40, 170.30, 207.90)
[benchmark]   Average Memory usage: 746.22 MB (752.84, 741.56, 745.46, 747.52, 743.72)
[benchmark]   Peak Memory usage: 1560.88 MB (1603.68, 1555.96, 1575.40, 1569.62, 1499.73)

So it does look like the node-sass really is the one that uses less memory, and takes less time to build. fibers is the second best. The average memory used in the fibers case seems to conflict with the results we got from the heap profiles though, here we see CLI8 with fibers using ~750mb versus the original CLI8 ~1000mb, but in the heap profile using fibers only reduced memory by ~20mb. I don't know why this is. Maybe it's because of the nature of averaging used memory, or maybe it's because there are more garbage collection cycles taking heap profiles. But at the end of the day node-sass seems to be the winner as far as performance goes.

I also found it weird that a big chunk of extra memory in the CLI8 heap profile seems to come from PostCSS processing. I wasn't expecting PostCSS to need to do anything different. I thought that maybe I just misinterpreted what that section was used for. But maybe the dart-sass output could be different from node-sass and that could cause more work for PostCSS. It was worth looking into so I added some code directly in sass-loader to write the results of sass compilation to disk. The output for dart-sass and node-sass seemed like it was the same really, aside for some whitespace, so I think I just misinterpreted the memory assigned to Sass processing as being assigned to PostCSS.

@johnpapa, as an aside, I did notice that most component CSS files in this project are importing mixin.scss. This file actually ~has no mixins and is just~ includes the whole material theme at 70kb (pre-optimizations) of toplevel CSS definitions. Component CSS is standalone and isolated from other CSS, so there is no deduplication or anything. When it's imported into component CSS, you're adding 70kb worth of CSS into that component, which is probably several times over the size of the component itself. @jelbourn has described importing non-mixin Sass this as an anti-pattern but it's not the sort of information that's obvious all round. We could do a better job of informing people about it, and warning them when component CSS is very large (cc @vikerman).

Another aside is that I also noticed that compiling just the single project via ng build 5-ngrx-end --prod actually compiles all .ts and .scss files inside src/. This happens because all projects in this workspace use the same ./src/tsconfig.app.json, and this tsconfig includes all ts files (except unit tests). Which is to say, compiling a single project here actually compiles all projects. The only way to truly address this is to have one tsconfig for each app. I wonder how common this mistake is. I wouldn't be surprised larger projects accidentally include extra TS files. Maybe we could warn users when we see the TS compilation has a lot more files than those that webpack loads (cc @vikerman, @clydin).

So I guess the takeaway here is that if you have a large project, you should install node-sass to your devDependencies. This will make building faster and take less memory.

Although it didn't seem to change between CLI7 and CLI8, the Angular compilation itself is still by far the largest contributor to the memory footprint, at around 230mb. Maybe we can fully drop all references to it when not building in watch mode, allowing Node to free up that memory. I'll explore that next.

filipesilva on 28 Jun 2019

👍7 ❤4 🚀1

@filipesilva just a short update: I tried using node-sass with angular 8 for super-productivity. The result is the same:

all builds work without sourcemaps
ng build --aot & ng serve both work with sourcemaps
ng build --prod throws the JavaScript heap out of memory error

I tried both Node 10 and 12. The results are the same.

johannesjo on 28 Jun 2019

@johannesjo ng build --aot and ng serve are vastly different than ng build --prod. I don't think there's anything to be gained by comparing them. Production builds will always use more memory than non-production builds.

When I benchmarked super-productivity I got these results for Node 10.16.0:

CLI version 8 with differential loading

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at /home/circleci/project/project)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 203022.00 ms (320720.00, 185220.00, 170860.00, 155420.00, 182890.00)
[benchmark]   Average Process usage: 1.35 process(es) (2.75, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.80 process(es) (5.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 164.51 % (154.16, 163.92, 168.56, 171.13, 164.77)
[benchmark]   Peak CPU usage: 557.11 % (650.00, 555.56, 520.00, 520.00, 540.00)
[benchmark]   Average Memory usage: 1333.09 MB (1709.25, 1234.43, 1236.67, 1225.88, 1259.21)
[benchmark]   Peak Memory usage: 2019.59 MB (2671.33, 1807.51, 1857.44, 1883.19, 1878.46)

CLI version 8 without differential loading

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at /home/circleci/project/project)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 87418.00 ms (91750.00, 94150.00, 92950.00, 77520.00, 80720.00)
[benchmark]   Average Process usage: 1.03 process(es) (1.13, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.60 process(es) (4.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 160.13 % (162.86, 155.83, 158.79, 161.31, 161.87)
[benchmark]   Peak CPU usage: 546.89 % (650.00, 544.44, 520.00, 510.00, 510.00)
[benchmark]   Average Memory usage: 1032.07 MB (1048.05, 1040.10, 1004.07, 999.71, 1068.42)
[benchmark]   Peak Memory usage: 1659.71 MB (1813.00, 1615.65, 1684.55, 1571.28, 1614.09)

CLI version 7

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at /home/circleci/project/project)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 89746.00 ms (145060.00, 78430.00, 77020.00, 77210.00, 71010.00)
[benchmark]   Average Process usage: 1.36 process(es) (2.81, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.80 process(es) (5.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 156.10 % (152.71, 155.33, 157.43, 156.50, 158.52)
[benchmark]   Peak CPU usage: 548.44 % (680.00, 522.22, 520.00, 520.00, 500.00)
[benchmark]   Average Memory usage: 1086.97 MB (1592.65, 963.42, 960.56, 971.31, 946.89)
[benchmark]   Peak Memory usage: 1702.50 MB (2539.70, 1438.24, 1606.59, 1474.61, 1453.35)

For production builds I didn't see a real difference between "CLI version 7" and "CLI version 8 without differential loading". There is a difference of ~200mb when using differential loading though. But you don't seem to be using differential loading currently (your root tsconfig has a es5 target).

We might be comparing different things though. I'm comparing using Angular 8 with both CLI 7 and 8. If you're testing with both Angular 7 and Angular 8 you might different results. What comparison are you making? Do you have two commits I can look at where only the version of Angular packages changed but ng build --prod went from completing successfully to failing with out of memory errors?

filipesilva on 28 Jun 2019

@filipesilva The crucial setting seems to be the buildOptimizer flag (or it is at least the one with the biggest impact on memory usage). When I disable it. The build runs fine regardless of the other config params even with sourceMap set to true.

I've no idea how all the internals work, so this is probably not too helpful, but my wild guesses about the cause would be the following:

The build optimizations are somehow affected by the existence of the sourcemaps or the related code inside the css
Sourcemap Extraction is run in parallel with the build optimizations
Sourcemap Extraction is run before build optimizations but memory is not cleaned up until the whole build task is run or vice versa
Both processes are somehow intertwined

Maybe there is some inspiration about the real cause in it ;)

johannesjo on 28 Jun 2019

@johannesjo both Build Optimizer and source maps need extra memory to process so that's expected. It's just more work to be done, and more intermediate data to store in memory. We are looking at moving the Build Optimizer step from app builds to library builds, but that would be in 9 only at best.

Edit: tagged @JohannesHoppe instead of @johannesjo. sorry about that!

filipesilva on 1 Jul 2019

👍3

Still working on this problem. Notes from the last few days below.

While discussing it with @clydin, he mentioned that the average memory isn't really what busts the limit. It's the peak memory. Average is ~750mb while peak is ~1500mb. We should focus on that instead.

First I need to figure out what's using memory at the peak. I guess it's uglify and build optimizer but I don't know. It's weird that the profiles I take never show peak usage though. I wonder I I'm doing something wrong there. I just started a process without any profiling and see memory at ~1200 during what seems like the ng compilation, then going down to ~900mb when loading files, then ~1300 and ~1500mb at the 90%+ stages. But my profiles show a ~800mb allocation max.

Maybe I need to control the timing of when I take these profiles. If that's it, then I need to find the exact time when usage is high and take a profile there. Using process.memoryUsage() as described in https://www.valentinog.com/blog/memory-usage-node-js/ might let me identify high memory usage periods. I can set a memory threshold and only take profiles at that time.

It might be that taking a heap profile triggers GC, or just that the increased running time prompts GCs more often. I can try to trigger gc right before each heap profile and see if the result is the same. https://stackoverflow.com/questions/27321997/how-to-request-the-garbage-collector-in-node-js-to-run shows how. Did that, and saw no real difference in the profiles. Whatever the profiles show seems to actually be retained.

I used setInterval with a log for process.memoryUsage() inside, and also kept an eye on the reported task manager memory usage, and noticed a couple of things. When I say that I saw 1234/5678 I mean 1234 was on the task manager and 5678 was on the process.memory logging.

The highest 1300/???? happened during the Angular Compiler phase, but only with dart-sass. With node-sass it lower at 900/700 and saw interval logging. I got almost no logging for that phase with dart-sass, I think because the setInterval never got on the event loop properly, I guess because a lot of things are sync? With node-sass I saw a lot of logging at that time. Maybe it's a native binding thing, it might free up the event loop. Anyway dart-sass seems to make resource loading in the angular compiler extra bad by piling on memory on an already expensive set of operations.

I saw some peaks of 1200/900 while progress showed loading individual files. I think that's Build Optimizer since its a js loader.

I barely saw any memory usage during Terser, but I bet that's because of the cache. I turned off the terser-webpack-plugin cache and saw 1100/940, but also a bunch of spawned processes between 40 and 400mb.

So as far as I could tell, the sass thing was actually a big part of reaching the peak because it runs during the ng compilation and piles on something that's already memory intensive.

I bet it doesn't help that we're doing webpack subcompilations for that either. This also means that even if we freed all references to memory using during the ng compilation, it wouldn't matter as much because that compilation is still part of the peak.

We already have plans to remove Build Optimizer so that doesn't bother me much

In regards to terser... Well if we had a smaller main process memory footprint, obviously the total would be reduced. But maybe we can also make that plugin smarter. It seems to run on chunks proper
https://github.com/webpack-contrib/terser-webpack-plugin/blob/a0d83170168e786004fa12b94525b2934a17a4f5/src/index.js#L416-L419
But we know terser can't break out of the webpack module loader so there's no point is processing whole chunks. Might as well process the individual webpack modules, that should be much smaller.

So important things to follow up:

improve terser-webpack-plugin performance (https://github.com/webpack-contrib/terser-webpack-plugin/issues/104)
improve sass-loader performance (https://github.com/webpack-contrib/sass-loader/issues/701)
reduce peak ngc memory usage
free ngc memory after emit
remove build optimizer

filipesilva on 3 Jul 2019

👍5 😄1

Finally, as the other had suggested, running node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod fixes the problem, and node.exe could use up to 5.5 GB memory for compiling my big fat NG app. Apparently Angular 8 build use much more memory than Angular 7 build.

now for debug build, I use:
node --max_old_space_size=8192 "node_modules/@angular/cli/bin/ng" build --base-href /ng/ --aot

for production build, I use:
node --max_old_space_size=8192 "node_modules/@angular/cli/bin/ng" build --configuration=production

However, I would prefer to use ng build ... rather than `node --max_old_space_size=8192 "node_modules/@angular/cli/bin/ng" build.

It will be nice that NG CLI team would make ng build read max_old_space_size somewhere in config.

BTW, with almost identical code base before moving to NG8, the node memory usage with NG7 build is 1.3 GB. I had seen the others had reported such too much memory usage in NG 8 build.

zijianhuang on 4 Jul 2019

@zijianhuang going from 1.3 to 5.5 is a lot more than in other projects I have tested. Is your project somewhere I could see to run some benchmarks on it? Also, what Node version are you using?

filipesilva on 4 Jul 2019

@filipesilva, If you could provide a secured upload location or your Email address, I may zip both the ng7 codes and the ng8 code so you could build both and compare. Node and npm are basically the latest.

zijianhuang on 4 Jul 2019

I'd love to have a look. WDYT of you having a private github project that you grant me access? The private ones are now free for up to three collaborators so you wouldn't pay anything. You can also remove access at any time.

filipesilva on 4 Jul 2019

In my case a big difference is having es2015 as target in the tsconfig.json, I wasn't able to use your benchmark tool (haven't found anywhere how to use it) but with time the max rss of es5 is 1.3GB with es2015 5.9GB, after going from node10+angular7 to node12+angular8 all of our builds failed because of the memory usage

alex88 on 4 Jul 2019

@filipesilva , invitation sent. Please also check my benchmark result in readme.txt.

zijianhuang on 5 Jul 2019

@alex88 you can find the benchmark tool I used in https://github.com/filipesilva/angular-cli-perf-benchmark#benchmark-package. If you want to provide me a repro via a private github repository, I'd love to have a look too.

@zijianhuang a big thank you for giving me a repro to look at. I had a look at it today using the benchmark package described in https://github.com/filipesilva/angular-cli-perf-benchmark.

One thing to be aware of is that the memory numbers collected by that tool include spawned processes. In the CLI we use a minifier package (terser-webpack-plugin) that spawns multiple processes, so it's useful to capture the memory the child processes use as well.

But that package also caches the minification results. So the first run does more work than subsequent ones. So when I benchmarked your packages I manually edited ./node_modules/@angular-devkit/build-angular/src/angular-cli-files/models/webpack-configs/common.js and changed cache: true to cache: false to disable that cache and get consistent numbers.

I also added these two npm scripts to help me benchmark things:

        "benchmark": "benchmark -- node --max_old_space_size=8000 node_modules/@angular/cli/bin/ng build --prod",
        "ng-high-memory": "node --max_old_space_size=8000 node_modules/@angular/cli/bin/ng build --prod",

I set the same memory limit for all projects because I know Node will behave differently near the limit. So this way we should get more accurate numbers.

I had to remove the Bounds import from avatar.component.ts because that gave me a build error in NG7. It was just used as a type so it was easy to remove. I replaced "target": "es2015" with "target": "es5" in ./tsconfig.json. This disables Differential Loading as described here. I know it's an important part of CLI8 applications but we are already looking at a different implementation of it that shouldn't require as much resources, so I didn't want to also benchmark it. This way I could compare mostly equal applications.

I noticed that your package.json between the two versions changes a lot of packages that aren't just the Angular packages. That might have an impact on the benchmarks. After trying to reproduce your initial numbers I'll also try to alter only Angular packages to see if I get different results.

Dev environment

In your reproduction your said:

Dev Environment:
* Intel Nuc i7, 16 GB memory, SSD
* Windows 10

Benchmarking tool: Process Explorer

I'm using my work machine (Win10, i7, 16GB RAM, SSD) which seems similar to your env. To benchmark I used the tool mentioned in https://github.com/filipesilva/angular-cli-perf-benchmark#benchmark-package. I'm also using Node 10.16.0 and npm 6.9.0.

Your test cases

In your reproduction your said:

Test case 1:
* Node 10.15.3
* npm 6.4.0
* NG 7 and NG CLI 7

Node memory usage peak: 1.3 GB.

I read from the other post that the default limit of memory usage of Node is 1.5 GB. The "NG CLI" command apparently is using such default.


Test case 2:
* Node 10.15.3 / 10.16.0
* npm 6.4.0 / 6.10.0
* NG 8.1 and NG CLI 8.1

Memory usage peak in the DEBUG build: 3.6 GB / 3.5 GB
Memory usage peak in the Production build: 3.3 GB / 3.4 GB


Test case 3:
* Locally installed Node 12.6 through "npm i"
* npm 6.9.0
* NG 8.1 and NG CLI 8.1

Memory usage peak: 5.5 GB

I'm not really going to try Test case 3 because we already know that Node 12 is using more memory in general (https://github.com/nodejs/node/issues/28205). I don't think there's much we can do about that.

Your Test case 1 is the CLI7/NG7/NG7deps scenario. I also say NG7deps because those dependencies are different from your NG8 project, which I'll say has NG8deps. Later on that might make a difference. Ideally we'd only change CLIx/NGx to get numbers.

So for CLI7/NG7/NG7deps (your Test case 1) I got:

$ ng version

Angular CLI: 7.3.9
Node: 10.16.0
OS: win32 x64
Angular: 7.2.15
... animations, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, platform-server
... router, service-worker

Package                            Version
------------------------------------------------------------
@angular-devkit/architect          0.13.9
@angular-devkit/build-angular      0.13.9
@angular-devkit/build-optimizer    0.13.9
@angular-devkit/build-webpack      0.13.9
@angular-devkit/core               7.3.3
@angular-devkit/schematics         7.3.3
@angular/cdk                       7.3.3
@angular/cli                       7.3.9
@angular/flex-layout               7.0.0-beta.24
@angular/material                  7.3.3
@angular/material-moment-adapter   7.3.3
@angular/pwa                       0.13.3
@ngtools/webpack                   7.3.9
@schematics/angular                7.3.3
@schematics/update                 0.13.9
rxjs                               6.5.2
typescript                         3.2.2
webpack                            4.29.0

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 node_modules/@angular/cli/bin/ng build --prod (at D:\sandbox\memory-debug\testngcli8\NG7Source)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 288429.80 ms (297613.00, 292811.00, 282489.00, 284134.00, 285102.00)
[benchmark]   Average Process usage: 2.65 process(es) (2.60, 2.69, 2.66, 2.64, 2.64)
[benchmark]   Peak Process usage: 8.00 process(es) (8.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 15.64 % (14.82, 16.84, 15.28, 15.27, 16.00)
[benchmark]   Peak CPU usage: 252.80 % (251.60, 300.00, 224.90, 261.00, 226.50)
[benchmark]   Average Memory usage: 1932.92 MB (1885.25, 1947.53, 1937.02, 1953.46, 1941.31)
[benchmark]   Peak Memory usage: 4024.00 MB (4115.83, 4092.19, 3943.85, 4018.98, 3949.16)



md5-d28c1dffbd11703a041dffc7b6a73087



$ ng version

Angular CLI: 8.1.0
Node: 10.16.0
OS: win32 x64
Angular: 8.1.0
... animations, cli, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, platform-server
... router, service-worker

Package                            Version
------------------------------------------------------------
@angular-devkit/architect          0.801.0
@angular-devkit/build-angular      0.801.0
@angular-devkit/build-optimizer    0.801.0
@angular-devkit/build-webpack      0.801.0
@angular-devkit/core               8.1.0
@angular-devkit/schematics         8.1.0
@angular/cdk                       8.0.2
@angular/flex-layout               8.0.0-beta.26
@angular/material                  8.0.2
@angular/material-moment-adapter   8.0.2
@angular/pwa                       0.801.0
@ngtools/webpack                   8.1.0
@schematics/angular                8.1.0
@schematics/update                 0.801.0
rxjs                               6.5.2
typescript                         3.4.5
webpack                            4.35.2



md5-1c09c7c9efd6bc8f010d05171d767994



[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 node_modules/@angular/cli/bin/ng build --prod (at D:\sandbox\memory-debug\testngcli8\NG8Source)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 253091.40 ms (281761.00, 251054.00, 242851.00, 244676.00, 245115.00)
[benchmark]   Average Process usage: 2.72 process(es) (3.34, 2.52, 2.61, 2.58, 2.54)
[benchmark]   Peak Process usage: 8.80 process(es) (12.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 12.94 % (15.28, 12.50, 12.20, 12.55, 12.18)
[benchmark]   Peak CPU usage: 210.62 % (223.50, 287.50, 173.50, 189.00, 179.60)
[benchmark]   Average Memory usage: 1997.28 MB (1980.34, 1920.54, 2051.03, 2025.27, 2009.23)
[benchmark]   Peak Memory usage: 4313.95 MB (4445.09, 4162.86, 4258.15, 4380.61, 4323.05)



md5-3ac01d1544d6859e233401c87995acf5



[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 node_modules/@angular/cli/bin/ng build --prod (at D:\sandbox\memory-debug\testngcli8\NG8Source)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 184542.00 ms (240928.00, 167820.00, 169525.00, 176387.00, 168050.00)
[benchmark]   Average Process usage: 1.31 process(es) (2.57, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 2.40 process(es) (8.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 11.67 % (11.08, 11.67, 11.74, 12.13, 11.73)
[benchmark]   Peak CPU usage: 173.76 % (268.80, 187.50, 167.20, 121.90, 123.40)
[benchmark]   Average Memory usage: 1569.17 MB (1997.59, 1421.48, 1607.34, 1424.31, 1395.14)
[benchmark]   Peak Memory usage: 2788.43 MB (4277.19, 2291.22, 2705.78, 2327.46, 2340.52)



md5-6a4dedbb221c789a67d1ba9659725dc1



[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 node_modules/@angular/cli/bin/ng build --prod (at D:\sandbox\memory-debug\testngcli8\NG8Source)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 448765.60 ms (450442.00, 441786.00, 453261.00, 442406.00, 455933.00)
[benchmark]   Average Process usage: 2.62 process(es) (2.63, 2.62, 2.65, 2.61, 2.61)
[benchmark]   Peak Process usage: 8.00 process(es) (8.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 11.07 % (11.65, 11.65, 11.10, 10.18, 10.78)
[benchmark]   Peak CPU usage: 280.28 % (307.70, 356.30, 237.40, 256.20, 243.80)
[benchmark]   Average Memory usage: 2484.00 MB (2467.08, 2548.74, 2480.70, 2471.22, 2452.25)
[benchmark]   Peak Memory usage: 5050.35 MB (5043.85, 5075.19, 5051.67, 5053.34, 5027.68)



md5-b0ecc6954a8df79193416e981408ff5e



[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 node_modules/@angular/cli/bin/ng build --prod (at D:\sandbox\memory-debug\testngcli8\NG8Source)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 246739.80 ms (222235.00, 246069.00, 244327.00, 252636.00, 268432.00)
[benchmark]   Average Process usage: 2.55 process(es) (2.55, 2.59, 2.58, 2.56, 2.46)
[benchmark]   Peak Process usage: 8.00 process(es) (8.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 12.83 % (11.53, 12.41, 12.92, 13.34, 13.97)
[benchmark]   Peak CPU usage: 229.42 % (287.40, 268.90, 197.00, 212.50, 181.30)
[benchmark]   Average Memory usage: 1947.94 MB (1941.51, 2076.98, 1938.72, 1926.15, 1856.36)
[benchmark]   Peak Memory usage: 4129.88 MB (4141.96, 4347.47, 4064.12, 4071.99, 4023.87)

This doesn't seem to be very different from the ES5 benchmark. @alex88 maybe when you used es2015 you were actually getting the DL builds? You should also not be using Node 12 for now because of https://github.com/nodejs/node/issues/28205.

@zijianhuang so to wrap up, I think the reason your project uses more resources with NG8/CLI8 than with NG7/CLI7 is that DL was turned on and that some builds were without cache. I really can't be sure of the second point because I don't know how you're caching things locally and in CI, but the numbers I got here seem to point to that.

For us the follow up from investigating your project's transition from Angular version 7 to 8 is that we really need to figure out what memory is retained between DL builds because that points to a leak, and that we need to have a more performant DL setup (@clydin has some ideas on that).

filipesilva on 5 Jul 2019

😄1

Hello @filipesilva , I have written a while back on this thread mentioning my benchmarks and what was not working for my case. It seems that the AOT is really not working for me. Is there any investigation being done in this area?

jsgoupil on 5 Jul 2019

@jsgoupil in https://github.com/angular/angular-cli/issues/13734#issuecomment-506760767 I saw a mimimum of 240mb and a maximum of about 1gb of memory being used in the AOT compilation. I tried for a couple of days to figure out exactly where but didn't have much success. There's still a couple more things I want to try though. And I also think the minimum is related to the "carry-over" memory I see on DL builds.

But that being said there is no getting around the fact that AOT does a lot more work and requires more resources. That's not a bug. It's just a much more involved build step. It might be requiring disproportionally more resources than non-AOT, but it will never require less.

Did you see the end comments in https://github.com/angular/angular-cli/issues/13734#issuecomment-506760767 about the compilation units? It might be that your tsconfig is including a lot of files you don't use as well, or that component sass is larger than you expect (if you use sass).

If you give me access to your code somehow I can take a look and see if I find anything odd.

filipesilva on 5 Jul 2019

@filipesilva , thank you very much for such detailed and inspirational analysis. I had done some minor changes to the build config, some of which are similar to you CLI team had done during the analysis. I had altered the browserslist:

    "browserslist": [
        "last 2 Chrome versions",
        "last 2 Safari versions",
        "last 2 Firefox versions",
        "not dead",
        "not IE 9-11"
    ],

to ensure only es2015 build is produced.

For the DEBUG build, the peak memory usage is 1.9 GB.
For the Production build, the peak memory usage is 2.0 GB.

This is actually a small jump from 1.3 GB comparing with build with NG7/NG CLI 7, which also produce only es2015 build. The default 1.5 GB memory limit of Node is the pitfall, however overall the small jump is reasonable. Similar to you had analyzed.

zijianhuang on 6 Jul 2019

@jsgoupil got in touch and let me have a look at his project. I turned off the terser cache (described in https://github.com/angular/angular-cli/issues/13734#issuecomment-508815407) and benchmarked the "ng build 3cconnect --prod" command:

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build 3cconnect --prod (at D:\sandbox\memory-debug\jsgoupil)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 258795.40 ms (280888.00, 250100.00, 241312.00, 256762.00, 264915.00)
[benchmark]   Average Process usage: 3.40 process(es) (3.41, 3.39, 3.42, 3.41, 3.39)
[benchmark]   Peak Process usage: 8.00 process(es) (8.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 11.59 % (13.45, 10.73, 10.52, 11.53, 11.75)
[benchmark]   Peak CPU usage: 236.56 % (237.60, 325.10, 168.60, 207.80, 243.70)
[benchmark]   Average Memory usage: 1523.76 MB (1547.40, 1471.40, 1544.40, 1562.83, 1492.76)
[benchmark]   Peak Memory usage: 2898.84 MB (2940.99, 2707.18, 2969.51, 3049.01, 2827.53)

This build has AOT and Build Optimizer turned off, and is using Differential Loading. The average memory usage is right at the maximum as well, which means the garbage collector is doing a lot of work to free up memory as the build progresses.

Tried turning on AOT and Build Optimizer and got OOM on the second DL build, during TerserPlugin. Used the node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build 3cconnect --prod --aot --build-optimizer command to build and it succeeded.

Benchmark for this build shows the following numbers:

$ benchmark -- node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build 3cconnect --prod --aot --build-optimizer
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build 3cconnect --prod --aot --build-optimizer (at D:\sandbox\memory-debug\jsgoupil)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 571885.00 ms (567385.00, 558561.00, 574671.00, 572119.00, 586689.00)
[benchmark]   Average Process usage: 2.93 process(es) (2.84, 2.90, 3.02, 2.89, 3.02)
[benchmark]   Peak Process usage: 8.40 process(es) (8.00, 8.00, 9.00, 8.00, 9.00)
[benchmark]   Average CPU usage: 11.00 % (10.82, 11.15, 11.26, 10.87, 10.89)
[benchmark]   Peak CPU usage: 333.82 % (381.40, 243.80, 272.00, 342.10, 429.80)
[benchmark]   Average Memory usage: 2997.71 MB (3094.71, 2923.23, 2899.82, 2987.55, 3083.24)
[benchmark]   Peak Memory usage: 5613.08 MB (5643.63, 5567.17, 5615.44, 5592.79, 5646.38)

I also tried without Build Optimizer turned on:

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build 3cconnect --prod --aot (at D:\sandbox\memory-debug\jsgoupil)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 395413.60 ms (396473.00, 384629.00, 401091.00, 396822.00, 398053.00)
[benchmark]   Average Process usage: 2.56 process(es) (2.61, 2.53, 2.53, 2.52, 2.62)
[benchmark]   Peak Process usage: 8.00 process(es) (8.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 10.42 % (10.50, 9.93, 10.76, 10.58, 10.32)
[benchmark]   Peak CPU usage: 297.14 % (307.80, 251.40, 293.70, 376.60, 256.20)
[benchmark]   Average Memory usage: 2483.32 MB (2481.63, 2471.55, 2518.96, 2451.15, 2493.35)
[benchmark]   Peak Memory usage: 5238.84 MB (5221.19, 5212.36, 5368.60, 5184.88, 5207.16)

I wasn't expecting Build Optimizer to impact either average or peak memory usage that much really. 500mb is a fair bit. We were already looking at a way to reduce the number of processed files in https://github.com/angular/angular-cli/pull/14998, which would help reduce the impact.

These numbers look much higher than the initial ones, but bear in mind that we increased the memory from the 1.4gb default to around 8gb. This means that there isn't as much pressure for the the garbage collector to free up memory, so things stay in memory for longer. For comparison, here is the initial build (without AOT or Build Optimizer) with an increased memory limit:

$ benchmark -- node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build 3cconnect --prod
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build 3cconnect --prod (at D:\sandbox\memory-debug\jsgoupil) 
[benchmark] Process Stats
[benchmark]   Elapsed Time: 253524.00 ms (256206.00, 245034.00, 245636.00, 243953.00, 276791.00)
[benchmark]   Average Process usage: 3.44 process(es) (3.41, 3.45, 3.45, 3.46, 3.42)
[benchmark]   Peak Process usage: 8.00 process(es) (8.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 11.65 % (12.18, 10.84, 11.67, 11.01, 12.55)
[benchmark]   Peak CPU usage: 200.90 % (200.00, 168.70, 231.20, 217.10, 187.50)
[benchmark]   Average Memory usage: 1972.97 MB (2010.47, 1927.40, 2014.74, 1939.05, 1973.17)
[benchmark]   Peak Memory usage: 3871.67 MB (3942.59, 3815.74, 3944.29, 3851.58, 3804.13)

So when using the same high memory limit, adding turning on AOT and Build Optimizer increased the average memory usage by 1GB and peak memory usage by 1.8GB.

Using just AOT increased build time by 70% (250->400 s), average memory usage by 25% (2000->2500), and peak memory usage by 36% (3800->5200 mb). Adding BO increased build time by 42% (400->570 s), average memory usage by 20% (2500->3000), and peak memory usage by 7% (5200->5600 mb).

I also tried to turn off Differential Loading by setting "target": "es5", in ./tsconfig.json while keeping AOT and Build Optimizer:

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build 3cconnect --prod --aot --build-optimizer (at D:\sandbox\memory-debug\jsgoupil)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 332036.80 ms (291110.00, 303767.00, 297923.00, 395777.00, 371607.00)
[benchmark]   Average Process usage: 2.87 process(es) (2.89, 2.86, 2.87, 2.86, 2.85)
[benchmark]   Peak Process usage: 8.00 process(es) (8.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 12.42 % (10.63, 11.52, 11.41, 14.30, 14.21)
[benchmark]   Peak CPU usage: 283.98 % (250.00, 237.40, 414.00, 207.70, 310.80)
[benchmark]   Average Memory usage: 2496.03 MB (2555.55, 2533.63, 2524.99, 2409.38, 2456.62)
[benchmark]   Peak Memory usage: 5084.00 MB (5386.41, 5283.17, 5257.22, 4586.35, 4906.83)

Without DL, memory used goes does by ~500mb, similar to https://github.com/angular/angular-cli/issues/13734#issuecomment-508815407. We're looking at improving that.

I tried to figure out how big the project was by listing files in src/ by extension:

$ find -type f | sed -e 's/.*\.//' | sort | uniq -c
      3 eot
      7 gif  
    478 html 
      1 ico  
      2 jpg  
      4 json 
     51 png  
      1 ps1  
    273 scss 
    470 svg  
   1302 ts   
      1 txt  
      3 woff 
      9 woff2
      1 xml

This does not include third party libs so it doesn't give us much insight as to the total compilation size. But I'm not surprised that this project needs the memory limit increased, there's a lot of TS files in here. I had a look at

@jsgoupil in https://github.com/angular/angular-cli/issues/13734#issuecomment-497365534 you mentioned that an Azure machine with 7gb couldn't build the app, which I find weird because 5.6gb was the maximum I saw on my Windows machine.

Maybe the CI machine is already using 1.4gb? That would make sense. You can probably check how much memory the machine has before the build. But you also said the machine has only 7GB and you are using --max_old_space_size=8192. I'm not really sure how Node behaves when you say it should use more memory than is actually available. So I wonder if that's a problem: you give it a bigger limit than available, so it thinks it can allocate all this memory, but when it's needed the system doesn't have it so you get an error. You do say in https://github.com/angular/angular-cli/issues/13734#issuecomment-497426391 that the error is different with and without the flag so that does point to a different issue. Maybe check how much memory is free and set a limit below that, like 5000 or 6000.

Regardless of the Azure machine you mentioned, it is impressive (in a negative way) that AOT compilation increases memory usage by 1GB average / 1.8GB peak. This is something that we are looking at in the general case. I think in your project it really is just a function of the amount of TS source files that need AOT compilation, I tried taking a profile of the memory usage and it mostly looked like the others I've looked at before from other projects.

I also noticed that you have some weird imports to ../../../node_modules/@angular/core instead of just @angular/core, and similar to RxJs. I think this makes no difference because resolution will still use the package.json there. But I thought it was odd.

filipesilva on 8 Jul 2019

👍1

If your application crash while building scss files try installing those dependencies node-sass, style-loader and sass-loader, this worked for me!

anyderre on 10 Jul 2019

❤2 👍2

Don't know if this is interesting information or not, but I had a lot of issues with versions of @angular-devkit/build-angular newer than 0.800.2 for the travis build recently producing the same error as above. Reverting back to 0.800.2 fixed the issue.

https://travis-ci.org/johannesjo/super-productivity/builds

johannesjo on 11 Jul 2019

same issue on ubuntu server 18.04, but this solved my problem:
node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod

Just dropping this in here but if anyone is using AWS codebuild make sure to update your build environment to be configured with more than the minimum 3GB memory, I followed the steps above but had to re-configure my environment with more memory.

In my case, reconfiguring the CodeBuild environment to use more than 3GB(i choose 15GB) was enough to remove the issue...i did not need to change memory settings in node...

ThomasEg on 15 Jul 2019

@ThomasEg , what environment do you have so you did not need to change memory settings in node?

Mine is Windows 10 with 16 GB memory, and changing the node max_old_space_size settings solve the problem:
node --max_old_space_size=8192 "node_modules/@angular/cli/bin/ng" build --configuration=production

while node actually used 2 GB memory.

I would suspect that your CodeBuild environment actually give node max_old_space_size behind the sense, while ng build apparently does not read max_old_space_size for node. Can you confirm that?

zijianhuang on 16 Jul 2019

@JohannesHoppe I'm not sure what branch/builds I should be looking at there. Can you link me specific builds where you noticed a difference? This might be related to your caching setup too. When you update Angular CLI you'll likely remove the minification cache that builds leave behind inside node_modules. So the first build without that cache needs more memory.

filipesilva on 16 Jul 2019

I've been trying to figure out if there's really a memory leak in Differential Loading (DL) builds. Below are my notes.

TLDR is that there doesn't seem to be any memory leak in DL, but when --max_old_space_size is used V8 will keep memory around. A build can finish successfully with --max_old_space_size=300 without erroring but will use up to 2500mb of memory if given --max_old_space_size=10000.

Trying to figure out if DL really stacks memory endlessly. Modified a DL build to double the entries in webpack-browser-config.js. Normal (2) builds uses 480 average/960 peak, 4 builds is 810/1700, 8 builds is 1400/2500. So each build leaves increases average by 200ish.

I don't quite get why peak also increases so much though. Peak should be average+something... right? Actually that's wrong. Because we already know the usage is going up over time. So the average during the first build is much smaller than the average during the last build. Kinda like an integral. So the number I should care about really is peak, because it represents things better for the multi-build case.

Also noticed that process.memoryUsage().heapUsed gave me numbers were much lower than what I saw on task manager. I found that odd. I also didn't disable the terser cache. I should do that to see if there's also a scaling cost there, after figuring out this initial scaling cost.

Tried doing global.gc every 30 seconds to figure out if memory was truly retained or if node was just very comfortable with the 10gb limit. Did it over 5 iterations. Without GC I got 1200/2150, with GC I got 1200/2050. So only a slightly smaller peak. This shows the memory can't be freed.

Took heap profiles and heap snapshots every 30s. Also tried the memory leak profile on chrome devtools. The heap profiles show almost no change between them. Very odd. Maybe I need to restart the profiler between samples? The snapshots fail to load too, with an error while building retainers. Will re-make them.

Tried profiling again with a lower sampling interval and didn't see anything different. All snapshots but the first still failed with the same message. Also tried just taking one snapshot at the end and got only a small snapshot (200mb).

Tried taking a snapshot manually in Chrome Devtools during the 8th (and last) build. Devtools said the process was using ~2000mb memory just before I took the snapshot, and then it showed only ~260mb. The snapshot was only 254mb.

So I thought it might be a GC thing. Tried limiting max_old_space_size to 1gb and it still built. Also built with 600mb and 400mb, but failed with 300 and 200. A single build run failed with 200 but succeeded with 300. A default DL build failed with 300 too.

So as far as I can tell there isn't any real amount of memory being kept because of DL. Maybe there's to 100 extra mb. A new app will use roughly the same memory (~300mb on the main process, plus up to 450mb on spawned Terser processes) for producing 1 build or 8 builds in the same command. But if I set the memory limit to a a very large number (10gb) then it will keep more memory around (up to 2.5gb) until it needs to be freed.

filipesilva on 16 Jul 2019

👍2

@filipesilva
Here are the logs for failing build:
https://travis-ci.org/johannesjo/super-productivity/jobs/556566074
I think key are the unintended changes to the package-lock.json made here:
https://github.com/johannesjo/super-productivity/commit/807a760d7d7cb9f18e706422a9f8f02af30342de

And there is the build for the commit that fixed the issue and the commit itself:
https://github.com/johannesjo/super-productivity/commit/5e6b7aaf234dbac286549a6ac74957ba15589cd8
https://travis-ci.org/johannesjo/super-productivity/jobs/557364217

(also you mentioned poor JohannesHoppe instead of me again :))

johannesjo on 16 Jul 2019

We were experiecing the same issue. It turned out, we imported the base scss files in most of our components. That was not really necassary.
Now without setting the max-old-space-size Node option, we were able to build our apps in production mode without the error, after removing the core file imports in our components, where it was not necessary.

We also switched from dart-sass to node-sass with npm install -DE node-sass and that worked as well, without changing anything.

Below my environment. Maybe this helps :)


     _                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/


Angular CLI: 8.1.1
Node: 10.16.0
OS: darwin x64
Angular: 8.1.1
... animations, cli, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... platform-server, router

Package                                    Version
--------------------------------------------------------------------
@angular-devkit/architect                  0.800.1
@angular-devkit/build-angular              0.801.1
@angular-devkit/build-optimizer            0.801.1
@angular-devkit/build-webpack              0.801.1
@angular-devkit/core                       8.0.6
@angular-devkit/schematics                 8.0.1
@angular/cdk                               8.0.2
@angular/flex-layout                       8.0.0-beta.26
@angular/material                          8.0.2
@ngtools/webpack                           8.1.1
@nguniversal/express-engine                8.1.1
@nguniversal/module-map-ngfactory-loader   8.1.1
@schematics/angular                        8.0.1
@schematics/update                         0.801.1
rxjs                                       6.5.2
typescript                                 3.4.5
webpack                                    4.35.2

mehrad-rafigh on 17 Jul 2019

🎉1

@filipesilva I doubt that it has anything todo with differential loading or that it will "stack" memory. we build in docker/k8s and we need to set max_old_space_size > 4096 even without DL. In Angular 7 or below (we used also "@angular-devkit/build-angular" < "~0.800.2") we had it set to 2048.

schmitch on 18 Jul 2019

Ran into this very same issue when calling "ng serve" w/ some custom configuration:

npm run env -s && ng serve --aot --configuration alpha

If you are just looking to get rolling, turning "buildOptimizer" and "optimization" flags in my configuration to false did the trick and I was able to build within seconds.

screen

CodeEpiphany on 22 Jul 2019

🎉1

@CodeEpiphany , I knew I knew, such config for faster build time and probably smaller build memory consumption is a trading for larger build output and slower run time performance, however, missing the point of the issue mentioned in the OP.

zijianhuang on 22 Jul 2019

I experienced this with dart-sass first. We managed to fix all of those scss issues, dart-sass had. It now happened with one of my apps using the sourceMaps Object. Setting sourceMaps to false fixed my issue

mehrad-rafigh on 26 Jul 2019

I had same issue with Angular 8. Windows 10
I guess different people have these issues because of different reasons.

My solution was :

node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod

Now again there are two things in this, either it worked because of max_old_space_size=8192 or may be because ive different local and global cli angular version. and using above command used the local/same version of ng

bkrajendra on 28 Jul 2019

👍1

I have a relatively small app, maybe 24 components, including routes, modals, and a few classes for state management, etc. And on a 16GB machine, even with all of the suggestions posted here, there's a 3/4 chance it will hit this bug. I don't understand how it can use this much memory and really don't understand why it isn't at least trying to page it out to the SSD (sure, it would be slow, but if it goes from not building at all, to building in 10 minutes, 10x the old build time, that's still pretty great).

This is with sourcemaps off, node getting the max_old_space_size setting, buildOptimizer off, optimization off.

3 versions ago, this same app could build with a 3GB machine. Should I really roll everything back to Angular 5 or am I better off just switching to Vue?

Senneseph on 30 Jul 2019

👎4

@Senneseph what is your node version? try node 10 instead of 12

maxisam on 30 Jul 2019

@maxisam I'm on 10.x until AWS decides to use a more recent version or I decide to create a custom Lambda Layer. But this should be working on 8.9+ according to the docs.

From https://www.npmjs.com/package/@angular/cli

Both the CLI and generated project have dependencies that require Node 8.9 or higher, together with NPM 5.5.1 or higher.

I was going to add a rant here, but I'm just going to send it to the bosses with a request to switch frameworks. It seems like this project is just in total chaos.

Here's something fun to try (8.1.3):

npm install -g @angular/cli
ng create new-project
cd new-project
ng update
"rxjs 6.4.0 -> 6.5.2 ng update rxjs"

Why does a brand new project already require a package update? 6.4.0 was in Jan. 6.5.2 was in May.

Angular CLI https://github.com/angular/angular-cli/blob/cae4a9dc4afb53292cb44ebbc644d33b7d8d159e/packages/schematics/angular/utility/latest-versions.ts

RxJs: '~6.4.0',

Angular https://github.com/angular/angular/blob/master/package.json

"rxjs": "^6.4.0",

Senneseph on 31 Jul 2019

👎5

@Senneseph dude, people here are still trying to fix it. No one ignores this issue. If you have one tiny issue and make you decide to quit Angular, good luck with other "framework"s (Vue and React are not frameworks, IMO). I am sure they are so perfect with 0 issue.

I have a project with 100+components with tons of ngrx classes and services.

My build server is on node v12 with pretty average hardware.

for es2015

Date: 2019-07-18T13:21:52.536Z - Hash: 9ba50a50277fc3c9fc2e - Time: 54734ms
for es5
Date: 2019-07-18T13:23:12.900Z - Hash: 93e6713a4d03978a67fd - Time: 80111ms

It takes a bit longer than v7 but it is not something crazy.

I switched my local dev machine back to node v10 on the very first day I upgrade to V8 but I haven't done anything to build server because I am confidence that CLI team will figure this out.

Here is a place for people to work on issues not a place to vent.

I don't even know what is wrong with the update? There is no breaking change on rxjs from 6.4.0 to 6.5.2.

maxisam on 31 Jul 2019

@maxisam

I definitely appreciate everyone's hard work. I initially chose Angular because it seemed to be the best organized and most holistically thought-out project.

The issue is not whether people are working hard on solving this problem. The issue is that it should have been solved before it was released and declared ready for public use.

As for the rxjs thing, there shouldn't be any breaking changes between the versions. The problem is that the comments read "// These versions should be kept up to date with latest Angular peer dependencies." and yet someone is just typing whatever they want into the field instead of running a script to extract the exact string. And then no one tested if it was working as intended.

I imagine the Angular team is stretched too thin.

Senneseph on 31 Jul 2019

👍1

@Senneseph

The issue is that it should have been solved before it was released and declared ready for public use.

You know this is BS right? There is no bug patch for Vue or React?

IMHO, V8 is ready. I have used it for a while. This issue doesn't really stop everyone. No one in our team(5+) has this issue. We have 10ish projects using Angular 7/8 with AOT and Build Optimizer on.

We do see the build time issue but nothing stop us to build.

Did you try to install node-sass, style-loader and sass-loader? (Mine works either way but you should try it.)

maxisam on 31 Jul 2019

Please keep the discussion on topic and review the code of conduct for behavior.

https://github.com/angular/code-of-conduct/blob/master/CODE_OF_CONDUCT.md

gnomeontherun on 31 Jul 2019

😄2

@maxisam

Thank you for the suggestion about Sass, but I am using plain CSS that is built in its own project. I didn't mention it earlier because the question of whether this was happening with plain CSS appears to be already answered.

Senneseph on 31 Jul 2019

@Senneseph any chance I could see that project? I don't think a 24 component project should use 16gb of memory. I haven't seen any project that uses that much memory at all really

There was a severe, but rare, but with typescript 3.4 (https://github.com/microsoft/TypeScript/issues/30050) that caused a lot of extra memory to be used in some cases with union types. We've just released support for TS 3.5 in Angular 8.2.0 with @angular-devkit/[email protected]. That might be what's affecting your project, you can try updating to those versions to see if it helps.

filipesilva on 31 Jul 2019

👍4

Thanks @filipesilva . Unfortunately, I can't grant access to the project itself. I will try the @angular-devkit/build-angular version you recommended and report back how it does in a week or so.

While I don't have many components, they are quite abstract. And while I'm not using the keyof operator directly, I do define a type that is a union of about 19 types. Perhaps somewhere I have combined this type with another in a bad way such that it explodes the tsc when it tries to perform normalization.

(if this is the reason that my angular app is normal on desktop browsers, but almost totally unresponsive when reactive forms are on the page, i will be really happy. For some reason, only Opera Mobile has the same responsiveness / cpu usage as desktop browsers.)

Senneseph on 1 Aug 2019

Last week, after merging a relatively small branch into the master branch, the build of my project started to experience this memory issue reported here. The interesting part is that the problem was happening on the CI server only, none of the developers was having trouble locally, even with --prod.

After reading @filipesilva 's comment above, I decided to update angular as follows:

$ ng update @angular/cli @angular/core
Using package manager: 'npm'
Collecting installed dependencies...
Found 60 dependencies.
Fetching dependency metadata from registry...
    Updating package.json with dependency @angular/cli @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/core @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/animations @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/compiler-cli @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/language-service @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/router @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/forms @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/common @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/platform-browser-dynamic @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular-devkit/build-angular @ "0.802.1" (was "0.801.2")...
    Updating package.json with dependency @angular/compiler @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/platform-browser @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency typescript @ "3.5.3" (was "3.4.5")...
UPDATE package.json (2341 bytes)

The problem is now fixed!

My impression is that the extra memory needed for DL dual builds, plus the fact that typescript was bumped to 3.4 which has the aforementioned problems regarding union type edge cases, started causing this extra memory consumption for some projects.

brunojcm on 12 Aug 2019

👍3

Hey all,

Just wanted to give you an update on the steps we (and others in the community) took to address this problem in Angular 8:

Added support for TypeScript 3.5 https://github.com/angular/angular/pull/31615
Improve performance when many lazy chunks are present https://github.com/webpack/webpack/pull/9472
Fix performance regression in Node 12 https://github.com/angular/angular/pull/31638
Warn when large component CSS is included https://github.com/angular/angular-cli/pull/15093
Warn when TS compilation includes unused files https://github.com/angular/angular-cli/pull/15061
Reduce amount of files processed by Build Optimizer https://github.com/angular/angular-cli/pull/14998
improved performance of Differential Loading builds https://github.com/angular/angular-cli/pull/15246
cleared up more resources on non-watch builds https://github.com/angular/angular-cli/pull/15292

These efforts are still under way:

Improve JS minifier performance https://github.com/webpack-contrib/terser-webpack-plugin/issues/104
Improve Sass processor performance https://github.com/sass/dart-sass/issues/744
Reduce memory used by the Angular Compiler
Remove pure_getter optimizations https://github.com/angular/angular-cli/issues/14316

Here's what you can do to take advantage of these fixes:

Update to @angular/[email protected] (together with the other framework packages)
Update to @angular/[email protected] and @angular-devkit/[email protected]
Update to [email protected]
Install node-sass as a dev dependency if you use Sass
Use Node 12.8

One special note is that we completely reworked how Differential Loading works. It should now be much faster, but can use more system memory depending on the numbers of cores you have. This is different from the node memory limit.

When you hit the node memory limit you see the FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory error. But if you hit the system memory limit you should see Fatal process OOM in insufficient memory to create an Isolate.

The new Differential Loading setup makes it rarer to hit the first, but if you have many cores but not a lot of memory, it can be more common to hit the second. We'd like feedback on @angular-devkit/[email protected] please, so that we can address this behaviour if it's a problem, by disabling parallelism in some cases or make it configurable.

filipesilva on 15 Aug 2019

👍28 ❤4 🎉3

I have the same error in Angular 8 and this solved my problem:

node --max_old_space_size=X node_modules/@angular/cli/bin/ng build --prod

WhereX = (2048 or 4096 or 8192 o..) is the value of memory

gokujy on 19 Aug 2019

👍6

@filipesilva just install Node 12.8 solved the issue . thanks.

yantrab on 26 Aug 2019

👍7 👎2

hello! very interesting thread
especially this :
https://github.com/angular/angular-cli/issues/13734#issuecomment-506760767
and this :
https://github.com/angular/angular-cli/issues/13734#issuecomment-521743839

Can someone explain this to me :

Install node-sass as a dev dependency if you use Sass

is this us "giving up" on dart-sass?

or is it a combination of using dart-sass as main compiler and node-sass as "support", if that works at all? (the context of whether or not npm i sass --save-dev/yarn add sass --dev would still be in use in this case was not given in the above guide).

what exactly would my package.json file look like and are there any other changes to make to my project?

I am personally upgrading my project to angular 8 and want to switch to dart-sass

EDIT :

are there any other changes to make to my project?

to answer at least that part of my own question : it seems dart-sass does not support the /deep/ scss tag.

however ::ng-deep works so it's the best of all worlds. I'm really liking dart-sass so far! so much less of a hassle when building.

tatsujb on 5 Sep 2019

I had this problem and this solution worked for me:

in the angular.json file, replace the value of the outputPath key with "dist"

Flavio-occhipinti on 8 Sep 2019

Just updating the node version from LTS to Current solved my issue.

akil3 on 9 Sep 2019

same issue. node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod still doesn't work.

iamcco on 10 Sep 2019

Faced crashes from a large application with lots of chunks. But it seems a problem in source map and has same error how in https://github.com/webpack/webpack-sources/issues/66

Tom910 on 10 Sep 2019

I had to upgrade to node 12 AND increase our codebuild instance memory from 3 to 7GB. Just to build an angular app.

ktersius on 12 Sep 2019

👎1

@filipesilva, I confirm that ng build is working well now with:

angular 8.2.2
angular/cli 8.2.2
node 12.4

ng build --configuration=production

angular.json

                    "configurations": {
                        "production": {
                            "optimization": true,
                            "outputHashing": "all",
                            "sourceMap": false,
                            "extractCss": true,
                            "namedChunks": false,
                            "aot": true,
                            "extractLicenses": true,
                            "vendorChunk": false,
                            "buildOptimizer": true,
                            "fileReplacements": [
                                {
                                    "replace": "src/environments/environment.ts",
                                    "with": "src/environments/environment.prod.ts"
                                }
                            ],
                            "serviceWorker": true
                        }

A few months ago when you tested my big fat SPA, the memory consumption of node was over 2GB, so I had to use:
node --max_old_space_size=8192 "node_modules/@angular/cli/bin/ng" build --configuration=production

Now with a larger codebase being built with ng build, the memory peak of node is 1.6GB.

Good work. :)

zijianhuang on 12 Sep 2019

👍3

Is there any other way to make this work on node 10?

stevefiedler on 12 Sep 2019

@stevefiedler , is there any reason why you have to stay on node 10?

zijianhuang on 12 Sep 2019

Well, jumping to 12 seemed to require a lot more changes for my application, and I have multiple applications to worry about. It should be working a step at a time

stevefiedler on 13 Sep 2019

@stevefiedler , I have a big fat Angular SPA started since Sep 2016 with Angular 2, and it seem I did not encounter any problem when moving to node 12. If you post the problem to StackOverflow, you may find more helps and I may have a look as well.

zijianhuang on 13 Sep 2019

@stevefiedler Out of pure curiosity. What problems do you have with your application when you change the compiler to Node 12?

manklu on 14 Sep 2019

I have the following environment:

Node 12.9.1
Angular 8.25 (from ng upgrade)

@angular-devkit/build-angular and build-ng-packagr of 0.803.3
@angular/cli of 8.3.3

When running ng build --prod, and watching the node process, I see similar behavior in the memory usage as described above, where it barely eclipses 2 GB of usage. However, at one point it just fails with the error message below. Seems to be crashing with webpack's usage of the acorn library.

I have a very large SPA that is utilizing sass, and i'm using node-sass instead of dart.

I'm seeing the following stack trace when trying to run ng build --prod:

ng build --prod
90% chunk assets processing
<--- Last few GCs --->

[32136:00000223FBDD0BD0] 321271 ms: Mark-sweep 1906.3 (2091.5) -> 1906.0 (2093.0) MB, 1306.9 / 0.0 ms (average mu = 0.187, current mu = 0.124) allocation failure GC in old space requested
[32136:00000223FBDD0BD0] 321985 ms: Mark-sweep 1912.7 (2093.0) -> 1910.5 (2093.0) MB, 656.7 / 0.1 ms (average mu = 0.152, current mu = 0.081) allocation failure GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 00007FF779B1224D]
1: StubFrame [pc: 00007FF779B75575]

Security context: 0x022679540919
2: parseExprOp [0000039FA6955EF1] [C:projectnode_moduleswebpacknode_modulesacorndistacorn.js:2034] [bytecode=0000007FB1EB9C01 offset=226](this=0x00876aee4d01 ,0x0183701f23d1 ,11588,0x0183701f23a9

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 00007FF778F5F0DF napi_wrap+121039
2: 00007FF778F051D6 v8::base::CPU::has_sse+34470
3: 00007FF778F05E96 v8::base::CPU::has_sse+37734
4: 00007FF7796F89AE v8::Isolate::ReportExternalAllocationLimitReached+94
5: 00007FF7796E0951 v8::SharedArrayBuffer::Externalize+833
6: 00007FF7795AEC8C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1436
7: 00007FF7795B812F v8::internal::Heap::ProtectUnprotectedMemoryChunks+1279
8: 00007FF7795B6614 v8::internal::Heap::PageFlagsAreConsistent+3204
9: 00007FF7795AC263 v8::internal::Heap::CollectGarbage+1235
10: 00007FF7795AAB04 v8::internal::Heap::AddRetainedMap+2356
11: 00007FF7795C7684 v8::internal::Factory::NewByteArray+100
12: 00007FF7796115D7 v8::internal::BasicBlockProfiler::Data::ResetCounts+8647
13: 00007FF7799194AC v8::internal::compiler::CodeGenerator::GenerateDeoptimizationData+124
14: 00007FF779919247 v8::internal::compiler::CodeGenerator::FinalizeCode+103
15: 00007FF77994FAEA v8::internal::compiler::LiveRange::End+746
16: 00007FF77995011F v8::internal::compiler::LiveRange::End+2335
17: 00007FF77964F0FA v8::internal::Compiler::FinalizeBackgroundCompileTask+922
18: 00007FF77964F583 v8::internal::Compiler::FinalizeOptimizedCompilationJob+931
19: 00007FF779639228 v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions+600
20: 00007FF779605CCA v8::internal::StackGuard::HandleInterrupts+1770
21: 00007FF779339432 v8::internal::interpreter::JumpTableTargetOffsets::iterator::operator=+5138
22: 00007FF779B1224D v8::internal::SetupIsolateDelegate::SetupHeap+575565
23: 00007FF779B75575 v8::internal::SetupIsolateDelegate::SetupHeap+981877
24: 00007FF779A8FCFC v8::internal::SetupIsolateDelegate::SetupHeap+41724
25: 0000039B9B7A5C28

tonyholt on 14 Sep 2019

Wanted to add a bit more of information. ng build seems to work fine. only when using the --prod flag is where the heap issue happens. Here's my prod configuration from the angular.json:

"production": {
"fileReplacements": [{
"replace": "apps/standard/src/environments/environment.ts",
"with": "apps/standard/src/environments/environment.prod.ts"
}],
"optimization": true,
"outputHashing": "all",
"sourceMap": false,
"extractCss": true,
"namedChunks": false,
"aot": true,
"extractLicenses": true,
"vendorChunk": false,
"buildOptimizer": true,
"vendorSourceMap": false
},

tonyholt on 14 Sep 2019

Thanks a lot @filipesilva for the exhaustive benchmark. Installing node-sass didn't pull us out but upgrading angular to v8.2.6 did the trick.

Still, isn't it weird that most of people reporting the JavaScript heap out of memory error get stuck at 92% precisely ? I would blame the Terser optimisation process for the sudden memory peak.

EkiGui on 16 Sep 2019

👍2

Is there any way to resolve this issue without updating node to current from LTS? In my project build gets generated in server. I can't ask all servers to update node.

akvaliya on 16 Sep 2019

@akvaliya node 12 will be LTS in less than a month. I can't see how a bug this intricate & wide reaching makes sense to tackle using a version that's about to be become 'semi-legacy'. https://nodejs.org/en/about/releases/

tomgruszowski on 16 Sep 2019

@tomgruszowski Yes. But i can't tell client to update all his servers.

akvaliya on 16 Sep 2019

@akvaliya Why are your application servers building the front-end?

benelliott on 16 Sep 2019

@benelliott Well i just push code to repo & testers from client team will pull & redeploy with build.

I don't this should be the point that why are we building fron-ent in server. Some client wants code to auto redeploy after commit. Like using jenkins.

akvaliya on 17 Sep 2019

for me the solution was editing my tsconfig (ionic4 and angular8)

i was playing around adding properties to tsconfig, towards the end i enabled source maps for my prod build inside angular.json.

i had to remove the sourceMap: true and sourceBase: "/" from tsconfig , after that --prod works like a charm

d1sco on 20 Sep 2019

👍4

"scripts": {
        "start": "node --max-old-space-size=4096 ./node_modules/@angular/cli/bin/ng serve",
        "start:prod": "node --max-old-space-size=4096 ./node_modules/@angular/cli/bin/ng serve --prod",
}

Above config solves for me

kumaresan-subramani on 24 Sep 2019

👍5 👎2

For us upgrading from Angular 8.2.0 to 8.2.8 did the trick and we no longer got the error. While allocating more memory did not work for us.

StevenSerrien on 26 Sep 2019

@filipesilva just install Node 12.8 solved the issue . thanks.

I have the same issue with 12.8 too.

irux on 30 Sep 2019

@filipesilva just install Node 12.8 solved the issue . thanks.

Was getting with debug and Node >12.8 fix for me

mindte on 1 Oct 2019

I have a custom project not using Angular but built with Webpack, and getting this error when I build with Webpack in production mode. I tried --max_old_space_size=8192 but no luck (I guess I need more).

I wonder what's actually the problem though, I bet differing frameworks/libs/apps may have similar Webpack configs that can cause this.

trusktr on 2 Oct 2019

I had the same issue and i did these steps:

ng update @angular/cli @angular/core
update node to v12.11.1
works fine with --max_old_space_size=8192
"@angular/core": "8.2.8", "@angular/cli": "^8.3.6"

Dev4mir on 2 Oct 2019

The specific cause of this error for me (in my non-angular project) is the use of uglifyjs-webpack-plugin. It uses a ton a memory. For now I disabled minification in production, and will have to figure out how to fix it later.

trusktr on 2 Oct 2019

I disabled minification in production

😨

This has been working for me for dev builds. I haven't seen this issue with production builds, yet.

"build:highmem": "node --max_old_space_size=8192 ./node_modules/@angular/cli/bin/ng build",

I only started to need this after updating Angular and CLI from 8.0.0 to the latest 8.2.8 and 8.3.6. Here's the PR where this was changed: https://github.com/angular/material.angular.io/pull/641

Splaktar on 2 Oct 2019

The specific cause of this error for me (in my non-angular project) is the use of uglifyjs-webpack-plugin. It uses a ton a memory. For now I disabled minification in production, and will have to figure out how to fix it later.

This is probably related to source maps. Try setting sourceMap to false in tsconfig.json (or, for those with angular projects, in angular.json).

trevorhreed on 3 Oct 2019

A Note For Angular/Windows/Docker Users

If you are trying to run 'npm run build' or something of the sort inside your Docker file and you are getting this error, as mention above, you want to use the following run command instead, increase Node.js' access to memory.

node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod

In addition, however, Docker is running a Linux VM in the background that will also have memory constraints that need to be addressed.

Navigate to 'C:UsersAppDataRoamingDocker' where profile name is the named of your logged in profile. There you will find a 'settings.json' file. Open it and adjust the 'memoryMiB' value to also be 8192. You will need to restart for the changes to take effect.

WARNING

Increasing the VM memory can have a drastic impact on your machine performance, so ensure that you have enough physical memory to do this before trying. If this solution works for you, you should then try to roll back the values as much as you can, by doing a process something like, halving the value initially, testing, halving it again if it works, or adding half of the new value back if it doesn't etc. This will help you minimize the impact on your machine.

You will also want to disable Docker as a start up program for Windows 10. Simply search "startup apps" in the task bar and select the corresponding option that appears. This way, you wont have a VM you are not using bogging down your system.

BasicExp on 8 Oct 2019

👍2

for me the solution was editing my tsconfig (ionic4 and angular8)

i was playing around adding properties to tsconfig, towards the end i enabled source maps for my prod build inside angular.json.

i had to remove the sourceMap: true and sourceBase: "/" from tsconfig , after that --prod works like a charm

Just removing sourceMap worked for me. Frustrating....

joelowrance on 8 Oct 2019

set NODE_OPTIONS=--max-old-space-size=30720 command worked for me.

sunilbcone on 14 Oct 2019

😄11 👍2 😕1

Had the same issue and upgrading node to 12.12 today let me ng build --prod

mac4281 on 17 Oct 2019

👍3

We're using Bitbucket pipelines and we can't increase the memory limit to 8GB. Also our application takes about 7-10min to build and we need the sourceMaps for our Sentry error reports.

UPDATE: running ng update @angular/cli @angular/core solved the problem.

Snesi on 17 Oct 2019

👍4

I updated the node.js to the latest LTS (at this moment 12.13.0) version and everything work perfectly. No need anymore set the max-old-space-size.

TarasKovalenko on 21 Oct 2019

👍12 👎4 🎉3

We had the same error on Windows-Comupters, caused by a misconfiguration in _angular.json_. outputPath was set to a path/drive which doesn't exist and it always ended up into the same error, in Angular 7/8 and with different versions of node.js (12.x).

"architect": {
  "build": {
    "options": {
      "outputPath": "Y:/apps/dist" // this caused "JavaScript heap out of memory"!
    }
  }
}

BeniFreitag on 23 Oct 2019

We're using Bitbucket pipelines and we can't increase the memory limit to 8GB. Also our application takes about 7-10min to build and we need the sourceMaps for our Sentry error reports.

UPDATE: running ng update @angular/cli @angular/core solved the problem.

Running the ng update worked for me as well. One of the minor point versions must have had a memory leak of some sort. Moving from 8.2.8 to 8.2.11 fixed the issue.

magicjj on 23 Oct 2019

👍1

With the latest versions of node is solved (at least for me). I recommend the team node to use in the package.json constructs Engines:
https://docs.npmjs.com/files/package.json#Engines

tailot on 8 Nov 2019

👎2 👍1

This works for me: https://stackoverflow.com/a/58916640/1077309

Sampath-Lokuge on 18 Nov 2019

I had a very large object (array of about 100.000 items) causing the builder to crash.

Removing that object solved the problem for me.

(As a workaround I now store the object as a string/json and parse it on runtime.)

joschne on 21 Nov 2019

[error]Error: Npm failed with return code: 3221226505

PritiMaurya on 25 Nov 2019

13 verbose stack Error: [email protected] dev_build2019: node --max_old_space_size=8384 ./node_modules/@angular/cli/bin/ng build --configuration=dev
13 verbose stack Exit status 3221226505
13 verbose stack at EventEmitter. (C:Program Filesnodejsnode_modulesnpmnode_modulesnpm-lifecycleindex.js:332:16)
13 verbose stack at EventEmitter.emit (events.js:210:5)
13 verbose stack at ChildProcess. (C:Program Filesnodejsnode_modulesnpmnode_modulesnpm-lifecyclelibspawn.js:55:14)
13 verbose stack at ChildProcess.emit (events.js:210:5)
13 verbose stack at maybeClose (internal/child_process.js:1021:16)
13 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)
14 verbose pkgid [email protected]
15 verbose cwd D:a1sPROP-VIVO-SPA
16 verbose Windows_NT 10.0.14393
17 verbose argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "run" "dev_build2019"
18 verbose node v12.13.0
19 verbose npm v6.12.0
20 error code ELIFECYCLE
21 error errno 3221226505
22 error [email protected] dev_build2019: node --max_old_space_size=8384 ./node_modules/@angular/cli/bin/ng build --configuration=dev
22 error Exit status 3221226505
23 error Failed at the [email protected] dev_build2019 script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 3221226505, true ]

PritiMaurya on 25 Nov 2019

Please provide me the solution for this build problem

PritiMaurya on 25 Nov 2019

👎8 👍1

@PritiMaurya , Try below command to generate build

node_modules/.bin/ng build --prod --aot --source-map=false --vendor-chunk=true --output-hashing none

kumaresan-subramani on 25 Nov 2019

@kumaresan-subramani I am getting error
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory after run above cmd

PritiMaurya on 25 Nov 2019

@kumaresan-subramani do you know about error
Npm failed with return code: 3221226505

I am getting this error while build on vsts

PritiMaurya on 26 Nov 2019

👎1

@kumaresan-subramani I am getting error
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory after run above cmd

You can try like this

node --max-old-space-size=4096 ./node_modules/@angular/cli/bin/ng build

kumaresan-subramani on 26 Nov 2019

Check your folder name. If your folder name having spaces, these kind of issues will generate. Rename without spaces and restart your machine. hope it will work.

roneyvia on 28 Nov 2019

I don't have any folder name with space

PritiMaurya on 28 Nov 2019

@PritiMaurya , Try solution suggested over this link. It might help.

kamlesh-cis on 2 Dec 2019

I don't have any folder name with space

kindly restart your machine.

roneyvia on 3 Dec 2019

Same with ng8 + node 10.16 + _not minitied_ build(found out that in my case Auth0Lock package causes the problem)
On node 12.13 everything builds ok

katyapavlenko on 18 Dec 2019

I'm facing this problem too. This is my config:

    _                      _                 ____ _     ___
   / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
  / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
 / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
/_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
               |___/

Angular CLI: 1.6.8
Node: 10.17.0
OS: darwin x64
Angular: 5.2.11
... common, compiler, compiler-cli, core, forms, http
... language-service, platform-browser, platform-browser-dynamic
... router

@angular/animations: 4.4.7
@angular/cli: 1.6.8
@angular-devkit/build-optimizer: 0.0.42
@angular-devkit/core: 0.0.29
@angular-devkit/schematics: 0.0.52
@ngtools/json-schema: 1.1.0
@ngtools/webpack: 1.9.8
@schematics/angular: 0.1.17
typescript: 2.5.3
webpack: 3.10.0

In the Stack Overflow I found this solution. It works, but I don't know if this approach seems correct or make sense.

jpcmf on 18 Dec 2019

I updated the node.js to the latest LTS (at this moment 12.13.0) version and everything work perfectly. No need anymore set the max-old-space-size.

This worked for me!

TAharshalcarpenter on 20 Dec 2019

👍7 👎4

I updated the node.js to the latest LTS (at this moment 12.13.0) version and everything work perfectly. No need anymore set the max-old-space-size.

This worked for me!

Updating to latest node.js worked for me too!

Sef1995 on 20 Dec 2019

👎4 👍3

I updated the node.js to the latest LTS (at this moment 12.13.0) version and everything work perfectly. No need anymore set the max-old-space-size.

This worked for me!

MerySantos on 2 Jan 2020

👎7 👍4

I updated the node.js to the latest LTS (at this moment 12.13.0) version and everything work perfectly. No need anymore set the max-old-space-size.

Worked for me too

umairhafeez-r on 5 Feb 2020

👎6 👍1

I updated the node.js to the latest LTS (at this moment 12.13.0) version and everything work perfectly. No need anymore set the max-old-space-size.

Worked for me too

I can confirm, that this is working.

Disane87 on 6 Feb 2020

👎8 👍2

Same problem here. Angular 7. Any Node 12.X version.

Works with Node 10.

davidgg on 13 Feb 2020

👀1 👍1

Upgrading to 12.x worked for me. Loving Angular less and less each day...

MatricomHQ on 16 Feb 2020

👍16 👎9 😄5

Every release I recall a presentation at NGConfg by one of the angular members stating something like:

"We have over 400 applications at google and we dogfood angular before we release it into the wild. If any of the applications have to change in order to adapt the update, we don't release it"

I think every single release required some type of update to the code.

As far as I remember, they said that they test the applications to make sure nothing breaks after necessary code upgrades. I believe, upgrading the code to take in the enhancements is anyways inevitable.

But this current issue has nothing to do with that. It's more about application size. If the application size is growing, you will have to increase the heap size for the node process.

umairhafeez-r on 5 Mar 2020

😕1 👍1

Just happened to me after updating from 9.0.3 to 9.0.5

rsivan on 7 Mar 2020

👍7

Hi All..
I was using ubuntu 18.04 EC2 instance to build docker images.
This issue is resolved as soon as i change the instance type to t2.medium from t2.micro.

rohithhasher on 5 Apr 2020

I was facing same issue for ng serve,

npm cache clean --force resolve this issue

tussharkhairnar on 15 Apr 2020

👎5

NodeJS Update to latest version solved it for me.

error505 on 1 May 2020

I have resolved by installing Nodejs V12 ,
I was facing issue with Nodejs V10

On Sat, 2 May 2020 at 01:59, Igor Iric notifications@github.com wrote:

Node JS Update to latest version solved it for me.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/angular/angular-cli/issues/13734#issuecomment-622554387,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AH6IJ4PME7KZY3J7J3ARITDRPMWLXANCNFSM4GY5SKAQ
.

tussharkhairnar on 2 May 2020

Heya all, this issue is pretty old by this point and we're mostly focused on resource usage issues on newer CLI version. This issue also contains several different problems that seem to manifest in similar ways, which makes it hard to zero-in on. If you're still seeing problems like this, please open a new issue with reproduction details.

filipesilva on 27 May 2020

👍2

use
"build:prod": "node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build --configuration=prod", in your package.json file

this will tell node to exceed default memory allocation by given size in command.

gayankasun on 11 Jun 2020

👎7

If it helps anyone else, when we switched from "@angular-devkit/build-angular": "0.803.26" to "@angular-devkit/build-angular": "0.803.27" it caused memory to run away during prod builds at ~43% build progress. It would get stuck at this step and escalate to >25GB of RAM rather than ~1.5GB.

When we switched back to "@angular-devkit/build-angular": "0.803.26", it worked again. 0.803.27 is a recent release around 6/11/20, so it would only explain new occurrences of this issue.

kyle-apex on 19 Jun 2020

👍3

Seems like the issue is back with v10 release. Using @angular-devkit/build-angular v0.1000.0 and @angular/* packages v10.0.0.

Node: v14.1.0
NPM: 6.14.5

Stack trace:

<--- Last few GCs --->

[13013:0x108008000]  2808381 ms: Scavenge 2003.1 (2051.0) -> 2002.5 (2051.5) MB, 4.0 / 0.0 ms  (average mu = 0.117, current mu = 0.045) allocation failure 
[13013:0x108008000]  2809520 ms: Mark-sweep 2003.3 (2051.5) -> 2002.1 (2051.0) MB, 1132.9 / 0.0 ms  (average mu = 0.073, current mu = 0.026) allocation failure scavenge might not succeed
[13013:0x108008000]  2809535 ms: Scavenge 2003.1 (2051.0) -> 2002.5 (2051.5) MB, 3.3 / 0.0 ms  (average mu = 0.073, current mu = 0.026) allocation failure 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100bc569b node::Abort() (.cold.1) [/usr/local/bin/node]
 2: 0x1000816b5 node::FatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0x10008181e node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 4: 0x100180909 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x1001808b3 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 6: 0x1002a0bfd v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 7: 0x1002a1f60 v8::internal::Heap::MarkCompactPrologue() [/usr/local/bin/node]
 8: 0x10029f947 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 9: 0x10029e112 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
10: 0x1002a60d4 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
11: 0x1002a612a v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
12: 0x10028417d v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/local/bin/node]
13: 0x1004e7b16 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
14: 0x10074fe99 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]
zsh: abort      npm run start

~~After upgrading Node.js to v14.4.0 the issue seems to be resolved.~~

Flyrell on 26 Jun 2020

👍10

@Flyrell I have the same issue with Nodejs 14.4.0 and Angular 10

a2441918 on 26 Jun 2020

👍11

@Flyrell I have the same issue with Nodejs 14.4.0 and Angular 10 but that not seems resolved

stingers on 28 Jun 2020

Yes, I was using the dev server for a little more today and the issue still exists.

Flyrell on 29 Jun 2020

👍1

Yes the issue still exists with angular 10 and nodejs 12.16.1 :(

gvsakhil on 1 Jul 2020

@Flyrell @gvsakhil Avoid commenting on closed issues that are referring to old versions. There is an opened issue #16860 for NG9, which would be more appropriate.

Note that there is even now an issue #18087 for Angular 10, which got closed so you can expect some improvements with next release.

superjulius on 1 Jul 2020

👍2

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{_This action has been performed automatically by a bot._}

angular-automatic-lock-bot[bot] on 1 Aug 2020

Angular-cli: Angular 7/8 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

🐞 Bug report

Command (mark with an `x`)

Description

🔬 Minimal Reproduction

🔥 Exception or Error

🌍 Your Environment

Most helpful comment

All 205 comments

`##[section]Starting: Command Line Script

Help : More Information

[command]"C:windowssystem32cmd.exe" /D /E:ON /V:OFF /S /C "CALL "d:a_temp3e7b4265-9fa3-47c6-b6b7-b690af8bcac6.cmd""

[error]Cmd.exe exited with code '134'.

[section]Finishing: Command Line Script`

Debugging the memory situation in CLI 8

Environment:

Measuring memory usage

Methodology

Results

new-project

Super-productivity

Results

cli-eight-project

super-productivity

awesome-angular-workshop

angular.io

spartacus

clarity

Conclusions

Dev environment

Your test cases

A Note For Angular/Windows/Docker Users

[error]Error: Npm failed with return code: 3221226505

Related issues

Angular-cli: Angular 7/8 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

🐞 Bug report

Command (mark with an x)

Description

🔬 Minimal Reproduction

🔥 Exception or Error

🌍 Your Environment

Most helpful comment

All 205 comments

`##[section]Starting: Command Line Script

Help : More Information

[command]"C:windowssystem32cmd.exe" /D /E:ON /V:OFF /S /C "CALL "d:a_temp3e7b4265-9fa3-47c6-b6b7-b690af8bcac6.cmd""

[error]Cmd.exe exited with code '134'.

[section]Finishing: Command Line Script`

Debugging the memory situation in CLI 8

Environment:

Measuring memory usage

Methodology

Results

new-project

Super-productivity

Results

cli-eight-project

super-productivity

awesome-angular-workshop

angular.io

spartacus

clarity

Conclusions

Dev environment

Your test cases

A Note For Angular/Windows/Docker Users

[error]Error: Npm failed with return code: 3221226505

Related issues

Command (mark with an `x`)