Angular-cli: v7.0.0 bugs and 4/5x slower serve & compile AOT

[elvisbegovic]

I hesitate to open this issue on angular page. I opened same issue title 2 week ago here without AOT : v7.rc.0 serving 2x - 3x slower than stable versions on my projects after moving to angular 7.0.0 all is slower (jit seems OK)

I'm submitting a...

[ x ] Performance issue
[ x ] Bugs

Current behavior

All was OK with cli7rc3 and angular 6.1.10
This morning after updating to angular 7.0.0 serving with JIT is ok but serving with AOT or build my application or generating i18n xmb file is 2-3 times slower

Expected behavior

Get build/extract i18n time like with cli7rc3 and angular 6.1.10

1. Minimal info of AOT serving

1. My project have >1200 files.
2. Before update node --max-old-space-size=8192 ./node_modules/@angular/cli/bin/ng serve --aot --sourceMap=false --disableHostCheck --port=4200 first build finish in 45seconds and Now blocking:
   >3mn => 11% building modules 9/10 modules 1 active ...ojects\i10\src\assets\styles\styles.css and rest steps seems ok It finish in 228737ms
3. Before update refresh(file edit) it finish in 30seconds and 3th refresh in 25seconds and Now it refresh in 200277ms and 3th refresh in WARNING : 381979ms it is >6 minutes

2. Minimal info of extract i18n

1. Extract is 3-4 times slower than before update

Here is my angular.json config file

I tried debug with ndb and share my profile data with ng s --aot but it seems not work ndb crash ( disk is highly used...):

Environment

Angular CLI: 7.0.1
Node: 10.12.0
OS: win32 x64
Angular: 7.0.0
... common, compiler, compiler-cli, core, forms, http
... language-service, platform-browser, platform-browser-dynamic
... router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.10.1
@angular-devkit/build-angular     0.9.0-rc.3
@angular-devkit/build-optimizer   0.9.0-rc.3
@angular-devkit/build-webpack     0.9.0-rc.3
@angular-devkit/core              7.0.1
@angular-devkit/schematics        7.0.1
@angular/cdk                      6.4.7
@angular/cli                      7.0.1
@angular/material                 6.4.7
@ngtools/webpack                  7.0.0-rc.3
@schematics/angular               7.0.1
@schematics/update                0.10.1
rxjs                              6.3.3
typescript                        3.1.3
webpack                           4.19.1

Others:
maybe related to : https://github.com/angular/angular/issues/26572 , https://github.com/angular/angular-cli/issues/12645

@filipesilva heya, sorry to make noise again if you consider this as angular compiler problem(as I think) close or move it please

[filipesilva]

@istiti did the size of you main bundle increase as well after the update? I've read a report that the angular compiler was being included in bundles which can cause longer builds (this is unintended).

Also cc @ocombe about the i18n error.

[filipesilva]

@istiti can you try @@angular-devkit/build-angular@~0.10.0 and tell me if the AOT rebuild times are still very long?

[hmagrini]

@filipesilva not OP, but I'm seeing similar long times with @angular-devkit/build-angular@~0.10.1 as well. Main bundle size increased by ~20%.

[elvisbegovic]

@filipesilva main bundle maybe about +0.3mb but maybe this is becasue code size changed... i can't really say it is bigger I find it is not increasing!

right I was with build-angular on 0.9.rc3 after updating using ng udpate --next and didn't realize this build-angular didn't automatically update so after move on 0.10.1 nothing change... same slowdown!

I remove package-lock, remove nodes_modules folder, npm i again on this package file

[filipesilva]

EDIT: that size difference between this repro and a new project seemed to be the router, so it didn't actually show a regression.

Original comment below.

---

@crisbeto provided a reproduction (https://github.com/filipesilva/cli-seven-size-regression) where the size regression can be seen:

$ ng build --prod

Date: 2018-10-19T10:23:51.856Z
Hash: c0ef836735b53b519186
Time: 31168ms
chunk {0} runtime.ec2944dd8b20ec099bf3.js (runtime) 1.44 kB [entry] [rendered]
chunk {1} main.9c9c99a7a2fa8cb359cc.js (main) 259 kB [initial] [rendered]
chunk {2} polyfills.c6871e56cb80756a5498.js (polyfills) 38.4 kB [initial] [rendered]
chunk {3} styles.3bb2a9d4949b7dc120a9.css (styles) 0 bytes [initial] [rendered]

Notice the main bundle is 259kB.

I tried to make a new project and actually couldn't see the regression, the size was pretty much the same as in v6 (170 vs 173kb):

$ ng build --prod

Date: 2018-10-19T10:11:55.402Z
Hash: ec058cf8c6e7c3d3766f
Time: 25655ms
chunk {0} runtime.ec2944dd8b20ec099bf3.js (runtime) 1.44 kB [entry] [rendered]
chunk {1} main.473563a0e53e983e2153.js (main) 173 kB [initial] [rendered]
chunk {2} polyfills.c6871e56cb80756a5498.js (polyfills) 38.4 kB [initial] [rendered]
chunk {3} styles.3bb2a9d4949b7dc120a9.css (styles) 0 bytes [initial] [rendered]

But since I have a repro I can follow up.

[elvisbegovic]

I confirm On new green project all seems okay @filipesilva

[dottodot]

My main bundle has gone from 1.33MB to 1.59 MB which makes v7 unuseable, 1.3Mb was already on the limit of what's acceptable and can't really afford it to be any bigger. Where has the extra 0.26MB come from?

[skreborn]

@filipesilva The difference between 173 kB and 259 kB is simply initializing the latter with the router module included. Started two fresh projects to confirm.

[skreborn]

Including the router module apparently adds an extra ~3400 (prettified) lines to the main bundle.

[filipesilva]

@dottodot I don't know where those extra 0.26 MB come from but I think they hold the answer to this problem.

Would you be willing to run some tests for me?

The first thing to figure out is if the increased size came from updating Angular itself or the CLI. We can figure that out by updating them separately.

By "Angular itself" I mean all @angular/* dependencies except @angular/cli. For v6 you can use 6.1.7 and for v7 you can use 7.0.0.

By CLI I mean @angular/cli and @angular-devkit/build-angular. For v6 you can use @angular/[email protected] and @angular-devkit/[email protected]. For v7 you can use @angular/[email protected] and @angular-devkit/[email protected].

I assume you're doing a production build. Can you tell me the size of the main bundle for these combinations of dependencies please?

• Angular v6 and CLI v6
• Angular v6 and CLI v7
• Angular v7 and CLI v7

This should help narrow down where the size changed.

[filipesilva]

@skreborn yeah it took me a while to figure that out, I've edited the original comment to reflect it :/

The size difference must be something else.

[dottodot]

@filipesilva no problem will try and get that info together will be a bit later on though.

[dottodot]

@filipesilva

Angular v6 and CLI v6 1.36 MB
Angular v6 and CLI v7 1.36 MB
Angular v7 and CLI v7 1.59 MB

So I'm assuming that means it's an angular issue rather than cli.

[skreborn]

@dottodot Is the increase in build time similarly noticeable between those versions?

[filipesilva]

@crisbeto also reports it happens when updating Angular v6 to v7, while keeping CLI at v7.

I'm trying to get to the bottom of it but it's been really hard what a reproduction. If anyone has a project where they see this and can share, it would really help.

[coryrylan]

A private project I tried to upgrade did increase in bundle size due to the compiler it looks like being bundled into the production output. I am hoping to try to recreate this is in a isolated small repo. The project is fairly large 60,000+ lines of TS. Here are the images of the webpack bundle analyzer before upgrading to 7 and after.

[dottodot]

@skreborn Angular v7 and CLI I can only build using the build as I get JavaScript heap out of memory

node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build client --prod

I'll try the builds again and measure the time.

[Marabyte]

I'm seeing the same as @coryrylan in my project

[filipesilva]

Just to give you all an update I've been most day going through the compiled source that @crisbeto provided, but didn't get very far.

All I can say at the moment is that something is pulling in part of the compiler but I can't find what that is. I couldn't find any references that invoked the compiler code by tracing the compiled source.

I have isolated the retained compiler exports though: ConstantPool, LiteralExpr, WrappedNodeExpr, jitExpression, R3ResolvedDependencyType, compileInjector, compileNgModule, compilePipeFromMetadata, makeBindingParser, parseTemplate, compileComponentFromMetadata, compileDirectiveFromMetadata, parseHostBindings, compileInjectable.

I don't yet have a repro that I can debug completely so if someone comes up with one, it would really help.

I'd like to say that I am sorry that this problem slipped through. It is a non trivial problem though, resulting from the interplay between imports on complex applications and dead come elimination. None of the apps and variations I've tried so far fave exhibited this behavior, so I don't think it is trivial to reproduce.

[dottodot]

@filipesilva It is worth us posting a list of packages installed to see if there's anything common in each app that it could be attributed to?

[filipesilva]

@dottodot off the top of my head I don't really think it's any external packages, but I might be wrong. If the problem ends up being related to third party libraries that would help, yes.

[filipesilva]

@coryrylan you mentioned you were trying to get a repro, and I know that's not easy starting at a large application so I wanted to give you an approach that might help:

• make a new git branch to keep debugging changes
• set "outputHashing": "none", and "sourceMap": true, in angular.json
• npm install -g source-map-explorer
• make a new npm script that runs build and source-map-explorer (like "test": "ng build --prod && source-map-explorer dist/latest-project/main.js",)
• try removing certain parts of the application and then running npm test, if @angular/compiler is still in the sourcemap then that part of the app is not relevant to the repro.

Starting from a full application, these are the parts I would try to remove in order:

• comment out lazy routes, as they are individual compilation units
• comment out non-lazy routes and delete their component imports
• comment out NgModule imports
• replace all templateUrl and template with template: '<div></div> to remove template compilation

This order should hopefully help eliminate big chunks of source code until the compiler stops appearing in the source-map-explorer visualization.

[filipesilva]

Apologies for my repeat comments (now deleted). GitHub was having an outage and I repeated them by accident.

[filipesilva]

@istiti and @crisbeto were kind enough to do some live debugging with me but we weren't able to get to the bottom of it.

Still managed to get some interesting insights though. In @istiti's project we could see that @angular/compiler was there, but went away when all the lazy routes were removed.

@crisbeto's project uses a custom webpack configuration and we removed everything until it was just a NgModule and an empty component but still saw @angular/compiler in the sourcemaps.

[elvisbegovic]

I don't want add irrelevant information at this step but :
I have 100+ lazy routes (loadChildren)
Build with ALL routes > I have compiler.js
Build without routes > I DON'T have compiler.js see sourcemaps graph

Build with only ONE loadChildren >I HAVE compiler.js see screenshot sourcemaps graph

@filipesilva

[filipesilva]

@istiti I think that's actually relevant, in the context of how modules are optimized.

Can you try doing something please?

First make a build that has compiler in the sourcemaps graph, like the one with one loadchildren.

Then open node_modules/@angular/compiler/fesm5/compiler.js and find lines 1181 to 1210:

var BoundPropertyMapping = (_a = {},
    _a[4 /* Animation */] = 4 /* Animation */,
    _a[1 /* Attribute */] = 1 /* Attribute */,
    _a[2 /* Class */] = 2 /* Class */,
    _a[0 /* Property */] = 0 /* Property */,
    _a[3 /* Style */] = 3 /* Style */,
    _a);
/**
 * A binding for an element property (e.g. `[property]="expression"`) or an animation trigger (e.g.
 * `[@trigger]="stateExp"`)
 */
var BoundElementPropertyAst = /** @class */ (function () {
    function BoundElementPropertyAst(name, type, securityContext, value, unit, sourceSpan) {
        this.name = name;
        this.type = type;
        this.securityContext = securityContext;
        this.value = value;
        this.unit = unit;
        this.sourceSpan = sourceSpan;
        this.isAnimation = this.type === 4 /* Animation */;
    }
    BoundElementPropertyAst.fromBoundProperty = function (prop) {
        var type = BoundPropertyMapping[prop.type];
        return new BoundElementPropertyAst(prop.name, type, prop.securityContext, prop.value, prop.unit, prop.sourceSpan);
    };
    BoundElementPropertyAst.prototype.visit = function (visitor, context) {
        return visitor.visitElementProperty(this, context);
    };
    return BoundElementPropertyAst;
}());

And change them to:

/**
 * A binding for an element property (e.g. `[property]="expression"`) or an animation trigger (e.g.
 * `[@trigger]="stateExp"`)
 */
var BoundElementPropertyAst = /** @class */ (function () {

  var BoundPropertyMapping = (_a = {},
    _a[4 /* Animation */] = 4 /* Animation */,
    _a[1 /* Attribute */] = 1 /* Attribute */,
    _a[2 /* Class */] = 2 /* Class */,
    _a[0 /* Property */] = 0 /* Property */,
    _a[3 /* Style */] = 3 /* Style */,
    _a);


    function BoundElementPropertyAst(name, type, securityContext, value, unit, sourceSpan) {
        this.name = name;
        this.type = type;
        this.securityContext = securityContext;
        this.value = value;
        this.unit = unit;
        this.sourceSpan = sourceSpan;
        this.isAnimation = this.type === 4 /* Animation */;
    }
    BoundElementPropertyAst.fromBoundProperty = function (prop) {
        var type = BoundPropertyMapping[prop.type];
        return new BoundElementPropertyAst(prop.name, type, prop.securityContext, prop.value, prop.unit, prop.sourceSpan);
    };
    BoundElementPropertyAst.prototype.visit = function (visitor, context) {
        return visitor.visitElementProperty(this, context);
    };
    return BoundElementPropertyAst;
}());

(moving var BoundPropertyMapping inside BoundElementPropertyAst).

Then build again. Do you still get compiler in the sourcegraph? If you don't, do you see see increased build time overall?

[elvisbegovic]

@filipesilva

both have compiler.js but size change a little bit

before replacing I get this graph and compiler.js in sourcmap is 201.85 KB

after replacing i get this graph and compiler.hs in sourcemap is 201.81 KB

[dhardtke]

I can confirm that even without any routes the compiler.js is included in our bundle as well.
(custom webpack config btw - for now I have found the workaround of declaring @angular/compiler as external in my webpack config)

[stevethemacguy]

I am using a custom Webpack build (i.e. not using Angular CLI). After updating from Angular 6.1.7 to 7.0.0, and from @angular-devkit/build-optimizer 0.8.3 to 0.10.2, my AOT build now includes the compiler. Please let me know if I can help in some way!

[filipesilva]

@stevethemacguy is your project open source or something that I could see?

For those using the CLI, could you try this please:

• open ./node_modules/@angular-devkit/build-angular/src/angular-cli-files/models/webpack-configs/browser.js
• find the following lines:

        resolve: {
            mainFields: [
                ...(wco.supportES2015 ? ['es2015'] : []),
               'browser', 'module', 'main'
            ]
        },

• add 'esm5' before the others (so 'esm5', 'browser', 'module', 'main')
• try building again

Does your build get faster and/or smaller?

[stevethemacguy]

@filipesilva The code is not open source or publicly released. I removed the Angular router imports completely and the bundle still includes the compiler. Also, I'm using ES2015 (i.e. ES6) modules in my TS compiler config.

I have a separate, smaller CLI repo that I use for testing and Webpack Bundle Analyzer shows the compiler in the main bundle, but the file size difference between Angular 6 and 7 is negligible, so I'm not actually sure if the CLI repo is including the compiler or if it's just an issue with Webpack Bundle Analyzer displaying things differently.

[stevethemacguy]

@filipesilva I tested adding 'esm5' to my CLI repo. The main bundle size increased from 567Kb to 694kb after adding the line. The build produced a new warning, but otherwise succeeded. Note: this is a stripped down repo and not my real app.

Before adding 'esm5':

After adding esm5:

[dottodot]

@filipesilva

I've also tried adding esm5 build time was about the same but I got a slight reduction in bundle size

1.59 MB before
1.55 MB after

also got the same error as above.

[exequiel09]

Angular Compiler was also included in my production build after updating to Angular v7 and Angular CLI v7

[crisbeto]

I managed to boil down my project to this that showcases the issue.

It has a single .ts file that looks like this:

import { enableProdMode } from '@angular/core';
console.log(enableProdMode);

[filipesilva]

I've been having a look at @crisbeto's reproduction. I've reduced it a bit further (https://github.com/filipesilva/cli-12646-custom-config) and think I have something that can help some configurations.

If you are using a custom webpack configuration, make sure you have turned off the node builtins (see https://github.com/filipesilva/cli-12646-custom-config/commit/c9f28d0632a5da114c57d3ff6667b870734b5622).

This is something we do on the CLI since v6. Having them on can prevent @angular/core from being concatenated via https://webpack.js.org/plugins/module-concatenation-plugin/.

Doing this prevents most of @angular/core and @angular/compiler from being retained. But not all. Still investigating why some is still retained.

This advice has no effect on CLI projects since they already have it turned off.

[filipesilva]

Hi all,

Just wanted to give another update. With a lot of help from @istiti and @crisbeto, I think I've identified what causes compiler to be included and builds to take more memory, time and be bigger.

This is related to https://github.com/angular/angular/issues/25645. In Angular v7, there is a direct dependency from @angular/core to @angular/compiler.

This should not be a problem because treeshaking should eliminate that dependency as it is not used anymore in the final code. The most advanced treeshaking optimizations are enabled through the joint work of Webpack's Module Concatenation, Terser/Uglify and Build Optimizer over a projects source code.

There are cases where module concatenation can be prevented, called bailouts. One such case is importing a module through require().

Specifically in this issue, whenever @angular/core is imported through require() anywhere in a project (including in dependencies), it will not be correctly optimized. Although this was still true of Angular v6, now with the new @angular/compiler import the consequences are greater and more code is retained.

More code being retained means bigger bundles, but also means that processing the bundles takes longer and needs more memory.

I've prepared two repos that show this:

• without the CLI just importing a module that uses require: https://github.com/filipesilva/cli-12646-custom-config
• with the CLI importing a library that uses require: https://github.com/filipesilva/cli-12646-cli-app

Now that we have identified the root cause of this problem, we are looking at solutions.

I really want to give a huge shoutout and thanks to both @istiti and @crisbeto. They helped a lot over the last few days by running tests, diagnostics and experiments on their code bases that lead to identifying the root cause.

Without that we still wouldn't have a reproduction, and without a reproduction there is little hope of fixing problems. Personally I believe making reproductions is the most important part of open source, because once a reproduction is identified anyone can try to work out a solution. But without a reproduction there's no hope of fixing truly a problem. So in cases where the reproduction is not trivial and projects in the wild are exhibiting issues, we really have to rely on help from those projects to figure out what's wrong.

[alexeagle]

Proposal:

• ng build --prod will check if @angular/compiler got retained in your optimized prod bundle
• if so, it prints something like "We detected a de-optimization affecting your production bundle size, run ng build --prod --verbose=optimizations
• This command uses information known to Webpack to find CJS imports that bring in @angular/core (could parse the stats file or add a new plugin after the graph is built)
• It prints something that helps the user take action to file an issue against the right library which caused the de-optimization

[stevethemacguy]

@filipesilva I see require("@angular/core") in a number of my dependencies (mostly in the included UMD files). Is there something I should do about that? I added "node: false" to the Webpack config, but it had no effect. If a more general solution is on it's way, then I can wait.

@dhardtke, How did you mark the compiler as external? I was able to exclude the compiler using the following, but the build fails:

externals: {
'@angular/compiler': '',
},

[dhardtke]

Same here, I too have a bunch of dependencies that have require() statements in them (e.g. https://github.com/tjoskar/ng-lazyload-image)...

@stevethemacguy I did this:

externals: {
    "@angular/compiler": JSON.stringify("@angular/compiler")
},

which essentially tells webpack to replace an import of "@angular/compiler" with a string. You might also use JSON.stringify("")

[stevethemacguy]

@dhardtke Nice. Seems like a sensible workaround for now. Thanks!

[100cm]

@dhardtke Nice work !

[alexeagle]

We discussed a plan yesterday with @mhevery that would allow us to cut the dependency from @angular/core to @angular/compiler by publishing a well-known symbol on window from the latter, and ensuring that either platform-browser-dynamic or maybe a new @angular/core/jit module was loaded first so that the compiler is present before the first Ivy component bootstraps. We are working to prioritize that.

[dereklin]

@dhardtke

Is externals a webpack config? where do you put this in an angular cli build?

[dhardtke]

I tried to look if there was a way to declare externals with the webpack config used by @angular/cli (https://github.com/angular/angular-cli/blob/master/packages/angular_devkit/build_angular/src/angular-cli-files/models/webpack-configs/common.ts) but couldn't find any :( So no, there is no way to do it with @angular/cli.

[zorrme]

I tried using https://github.com/meltedspark/angular-builders, but did not help. I have a universal app with lots of lazy modules

[hakimio]

@dereklin See _100cm_ comment here.

[thekiba]

@filipesilva In typescript version 3.1 and more, was found a bug with performance regression. I think the problem is the same.

Issue: Microsoft/TypeScript#28025
PR: Microsoft/TypeScript#28028

[filipesilva]

@thekiba I think the problem you are describing is slightly different, and is the root cause of https://github.com/angular/angular/issues/26674. In fact I'm pretty sure it's the same because both mention moduleNameResolvesToAmbientModuleInNonModifiedFile specificially.

I agree they are similar problems because both affect performance, and affect them with the same dependency changes since Angular 7 depends on TS 3.1. But the TS issue is broader and the issue here needs to also be solved for a full resolution in CLI projects.

[alexeagle]

https://github.com/angular/angular/pull/26734 will fix this from framework side. We are figuring out if that can go in the next 7.0.x release of Angular to resolve this issue next week.

[seangwright]

Our CI build in Azure Pipelines fails with

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Our smaller apps still build ok, but larger ones are using too much memory for the hosted agents.

Hopefully the combination of fixes for the above mentioned Typescript regression and this framework change will resolve this issue.

[aalbericio]

Same here. We have seen that production builds use more memory than in previous versions.

This should be addressed as soon as possible but, in the meantime, you can increase the -max-old-space-size to something around 4gb.

[mko]

As a note: we have noticed that production builds aren't the only thing that suffer from this. With a sufficiently large application, ~this~ _a JS Heap out of memory error_ (which was how I got to this issue page from #12645 ) happens 100% reliably with the application watcher. Increasing the heap limit as suggested in a few places definitely delays the error, but it looks like there might be a memory leak in the build->watch->detect change->rebuild loop.

It should be noted that it appears to happen mostly in the SourceMapDevToolPlugin step of the rebuild. Not sure how to provide a reproduction as the issue appears to be purely related to the size of the application and number of independent modules. We have multiple projects that this happens on and the smaller (~1650 modules) fails after more rebuilds than our larger (~3400 modules) project with equivalent memory settings.

[mordka]

I agree with @mko. We have a larger app which fails to rebuild when running using ng serve. I identified the problem that compiler.js is included in the bundle. I expected that this PR angular/angular#26734 will fix it but it's merged and as of version 7.0.2 the issue still remains - compiler.js is included in the bundle. This blocks our upgrade to Angular 7.

[filipesilva]

The @angular/[email protected] releaseincludes https://github.com/angular/angular/pull/26734 and I could verify with the repros in https://github.com/angular/angular-cli/issues/12646#issuecomment-432292721 that the compiler is not retained anymore. This should address compiler being retained and part of the longer build time.

On the TS side, Microsoft/TypeScript#28028 is merged but not yet released. We are waiting for it to be released in 3.1.x. When that is out, it should overall address longer build and rebuild times.

[filipesilva]

@mordka I'm not sure why the compiler is still included in your bundle. I can give you instructions on how to obtain a log that lists the reasons though.

• first make sure you are using @angular/[email protected], Typescript 3.1.x and CLI 7.x
• then open node_modules/@angular-devkit/build-angular/src/angular-cli-files/models/webpack-configs/stats.js
• replace the verboseWebpackOutputOptions variable with below:

const verboseWebpackOutputOptions = {
  // The verbose output will most likely be piped to a file, so colors just mess it up.
  colors: false,
  usedExports: true,
  maxModules: Infinity,
  optimizationBailout: true,
  reasons: true,
  children: true,
  assets: true,
  version: true,
  chunkModules: true,
  errorDetails: true,
  moduleTrace: true,
};

• run ng build --prod --verbose > log.txt

This should make a log showing all the modules and the reasons for their inclusion. File paths will be shown here. Can you send post that log here?

[quanterion]

I updated to @angular/[email protected] ng serve --aot works well first time first rebuild is as slow as initial build, second rebuild resulted in CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

The same project with 6.x branch of angular and cli works perfectly for hours of development

[elvirdolic]

I can confirm that with angular 7.0.2 our bundle size is now almost the same compared to 6.x. With 7.0.1 it was about 300kb more. Main bundle. —prod build is about 2x slower then before from 8min to 16min hope for this typescript fix

[seangwright]

To add to what is being mentioned about issues with the build/serve, I'm currently running into both slower re-compilation and failing re-compilation.

Here are my package versions.

Angular CLI: 7.0.3
Node: 10.13.0
OS: win32 x64
Angular: 7.0.2
... animations, cdk, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.10.3
@angular-devkit/build-angular     0.10.3
@angular-devkit/build-optimizer   0.10.3
@angular-devkit/build-webpack     0.10.3
@angular-devkit/core              7.0.3
@angular-devkit/schematics        7.0.3
@angular/cli                      7.0.3
@angular/flex-layout              7.0.0-beta.19
@ngtools/webpack                  7.0.3
@schematics/angular               7.0.3
@schematics/update                0.10.3
rxjs                              6.3.3
typescript                        3.1.5
webpack                           4.19.1

My initial ng serve --aot builds correctly but after the first save I see the following result in the console.

i ｢wdm｣: Compiled successfully.
i ｢wdm｣: Compiling...

Date: 2018-11-01T15:37:56.029Z - Hash: 50c9f255f7875ecf1705 - Time: 17543ms
17 unchanged chunks

ERROR in Internal Error: The structure of the program changed during codegen.
i ｢wdm｣: Failed to compile.

[mordka]

@filipesilva I'm sorry, I forgot to clean package-lock.json and it appears the compiler.js was only in the dev build which is correct.
It works now! :confetti_ball:
Builds are not failing and build times are fast again.

To anyone concerned, ensure to rm package-lock.json and rm -rf node-modules. Also followed by ng update @angular/cli @angular/core

[quanterion]

I have tried just released angular-cli 7.0.4 and TypeScript 3.1.5. Build times are great. But second rebuild in ng serve results in out of memory error.

After first build a have 1GB memory usage by node.js process, after the second one its 1.4 Gb and third rebuild is OOM. Between each build I just commented one line in one ts file.

PS: I have remove node_modules and yarn.lock to ensure correct testing

Angular CLI: 7.0.4
Node: 10.6.0
OS: win32 x64
Angular: 7.0.2
... animations, cdk, common, compiler, compiler-cli, core, forms
... language-service, material, platform-browser
... platform-browser-dynamic, router

Package Version

@angular-devkit/architect 0.10.4
@angular-devkit/build-angular 0.10.4
@angular-devkit/build-optimizer 0.10.4
@angular-devkit/build-webpack 0.10.4
@angular-devkit/core 7.0.4
@angular-devkit/schematics 7.0.4
@angular/cli 7.0.4
@angular/flex-layout 7.0.0-beta.19
@ngtools/webpack 7.0.4
@schematics/angular 7.0.4
@schematics/update 0.10.4
rxjs 6.3.3
typescript 3.1.5
webpack 4.19.1

PS2: in my yarn.lock a can see TypeScript 3.1.3 among angular-cli build dependencies. May be TypeScript should be upgraded in cli repo?

[zorrme]

Typescript 3.1.6 has been released to fix https://github.com/Microsoft/TypeScript/pull/28028

[filipesilva]

[email protected] was just released which should include the performance fix. The combination of @angular/[email protected] and [email protected] should take care of the performance regressions as originally detailed in this issue.

However there are also some reports of a memory leak over rebuilds (@mko, @quanterion), and an odd error about TS program structure (@seangwright). Are these happening only on AOT, or also outside of AOT?

[seangwright]

@filipesilva When I change to JIT for ng serve I do not see the The structure of the program changed during codegen. error. It's only during AOT.

I updated to

"@angular/core": "7.0.2",
...
"@angular-devkit/build-angular": "0.10.4",
"@angular/cli": "7.0.4",
...
"typescript: "3.1.6"

and I am no longer seeing the The structure of the program changed during codegen. error. Build times also seem to be far far more reasonable.

My CI build was constantly throwing Out of Memory exceptions, so I'm going to rerun with this package updates to see if it will start working again.

Thanks for all the work on fixing these issues!

[alexeagle]

This issue was specifically about bundle size and performance, which we think are fixed, and I've noted this in the release notes for https://github.com/angular/angular-cli/releases/tag/v7.0.4

I think we should close this issue, thanks everyone!
If there's a memory leak, that's a separate issue and should be opened separately.

[filipesilva]

Heya, just wanted to mention that I tried to reproduce the memory leak with the new @angular/core and typescript, and couldn't see any. If someone is still experiencing the memory leak please open a new issue and ping me on it.

[aalbericio]

Back to normal build times with the new CLI 7.0.4 and TypeScript.

"_JavaScript heap out of memory_" issue is still present in production builds though...

[hakimio]

@aalbericio Did you update "@angular/core" to version 7.0.2?

[aalbericio]

@hakimio Yes, that was all done by ng update but, anyway, I've just double-checked it and I'm in: @angular/[email protected]

[perjerz]

with ng build --prod --vendor-chunk --progress false "portal"

Angular CLI: 7.0.4
Node: 11.1.0
OS: linux x64
Angular: 7.0.2
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router, service-worker

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.10.4
@angular-devkit/build-angular     0.10.4
@angular-devkit/build-optimizer   0.10.4
@angular-devkit/build-webpack     0.10.4
@angular-devkit/core              7.0.4
@angular-devkit/schematics        7.0.4
@angular/cli                      7.0.4
@ngtools/webpack                  7.0.4
@schematics/angular               7.0.4
@schematics/update                0.10.4
rxjs                              6.3.3
typescript                        3.1.6
webpack                           4.19.1

Still JavaScript heap out of memory

[maxime1992]

Same here after full upgrade Angular (7.0.3) and TS (3.1.6) today:

$ ng build --source-map false --project frontend --prod
 92% chunk asset optimization TerserPlugin

#
# Fatal error in , line 0
# API fatal error handler returned after process out of memory
#
#
#
#FailureMessage Object: 0x7f39243649b0

#
# Fatal error in , line 0
# API fatal error handler returned after process out of memory
#
#
#
#FailureMessage Object: 0x7f39253669b0Illegal instruction (core dumped)

If I turn off the optimization in angular.json it's building just fine though.

[aalbericio]

Same here.

Tested using the same versions described above.

Anyway, our project must be very close to that point where the build can either fail or succeed though. I'd say that it fails 90% of the cases vs 10% of successes.

The question is: is there anything (probably with the TerserPlugin) that can make our build jump the red line of default memory needs?

What has changed in v7 that can make this build cross that line?

Is it expected that we need to allocate more memory for our v7 prod builds?

[filipesilva]

To be honest I expected terser to require less memory than uglify-es, because of existing bugs that were fixed.

Are you using the additional memory flag? You can use it on a npm script like this:

"ng-high-memory": "node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng",

Increasing the memory limit is something you should expect to do at some point. Node processes have a default memory limit of about 1.7gigs. When a node process starts getting close to the memory limit it needs to spend more and more time doing garbage collection to free up memory, which in turn makes it run more slowly.

I wouldn't be terribly surprised if the projects that are now need more memory were nearly at the maximum before, and different code structures ended up demanding more memory...

Also what can happen is that the inner module closures (the module concatenation thing I mentioned before) managed to get more code in a single module and thus require more memory to optimize, with potential better savings.

[maxime1992]

Filipesilva... Please accept some Angular love

I just came across your comment to fix that issue but it actually fixed way more than that.

Working on a huge project, I had to turn off source maps when building in prod mode (which was a shame because we're reporting our errors to sentry and before we were able to point out where in our TS code the error occurred instead of compiled JS!).

But that's not it. Recently, I got crashes when building prod so I also had to turn off optimization...

And build, serve, linting... Pretty much everything was slow. It's waaaay faster now :heart: :tada: :fire:

Thanks!

[aalbericio]

Thanks for your clarification, Filipe.

Now that we know that this is an expected behavior, I can now be confident in increasing that memory setting.

I didn't want to do that before since I always expect new versions to be faster, better and use less memory :)

[kzc]

To be honest I expected terser to require less memory than uglify-es, because of existing bugs that were fixed.

@filipesilva No memory improvement was made in terser over uglify-es. Perhaps in the course of fixing bugs the memory requirement may have actually increased in some cases. If you manage to isolate a terser minify() JS input that has significantly larger memory requirements than uglify-es with the same minify options, put the JS in question into a gist and create a new terser issue.

The one dramatic improvement for the minify time regression in collapse_vars was capping the upper bound of quadratic complexity to save CPU time - not memory.

what can happen is that the inner module closures (the module concatenation thing I mentioned before) managed to get more code in a single module and thus require more memory to optimize, with potential better savings

That's probably what's going on. More code was likely fed to terser in a more recent version of webpack or angular-cli. Perhaps there are webpack dials to limit the size of chunks.

[kzc]

Also keep in mind that one must take into account the aggregate memory consumed by angular-cli, webpack and the minifier. Because the minifier is the last to run it often is blamed for problems occurring in earlier phases of the build pipeline. Webpack had several memory problems related to chunks in the past as I recall.

[filipesilva]

@kzc thanks for chiming in, it's always nice to have your uglify insight!

I kinda assumed that the memory usage came hand in hand with the CPU consumption since as one gets closer to the memory limit more and more time is spent on garbage collection. But I don't really have the logs of any particular project... source files for these bigger projects is hard to come by.

I had a report in https://github.com/angular/angular-cli/pull/11996 of a 60% reduction in build time when moving to terser though. Maybe that was just the regression you mentioned.

Anyway if I come across specific input that exhibits odd resource consumption I'll file an issue.

In regards to the aggregate memory usage, we do use the parallel processing in https://github.com/webpack-contrib/terser-webpack-plugin#parallel. This ends up running individual chunks through terser in a separate process, so the overhead from other tooling is reduced.

However the chunks are rarely of similar size. Most of the time there's a single huge chunk and several small ones. I think the huge chunk is the one that busts the memory limit.

Now that I think about it, I should check if the parent processes memory limit is being passed to the worker processes there...

[kzc]

Here's an example of a webpack process being very close to the Node heap threshold and a webpack + minifier upgrade just put it over the limit: https://github.com/webpack-contrib/uglifyjs-webpack-plugin/issues/188

I kinda assumed that the memory usage came hand in hand with the CPU consumption since as one gets closer to the memory limit more and more time is spent on garbage collection.

True, CPU and memory use are often correlated but minifiers (and probably most optimizing compilers in general) spend the majority of their time looking for optimization opportunities in the AST that may or may not exist. These AST searches don't usually create much in the way of long lived heap objects.

Recent V8 versions reportedly have better GC algorithms, but there also could be regressions in some cases. So it would be worth testing different Node versions when encountering a memory issue.

[volago]

In my case everything was fine after ng update to v7. But once I updated devDependencies:

the production build time increased about 3 times:

[filipesilva]

@volago can you try the latest @angular-devkit/[email protected] instead please? The version you are using is old.

[volago]

@filipesilva I've tried with 0.6.8 because it's marked as LTS. Latest (0.10.6) as well as next (0.11.0-rc.0 ) also slows down the build.

[filipesilva]

@volago is your project somewhere I can look at?

[volago]

@filipesilva is't private, sorry.

[Hypenate]

Just hopping in on this.
I was getting an error (call_and_retry_last allocation) when I wouldng build --prod my project, ng serve and ng build worked fine.
The ng build --prod would run forever until it threw this error.

I updated to the latest node.js and turned sourceMap: true to sourceMap: false in both tsconfig.js and angular.json.

[inthegarage]

Instead of the number 8000 (max_old_space_size) being bandied about, our medium sized project needed a value of 16000 to get going. This 1/2ed the time taken for the build.

[fxck]

Ever since updating to

"@angular-devkit/build-angular": "~0.11.0",
"@angular/cli": "7.1.0"

serve/jit rebuild times got like ten times slower..

(running it with "start:jit": "node --max_old_space_size=16384 ./node_modules/@angular/cli/bin/ng serve")

[jtyagi]

@filipesilva getting ERROR in Internal Error: The structure of the program changed during codegen..
while creating build using ng build --prod. it is working fine for ng build or ng serve.
while upgrading my application to angular 7.
"@angular/cli": "^7.1.0",
"@angular/compiler-cli": "~7.1.0"
"@angular/core": "~7.1.0"

angular.json configuration
"production": {
"optimization": true,
"outputHashing": "all",
"sourceMap": false,
"extractCss": true,
"namedChunks": false,
"aot": true,
"extractLicenses": false,
"vendorChunk": false,
"buildOptimizer": true,
"fileReplacements": [
{
"replace": "src/environments/environment.ts",
"with": "src/environments/environment.prod.ts"
}
]
}

[ggsandro]

Same here

[HarelM]

Just upgraded from 6.0 to 7.1 and build times of both production and incremental became unreasonable.
I have opened a separate issue here: #13112 which contains all the relevant data.
It seems that I'm now also getting javascript heap as described here: #5618.
Can be seen in latest appveyor build:
https://ci.appveyor.com/project/IsraelHikingHost/site
Should I downgrade back to 6? Does anyone in this thread have an experience with a version that is working as expected?

[no-more]

Is it not possible to enable cache like cache-loader? Maybe it have already been asked and answered but when I was using webpack + ts-loader + cache-loader + thread-loader I was able to reduce my build time by 3-4 and rebuild time by almost 10. Watch was also greatly improved.

I had to switch to angular-cli to benefit AOT witch is not an option (JIT was taking over 30s on an high end computer).
I really like angular cli and schematics. But build, rebuild and watch times get me crazy.

If it was possible to improve build time with webpack it should be possible to do so with angular-cli, no ?

Thanks

[parliament718]

@HarelM cli v 7.1 was unbearably slow for me. Reverting to 6.2.8 gave me a drastic improvement

[aalbericio]

I'd like to ask a little question regarding the future...

What is the expected impact of Webpack 5 in this issue? Is there any numbers Google can share on this?

Thanks

[matheusdavidson]

Also have slower live reloading with angular 7. Was way quicker with angular 6.

[Meligy]

Has anyone checked if the situation is different with Bazel running the compilation?

As explained by @mgechev in his post Introducing Bazel Schematics for Angular CLI.

[filipesilva]

Just wanted to mention there was a regression in 7.1 (https://github.com/angular/angular-cli/issues/13102), and we have a fix on the way for it.

[StuartMc77]

Has anyone got a definite fix for this yet? I created a new Angular 7 project (version details below) and running any ng commands while in the app folder is taking a ridiculous about of time, for example, running "ng --version" outside the app folder takes about 5 seconds, running it INSIDE the app folder took around 10 minutes.

VERSION DETAILS (ng --version):
Angular CLI: 7.2.1
Node: 8.11.3
OS: win32 x64
Angular: 7.2.0
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router

Package Version

@angular-devkit/architect 0.12.1
@angular-devkit/build-angular 0.12.1
@angular-devkit/build-optimizer 0.12.1
@angular-devkit/build-webpack 0.12.1
@angular-devkit/core 7.2.1
@angular-devkit/schematics 7.2.1
@angular/cli 7.2.1
@ngtools/webpack 7.2.1
@schematics/angular 7.2.1
@schematics/update 0.12.1
rxjs 6.3.3
typescript 3.2.2
webpack 4.23.1

UPDATE
Through trial and error I have discovered my problem is something to do with the IDE I am using, VS Code. When it is closed I can run any ng command, i.e. --version or serve, and it runs as quickly as I expect. When VS Code is open, AND my angular apps folder is opened in it, it slows down again.

Currently debugging trying to pinpoint what VS Code is doing to slow it down.

[subodhkumar]

To be honest I expected terser to require less memory than uglify-es, because of existing bugs that were fixed.

@filipesilva No memory improvement was made in terser over uglify-es. Perhaps in the course of fixing bugs the memory requirement _may_ have actually increased in some cases. If you manage to isolate a terser minify() JS input that has significantly larger memory requirements than uglify-es with the same minify options, put the JS in question into a gist and create a new terser issue.

The one dramatic improvement for the minify time regression in collapse_vars was capping the upper bound of quadratic complexity to save CPU time - not memory.

what can happen is that the inner module closures (the module concatenation thing I mentioned before) managed to get more code in a single module and thus require more memory to optimize, with potential better savings

That's probably what's going on. More code was likely fed to terser in a more recent version of webpack or angular-cli. Perhaps there are webpack dials to limit the size of chunks.

@filipesilva & @kzc Thanks for providing details on this.

Just wanted to know where can we find the webpack dials to check the chunk size.
Shouldnt the size of chunks be limited not to cross node limit, as it cause errors(I agree on the performance reason though).. a more info on the same would help.

Thanks again!

[mp54214]

With Angular 8 I had the same issue. I updated "@angular-devkit/build-angular": "^0.800.6"
to latest "@angular-devkit/build-angular": "^0.801.0" and ng serve build / rebuild times dropped down to what I could call normal (i.e. very quick).

[angular-automatic-lock-bot[bot]]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{_This action has been performed automatically by a bot._}