Angular-cli: Slow build: loadNgStructureAsync takes 73 secs

[vivainio]

🐞 Bug report

Command (mark with an x)

• [ ] new
• [x] build
• [ ] serve
• [ ] test
• [ ] e2e
• [ ] generate
• [ ] add
• [ ] update
• [ ] lint
• [ ] xi18n
• [ ] run
• [ ] config
• [ ] help
• [ ] version
• [ ] doc

Is this a regression?

Yes, Angular 8 was not blocked here.

Description

• Enable benchmarking in:

....\node_modules/@ngtoolswebpack\src\benchmark.js

(const benchmark = true)

Observe that there is a clear slow phase in the build:

NgccProcessor.processModule.ngcc.process+webpack-sources: 3.247ms
NgccProcessor.processModule.ngcc.process+yargs: 3.237ms
NgccProcessor.processModule.ngcc.process+source-map: 2.713ms
AngularCompilerPlugin._createOrUpdateProgram.ng.createProgram: 13105.799ms
30% building 13/13 modules 0 activeAngularCompilerPlugin._createOrUpdateProgram.ng.loadNgStructureAsync: 73171.067ms   <=== BOOOM
AngularCompilerPlugin._make.resolveEntryModuleFromMain: 21.720ms
AngularCompilerPlugin._listLazyRoutesFromProgram.listLazyRoutes: 77.186ms
AngularCompilerPlugin._emit.ng.getNgStructuralDiagnostics: 0.024ms

🔬 Minimal Reproduction

Not currently. This is our proprietary product.

🌍 Your Environment

Angular CLI: 9.0.7
Node: 10.16.0
OS: win32 x64

Angular: 9.0.7
... animations, cli, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... platform-server, router
Ivy Workspace: Yes

Package                            Version
------------------------------------------------------------
@angular-devkit/architect          0.901.7
@angular-devkit/build-angular      0.901.7
@angular-devkit/build-ng-packagr   0.901.7
@angular-devkit/build-optimizer    0.901.7
@angular-devkit/build-webpack      0.901.7
@angular-devkit/core               9.0.7
@angular-devkit/schematics         9.0.7
@angular/cdk                       9.2.4
@angular/material                  9.2.4
@ngtools/webpack                   9.1.7
@schematics/angular                9.0.7
@schematics/update                 0.900.7
ng-packagr                         9.0.0
rxjs                               6.5.4
typescript                         3.7.5
webpack                            4.42.0

Anything else relevant?

I downgraded to 9.0.7 as hinted by https://github.com/angular/angular/issues/36272. Build time with 9.1.9 was even worse.

[alan-agius4]

What were the timings for loadNgStructureAsync in Angular 8?

[vivainio]

loadNgStructureAsyng is actually not there in Angular 8 output.

Here's the beginning of the profile from Angular 8.3.0:

AngularCompilerPlugin._setupOptions: 661.030ms
AngularCompilerPlugin._createOrUpdateProgram.ts.createProgram: 11604.765ms
AngularCompilerPlugin._make.resolveEntryModuleFromMain: 8.484ms
AngularCompilerPlugin._listLazyRoutesFromProgram.createProgram: 7.179ms
AngularCompilerPlugin._listLazyRoutesFromProgram.listLazyRoutes: 3217.039ms
AngularCompilerPlugin._emit.ts.getOptionsDiagnostics: 0.185ms
AngularCompilerPlugin._emit.ts.gatherDiagnostics.ts.getSyntacticDiagnostics: 1.966ms
AngularCompilerPlugin._emit.ts.gatherDiagnostics.ts.getSemanticDiagnostics: 11583.725ms
AngularCompilerPlugin._emit: 22501.209ms
AngularCompilerPlugin._update._emit: 22502.043ms
AngularCompilerPlugin._update: 38016.477ms
AngularCompilerPlugin._make: 38017.190ms

after _make, it stops "hanging" and starts quickly compiling ts files.

[vivainio]

For completeness, here's new snippet from Angular 9.0.7, showing getSemanticDiagnostics etc, and how loadNgStructruceAsyng looks like entirely new slow phase introduced in Angular 9.

AngularCompilerPlugin._createOrUpdateProgram.ng.createProgram: 13101.918ms
AngularCompilerPlugin._createOrUpdateProgram.ng.loadNgStructureAsync: 73200.319ms
AngularCompilerPlugin._make.resolveEntryModuleFromMain: 24.710ms
AngularCompilerPlugin._listLazyRoutesFromProgram.listLazyRoutes: 74.687ms
AngularCompilerPlugin._emit.ng.getNgStructuralDiagnostics: 0.167ms
AngularCompilerPlugin._emit.ng.getTsOptionDiagnostics: 0.286ms
AngularCompilerPlugin._emit.ng.getNgOptionDiagnostics: 0.057ms
AngularCompilerPlugin._emit.ng.gatherDiagnostics.ng.getTsSyntacticDiagnostics: 2.743ms
AngularCompilerPlugin._emit.ng.gatherDiagnostics.ng.getTsSemanticDiagnostics: 13985.054ms
AngularCompilerPlugin._emit.ng.gatherDiagnostics.ng.getNgSemanticDiagnostics: 3502.293ms

[vivainio]

Possibly related (based on git logs):

https://github.com/angular/angular/pull/34792

cc @alxhub

[alan-agius4]

Hi @vivainio,

The main difference between version 8 and version 9. Is that AOT is turned on by default. Hence, that is why loadNgStructureAsync is not listed in the above version 8 logs, because with version 8 you are running a JIT build.

Indeed loadNgStructureAsync, is taking a lot of time in your case. However, to investigate this we'll be needing a reproduction. Can you please share a reproduction, even privately? thanks.

[vivainio]

@alan-agius4 I'll see if I can deliver a private repro. Where / how should I deliver that securely?

[alan-agius4]

You can share that via email.

[elvisbegovic]

Hi,
This should be related to https://github.com/angular/angular/issues/35906 or this earlier https://github.com/angular/angular/issues/34867 🤔 ?

[alan-agius4]

@elvisbegovic, the shared issues are related to incremental recompilations. The above looks like a cold build.

[vivainio]

Yes, this is a cold build.

At this time, I'm working around this problem by disabling aot compiler, i.e. passing "--aot false" to "ng build". Indeed there is no loadNgStructureAsync freeze anymore, and app seems to still work :).

[elvisbegovic]

Ha ok.

Without AoT build/rebuild Is always faster, but you will certainly lost time on browser side « refresh is longer » on medproject. And without map files even faster.

Any chance to try&update on current —next aot/ivy of course? (I would update node too and if app is mainly chunked (lazyload) I would start to disable some chunks and reduce deps)

[vivainio]

@elvisbegovic Node update won't work because I'm on Windows & node-sass doesn't have precompiled binaries for latest node 12 (ones that would work anyway).

With disabled aot, I pay for few seconds on refresh, vs over minute on build time so that's a tradeoff I'm currently happy to make.

With "current" aot/ivy, do you mean Angular 10 prereleases?

[vivainio]

Update: tried with Angular 9.1.10 with AOT on, still slow:

NgccProcessor.processModule.ngcc.process+source-map: 1.159ms
AngularCompilerPlugin._createOrUpdateProgram.ng.createProgram: 14352.329ms
AngularCompilerPlugin._createOrUpdateProgram.ng.loadNgStructureAsync: 77529.531ms
AngularCompilerPlugin._make.resolveEntryModuleFromMain: 28.754ms
AngularCompilerPlugin._listLazyRoutesFromProgram.listLazyRoutes: 67.952ms
AngularCompilerPlugin._emit.ng.getNgStructuralDiagnostics: 0.027ms
AngularCompilerPlugin._emit.ng.getTsOptionDiagnostics: 0.584ms
AngularCompilerPlugin._emit.ng.getNgOptionDiagnostics: 0.079ms

[JoostK]

I would love to poke around in a reproduction. The timing logs indicate that this concerns a very large TS compilation unit, probably with hundreds of components. A reproduction would allow seeing where the AOT compiler spends its time, hopefully exposing the areas that we should focus on to improve performance.

That being said, large compilation units will result in slow builds at some point. It's therefore not guaranteed that there any improvements possible to be made.

---

Chatting with Alan led us to the ability to run just ngc instead of using the Angular CLI. This will exclude Webpack and SASS compilation so it gives a better picture of how much time is spent in the Angular compiler itself (together with the TypeScript compiler). You should be able to run .\node_modules\.bin\ngc -p tsconfig.app.json or wherever the appropriate tsconfig is stored.

[vivainio]

@JoostK

Running ngc on our tsconfig.app.json takes 65 secs (same time on repeated runs). Is there a way to diagnose where it's spending the time, or is 65 sec "ok"?

I can try some things myself before sharing the repro, and would rather share diagnostics dumps than the full source code.

[JoostK]

@vivainio ngc doesn't yet implement the --diagnostics flag (that's on the horizon) but running a bare tsc compile could provide insight into what Angular is actually doing. Use the same command as before, replacing ngc with tsc.

By the CLI timing logs I expect tsc to take roughly 46 seconds. This means that roughly 20 seconds is spent in the Angular compiler, far less excessive than the 73 seconds reported in the loadNgStructureAsync step. This can be explained by SASS compilation which is not done with bare ngc, so it could well be that you are dealing with large amounts of SASS that needs to be compiled.

[vivainio]

@JoostK tsc took only 29seconds. We have ~ 400 SASS files, which is way too much, and SASS compilation has looked like the laggy part here.

I tried replacing all the scss files with blank files, and indeed loadNgStructureAsync takes only 26 secs after that.

Would it be fair to assume then that SASS is the culprit?

If anything good comes out of this (apart from accelerating my urgency in ditching SASS for plain CSS ;-), it would be pretty great if a future update to the cli placed the blame correctly.

[JoostK]

@vivainio Thanks for confirming. I realize now that the workload for tsc is a bit smaller when no AOT compilation has taken place, so that may be why my guess was a bit off (it was derived from createProgram + getSemanticDiagnostics + emit). Your finding is still super valuable as it indicates that SASS compilation takes approx 2/3 of analysis (loadNgStructureAsync), but this is not visible from looking at the logs.

I am not familiar with the CLI's perf reporting logic, it might be possible to trace SASS compilation timings separately. It is currently part of loadNgStructureAsync as that is where the Angular compiler is parsing templates and reading stylesheets, therefore invoking the SASS compiler.

[clydin]

On the Sass and Node.js topic, Angular CLI 8+ no longer uses node-sass by default. Active development of Sass is now focused on dart-sass (on npm sass) and it is now the default Sass preprocessor for the Angular CLI. Although not common, if encountering inadequate performance with dart-sass, the fibers package can be used. The CLI, and in turn dart-sass, will automatically use it if it is install in the project.

Node.js 12+ is highly recommended especially if encountering memory and/or performance problems. In general, it will provide better performance and use less memory. One issue in particular with older versions of Node.js was a performance de-optimization that was triggered by certain looping constructs that contained polymorphic objects. The memory usage improvements in Node.js 12 can also provide significant overall performance benefits as well. When Node.js is close to its memory limit, the garbage collector will run repeatedly and cause less time to be available for actual code to be run.

In regards to optimizing for Sass (or any other preprocessor), one area to investigate is to ensure that excess rules are not being included in each component stylesheet. This can lead to increased build times as well as an increased final application size. In general, Sass imports should be limited to mixins and functions when possible.

[vivainio]

@clydin thanks for the summary of the situation.

• Problem with dart-sass is not the performance, but rather the fact that it's incompatible with some of the features that work with node-sass (and we are currently using). Migrating to it is a bit of an engineering project, esp. since we have multiple semi-independent projects sharing the toolchain and libs.

• Problem with updating to Node 12 is that we are on Windows, and precompiled libsass binaries used by node-sass were not downloadable for Node 12 (as of few days ago).

These are all issues to be tackled. I'm currently thinking of

1) reducing use of SASS by porting most components to use CSS (I kinda scripted this already, variable substitution and just calling node-sass & storing the output)

2) Porting the stuff that needs to be in SASS to use dart-sass, and switch to that

3) Update to node 12.

Thanks everyone for helping and listening patiently :). I'm closing this as resolved.

[angular-automatic-lock-bot[bot]]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{_This action has been performed automatically by a bot._}