Flow: Development Mode Performance Regression 14.x vs 19

[knoobie]

Update:

Further investigation https://github.com/vaadin/flow/issues/10281#issuecomment-802839323

|Topic | Pull Request |
| --------------------------------------- |:--------------------------------------:|

| Service Worker Generation | https://github.com/vaadin/flow/pull/10361 (20.x / 7.x) & https://github.com/vaadin/flow/pull/10384 (19.x / 6.x) |

Original Ticket:

Description of the bug / feature

Starting an application (v19) with mvn downloaded from start.vaadin.com takes a lot more time than the same application with v14. On my machine 3x slower..

Minimal reproducible example

V14 Example

• Download v14 app from https://start.vaadin.com
• Run it with mvn
• Application launches fast

Example attached: my-app(1).zip

V19 Example

• Download v19 app from https://start.vaadin.com
• Run it with mvn
• Application launches slow

Example attached: my-app(2).zip

Expected behavior

Both application have the same starting time.

Actual behavior

v14 - 7 sec webpack execution

2021-03-13 09:32:19.629  INFO 22488 --- [nio-8080-exec-1] c.v.f.s.DefaultDeploymentConfiguration   : 
Vaadin is running in DEBUG MODE.
When deploying application for production, remember to disable debug features. See more from https://vaadin.com/docs/
2021-03-13 09:32:21.440  INFO 22488 --- [nio-8080-exec-1] c.vaadin.flow.spring.SpringInstantiator  : The number of beans implementing 'I18NProvider' is 0. Cannot use Spring beans for I18N, falling back to the default behavior
2021-03-13 09:32:26.009  INFO 22488 --- [         task-2] dev-webpack                              : Started webpack-dev-server. Time: 6972ms

v19 - 21 sec webpack execution (and a lot of unrelated logging)

2021-03-13 09:33:18.310  INFO 22804 --- [onPool-worker-1] com.vaadin.flow.server.DevModeHandler    : Starting webpack-dev-server

------------------ Starting Frontend compilation. ------------------
2021-03-13 09:33:18.671  INFO 22804 --- [onPool-worker-1] com.vaadin.flow.server.DevModeHandler    : Running webpack to compile frontend resources. This may take a moment, please stand by...
[....]
[2021-03-13 09:33:39.850  INFO 22804 --- [onPool-worker-1] com.vaadin.flow.server.DevModeHandler    : Started webpack-dev-server. Time: 21539ms

Versions:

- Vaadin / Flow version: 14.x & 19.
- Java version: 8
- OS version: Ubuntu
- IDE (if applicable): Maven

[knoobie]

@pleku well-meaning ping because I think this is a huge regression (300%) and results in a really bad DX

[pleku]

@knoobie this has been discussed internally but there is no conclusion yet. We're tracking the cold/warm build times internally too but this was noticed there only when the final release was done (as apparently it cannot be done reliably with prereleases). One performance test we have for webpack build is still having the same threshold of 11s (or so) since 3.0.

It was sort of ruled out already that the change should not be caused, at least entirely, by switching to use npx nor the theming features. Npx was tested to actually make things faster. And our internal regression test (that also shows build slowdown) doesn't even have the theming features in use. The PWA features were moved from Java runtime to the frontend build, so that might have some impact. But this needs further investigation..

I think it is up for @vlukashov and @mstahv to agree who starts to take a look at this and when. I don't know if this can be "fixed" or if we need to rework the whole build like we've tried to do with investigating webpack 5 module federation (didn't work for web component / polymer reasons) or Snowpack. Personally I would like to get rid off compiling things that are not changed as those are slowing down _every single build we have_ but that might be pointless if there is a better alternative. Personally I also feel we would benefit from having dedicated team for the frontend build, but that is still just a personal opinion for now.

It has been already known that since 3.0+ the fusion introduced changes (like TS compilation that occurs with Flow Lit templates too) makes things slower https://github.com/vaadin/flow/issues/7596 and the issue has some pointers what to look at, but the impact there was from 5s -> 10s.

[knoobie]

@pleku thanks for your valuable feedback and insights! I try to make some more tests the next days with other versions / less features and see if I can narrow down the issue a bit.

Personally I would like to get rid off compiling things that are not changed as those are slowing down every single build we have but that might be pointless if there is a better alternative.

That would probably help a lot with recurring builds! Sadly webpack is a total Blackbox for me so I can't really help on this part..

[Artur-]

I tested this using the production build as follows

for run in `seq 1 3`; do rm -rf v14 && npx @vaadin/cli init v14 && cd v14 && mvn install -Pproduction > ../v14-run-$run && cd .. ; done
for run in `seq 1 3`; do rm -rf v19 && npx @vaadin/cli init --latest v19 && cd v19 && mvn install -Pproduction > ../v19-run-$run && cd .. ; done

The result is

$ grep "Total time" v14-run-*
v14-run-1:[INFO] Total time:  01:05 min
v14-run-2:[INFO] Total time:  01:10 min
v14-run-3:[INFO] Total time:  01:05 min

$ grep "Total time" v19-run-*
v19-run-1:[INFO] Total time:  01:01 min
v19-run-2:[INFO] Total time:  59.422 s
v19-run-3:[INFO] Total time:  57.877 s

So is this related to my machine vs your machine or production vs development mode?

[knoobie]

@Artur- I haven't checked the production build time because it runs on our CI server. I tested the development mode because that's what a developer is running locally most of the time.

[mstahv]

Both matter, but the development mode build time is much more important.

[knoobie]

@Artur- I run the production build with the following result:

v14: 16 seconds
v19: 32 seconds

Environment: Lenovo T470s with Ubuntu LTS (3 year old business laptop)

[Artur-]

Tried the same on a Windows desktop Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 3401 Mhz, 4 Core(s), 8 Logical Processor(s)

When run on an old HDD

v14-run-1:[INFO] Total time:  03:45 min
v14-run-2:[INFO] Total time:  02:54 min
v14-run-3:[INFO] Total time:  02:20 min
v19-run-1:[INFO] Total time:  01:56 min
v19-run-2:[INFO] Total time:  01:08 min
v19-run-3:[INFO] Total time:  01:08 min

when run on an SSD

v14-run-1:[INFO] Total time:  01:53 min
v14-run-2:[INFO] Total time:  01:48 min
v14-run-3:[INFO] Total time:  01:48 min
v19-run-1:[INFO] Total time:  01:46 min
v19-run-2:[INFO] Total time:  01:03 min
v19-run-3:[INFO] Total time:  01:02 min

[knoobie]

@Artur- did you run production mode or development mode?

[Artur-]

This was the same mvn install -Pproduction

If I run only webpack in the generated projects, then I see a difference (this time on a Mac):

for a in `seq 1 3`; do time node_modules/.bin/webpack > /dev/null ; done` 

real    0m7.396s
V19
real    0m10.760s

[knoobie]

Created two screen casts of my runs - both builds happened with no mvn clean before and a proper node_modules.

V14:
https://drive.google.com/file/d/149phuBeqjGyscn9w05snwlV6fSuX0vUR/view?usp=sharing

v19:
https://drive.google.com/file/d/1vJtj68IlCmxafQ2kkPhuQHzbr9sbRjcn/view?usp=sharing

[vlukashov]

I was able to reproduce and can confirm the issue. The frontend build time in v19 has increased 3.5x - 4x compared to that in v14.5.

webpack-dev-server startup time when running a default Flow project generated from https://start.vaadin.com:
| Vaadin version | Win 10 Pro | WSL2 | MacOS 10.14 |
| --- | --- | --- | --- |
| 14.4.9 | 3.0 sec (100%) | 2.5 sec (100%) | 3.1 sec (100%) |
| 16.0.4 | 9.4 sec (316%) | 5.7 sec (228%) | 7.0 sec (228%) |
| 17.0.11 | 7.8 sec (265%) | 5.7 sec (227%) | 6.9 sec (226%) |
| 18.0.7 | 8.5 sec (288%) | 6.1 sec (245%) | 7.5 sec (243%) |
| 19.0.0 | 12.3 sec (416%) | 8.5 sec (340%) | 11.3 sec (369%) |

Projects: v14.zip, v16.zip, v17.zip, v18.zip, v19.zip.

Why it was not detected by an automatic regression test nor noticed by one of Vaadin developers is another question, but let's focus at fixing the build time regression here. I could speculate, without having done a proper research, that the time increase could be due to (i) adding TypeScript compilation in V15 and (ii) adding Service Worker generation in V19. These are assumptions which need validation.

In Flow apps both TypeScript and Service Worker support bring little value, and the build time increase would be hard to justify if it is indeed due to adding these 2 features.

[haijian-vaadin]

Did some preliminary experiments, found that the client-side bootstrapping and service worker has a significant impact on the performance, the impact of theme is not that significant.

On my Mac Laptop:

• The attached v19 app takes 9.1s to build the frontend
• The time drops to 6.6s by removing the service worker plugin alone
• The time drops to 4.2s by removing the client-side bootstrapping (setting useClientSideIndexFileForBootstrapping to false in webpack.generated.js file)
• The time drops only about 0.5s by removing theme related code in the webpack.generated.js file

[haijian-vaadin]

Next step is make a 2 day time-boxed research to identify the root causes and possible solutions.

[platosha]

I researched for the performance improvement ideas using the https://github.com/stephencookdev/speed-measure-webpack-plugin when running node node_modules/webpack/bin/webpack.js --mode=development.

The starting numbers

 SMP  ⏱  
General output time took 8.43 secs

 SMP  ⏱  Plugins
InjectManifest took 6.38 secs
CompressionPlugin took 0.384 secs
StatsPlugin took 0.136 secs
WebpackManifestPlugin took 0.134 secs
HtmlWebpackPlugin took 0.006 secs
ApplicationThemePlugin took 0.005 secs

 SMP  ⏱  Loaders
ts-loader took 4.9 secs
  module count = 7
modules with no loaders took 3.4 secs
  module count = 694
lit-css-loader, and 
extract-loader, and 
css-loader, and 
@vaadin/theme-loader took 0.451 secs
  module count = 4
html-webpack-plugin took 0.015 secs
  module count = 1

Improvements

Recommend: using {transpileOnly: true} option for ts-loader

Gives the best gain by far from the rest. In combination with fork-ts-checker-webpack-plugin for error checking:

Before:

InjectManifest took 6.38 secs
...
ts-loader took 4.9 secs
  module count = 7

After:

InjectManifest took 2.29 secs
ForkTsCheckerWebpackPlugin took 0.004 secs
...
ts-loader took 0.601 secs
  module count = 7

Note: InjectManifest plugin time also decreases.

Recommend: removing flow-frontend/**.ts from TypeScript compilation

We currently ship both source + compiled .js + .d.ts files in target/flow-frontend modules, and the *.ts sources are actually picked up for compilation because we have resolve: { extensions: ['.ts', '.js'] } set in the generated webpack config. I tried simply swapping for ['.js', '.ts']. This brings TypeScript module count down and improves ts-loader time even more.

Before:

ts-loader took 0.601 secs
  module count = 7

After:

ts-loader took 0.183 secs
  module count = 3

The .ts-first extensions configuration is the one recommended in webpack docs, however. We should consider removing the source .ts files from the target/flow-frontend as a way of doing this optimisation.

Maybe: removing resolve.modules entries

Webpack docs recommend keeping those at minimum. We currently have

  resolve: {
    modules: [
      'node_modules',
      flowFrontendFolder,
      ...projectStaticAssetsFolders,
    ],
   }

Removing the latter two was relatively easy in V20 Flow project from the starter for me. The flowFrontendFolder entry was only necessary for the generated theme files, where we could replace it with the existing Frontend/ directory alias, and the ...projectStaticAssetsFolders was never used.

The impact of this is harder to measure with together other improvements applied, does not yield much to my eye. However, with default ts-loader setup, this seems to reduce total time for ~10%:

Before:

General output time took 8.43 secs

After:

General output time took 7.61 secs

Maybe: skipping InjectManifest plugin

In V20, the frontend build generates the service worker, whereas in V14 the service worker is served from Java runtime. Let us see how much it costs for frontend build time, and try skipping InjectManifest plugin:

Before:

General output time took 8.43 secs
...
ts-loader took 4.9 secs
  module count = 7
modules with no loaders took 3.4 secs
  module count = 694

After:

General output time took 6.36 secs
...
ts-loader took 3.3 secs
  module count = 6
modules with no loaders took 3.21 secs
  module count = 602

Again these numbers are taken without ts-loader improvements applied. With transpileOnly: true option for ts-loader, the impact was hardly noticeable for me.

[knoobie]

Some findings from my testing after the chore: skip service worker plugin for non PWAs. (https://github.com/vaadin/flow/pull/10384)

|Version | Time |
| --------------------------------------- |:--------------------------------------:|
| v19.0.0 | 21539ms |
| v19.0.3 | 17526ms (- ~20%) |

After adding <vaadin.useDeprecatedV14Bootstrapping>true</vaadin.useDeprecatedV14Bootstrapping> it drops to 9853ms (- ~60%) but the styling isn't properly applied anymore.

deprecatedV14Bootstrapping:

normal starting: