Flow: Major issues with NPM, are WebJars still an option?

[heruan]

After two months since the upgrade from V13 to V14, I'm still having major issues related to NPM, especially in CI/CD:

1. need to maintain two separate dependency trees (Maven + NPM);
2. for one reason or the other, not all dependencies are detected;
3. at some point the app breaks and you need to delete node_modules;
4. the generated package.json refers to the build directory, which is bad per-se and could also be a serious problem when in CI/CD the build directory is different;
5. also the generated package-lock.json contains references to the build directory, which makes it far more difficult to maintain;
6. if you have multiple modules with integration tests, you start to have multiple package.json, package-lock.json, webpack.config.js, etc. files per each module, all with these 1–5 issues.

Question is: are WebJars still an option to consider? They'll make Java developers and DevOps happier, especially for having a single dependency tree to maintain.

[heruan]

Let me report here @pleku's thorough answer from Gitter:

Webjars obviously had some benefits over npm but it was also very fragile and not really enterprise grade as it is kind of "hobby project" and there were several occasions when there appeared severe regressions from some changes that were not suppose to break anything.

To be clear: we're not happy with how the npm build is working and have made already a change for version 14.1 to improve the development time turnaround time significantly for the "worst case" projects.
But this is not enough, we're still looking at improving things with npm & webpack in terms of experience, stability and performance. E.g. we just discovered e.g. that there is a bug in npm 6.11.0 - 6.11.2 that breaks Vaadin apps when updating maintenance version. Or just freezes during npm install. This is not acceptable.
We are currently investigating several options on how to improve the situation, like switching the frontend resource management from npm -> pnpm or even how to get rid of webpack&npm for development time (without webjars).
But for making a production build, npm&webpack will be needed as it makes no sense for us to build our own tooling for this.
I also don't see Maven becoming the defacto standard for distributing frontend resources.

[heruan]

I didn't know WebJars were a "hobby project", given that I agree it's not a reliable solution.

Glad to know there is work in progress about this! Still I'm concerned with the issues I've listed here, as it's not just bugs or performance but maintainability.

[…] even how to get rid of webpack&npm for development time

This is very interesting, especially for testing and CI/CD. Is there an issue to follow?

[pleku]

This is very interesting, especially for testing and CI/CD. Is there an issue to follow?

For CD (and probably for CI too) you need to go with production mode, and production mode will require full frontend build and that means that npm & webpack are needed. Supposedly if there are no changes on the frontend side of things (not a single frontend dependency/file has changed), there would not need to do any build again if the old artifact can be retained, but this feels to me like an optimization not worth making.

The changes that we are looking for will mostly help development time. But switching to pnpm and/or bundling some of the frontend resources into our .jars might make a difference in CI&CD too because there would be less things to download.
Removing webpack would only be for development time, and I'm being told that it does involve some nasty hacks to be applied on the browser. For production mode, we really want webpack to make the build as it does some many things for us out-of-the-box and we can even further leverage it to allow spitting down the frontend resources into chunks based on e.g. routes (to be done #5537).
Removing npm for development time without building everything ourselves (or using webjars) would mean using unpkg, but that has some issues that we would need to solve too. Also the future of that solution is a bit unsure so it could be risky to use it.

1. need to maintain two separate dependency trees (Maven + NPM);

As said, unfortunately Maven does not have all the frontend resources available and probably never will. Webjars could work even with serving npm based packages, but that would again require that everyone goes and creates a proper type of webjar from the frontend dependency they need, wait for a while to get it live, if there are no issues... so, unfortunately, we don't see webjars as a future proof solution.

1. for one reason or the other, not all dependencies are detected;
2. at some point the app breaks and you need to delete node_modules;

It should be never required for you to delete anything yourself.
That situation is a bug in our (or even in npm code) and needs to be fixed. One such was just discovered #6743 and I hope everyone keeps on reporting these issues so we may fix them.
All dependencies should be detected. We've fixed plenty of issues regarding this and if we are still missing something, please ticketize.

1. the generated package.json refers to the build directory, which is bad per-se and could also be a serious problem when in CI/CD the build directory is different;

To my (limited) understanding, 4) should be just about adding some more configuration and nothing more, if we are missing some configuration that you need to make production build, please ticketize.

1. also the generated package-lock.json contains references to the build directory, which makes it far more difficult to maintain;

I'm not sure what maintaining you mean here. You have customized it ? Do you have it in version control ? The prepare-frontend goal can be used to rebuild this even every build if needed in CI.

1. if you have multiple modules with integration tests, you start to have multiple package.json, package-lock.json, webpack.config.js, etc. files per each module, all with these 1–5 issues.

If everything works smoothly with the current tooling, there should not be any issues with having multiple modules. Except that npm makes the build very slow. For this using pnpm should help as it is faster to begin with and it is using caching out-of-the-box.

[heruan]

Thanks Pekka, you shed light on most of my concerns. About the build directory, let me explain myself well: as far as I understand, package.json and package-lock.json should be in version control; is that right? If it is, package.json might contain:

  "dependencies": {
    "@polymer/polymer": "3.2.0",
    "@webcomponents/webcomponentsjs": "^2.2.10",
    "@vaadin/flow-deps": "./target/frontend"
  }

Where "@vaadin/flow-deps": "./target/frontend" is problematic, since the build directory (here ./target) could be different/elsewhere in a CI/CD enviroment; plus, there are references to it also in package-lock.json.

So, should these files be in version control or not? If yes, I suggest a different path for frontend, not being in the build directory (e.g. in ./frontend, like ./node_modules).

[pleku]

So, should these files be in version control or not? If yes, I suggest a different path for frontend, not being in the build directory (e.g. in ./frontend, like ./node_modules).

Yes you're right, they should be in version control. Just checking if you had something else going on in CI.

IIRC /target is used for @vaadin/flow-deps, the dependencies discovered to be used in the application, since it will be automatically cleaned with mvn clean and is also very convenient for us.

If you have another build directory we should output things to and then refer to from package.json and package-lock.json, then we are probably lacking that configuration and (I hope) it should be relatively straight forward to add.

[heruan]

If you have another build directory we should output things to and then refer to from package.json and package-lock.json, then we are probably lacking that configuration and (I hope) it should be relatively straight forward to add.

Problem is that if the build directory is different in each build environment, you should use an environment variable in package.json and package-lock.json to refer to it (which brings a lot of side effects).

IMHO there should be no references to the build directory in any versioned file, so I have an idea about:

IIRC /target is used for @vaadin/flow-deps, the dependencies discovered to be used in the application, since it will be automatically cleaned with mvn clean and is also very convenient for us.

It would be convenient to clean also node_modules on mvn clean, and since node_modules and frontend both are created by the Maven plugin, why don't use a goal bound to the clean phase to clean them both? This way frontend does not need to stay in the build directory anymore for that purpose.

[Legioth]

mvn clean is only one half of the reason for why those files are located inside /target/. The other reason is that those generated files should not be in version control. /target/ is already by convention included in .gitignore and similar, so the developer doesn't need to add yet another entry.

[heruan]

Well, the same holds for node_modules then.

I'm not a Maven guru, but I guess both node_modules and frontend (or whatever) can be cleaned hooking the plugin to the clean phase regardless of their location.

The .gitignore file in the starters will include that directory as it already includes node_modules, drivers, etc.

[heruan]

@Legioth Any additional thought about the issue with the build directory referenced in versioned files? We're still suffering from this in our test environments.

[Legioth]

We are currently investigating the possibility adding dependencies based on @NpmModule annotations directly to the top-level package.json file instead of a separate generated file that needs to be referenced from the main file. This is discussed in https://github.com/vaadin/flow/issues/7018.

[pleku]

I'm not sure if there is still something left open with the question - the answer was No 😄

Using pnpm (#6966) and the single package.json are already WIP (#7018) . And those are not the only changes we are looking at, but we have not yet decided the next steps after those.

I'm closing this issue. A newer one can be opened or just comment here if there is still something that was missed.