Generator-jhipster: Use npm ci instead of npm install in frontend-maven-plugin

[alexander-redmann-teaching]

Overview of the issue

When an project is created by Jhipster by default the frontend-maven-plugin use install for building the project instead of npm ci.

                     <plugin>
                        <groupId>com.github.eirslett</groupId>
                        <artifactId>frontend-maven-plugin</artifactId>
                        <version>${frontend-maven-plugin.version}</version>
                        <executions>
                            <execution>
                                <id>install node and npm</id>
                                <goals>
                                    <goal>install-node-and-npm</goal>
                                </goals>
                                <configuration>
                                    <nodeVersion>${node.version}</nodeVersion>
                                    <npmVersion>${npm.version}</npmVersion>
                                </configuration>
                            </execution>
                            <execution>
                                <id>npm install</id>
                                <goals>
                                    <goal>npm</goal>
                                </goals>
                                <configuration>
                                    <arguments>install</arguments>
                                </configuration>
                            </execution>
                            <execution>
...

The problem is that everytime npm install is executed the package-lock.json is newly created if the node_modules directory is missing

Motivation for or Use Case

If I want to make an release with Jenkins (for example with jgitflow or maven release plugin). The Jenkins check out the source code without the node_modules directory, but when running the Maven script the package-lock.json would be touched and the release would be failing, because a sourcecode file is changed.
The reason is the behaviour of the npm install call.

Reproduce the error

Commit everything to git and delete all ignored directories by git and run mvn clean package The package-lock.json is changed.

Suggest a Fix

Change install to ci as argument in the frontend-maven-plugin at the npm goal

                            <execution>
                                <id>npm install</id>
                                <goals>
                                    <goal>npm</goal>
                                </goals>
                                <configuration>
                                    <arguments>ci</arguments>
                                </configuration>
                            </execution>

NPM documentation for npm ci
A second advantage is that npm ci is faster than npm install

JHipster Version(s)

5.5.0

JHipster configuration

Build Jhipster project as Maven project. But I think Gradle has the same behaviour

[PierreBesson]

Yes npm ci should be used everywhere, the package-lock should be changed only when the user is actually installing/updating packages. Can you do the PR ?

[ruddell]

Like Pierre said, the package-lock.json only changes during npm install when your package.json has different versions in it than the lock file. This doesn't happen if no versions change. You should commit your updated package-lock.json file to git.

The only issue with this change is that npm ci requires an existing package-lock.json which will fail for apps generated by JHipster Online or using the --skip-install flag

[jdubois]

Indeed, we don't generate this file yet (it's hard), so we can't use 'npm ci' at the moment.

[BhawaniSingh]

We can use npm shrinkwrap, after npm install. Next time whenever we do the npm install, npm will install the exact dependency tree as described by package-shrinkwrap.json

Doc link: https://docs.npmjs.com/cli/shrinkwrap https://docs.npmjs.com/files/shrinkwrap.json

[deepu105]

@BhawaniSingh not sure how it will help in this case, if you generate a project and run a CI on it won't you have the same problem? normally we expect users to run npm install etc locally(as part of the generation or as part of server run) and commit the generated package.json, if they don't do that then this kind of issues will crop in.

IMO the best solution is to document the proper practice to setup CI as below

• make sure package-lock.json is created locally and commit that
• run npm test locally to create snapshots and commit that related to #8612
• setup CI after this to avoid these kinds of issues

[deepu105]

@jdubois @jhipster/developers I would propose docs as above and changing this to use npm ci, of course if someone tries to generate a project with jhipster --skip-install and tries to run it in CI it will fail but thats fine, we are all about best practices and not about supporting whatever people like to do.

[deepu105]

@jdubois sorry missed the jhipster-online part. We need to find a solution for that

[alexander-redmann-teaching]

what if you setup an extra maven profile before the prod profile (the online one with the install argument) and the activation is the missing package-lock.json.
Something like that:

        <profile>
            <id>generate-package-lock.json</id>
            <activation>
                <file>
                    <missing>./package-lock.json</missing>
                </file>
            </activation>
            <build>
                <plugins>
                    <plugin>
                        <groupId>com.github.eirslett</groupId>
                        <artifactId>frontend-maven-plugin</artifactId>
                        <version>${frontend-maven-plugin.version}</version>
                        <executions>
                            <execution>
                                <id>install node and npm</id>
                                <goals>
                                    <goal>install-node-and-npm</goal>
                                </goals>
                                <configuration>
                                    <nodeVersion>${node.version}</nodeVersion>
                                    <npmVersion>${npm.version}</npmVersion>
                                </configuration>
                            </execution>
                            <execution>
                                <id>npm install</id>
                                <goals>
                                    <goal>npm</goal>
                                </goals>
                                <configuration>
                                    <arguments>install</arguments>
                                </configuration>
                            </execution>
                        </executions>
                    </plugin>
                </plugins>
            </build>
        </profile>
        <profile>
            <id>prod</id>
...

And change the install argument in the prod profile to ci. That would generate the package-lock.json on the first run but later only runs ci.
That has the little disadvatange that on the first run npm is build the project twice.

[jdubois]

I'd rather spend time generating the package-lock.json as this will solve everything -> we need a specific ticket for this, not sure when I have time to work on this

[PierreBesson]

@alexander-redman, it seems to me that you are just mimicking the default npm install behavior but with maven instead !
I know you really want to benefit from the speed benefit of npm ci but our first concern is to work out of the box for all setups.

Maybe for now, we could leave the npm ci option commented out to signal to users that they can use it. Like this:

                                <configuration>
                                    <arguments>install</arguments>
                                    <!--
                                   note: with a valid package-lock.json for your project you can use npm ci instead of npm install
                                    <arguments>ci</arguments>
                                    -->
                                </configuration>

[alexander-redmann-teaching]

@PierreBesson Nope the speed is not the problem. The problem is the modification of the package-lock.json when checking out a jhipster project from git (like a jenkins will do) and running mvn -P prod clean package
The proposed profile only fixes the problem with jhipster-online and will also work out of the box ;)
The problem with the comment is that nobody will notice it, because it somewhere around line 1000 and maybe the most one don't know that npm is called by maven in some cases.

[PierreBesson]

You are right... It's really stupid that they recreate the package-lock.json, the package-lock feature is really not at all working like user expect it to. It's 100% bullshit marketing from NPM : "Look we are doing lockfiles just like yarn".

[alexander-redmann-teaching]

Its in many ways mysterious what, how and why npm and angular do the things like they do ;)

[cbornet]

For Angular, we have only have one option (websocket) that has an effect on prod dependency. So that's 2 possible package.json installed with --production. If we also need to generate the devDependencies, then we have 5 options (protractor, skipCommitHook, enableTranslation, useSass, maven) plus all the marketplace modules. So that's 64 combinations if we don't consider the modules.
What I'm unsure is : since npm doesn't flatten the dependencies, if we generate a package-lock.json from a package.json that contains all the possible dependencies, is it valid for all the package.json that we generate ? If it is we could generate this uber package-lock.json.

[deepu105]

I really don't want to generate package locks ourselves. Lets just document
stuff.

I still don't understand why you can't just commit your package.json before
running CI and then use npm ci to run it.

We just need to find a solution for Jhipster online, or document that as
well asking user to do nom install first

On Wed, 24 Oct 2018, 11:39 am Christophe Bornet, [e-mail]
wrote:

For Angular, we have only have one option (websocket) that has an effect
on prod dependency. So that's 2 possible package.json installed with
--production. If we also need to generate the devDependencies, then we
have 5 options (protractor, skipCommitHook, enableTranslation, useSass,
maven) plus all the marketplace modules. So that's 64 combinations if we
don't consider the modules.
What I'm unsure is : since npm doesn't flatten the dependencies, if we
generate a package-lock.json from a package.json that contains all the
possible dependencies, is it valid for all the package.json that we
generate ? If it is we could generate this uber package-lock.json.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/jhipster/generator-jhipster/issues/8642#issuecomment-432587377,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABDlF_MIBfyPJ6DhpG8DT8yf3c_sBbJ5ks5uoDVJgaJpZM4X07Ub
.

[BhawaniSingh]

@deepu105 What I was trying to say, npm install install & updates the package lock file, but if user or generator fire up npm shrinkwrap, it will generate a new file pacakage-shrinkwrap.json, and when "package-shrinkwrap.json" is present, npm will skip the lock file and use shrinkwrap instead to download the dependencies, and probably fixes the CI version mismatch error

When generator fires npm install command to install dependencies, we can configure the generator to fire npm shrinkwrap command.

I'm not saying lock file should be generated by the generator, we just document this in case user CI server encounter version mismatch error.

And yes its better to document then to fix all user scenarios

I'm sorry, I wasn't able to explain last time.

PS: I never encountered this CI situation as described

[deepu105]

@BhawaniSingh sorry I'm still not following. shrinkwrap in npm is the exact same thing as package-lock, the only difference is that the shrinkwrap file is included in the tarball when you publish the project to npm, in this case, that is not applicable since its a java project, so I don't see the relevance of shrinkwrap and I still don't see how its a solution. Apologies if I missed anything.

[BhawaniSingh]

@deepu105 I don't think so npm shrinkwrap is just for publishing,

I've used for version lockdown of dependencies just like package-lock.json, but the main difference is if there's package-lock.json file and we do npm install, there are chances that package-lock.json get updated, but if npm-shrinkwrap.json file is present, npm will install all the dependencies listed in the npm-shrinkwrap

https://ivanjov.com/what-is-npm-shrinkwrap-and-why-you-should-start-using-it-right-now/

[deepu105]

I'll take a look and see if my understanding is different.

On Wed, 24 Oct 2018, 1:40 pm Bhawani Singh (Ghost), <
[email protected]> wrote:

@deepu105 https://github.com/deepu105 I don't think so npm shrinkwrap
is just for publishing,

I've used for version lockdown of dependencies just like
package-lock.json, but the main difference is if there's package-lock.json
file and we do npm install, there are chances that package-lock.json get
updated, but if npm-shrinkwrap.json file is present, npm will install all
the dependencies listed in the npm-shrinkwrap

https://ivanjov.com/what-is-npm-shrinkwrap-and-why-you-should-start-using-it-right-now/

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jhipster/generator-jhipster/issues/8642#issuecomment-432621665,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABDlF_brJsIb9BNDXokECx9GJovbYEQiks5uoFGpgaJpZM4X07Ub
.

[pascalgrimaud]

Just testing locally npm install and npm ci, I have some strange behavior, and no improvement at all. With an existing node_modules:

• npm install -> 23s
• npm ci -> 1m5

Maybe the improvements are about consistency and not speed.

So switching to npm ci instead of npm install will require a lot of tests / use cases. We need to think about how to generate the package-lock.json, and don't forget to check Yarn support too.
For now, I suggest to simply document the improvement with prerequisites.
So our users can improve the packaging / continuous integration process, if they want.

[pascalgrimaud]

IMO, best place: https://www.jhipster.tech/tips/

[deepu105]

I would place it in the ci/cd docs may be instead of a tip

On Thu, 25 Oct 2018, 8:55 am Pascal Grimaud, [e-mail]
wrote:

IMO, best place: https://www.jhipster.tech/tips/

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jhipster/generator-jhipster/issues/8642#issuecomment-432936196,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABDlF6gg0vLVmx2OUwQBvvkaKo7DOfz6ks5uoWB-gaJpZM4X07Ub
.

[jdubois]

OK if that's slower with npm ci there's really no point to have this by default.

I did the same analysis as @cbornet did on this comment and I totally agree. For me the most important thing is to generate a correct package-lock.json. We still have to find a good way to do it, but it shouldn't be overly difficult as we don't have that many options (outside of the JHipster modules).

For the two reasons above, I'm closing this, and if you find this useful, indeed a tip is much welcome.

[pascalgrimaud]

After some discussion with @hcapitaine and @sahbi-ktifa today (see https://blog.npmjs.org/post/171556855892/introducing-npm-ci-for-faster-more-reliable), npm ci is better and faster only when there is no node_modules folder but needs a package-lock.json
So yes, it is the best choice for CICD, when package-lock.json is already generated.

So we need to document it.

[wmarques]

Sorry to comment on closed issue but I discovered npm ci recently. For me on developer laptop we need to keep the npm install command as it generates the lockfile when the package.json changes.
The usage of npm ci is more for the CI envs, when you clone the project and want to install your deps to build your project. So I would add it for the CICD sub-gen maybe, and also add it in our travis builds as it seems faster in those cases.

[pascalgrimaud]

@wmarques :

• we could use npm ci, in our CI for generator-jhipster -> I will try it
• about CICD subgen, npm ci will improve the speed, but there is still some issues:
  
  
  
  • generate a project with start.jhipster.tech
  
  
  • generate a continuous integration (Travis, Jenkins, Gitlab or Azure)
  
  
  • launch builds
  
  
  • it will fail because of missing package-lock.json
  
  

I think we simply need to document it (or put in comment) and people who want to use npm ci can replace 1 line in our generated code.

[pascalgrimaud]

I don't understand why npm doesn't do this, for npm ci :
if there is no package-lock.json -> launch npm install
else npm ci

Maybe we could do it in our package.json config ?

[PierreBesson]

@pascalgrimaud It's because npm is trying to upgrade your dependencies to their latest patch version.
Or we could have a maven execution that would run npm install --package-lock-only if the package-lock is missing.

[deepu105]

@pascalgrimaud @PierreBesson then isn't it simply sufficient if we run npm install --package-lock-only when generating projects via jhipster online? then we could sue npm ci from CICD always

[pascalgrimaud]

jhipster-online could use npm install --package-lock-only but, it will use bandwitch and we need to modify the --skip-install flag, or adding a new one.

Locally, it tooks more than 30sec:

12:34:24 in ~/tmp/03-toto on  master 
➜ npm install --package-lock-only
npm WARN deprecated [email protected]: No longer maintained, please upgrade to swagger-ui@3.
npm WARN deprecated [email protected]: 🙌  Thanks for using Babel: we recommend using babel-preset-env now: please read babeljs.io/env to update! 
npm WARN deprecated [email protected]: Package no longer supported. Contact [email protected] for more info.
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.2.2 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN @fortawesome/[email protected] requires a peer of @angular/common@^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @fortawesome/[email protected] requires a peer of @angular/core@^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @ng-bootstrap/[email protected] requires a peer of @angular/common@^6.1.0 but none is installed. You must install peer dependencies yourself.
npm WARN @ng-bootstrap/[email protected] requires a peer of @angular/core@^6.1.0 but none is installed. You must install peer dependencies yourself.
npm WARN @ng-bootstrap/[email protected] requires a peer of @angular/forms@^6.1.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of @angular/core@^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of @angular/router@^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of @angular/core@^5.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of [email protected] - 3 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of popper.js@^1.14.3 but none is installed. You must install peer dependencies yourself.

added 2219 packages from 25 contributors and audited 66113 packages in 37.188s
found 2 low severity vulnerabilities
  run `npm audit fix` to fix them, or `npm audit` for details

[deepu105]

I think 30s is not bad, it might be faster on Google cloud

Thanks & Regards,
Deepu

On Mon, Oct 29, 2018 at 12:36 PM Pascal Grimaud [e-mail]
wrote:

jhipster-online could use npm install --package-lock-only but, it will
use bandwitch and we need to modify the --skip-install flag, or adding a
new one.

Locally, it tooks more than 30sec:

12:34:24 in ~/tmp/03-toto on  master
➜ npm install --package-lock-only
npm WARN deprecated [email protected]: No longer maintained, please upgrade to swagger-ui@3.
npm WARN deprecated [email protected]: 🙌 Thanks for using Babel: we recommend using babel-preset-env now: please read babeljs.io/env to update!
npm WARN deprecated [email protected]: Package no longer supported. Contact [email protected] for more info.
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.2.2 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN @fortawesome/[email protected] requires a peer of @angular/common@^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @fortawesome/[email protected] requires a peer of @angular/core@^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @ng-bootstrap/[email protected] requires a peer of @angular/common@^6.1.0 but none is installed. You must install peer dependencies yourself.
npm WARN @ng-bootstrap/[email protected] requires a peer of @angular/core@^6.1.0 but none is installed. You must install peer dependencies yourself.
npm WARN @ng-bootstrap/[email protected] requires a peer of @angular/forms@^6.1.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of @angular/core@^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of @angular/router@^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of @angular/core@^5.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of [email protected] - 3 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of popper.js@^1.14.3 but none is installed. You must install peer dependencies yourself.

added 2219 packages from 25 contributors and audited 66113 packages in 37.188s
found 2 low severity vulnerabilities
run npm audit fix to fix them, or npm audit for details

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jhipster/generator-jhipster/issues/8642#issuecomment-433878562,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABDlF1gXtn56RpudqTCYGdIAFzHItbxJks5upugggaJpZM4X07Ub
.

[rubensa]

Like Pierre said, the package-lock.json only changes during npm install when your package.json has different versions in it than the lock file. This doesn't happen if no versions change. You should commit your updated package-lock.json file to git.

The only issue with this change is that npm ci requires an existing package-lock.json which will fail for apps generated by JHipster Online or using the --skip-install flag

@ruddell This is no longer true since npm 5.1.0, because now the generated module tree is a combined result of both package.json and package-lock.json. (Example: package.json specifies some package with version ^1.1.0; package-lock.json had locked it with version 1.1.4; but actually, the package is already available with version 1.1.9. In this case npm i resolves the package to 1.1.9 and overwrites the lockfile accordingly, hence ignoring the information in the lock file.)

see: https://github.com/npm/npm/issues/18103

I think that npm ci should be used (imagine the case that your pom.xml file is modified every time you run a mvn compile) to allow predictability and reproducibility. npm-shrinkwrap is another very good alternative to npm ci.

@jdubois That JHipster Generator creates the package-lock.json is not the solution as any execution of npm install (throw frontend-maven-plugin) will override the generated file.

[PierreBesson]

@rubensa We are not saying that generating the package-lock.json will fix npm install but without it we can't use npm ci in the frontend-plugin config because it will fail the first time (and it's important that JHipster apps work out of the box).

[rubensa]

@PierreBesson I don't know if I'm missing something...

As far as a know, JHipster runs npm install when you are building for the first time running, for example, jhipster import-jdl Model.jdl. So package-lock.json is generated.
The problem is that if you, now, run a ./mvnw, frontend-maven-plugin runs again npm install that overrides package-lock.json.
This can be avoided if the generated pom.xml specifies:

                                <configuration>
                                    <arguments>ci</arguments>
                                </configuration>

both in npm install execution for webpack and prod profiles.

PS: Oh!!! Now I think that I understand the problem... once the project exists and is checked out from a repo, jhipster is not executed and the build (for example in a CI building) fails. But this can be solved with @alexander-redmann-teaching profile suggestion. Isn't it?
PS2: Ummm... Thinking again... This problem does not exists as package-lock.json exists in git repo when checked out by others (including CI server), so... I'm still missing something here.

[pascalgrimaud]

@rubensa : did you try jhipster-online at https://start.jhipster.tech ? You can generate a JHipster project without installing it locally. But with this workflow, there is no package-lock.json and that's the problem.