Generator-jhipster: Automatic Webpack build from Gradle/Maven wastes time

[kaidohallik]

Overview of the issue

In #8033 was agreed conditional automatic front end compilation from Maven/Gradle in dev profile and this was implemented in #8038
For Gradle this automatic front end compilation was made more frequent in #8451

I feel that these changes degrade the developer experience at least for those developers who are working daily with the project generated by JHipster, because there are cases where parallel front end build is executed if developer runs (as suggested in https://github.com/jhipster/generator-jhipster/blob/master/generators/common/templates/README.md.ejs#L69-L74):

• in one terminal npm start
• in second terminal ./gradlew or ./mvnw

Starting from #8038 these cases are at least:

• after cloning fresh version from Git
• after deleting build/targetfolder
• after doing prod build

For Gradle, #8451 adds the following case:

• any front end file is modified between 2 executions of ./gradlew

Parallel builds have 2 negative aspects:

• the question arises: does front end build from Gradle/Maven breaks version built by npm start, to be sure no side effects I end up with killing npm start and starting it again after back end is up and running
• back end starts too slowly, because it's doing unnecessary front end build

Motivation for or Use Case

I don't want to loose time for waiting while back end compiles front end if front end is already handled by npm start.

Reproduce the error

Delete build/target folder and then run in parallel terminals:

• npm start
• ./gradlew or ./mvnw

Related issues

Suggest a Fix

Revert #8451 (this is already reverted for prod profile in #8472)

Consider reverting #8038 and instead maybe try the following:

• is it possible to solve this as @jdubois suggested: https://github.com/jhipster/generator-jhipster/issues/8033#issuecomment-408411870
• improve https://github.com/jhipster/generator-jhipster/blob/master/generators/common/templates/README.md.ejs - try clearly explain that if this is not freshly generated project and user wants test application in one terminal in dev profile without need to change the app then Gradle/Maven process must be started with -Pwebpack

JHipster Version(s)

[email protected] C:\projects\teest\jhipster-diff
`-- [email protected]  -> C:\projects\kaido-github\generator-jhipster

##### **JHipster configuration, a `.yo-rc.json` file generated in the root folder**

.yo-rc.json file

{
  "generator-jhipster": {
    "promptValues": {
      "packageName": "com.mycompany.myapp",
      "nativeLanguage": "et"
    },
    "jhipsterVersion": "6.2.0",
    "applicationType": "monolith",
    "baseName": "jhipster",
    "packageName": "com.mycompany.myapp",
    "packageFolder": "com/mycompany/myapp",
    "serverPort": "8080",
    "authenticationType": "jwt",
    "cacheProvider": "ehcache",
    "enableHibernateCache": true,
    "websocket": false,
    "databaseType": "sql",
    "devDatabaseType": "h2Disk",
    "prodDatabaseType": "mysql",
    "searchEngine": false,
    "messageBroker": false,
    "serviceDiscoveryType": false,
    "buildTool": "gradle",
    "enableSwaggerCodegen": false,
    "jwtSecretKey": "bXktc2VjcmV0LXRva2VuLXRvLWNoYW5nZS1pbi1wcm9kdWN0aW9uLWFuZC10by1rZWVwLWluLWEtc2VjdXJlLXBsYWNl",
    "embeddableLaunchScript": false,
    "useSass": true,
    "clientPackageManager": "npm",
    "clientFramework": "angularX",
    "clientTheme": "none",
    "clientThemeVariant": "",
    "testFrameworks": [],
    "jhiPrefix": "jhi",
    "entitySuffix": "",
    "dtoSuffix": "DTO",
    "otherModules": [],
    "enableTranslation": true,
    "nativeLanguage": "et",
    "languages": [
      "et",
      "en",
      "ru"
    ],
    "blueprints": []
  }
}

JDL for the Entity configuration(s) entityName.json files generated in the .jhipster directory

JDL entity definitions

Environment and Tools

java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

git version 2.21.0.windows.1

node: v10.15.3

npm: 6.9.0

yeoman: 2.0.6

yarn: 1.15.2

Docker version 19.03.2, build 6a30dfc

docker-compose version 1.24.1, build 4667896b

Browsers and Operating System

Windows 10

• [x] Checking this box is mandatory (this is just to show you read everything)

[atomfrede]

I would rather not remove the up2date check as e.g. running that when a frontend file has changed is imho totally correct and when doing e.g. only bootRun the frontend should be build conditionally.

I agree when doing frontend via npm thats not great to have, but maybe we can add a dedicated task which excludes the frontend build (or you can do bootRun -x webpackBuild, but tested).

[kaidohallik]

I think that developing front end in the following manner is ineffective:

• change front end file
• rerun ./gradlew

As readme states, the normal development flow is to run npm start and ./gradlew in the parallel terminals.

I personally can handle this, I'm already doing these 2 actions instinctively together:

1. generate app
2. delete webpackBuildDev related lines from the profile_dev.gradle

But I don't know does this parallel build bothers also others or not.

[atomfrede]

I think that developing front end in the following manner is ineffective:

* change front end file

* rerun ./gradlew

Totally agree, that is not the idea.

As readme states, the normal development flow is to run npm start and ./gradlew in the parallel terminals.

I will try around with excludes flag and update the documentatio (or change the default task). What do you think?

[kaidohallik]

If you change default task to exclude webpackBuildDev then you are basically reverting #8038 for Gradle. And even more, not providing back previously existed -Pwebpack option. So I think this is not a good idea.

I liked solution before #8038 and #8451 where ./gradlew and ./mvnw didn't run Webpack and the following commands run: ./gradlew -Pwebpack and ./mvnw -Pwebpack.

The idea of the #8038 was to decrease JHipster team maintenance burden, stop receiving complains about blank screen because people didn't read readme and didn't run Gradle/Maven with -Pwebpack if this was needed.

Blank screen complains are gone now, but the price is that there are side effects described in the initial comment.

My initial suggestion was to revert these changes and try to improve README.md, maybe before the block with npm start and ./gradlew or ./mvnw try to explain when it's needed to run ./gradlew -Pwebpack or ./mvnw -Pwebpack instead.

One possible alternative solution:

• introduce parameter no-webpack for Gradle and Maven
• mention in the README.md the occasions where the parallel Webpack build is happening and mention that this can be avoided using -Pno-webpack

Or if you think that this problem is not worth attention then we can just close this. I personally know what to do (I'm just deleting webpackBuildDev logic as I don't need it).

[mraible]

I like the current solution. It works better in demos because you only have to run one command. However, it is a pain that the Java build process seems to run npm install again.

It's also funny to me that it takes longer to compile the TypeScript code than Java code. 😄

On Sep 6, 2019, at 13:28, Kaido Hallik [e-mail] wrote:

If you change default task to exclude webpackBuildDev then you are basically reverting #8038 for Gradle. And even more, not providing back previously existed -Pwebpack option. So I think this is not a good idea.

I liked solution before #8038 and #8451 where ./gradlew and ./mvnw didn't run Webpack and the following commands run: ./gradlew -Pwebpack and ./mvnw -Pwebpack.

The idea of the #8038 was to decrease JHipster team maintenance burden, stop receiving complains about blank screen because people didn't read readme and didn't run Gradle/Maven with -Pwebpack if this was needed.

Blank screen complains are gone now, but the price is that there are side effects described in the initial comment.

My initial suggestion was to revert these changes and try to improve README.md, maybe before the block with npm start and ./gradlew or ./mvnw try to explain when it's needed to run ./gradlew -Pwebpack or ./mvnw -Pwebpack instead.

One possible alternative solution:

introduce parameter no-webpack for Gradle and Maven
mention in the README.md the occasions where the parallel Webpack build is happening and mention that this can be avoided using -Pno-webpack
Or if you think that this problem is not worth attention then we can just close this. I personally know what to do (I'm just deleting webpackBuildDev logic as I don't need it).

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[ruddell]

For Maven, you can exclude the webpack profile by running ./mvnw -P-webpack which prevents it from running. Not sure how to do that for Gradle.

[atomfrede]

I need to test it, but you can do the same via ./gradlew -x webpackBuildDev I think. So I would say we should clarify our documentation.

@mraible Are you talking about maven? Gradle should detect npm install as up-to-date and not execute it again (but will verify that)

[duderoot]

@ruddell that's a nice hint. I was also having pipeline improvements in my ToDo's. Now with the hint from @ruddell we can skip UI tasks in the BE test for example. I also ask my self why is still need for installation of the ui tools here since we are using a JHipster image which already has the tools installed e.g. npm, node, etc.
I created a PR #10367 with those changes.

[atomfrede]

I would suggest we adapt the documentation. When doing the following in 3 terminals it works "optimal" I would say:

• ./gradlew serves the server side application
• ./gradlew compileJava -x webpackBuildDev -t recompiles the java code on change without doing the npm stuff
• npm run start compiles and reloads the frontend

The complete cycle (changing a java file until the application is restartet via boot dev tools) takes ~2 - 3 seconds.

I would change the documentation like this:

EDIT (before it was ./gradlew which is nonsense)

Run the following commands in three separate terminals to create a blissful development experience where your browser
auto-refreshes when files change on your hard drive.

    ./gradlew bootRun -x webpackBuildDev
    ./gradlew compileJava -x webpackBuildDev -t
    npm start

[kaidohallik]

If there is a command ./gradlew included in the main workflow then nothing changes.
When running ./gradlew then again and again unnecessary front end build is done by Gradle if some front end source file is changed between 2 executions of ./gradlew, which wastes time and is very annoying.
Additionally task npmInstall is currently executed always, even if no change in FE files. This is not wasting time a lot if no new packages, but even couple of seconds doing unnecessary work is not good.

./gradlew compileJava -x webpackBuildDev -t is a great hint, this should be in the documentation.

Currently one additional drawback I didn't mention in the initial comment is that if using Gradle then initially front end is always built 2 times:

• if generating app with JHipster then front end is installed and built
• if running ./gradlew then this is done again, because in generation process front end is installed and built without Gradle and Gradle has not yet information about input and output of webpackBuildDev

If testing this 3 terminal workflow then in my Windows 10 environment Spring DevTools Restart is not working. I had to start app using main class in IntelliJ Idea to make it work as described here: https://dev.to/suin/spring-boot-developer-tools-how-to-enable-automatic-restart-in-intellij-idea-1c6i. Is this working in your machine in 3 different terminals without using IDE?

If Spring DevTools Restart would work in terminal then blissful with the current codebase for me would be:

    ./gradlew -x webpackBuildDev
    ./gradlew compileJava -x webpackBuildDev -t
    npm start

But maybe mention also, that ./gradlew is for example meant for testing/demo JHipster apps cloned from Git without need to change FE part and for 1 terminal development, mention that this checks for FE changes and if found then recompiles.

One question with these commands is: why default workflow commands should be so awkward, why these should include -x webpackBuildDev. Maybe better is to use for default workflow simple ./gradlew and for one time demo/testing and 1 terminal development ./gradlew -Pwebpack (change Gradle config for that), which runs webpackBuildDev conditionally using Gradle's great up-to-date check mechanism.

[atomfrede]

If there is a command ./gradlew included in the main workflow then nothing changes.
When running ./gradlew then again and again unnecessary front end build is done by Gradle if some front end source file is changed between 2 executions of ./gradlew, which wastes time and is very annoying.

I agree not good, but doing that helps not so experienced developers with our tooling. Advanced use cases can be done differently of course. What I can imagine is a special task runServer which excluded the webpack part maybe.

That is true (also noticed that), but to be honest I didn't see any problem with that, but we can try to do the initial build with gradle or maven which will solve the issue.

Additionally task npmInstall is currently executed always, even if no change in FE files. This is not wasting time a lot if no new packages, but even couple of seconds doing unnecessary work is not good.

Agree, but I would consider this very minor.

./gradlew compileJava -x webpackBuildDev -t is a great hint, this should be in the documentation.

Currently one additional drawback I didn't mention in the initial comment is that if using Gradle then initially front end is always built 2 times:

* if generating app with JHipster then front end is installed and built

* if running `./gradlew` then this is done again, because in generation process front end is installed and built without Gradle and Gradle has not yet information about input and output of `webpackBuildDev`

If testing this 3 terminal workflow then in my Windows 10 environment Spring DevTools Restart is not working. I had to start app using main class in IntelliJ Idea to make it work as described here: https://dev.to/suin/spring-boot-developer-tools-how-to-enable-automatic-restart-in-intellij-idea-1c6i. Is this working in your machine in 3 different terminals without using IDE?

I only used 3 terminals without ide(a) and it worked well. Maybe you can try ./gradlew bootRun -x webpackBuildDev?

If Spring DevTools Restart would work in terminal then blissful with the current codebase for me would be:

    ./gradlew -x webpackBuildDev
    ./gradlew compileJava -x webpackBuildDev -t
    npm start

But maybe mention also, that ./gradlew is for example meant for testing/demo JHipster apps cloned from Git without need to change FE part and for 1 terminal development, mention that this checks for FE changes and if found then recompiles.

One question with these commands is: why default workflow commands should be so awkward, why these should include -x webpackBuildDev. Maybe better is to use for default workflow simple ./gradlew and for one time demo/testing and 1 terminal development ./gradlew -Pwebpack (change Gradle config for that), which runs webpackBuildDev conditionally using Gradle's great up-to-date check mechanism.

The default command includes the webpack build because otherwise there would no single command to run the application, so this is fine for me. I don't adding more -P flags (again) as they are not discovereable (via ./gradlew tasks or composable). I even would like to remove the -Pdev and -Pprod and replace them with special tasks, but thats a different topic.

[kaidohallik]

I agree not good, but doing that helps not so experienced developers with our tooling. Advanced use cases can be done differently of course. What I can imagine is a special task runServer which excluded the webpack part maybe.

I agree that it's important to keep JHipster as simple as possible.

But I disagree that avoiding unnecessary webpack builds is advanced workflow. I think that this should be at least mentioned in the docs how to avoid unnecessary webpack builds, so that people who read docs get information how they can develop in the quickest way (and can customize config if they don't want webpack builds from Gradle and want to keep commands to run as short as possible).

I only used 3 terminals without ide(a) and it worked well. Maybe you can try ./gradlew bootRun -x webpackBuildDev?

This didn't work either. But if I deleted https://github.com/jhipster/generator-jhipster/blob/master/generators/server/templates/build.gradle.ejs#L107-L123 then Spring DevTools Restart started to work also in terminal in Windows.

I don't adding more -P flags (again) as they are not discovereable (via ./gradlew tasks or composable). I even would like to remove the -Pdev and -Pprod and replace them with special tasks, but thats a different topic.

Good point.

[atomfrede]

Are you on windows?

[kaidohallik]

Yes.

[atomfrede]

Ah I see :( Now I understand why you are so annoyed by the frontend build. It takes (sadly) much more time under windows from my experience

[pascalgrimaud]

let's improve our CICD sub gen, and our documentation in readme and at jhipster.tech

[atomfrede]

:+1: Will the documentation part and have a look at the ci/cd part when preparing my upcoming talks.

[kaidohallik]

I would suggest also to rename Gradle task webpackBuildDev to webpack then this is aligned with prod profile and with Maven and it's easier to remember and run ./gradlew -x webpack than ./gradlew -x webpackBuildDev.

The reason why this task name is webpackBuildDev is: if the switch -Pwebpack was used then it was mandatory to use different task and switch names because Gradle's project.hasProperty("webpack") evaluated always to true if the task with the name webpack existed.

[pascalgrimaud]

@atomfrede @kaidohallik : what is the state of this ticket ? Are we all ok with the different PRs ?
In this case, can this ticket be closed ?

[atomfrede]

I think we can close this imho. We have updated the documentation for the advanced workflow with gradle, will have better generated pipelines soon and I don't think we should remove the automatic webpack execution when building the jar archive. @kaidohallik Are you ok with that?

[duderoot]

@pascalgrimaud @kaidohallik @atomfrede from what I saw PR #10398 was only fixing the build for gradle. Are contributions related to the speed of pipeline for maven in other PR?

[kaidohallik]

@duderoot Maven part was optimized by #10392 and #10398 was follow-up for Gradle.

@atomfrede I think documentation is not updated. As Gradle continuous build results to infinite rebuild in some JHipster configurations then I think ATM we can't suggest this. So maybe ATM just change

    ./gradlew<% } %>

to

    ./gradlew -x webpack<% } %>

in https://github.com/jhipster/generator-jhipster/blob/master/generators/common/templates/README.md.ejs#L94

[atomfrede]

Ah yes you are right, I just updated the documentation on the website. Will update the readme accordingly. Can you pinpoint which configurations result in infinite rebuilds?

[kaidohallik]

@atomfrede One configuration (JDL file provided) is described in this comment: https://github.com/jhipster/generator-jhipster/pull/10398#issuecomment-531774765

And in some other ticket was declared that using Open API option also results in infinite rebuild.

[kaidohallik]

I created PR for GitHub Pages https://github.com/jhipster/jhipster.github.io/pull/825

[kaidohallik]

I created PR for readme: https://github.com/jhipster/generator-jhipster/pull/10618