Generator-jhipster: can I please get some help reviewing/improving the Jenkins X build pack for JHipster?

[jstrachan]

Jenkins X build pack

I've take a stab at putting together a build pack for JHipster but I wanted some guidance. I took the generated Jenkinsfile from jhipster as a base

We're using a docker container in Jenkins X to host all the build tools (maven, npm, yarn, git etc) so that when we do CI/CD pipelines in Jenkins X we don't have to wait for a while for npm/yarn to install etc.

Here's the current docker image for the build container: https://github.com/jenkins-x/builder-maven-node-yarn/blob/master/Dockerfile#L3-L14

I'm a bit of a jhipster noob but most samples I see use the mvn plugin to download + install npm/yarn - I'm trying to switch the pipeline steps over to directly use npm/yarn when that makes sense - then use mvn for the mvn-native stuff - I'm just not sure what those steps should be ;)

Here's the current pipeline steps for the PR pipeline:
https://github.com/jstrachan/hipster3/blob/master/Jenkinsfile#L25-L36

then for the release pipeline:
https://github.com/jstrachan/hipster3/blob/master/Jenkinsfile#L77-L90

I wonder does anyone know the canonical CLIs using yarn/npm instead of the mvn plugin for the angular/jhipster parts?

Motivation

The idea is to try automate CI/CD on kubernetes for JHipster apps with Preview Environments of changes for each pull request and longer term provide an awesome developer UX for creating/editing code before you are ready to submit a PR and trigger CI/CD.

Review Required

I'm mostly just looking for validation the current build image is OK; does it need anything else?

Then which commands should we run inside the build image when doing a Pull Request pipeline (test + verify then build the docker image - then the same again for when releasing (where we wanna release all the artifacts with a release version etc).

JHipster Version(s)

4.14.4

JHipster configuration

[email protected] /workspace/jenkins-x-rocks/hipster3
└── [email protected]

##### **JHipster configuration, a `.yo-rc.json` file generated in the root folder**

.yo-rc.json file

{
  "generator-jhipster": {
    "promptValues": {
      "packageName": "com.mycompany.myapp"
    },
    "jhipsterVersion": "4.14.4",
    "baseName": "hipster3",
    "packageName": "com.mycompany.myapp",
    "packageFolder": "com/mycompany/myapp",
    "serverPort": "8080",
    "authenticationType": "jwt",
    "cacheProvider": "no",
    "websocket": false,
    "databaseType": "sql",
    "devDatabaseType": "h2Disk",
    "prodDatabaseType": "postgresql",
    "searchEngine": false,
    "messageBroker": false,
    "serviceDiscoveryType": false,
    "buildTool": "maven",
    "enableSocialSignIn": false,
    "enableSwaggerCodegen": false,
    "jwtSecretKey": "replaced-by-jhipster-info",
    "clientFramework": "angularX",
    "useSass": false,
    "clientPackageManager": "yarn",
    "applicationType": "monolith",
    "testFrameworks": [],
    "jhiPrefix": "jhi",
    "enableTranslation": false,
    "enableHibernateCache": false,
    "appsFolders": [
      "hipster3"
    ],
    "directoryPath": "../",
    "dockerRepositoryName": "",
    "dockerPushCommand": "docker push",
    "kubernetesNamespace": "default",
    "kubernetesServiceType": "Ingress",
    "ingressDomain": "default.192.168.99.100.nip.io"
  }
}

JDL for the Entity configuration(s) entityName.json files generated in the .jhipster directory

JDL entity definitions

Environment and Tools

java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)

git version 2.15.1 (Apple Git-101)

node: v8.11.2

npm: 5.6.0

yarn: 1.6.0

Docker version 17.09.1-ce, build 19e2cf6

docker-compose version 1.17.1, build 6d101fb

Browsers and Operating System

This is running inside kubernetes (GKE with k8s 1.9.x)

• [X] Checking this box is mandatory (this is just to show you read everything)

[PierreBesson]

I'm not sure if this is such a good idea to use the global node/npm/yarn because the frontend is quite sensible on those versions and so it's better if they be specified in the pom for reproducibility.

[jstrachan]

I get the desire to let the app/git source be in charge of specifying all the versions of tools & dependencies. Though the flip side is running mvn to run java code to install npm & yarn on every build in CI/CD & devpod slow service things down hugely.

There can be many images used to do the builds & we could easily lazily create them on demand given an arbitrary set of versions of mvn, npm, yarn, jhipster etc. ie we could specify those versions somewhere in source code (eg package.json?) & the pipeline could lazily make a docker image lazily if it’s required - but most of the time those versions won’t change so we can have a cached fast build startup time?

[sendilkumarn]

I like this l(cr)azy idea and of course it will increase the devpod start up time.

Specifying them on the package.json will not be a good idea. Is it possible to use it straight from the gradle.properties and use that as an indicator.

@PierreBesson Most of the times this should not have any problems, But yeah I do agree that we had some bizzare issues related to this node / npm versions.

@jstrachan Do you think is it possible to configure the pipeline such that it takes the longer route (the route with npm install via maven / gradle) when needed ?

[jstrachan]

@sendilkumarn happy to store the properties in any file you think is best - so gradle.properties orpom.xml or whatever sounds great to me. BTW whats the split between gradle v maven in the JHipster community; is there a strong preference? Happy to use either really. We could support a build pack for both so we leave it to the developer to pick.

We could easily have a flag that lets the pipeline work in slow or fast mode; so folks can keep it real and use mvn / gradle for installing everything; or use the fast lazy pre-built docker images etc.

On the versions of node/npm/yarn stuff; I wonder if we can figure out a way to lock known combinations down; e.g. for a given jhipster version there's known node/npm/yarn versions that get inherited or something? I guess if JHipster can define the versions it thinks are good; we could try keep the Jenkins X build packs in sync with those versions?

[deepu105]

@jstrachan I agree that it could speed things up on multiple build cases. While I understand @PierreBesson's concern about version issues, I personally don't think its a huge issue. I always prefer using npm/yarn directly in my day 2 day work, while many of my colleagues use the gradle plugin. So far I haven't seen any major issues. Since jhipster is not a nodejs application the reproducibility of the front end deps is not a big deal especially since we use fixed versions.

So IMO using the tools directly on CI shouldn't be a big issue as the original idea of using the frontend maven/gradle plugin was to rely on one build tool and for complete independent builds from scratch in any environment.

Here is a way to do that for Gradle for example

yarn && yarn webpack:prod && ./gradlew -Pprod -x webpack -x webpack_test

[gmarziou]

BTW whats the split between gradle v maven in the JHipster community; is there a strong preference?

Last month, it was 88% maven, 12% gradle if I interpreted correctly google analytics report.

[sendilkumarn]

for a given jhipster version there's known node/npm/yarn versions that get inherited or something? I guess if JHipster can define the versions it thinks are good; we could try keep the Jenkins X build packs in sync with those versions?

We already have this in gradle.properties and pom.xml

@gmarziou we should make gradle default 😜

[PierreBesson]

Just some additional info on using the native yarn/npm vs the one provided by the frontend-plugin.
From https://github.com/eirslett/frontend-maven-plugin#what-is-this-plugin-not-meant-to-do

Notice: This plugin does not support already installed Node or npm versions. Use the exec-maven-plugin instead.

So it seems that at least for our maven build using the system node/npm/yarn binary is not supported.

[jdubois]

I spoke yesterday with @cyrille-leclerc and I'd really like this ticket to move forward: @jstrachan I understand that you are based in London, would you be available to make a trip to Paris to one of our monthly JHipster hackathon? There will be @PierreBesson and @pascalgrimaud who are our 2 best experts on this, and I'm pretty sure that would be a very productive day.

[mraible]

I spent the last couple days trying to get Jenkins X to work with a project that uses the frontend-maven-plugin. I started by importing my project, but since it doesn't have a package.json or pom.xml in the root directory, it didn't work. I've since figured out how to fix, and the sample Jenkins file you posted @jstrachan will likely be a good help.

Currently, I'm stuck on node-sass installing because of a permission denied error. If I could get past this, I could probably get the JHipster pipeline working.

https://groups.google.com/forum/?nomobile=true#!topic/jenkinsci-users/RydF3PjbpMI

The thing I'll need after I get past this is how to run Protractor tests as part of the build. Here's what I'm currently hoping will work (it does in Travis):

// artifact has been built, run Spring Boot app
sh "cd holdings-api && java -jar target/*.jar &"
// then run Protractor tests
sh "DISPLAY=:99 cd crypto-pwa && npm run e2e" 

[jdubois]

@mraible concerning node-sass the very good news is that it's going away in #8140 (what he pain this thing was!!!). So no more issue very soon.

As this has been opened for 1.5 months with no more work, I'm closing this -> @jstrachan if you need help just ping me. One good solution would be that you come to one of our "JHipster days" at Ippon. You will be with a big part of the core team, including @pascalgrimaud @PierreBesson (who should be the best to help you). We're in Paris, and from my understanding you are in London, so that's pretty easy to do that trip.

[mraible]

FWIW, I was able to get Spring Boot + Angular working and wrote a blog post about it.

https://developer.okta.com/blog/2018/07/11/ci-cd-spring-boot-jenkins-x-kubernetes

[yelhouti]

@jstrachan I would like to help work on this, are you still interested?