Generator-jhipster: Create jhipster-lib snapshots and use them at CI.

[mshima]

Overview of the feature request

Our current test approach is to clone jhipster-lib, build and use it.
We should consider to create snapshots for each push and deploy to github packages.
Those snapshots could be used at generator-jhipster tests.

Motivation for or Use Case

Benefits:

• Reduce CI builds by 30~40 seconds each.
• Allows to simplify big dependencies update like spring boot major version bump. It could be done gradually.
  Eg: create snapshots for jhipster-lib -SB23 and add an option to use it.
• Sonatype snapshots are daily, so not suitable for our goal.

Limitations:

• Github packages requires authentication to read the packages
  
  
  
  • Not sure it's worth to migrate azure builds.
  
  
  • We should keep using daily snapshots at non github actions generated projects.
    
    
  
  

Related issues or PR

• [ ] Checking this box is mandatory (this is just to show you read everything)

[pascalgrimaud]

Our current test approach is to clone jhipster-lib, build and use it.

I like this approach, because we can reproduce the issue, with the exact commit of jhipster lib and generator-jhipster.

By using snapshot version, we won't be able to reproduce the exact build, if there is a merge in jhipster lib : the previous snapshot will be erase.

The main problem would be the different approach with Azure as the github token is not available there.

So for now, I'd prefer to keep our current system.
But let's discuss ! :-)

[mshima]

Our current test approach is to clone jhipster-lib, build and use it.

I like this approach, because we can reproduce the issue, with the exact commit of jhipster lib and generator-jhipster.

Confused about this.
Even if a snapshot was deleted it would be reproducible.

By using snapshot version, we won't be able to reproduce the exact build, if there is a merge in jhipster lib : the previous snapshot will be erase.

A public package version is never deleted from github packages.
https://docs.github.com/en/packages/publishing-and-managing-packages/deleting-a-package
I don’t know about public package snapshots.

The main problem would be the different approach with Azure as the github token is not available there.

We could create a token with read package permission only and use it.
Anyway not allowing unauthenticated read is my main concern.

[pascalgrimaud]

Some packages have been published:
https://github.com/jhipster/jhipster/packages

Let's see if it works well.

[DanielFran]

We need to update the generator to use this repository

[pascalgrimaud]

I'd like to do some test locally before merging this and find a way to know which version I download from github packages, with the exact commit

[DanielFran]

Agree.
@mshima I just restarted the tests in the PR #12252 to see if it uses the current snapshot

[mshima]

@DanielFran last commit broke it.
Since there were no way to test it 😆

[mshima]

@pascalgrimaud printing the snapshot version isn’t enough?
In this case 3.10.0-20200818.092844-1

Uploaded to github: https://maven.pkg.github.com/jhipster/jhipster/io/github/jhipster/jhipster-parent/3.10.0-SNAPSHOT/jhipster-parent-3.10.0-20200818.092844-1.pom (19 kB at 18 kB/s)

It could be traced back to the commit.

[mshima]

We could try to add the commit hash to the version.

[pascalgrimaud]

Currently, when I want to reproduce a failure in the CI:

• jhipster lib
  
  
  
  • get the exact commit in logs
  
  
  • update my local project with upstream
  
  
  • git checkout "commit reference"
  
  
  • mvn clean install -Dgpg.skip
  
  
• generator-jhipster
  
  
  
  • do the same
  
  
  • get the exact commit in logs
  
  
  • git checkout "commit reference"
  
  
  • npm ci / npm link
  
  
• use sample from test-integration folders, and regenerate the same project with same entities
• launch backend tests
• it will use my local SNAPSHOT of jhipster lib

Now, with these changes, if I understand well, to reproduce the exact CI:

• jhipster lib
  
  
  
  • no need to use my local project, as it will download snapshot version
  
  
  • but I need to add a custom settings.xml or edit the pom.xml / build.gradle to include specific part (to be the same than the CI)
  
  
• generator-jhipster
  
  
  
  • get the exact commit in logs
  
  
  • git checkout "commit reference"
  
  
  • npm ci / npm link
  
  
• use sample from test-integration folders, and regenerate the same project with same entities
• launch backend tests
• it will download a SNAPSHOT version of jhipster lib -> what can I do to get the exact version, used by the CI during the failed build ? Supposing, I try to reproduce few days later, there can be some commits in jhipster lib project, so new SNAPSHOT version ?

Again, I just want to discuss more about it, I'm not against your proposal, @mshima
Thanks a lot for all your work for the project :-)

[mshima]

• it will download a SNAPSHOT version of jhipster lib -> what can I do to get the exact version, used by the CI during the failed build ? Supposing, I try to reproduce few days later, there can be some commits in jhipster lib project, so new SNAPSHOT version ?

@pascalgrimaud maybe changing the pom property from

<jhipster-dependencies.version>3.10.0-SNAPSHOT</jhipster-dependencies.version>

to

<jhipster-dependencies.version>3.10.0-xxxx-x</jhipster-dependencies.version>

This will force jhipster-framework artefact, not sure if this could bring a different version of jhipster-dependencies artefact.

To get snapshots from github needs to add to ~/.m2/settings.xml

<profiles>
  <profile>
      <id>github</id>
        <activation>
          <activeByDefault>true</activeByDefault>
      </activation>
      <repositories>
        <repository>
          <id>github</id>
          <name>GitHub OWNER Apache Maven Packages</name>
          <url>https://maven.pkg.github.com/jhipster/jhipster</url>
        </repository>
      </repositories>
    </profile>
</profiles>

<servers>
  <server>
    <id>github</id>
    <username>user</username>
    <password>token with read package permission</password>
  </server>
</servers>

If think if we print the git commit that the snapshot was generated with, then you could keep our old method.

[pascalgrimaud]

Another idea:
I think we can still clone jhipster lib and display git log
There is a big chance the snapshot would be similar to the git log

[mshima]

Another idea:
I think we can still clone jhipster lib and display git log
There is a big chance the snapshot would be similar to the git log

Should work for the majority of the cases.
Will not work if the build jhipster-lib build failed so the artefact was not generate or used a different version.

[pascalgrimaud]

still not totally convinced but we can give a try and see if it works well
are you confident enough @mshima ?

[mshima]

still not totally convinced but we can give a try and see if it works well
are you confident enough @mshima ?

@pascalgrimaud I think it’s good enough, but we don’t need to merge this until there is a bigger motive like: https://github.com/jhipster/generator-jhipster/pull/11742#issuecomment-678758820

[mshima]

@pascalgrimaud @DanielFran, I prefer using snapshots instead of building each time.
But we should close this if you prefer to keep current workflow.