Generator-jhipster: Generation of new liquibase changelogs when updating entity

[ntorionbearstudio]

Overview of the feature request

Generate new liquibase changelogs when updating one entity and don't rewrite previous changelogs linked to this entity.

One first possible implementation would be to add these options to jhipster entity Entity command tool for the question "Do you want to update the entity ?" :

Yes, add more fields and relationships and write new changelogs for database updates

Yes, remove fields and relationships and write new changelogs for database updates

And then, new changelogs files for liquibase will be created : updated_entity_Entity.xml and updated_entity_constraints_Entity.xml

We can also extend this feature to import_jdl tool, when a new JDL schema is imported, we could make a diff between current fields/relationships and new fields/relationships from JDL schema for each entity and generate corresponding liquibase changelogs files.

Motivation for or Use Case

When you have an existing jHipster project in production and you want to update one current entity, you have to manually create a new liquibase file for updating your database.
The idea with this feature is to generate a new liquibase file for new or removed fields/relationships when you update one entity and save time.

Related issues or PR

I didn't find a similar feature request. I'm available to work on this feature.

• [x] Checking this box is mandatory (this is just to show you read everything)

[gmarziou]

Previous work that was never finished: #6016

[pascalgrimaud]

how do you manage to remember this, @gmarziou ?!
nice link to this ticket !

[gmarziou]

:) Because I remember answering several times about this highly requested feature on SO.

[ntorionbearstudio]

Hi, I tried a small implementation of new changelogs generation when updating entity with jhipster entity EntityName command line tool. I didn't write test for the moment and I don't know if my implementation will work well with all cases but I tested successfully for the moment :

• Add of new fields
• Add of new relationships
• Drop of fields
• Drop of relationships

After selection of 'Add' or 'Remove' in options for update type, the user is asked if he wants to generate a new changelog for updating database.
And after, new changelogs files for liquibase are created : updated_entity_Entity.xml and updated_entity_constraints_Entity.xml

Here a demo video :
https://drive.google.com/file/d/1AfQGmgbZJ5sE9oRNvYnpdv0uB7gXVQq2/view?usp=sharing

Tell me if I can open a pull request for this first part in order to have feedbacks on it, thanks.

[gmarziou]

Thanks @ntorionbearstudio for this excellent work, the idea of the video is excellent too.

• Does it work also when updating the entity through import-jdl?
• I would rather always create a new separate changelog because this is how Liquibase is supposed to be used with immutable changelogs. I often answered to JHipster users on stackoveflow about liquibase checksum errors due to modified changelogs. It also helps when you must migrate data.

[mshima]

Tell me if I can open a pull request for this first part in order to have feedbacks on it, thanks.

@ntorionbearstudio you definitively should open the pull request, preferably as Draft Pull Request.

[ntorionbearstudio]

• Does it work also when updating the entity through import-jdl?

@gmarziou Not for the moment, I only implemented this in jhipster entity CLI part for now

[pascalgrimaud]

@ntorionbearstudio : I saw your video and it's impressive ! good job :)

Now, I'd like to know what you have in mind for JDL ? Is it possible or not ? Maybe use a flag ?

[pascalgrimaud]

I changed the label as it's not a simple enhancement but a very good feature.
I added a bounty too, as if we manage to achieve this, it will help a lot of users

[mshima]

@MathieuAA I am wondering on how to add this feature to jdl?
Looks like https://github.com/jhipster/generator-jhipster/pull/6016 was doing diff between the imported code and the new code.

[MathieuAA]

I like this approach

• returning a diff when importing JDL content
• this diff could be read in the generator and the file handling it would do its magic to do the necessary

This implies knowing the format of this diff, maybe something like:

• new: { entity, field, etc }
• updated
• deleted

[ntorionbearstudio]

Yes I think that a diff between previous and new imported JDL schema could work (I don't well import JDL part so maybe I'm wrong).

I was thinking to add a new option to import-jdl command line like this :

jhipster import-jdl jhipster-jdl.jh --new-changelogs

With this option import-jdl know that he has to do a diff between previous and new JDL schema, and generate some lists to send information to jhipster generator :

• newEntities
• updatedEntities

inside an updated entity :

• newFields
• newRelationships
• removedFields
• removedRelationships

Work on #6016 will help us to do diff between JDL schemas.

I will discover in the next days how import JDL works in the source code to have a better vision on it

[MathieuAA]

And how would the generator know there's a diff to be read afterwards?

[gmarziou]

Why passing a --new-changelogs option?

Let's embrace changelog immutability and always create a new changelog for each change.

This is a core concept of Liquibase and the reason why changelogs are checksumed.

[mshima]

I just pushed an updated version of this feature:

• It creates a new generator called liquibase. why?
  Workflow current implementation app -> entity -> entity-server doesn't work for this feature.
  Had to change to app -> liquibase (each changelog) -> entity -> (entity-server & entity-liquibase), when no changelog is found it fallback to app -> entity -> entity-server.
• Prompts: jhipster liquibase show prompts to add entities and change entities.
  One changelog is bound to one entity. It can create multiples changelogs on one run.
• Merging: jhipster liquibase --merge allows to merge multiples changelogs from one entity.
  TODO: Merge to the root.
• Drop: jhispter liquibase --drop allows to drop a changelog.
• Custom changelog: jhispter liquibase --custom-changelog foo creates a file on master.xml for custom changelogs.
  TODO: Change to --snapshot and create a new start point for an entity, so it could do some advanced stuff, like change table name, change field type, fields and relationships should not added or removed here.
• JDL: jhispter liquibase --import-jdl foo.jdl
  Why not integrate into jhispter import-jdl:
  
  
  
  • liquibase generator is experimental
  
  
  • Simplicity
  
  
  • jhipster import-jdl writes directly to the file, I changed jhipster-core to don't write, just return.
    
    So I can compare the changes and generate changelogs.
  
  
  • I don't have plan to implement multiples jdls.
  
  

Others todo:

• Implement tests: Ideas?? 🤔
• Dropping relationships and related constraints isn't implemented.
• Fake data.
• Liquibase tags

[gmarziou]

@mshima why mutating changelogs (change, merge, drop)? I'm really concerned about violating Liquibase immutability principle.

[mshima]

@gmarziou

• Great for testing the generator.
• Useful for a simplified production changelog, the less files the better IMO. Dev dbs always can rebased if needed. liquibase --rollback will do the trick here.
• The developer should take care about this.

[mshima]

@gmarziou creating liquibase tags should be useful here.
jhipster liquibase --create-tag v1.0.0
Drop and merge should ignore every change before any tag.

[gmarziou]

@mshima

The developer should take care about this.

OK but tools should promote safest practices, so default behavior should be to generate immutable changelogs. Then for developers who know what they're doing, they can use the options you propose.

Let's compare it with rewriting history in git: it's very useful but it can be dangerous if you do it on a branch that has been exported and used by others. This is why default behavior for git push is to reject rewritten history, then it's up to developer to use --force option. It's the same for Liquibase once a changelog has been pushed to production.

[MathieuAA]

JDL: jhispter liquibase --import-jdl foo.jdl
Why not integrate into jhispter import-jdl:
- jhipster import-jdl writes directly to the file, I changed jhipster-core to don't write, just return.

How so? If it's just a matter of adding a new flag, it's pretty easy.

[mshima]

JDL: jhispter liquibase --import-jdl foo.jdl
Why not integrate into jhispter import-jdl:

• jhipster import-jdl writes directly to the file, I changed jhipster-core to don't write, just return.

How so? If it's just a matter of adding a new flag, it's pretty easy.

The jhipster-core part is easy.
The cli part can be complex, not sure, probably it should be rewritten as a generator.

And what I mean is that this is not the focus right now. The liquibase generator should be mature first.

[ntorionbearstudio]

Hello, I discovered JDL import part in source code during the last days.
So, I tried to link it with what I have already done on jhipster entity command line tool and I managed to make it work with JDL importation. Here is the demo link with JDL schema update :

https://drive.google.com/file/d/1uV-mJdPHdpKc8GYVJaePZT2unVYHpE-4/view?usp=sharing

How it works :

• User imports JDL schema updated
• All current entities JSON description (before update) are duplicated (Entity-previous.json)
• For each entity :
• Jhipster generator loads new entity description (Entity.json) and if exists, previous entity description (Entity-previous.json)
• Jhipster generator makes diff between previous and new entity descriptions and generates these lists :

newFields
newRelationships
removedFields
removedRelationships

These lists are used by previous implementation on jhipster entity command line to generate corresponding new changelogs for Liquibase.

This solution is incremental, there is no way for the user to merge or manage his changelogs like @mshima propose. Jhipster generator will only generate required new changelogs corresponding to user updates on entities to make it work in production.

To my mind @mshima solution is far more interesting to allow user to do specific and more avanced operations on liquibase changelogs, my solution could be used for simple cases when user only wants to update his entites without have to manage liquibase changelogs by himself

[amatosg]

this looks awesome!

I have a question, how does it behaves when editing the same entity multiple times?

[Tcharl]

What about relying on the liquibase maven (or gradle) plugin at the end of an existing change?

• Running mvnw verify Before regenerating an existing implementation
• Generating, and not cleaning the target folder
• Running mvnw compile on newly generated code
• Calling the liquibase Mojo

Seems simpler to answer the use case and I would prefer to rely on a tool that has been made for that than a bake our own solution which will hardly achieve the same.

I also like the idea to promote the same approach (generating diff) for jhipster jsons: doing so, the framework could become more incremental than it is in its generation strategies. But it should be expressed in a dedicated issue and fill another need :-).

[ntorionbearstudio]

this looks awesome!

I have a question, how does it behaves when editing the same entity multiple times?

Hello @amatosg, there is no problem for updating multiple times the same entity, because each time Entity.jsonand Entity-previous.json will be updated and a new changelog will be created

[danielpetisme]

Regarding the new sub generator can we use a more generic name, like data-migration or kind of.
2 main reasons:

• in jhipster we also use data migration for NoSQL (which does not rely on liquibase) and this subgen could be use too (I know NoSQL is not the more used DB choice...)
• The blueprints may use other tools (I'm thinking about .Net which provides a built-in migration tool).

At the end it's just a name but I would prefer to stick with concepts/pattern/layer names in the sub gen naming and let the code use whatever implementation (like client vs angular, server vs spring, etc).

WDYT

[amatosg]

@ntorionbearstudio how is this going on? did you have time to finish? maybe a PR would be nice :)

[mshima]

@amatosg you can try https://github.com/jhipster/generator-jhipster/pull/11423

[ntorionbearstudio]

@ntorionbearstudio how is this going on? did you have time to finish? maybe a PR would be nice :)

Hello @amatosg I opened this PR : https://github.com/jhipster/generator-jhipster/pull/11419 I'll try end it soon (with tests)

[amatosg]

@mshima thanks but I think @ntorionbearstudio is a cleaner approach. The current behavior doesn't take any parameters (like versioned-database or --custom-changelog) and I don't see why new options or parameters are needed, it over-complicates the feature IMHO.

The current logic is to update the files and this new approach is to create a new file and add the reference to the master file (which is something I have to do manually very often).

Of course this is just a personal preference on how the feature should behave.

[mshima]

@amatosg I don’t get why it’s not clean. jhipster versioned-database --init liquibase is because I don’t want to make it default now. Once is called no more command is needed, neither for import-jdl nor for prompts.

--custom-changelog is to register your manually added file, so you don’t need to manually change master file everytime.

[amatosg]

@mshima well, I didn't say it's not clean, I said it's _cleaner_. When you say you don't want to make it default _now_, I understand and it will become default in a near future? I think this could lead to confusion. Again, this is just my opinion.

I have a couple of questions that I think @ntorionbearstudio addresses in his PR.

• The current logic is to created a versioned db. Why would I need to call jhipster versioned-database --init liquibase? Wouldn't it make more sense to call something like unversioned-database?

• Do I call jhipster versioned-database --init liquibase before o after the project has been generated? (Either way I have to make 2 calls to jhipster, right)

I think adding --custom-changelog is overcomplicate a simple functionality as adding one line to the master.xml is fairly simple.

[mshima]

@ntorionbearstudio I just released the blueprint generator-jhipster-liquibase. Can you try it?
If you still planning working on your PR, I should create a new issue to track incorporating generator-jhipster-liquibase into core generator-jhipster.

Just run jhipster --blueprints liquibase and it will generate a changelog for every field added/removed and relationship added/removed.

[ntorionbearstudio]

Related to this comment: https://github.com/jhipster/generator-jhipster/issues/11398#issuecomment-597622692, I also need changements on jhipster-core to make the new changelogs generation work on JDL importation

PR -> https://github.com/jhipster/jhipster-core/pull/469/files

[avdev4j]

hi @ntorionbearstudio
It would be awesome to have this feature in v7.

Also, do we plan to let the choice to the user to choose between adding liquibase diff or rewrite the old changelogs? (it could be useful).

[mshima]

@avdev4j since this PR is kind of stateless, then it's not possible to regenerate a old changelog if one partial changelog exists.
Ex:
1) Generate Foo inside initial changelog: eg. 20200601.
2) Add newField field to Foo inside second changelog: eg. 20200603.

20200601 -> 20200603 diff cannot be reconstructed if the history is not saved.

@avdev4j what can you tell me about https://github.com/jhipster/generator-jhipster/pull/11833?
This is required for implementing incremental liquibase changelog https://github.com/mshima/generator-jhipster-liquibase.

A new implementation of https://github.com/jhipster/generator-jhipster/pull/11833 is inside blueprint https://github.com/mshima/generator-jhipster-customizer/tree/master/lib.

If you agree with lazy calculating entity/field/relationship data I can create PRs.

[ntorionbearstudio]

Hello, sorry for the delay on this feature. Currently I have 2 PRs for this feature :

• https://github.com/jhipster/generator-jhipster/pull/11419 : I need help for tests, I don't know how it works inside the project, I wrote a comment on it inside the PR

• https://github.com/jhipster/jhipster-core/pull/469 : I have feedbacks to treat and @mshima idea to apply, on this too I need maybe some help

[MathieuAA]

@ntorionbearstudio for the JCore part, do ping me for help :)

[ntorionbearstudio]

Hello, sorry for the delay on this feature. Currently I have 2 PRs for this feature :

• #11419 : I need help for tests, I don't know how it works inside the project, I wrote a comment on it inside the PR
• jhipster/jhipster-core#469 : I have feedbacks to treat and @mshima idea to apply, on this too I need maybe some help

Hello I'm working on it this week to complete the feature

[ntorionbearstudio]

My work on it was done with some tests on new generated changelogs : https://github.com/jhipster/generator-jhipster/pull/11419 but like I have written on the PR, with @mshima 's work on https://github.com/jhipster/generator-jhipster/issues/12178 this issue can be considered as closed I think

[DanielFran]

@pascalgrimaud Can you confirm this can be closed?

[pascalgrimaud]

@DanielFran : I don't know, I wanted to test this feature but didn't have the time yet, because of preparing JHipster Code... Only @mshima knows :)

[mshima]

@pascalgrimaud @DanielFran I think so.

[DanielFran]

Ok. So let's close it