Generator-jhipster: Improved JDL and liquibase diff based workflow

[gimoh]

• Overview of the request

I'd like to be able to use JDL as the source of truth for the object models and schema.

I'm opening this ticket to discuss issues and solutions, I'll submit PRs for any changes that come out of this.

Roughly the improvements I'd like to look into are:

• [ ] prevent entity generator from updating existing _added_entity_ changelogs (as applying these will always fail unless the table hasn't been created yet)
• [ ] snapshot entity JSON files (in .jhipster directory) before running import-jdl or entity
• [ ] generate a liquibase changelog after import-jdl or entity finishes (would be best if it was possible to run it after all entities have been updated)

• [ ] inject the newly generated changelog into master.xml

• Motivation for or Use Case

JDL and the entity generator are great features provided by JHipster. It would be great if the full workflow of using JDL was more streamlined - I'm thinking:

• hack on your JDL (.jh file)
• run import-jdl
• review, potentially restore customisations and commit

Currently after modifying the JDL you need to revert some changes, run liquibase task to generate a changelog, register that changelog and finally review and commit.

• Related issues

3136, #3510, #4104 (kinda)

• Suggest a Fix

I'd appreciate comments, as I said I'm going to submit PRs for anything I get done unless conclusion is to put it into a separate module, etc.

For starters I'm wondering what does everybody think about the first task of preventing the entity generator from updating existing _added_entity_ changelogs. I see this is currently done unconditionally but I'm thinking it would be better if it was only done when an entity is actually created not when it's updated. Any thoughts?

[deepu105]

This is something I was considering as well when I originally coded the entity editing feature but I decided not to do it at that time as its more complex than it looks.

• For entity generator, it's possible to generate new changelogs when fields & relationships are added/removed though not very simple to implement it is feasible. So when you add new data to entity you will create a new changelog with that data and same when you remove fields. However handling relationships and changes to a related entity could be tricky
• Now for the import JDL its far more complex as JHipster wouldnt know what kind of changes were made in JDL and hence it would require a complex diff implementation to check against existing entity json files and the newly generated files then creating changelogs from that.

If you are up for a challenge I'm in favor of this as I agree that this will improve the developer experience

[gimoh]

I know exactly what you mean, this was my original plan (to have the generator create new changelogs) but I quickly realised it would need to maintain a before and after state of the json representations of entities and then figure out a diff. So in a way this seems like re-implementing what liquibase diff already does.

And given that from what I understand, many people here already use liquibase diff rather successfully, I'm now leaning to build on that, at least in the short term.

What do you think? Is it reliable enough to use it?

[deepu105]

We normally like to give the developers out of the box experience. that means everything working OOB. Liquibase diff would need the user to configure credentials and stuff to work and it's not that widely used, to be Frank. So I wouldn't rely on that in the generator itself. So no

[deepu105]

using liquibase diff would also break hot reloading workflow.

[gimoh]

I actually thought about it some more yesterday and thought that this approach could be flaky as liquibase also relies on the dev DB being in the correct state, so e.g. someone running clean and then running diff will produce a changelog with everything added in. So it relies on correct sequence of events, not to mention there's still a couple of issues with it and H2 DB.

Anyway, I also thought doing it directly in JHipster may not be that difficult. Essentially I'm thinking if we can snapshot the entity generator variables (like fields or relationships) before and after the generator modifies them, it should be straightforward to figure out what's been added/removed/modified.

And because the import-jdl also just modifies the entity json files and relies on the generator to do the rest, it should handle this case too.

The only thing is import-jdl modifies the .json files on disk before calling the generator, so any snapshotting would need to be done on the files not the in-memory data.

[deepu105]

@gimoh can you try to do a POC first?

[gimoh]

@deepu105 yip, working on it :)

[jdubois]

@gimoh if you have a working POC, then I'm eager to test it.
Meanwhile, this looks really complicated to do, and I agree with @deepu105 's arguments. So I'm closing this as to me it doesn't look like a great idea, but I'll be happy to be proven wrong with your POC.

[gimoh]

First POC: https://github.com/gimoh/jhi-entity-diff-to-changelog

As it is only a POC it only handles adding string columns to a table. And you have to do the snapshotting manually.

Even for that I had to replicate a bit of functionality from the generator so for going forward I think it would be best if we agreed on the approach so I can propose changes directly against the generator.

OK to re-open?

[jdubois]

Yes, re-opened!
But I'll need time to review it

[gimoh]

Cool! Any questions let me know sure!
The idea is to generate diff of the entity json files and then generate liquibase operations based on diff operations (this is using JSON patch as a result of the comparison).

[deepu105]

@gimoh interesting. I had a quick look and seems good. But we need to do this in memory without using before and after folders. It should be possible since with jhipster-core you can have the json streamed as output using convertToJHipsterJSON. So you could stream the output and compare it against the current .jhipster folder content before writing it using exportToJSON and then use the compared output to create changelogs.

I'll take a closer look later and let you know

[gimoh]

That should be OK, the diff is calculated from the in-memory objects anyway (it reads the entities in here), so it would be enough to just read it in before it's changed.

The question is where to read it in.

Could do it in the entity generator, but that won't work for import-jdl as it modifies the .jhipster JSON entities before calling the generator. If you do it in import-jdl it doesn't cover using the entity generator directly.

Unless it's in a library, called from both places?

[deepu105]

It should be a function in generator base anyways, but I think the implementation can be in the entity generator so that it works on a per entity basis and import-jdl, in the end, will call entity generator anyway. But if you find it complicated then feel free to use the function in both entity generator and import-jdl but then you need to avoid it being run twice.

[gimoh]

It should be a function in generator base anyways

Right, that makes sense

import-jdl, in the end, will call entity generator anyway

the thing is it only does that after modifying the .jhipster JSON files

unless we make it load the ("before" snapshot of) entities and store them / pass them to the entity generator somehow?

[deepu105]

Oh yes, in that case, I think you can implement this first in import-jdl then we can find a way to use it in entity generator as well, shouldn't be hard

[gimoh]

Cool, works for me :smile:

[gimoh]

So I've been scratching my head a bit today and I thought I'll drop a comment here to see if what I'm seeing is expected.

I've been trying to read entities from the .jhipster directory and I was using generator's this.getExistingEntities(), this works fine the first time, but it turns out when I was using it second time after the entities have been modified on disk (i.e. after generateEntities()) it was returning stale results.

This turns out to be because getExistingEntities() uses yeoman's this.fs.readJSON() to read the files, which apparently uses an in-memory filesystem (reads the files once, then uses the cached version), however when the entities are modified in parseJDL() it uses jhipster-core which uses the real fs directly, so not going through the in-memory one.

Now that I know what's going on I can just use jhipster-core to read them in, but I thought I'll check here if this was expected, as it's a somewhat surprising behaviour.

[deepu105]

Yes it is indeed. Can you change the method to use global fs instead of this.fs?

[gimoh]

Sure, I'll propose a separate PR.

[adnans-coviot]

So, what is currently the best advice for managing changes to entity model? JHipster documentation advises to either do the change through sub-generator, or starting with manually modifying .json files, but this breaks the sync with JDL file and it is very difficult for big changes.
If there is an option for import-jdl tool to generate only .json files, rest can be better manageable. I can discard files that has no changes to begin with.

[deepu105]

@adnans-coviot goode idea. Its easy to add a flag to support this say --json-only let me take a look

[deepu105]

@gimoh FYI https://github.com/jhipster/generator-jhipster/commit/f3b731fcb3ddee74b54375b3c552c7849693c02d

[deepu105]

@adnans-coviot the flag --json-only is added as well. Available in master

[deepu105]

@adnans-coviot only entities that have changed will be regenerated now, if you want to generate everything like before then you need to pass --force flag

[jdubois]

I'm closing this as this as been stalled for 5 months.
I'm really sorry about that, but we have 44 opened issues, and I just can't follow everything.