Generator-jhipster: Support for rich document models for mongodb (embedding and referencing)

[ivangsa]

Hi,

I would like to add support for rich documents a relations for mongodb to jhipster

I think it would be easy to add support for both embedded/nested objects and collections as well as references between entities

I've been doing some work but I'm quite new to js development and I'm stuck with debugging some errors and finding my way with mocha tests..

https://github.com/ivangsa/generator-jhipster/issues/1

Let me know what you think about this...

Regards
Ivan

---

I would like to implement support for entity relations with mongodb, to generate microserices from jdl to cover two use cases:

1.- Support complex document models: embedded and nested objects and collections.
2.- Support documents with relations using @DBRef

I think the JDL would need extending to acommodote both embedding and referencing... Allow adding fields with an entity as fieldType, very much as the Enums are defined now, using jpa @Embeddable/@Embedded as field configuration for sql backend, benefiting sql backend as well as mongodb.
entity Person {
name String,
address Address
}
entity Address { ... }

(There is a small problem when using @Embeddable/@Embedded is not posible mark the embedded Id as transient, but small tweaks on generated code can acomodate the mappings for exact needs..)

This is how I'm planning to interpret the JDL as it is right now plus the 'embedded field' extension:

Embedded (for embedding entities)

• Unidirectional: @Field (@Embeddable/@Embedded for sql)
  entity Person {
  name String,
  address Address
  }
  entity Address { ... }

One-to-One (for entities references)

• Bidirectional: @DBRef for both left and right side fields. @JsonBackReference on right side.
  relationship OneToOne {
  Car{driver} to Driver{car}
  }
• Unidirectional: @DBRef on left side
  relationship OneToOne {
  Citizen{passport} to Passport
  }

One-to-Many

(Owner has none, one or more Cars) For embedding collections of entities with backreference.

• Bidirectional: @Field for left side, @DBRef for right side. @JsonBackReference on the non-collection (right) side.
  relationship OneToMany {
  Owner{car} to Car{owner}
  }
• Unidirectional: (not supported)

Many-to-One

• Bidirectional: For embedding collections of references: @DBRef on both sides. @JsonBackReference on the non-collection side.
  relationship ManyToOne {
  Car{owner} to Owner{car}
  }
• Unidirectional (where the car know its owner): @DBRef on the left side field.
  relationship ManyToOne {
  Car{owner} to Owner
  }

Many-to-Many

• Bidirectional: using @DBRef on both sides. Not sure about @JsonBackReference/@JsonIgnore of how much sense does it make.
  relationship ManyToMany {
  Car{driver} to Driver{car}
  }

I believe this would cover a wide range of posibilities for generating rich document models for mongodb microservices.

I found this was proposed before but not work was done yet..
https://github.com/jhipster/generator-jhipster/issues/4079

• [x] Checking this box is mandatory (this is just to show you read everything)

[iceddante]

I too opened an issue a while back for this. The objection then, which is still relevant with your post, is that these changes would not be compatible with the front-end generation. Not without quite a bit of development and introducing a lot of complexity to front-end templates anyway. I suppose requiring headless generation is one workaround. Creating a separate module to do a NoSQL JDL import is a different possibility. Happy to work this with you if you're interested

[ivangsa]

Hi

I was offline these days,

I understand that the gui generators were developed for jpa relations

I think it's worth give it a try.. The main purpose is to use it for microservices but once it can generate the backend we can see how the front ends works with it

There is a branch in my forked repo (mongodb-with-relations) with work in progress, there are still some tests that doesn't work
https://github.com/ivangsa/jhipster-core
https://github.com/ivangsa/generator-jhipster

you can give it a try, I'll be glad to work with you developing this

Thank you

[ivangsa]

Hi

this work is already in progress, there is plenty of work and testing to be done
if you want to join you can try it following these steps

git clone https://github.com/ivangsa/jhipster-core.git
git clone https://github.com/ivangsa/generator-jhipster.git

cd jhipster-core
git co mongodb-with-relations
yarn link

cd ../generator-jhipster
git co mongodb-with-relations
yarn link jhipster-core

cd ..
mkdir mongodb-with-relations
yarn link jhipster-core && yarn link generator-jhipster

node_modules/generator-jhipster/cli/jhipster.js
node_modules/generator-jhipster/cli/jhipster.js import-jdl <some-model.jdl>

Just drop me and email to ivangsa at gmail.com to start a conversation..

Cheers
Ivan

[jdubois]

@ivangsa great to see this going forward, this was requested by so many people! I'll tweet about this to try to attract more contributors

[PierreBesson]

@jdubois Maybe we could offer a reward for people who help on this feature as we discussed before on the mailing list about the "bug bounty" program.

[jdubois]

Agreed @PierreBesson - bounty added :-)

[jdubois]

So even with the $100 bounty this is still stuck, as it already happened with #4079

This is why I'm closing this -> if anybody is motivated, I will gladly re-open it and add back the label, of course.

[ivangsa]

This will be done,
its just summer time, no time to be indoors..

[jdubois]

Thanks @ivangsa , I'm re-opening!

[iceddante]

@ivangsa I am free as well! Cloning your fork and emailing you to see how I can help

[ivangsa]

Thanks Randeep, I got your email.. let's do it

[ivangsa]

I have already generated some projects that uses mongodb with relations with different options that seems to work nicely

https://github.com/ivangsa/jhipster-mongodb-sample-projects

Can you add me to the google group so I can share news and ask technical questions..

[jdubois]

@ivangsa the good place to discuss is here, not the mailing list. The mailing list is only for core dev team members, and it's not good to link code, discuss with everyone, etc... We do all the work on GitHub, even if sometimes the issues because too big!

[ivangsa]

Ok,

my next plan is to start testing adding new relationships with prompts which I didn't test yet at all..

projects generated with jdl already work nicely without any tweaks
many-to-one, many-to-many, embedded/one-to-one are tested with/out dtos, pagination...

and gui code seems compatible with mongodb persistence,
I didn't need to tweek anything above dtos layer, so microservices are expected to work as well

Code is already merged with v5.2.1 in a single commit for easy review
https://github.com/ivangsa/generator-jhipster/tree/mongodb-with-relations-v5.2

and sample projects are as they were generated, no tweaks..

If anyone would like to give it a shot there are quick instructions in the readme.md
https://github.com/ivangsa/jhipster-mongodb-sample-projects

I may ask some questions about nested forms here, but later on..

[ivangsa]

yes, @Embedded is a jpa annotation and is not being used for mongodb.. it was added only for sql..
(what I was expressing with 'embedded' for mongodb was an entity configured with @Field)

Now that full working projects can be generated I would like to rethink how to express relationships for both @Field/@DBRef so every option can be generated

I belive now it would be good idea, for much simpler, to get rid of that and use only OneToOne to embbed/nest entities (like Customer{address Address})

But, in this example, if address is to be saved embedded (as one @Document) it makes sense for address fields to be present in Customer .html forms...

Now that working projects can be generated end to end is a good time to re-think how to express and generate those relationships

you already know this the .ejs template to play with
generator-jhipster/generators/entity-server/templates/src/main/java/package/domain/Entity.java.ejs

let me know of any ideas that make sense based on your real projects experiences...

[randeepbydesign]

I spoke with Ivangsa about this. I think we need to modify the JDL syntax to include a reference to an "embedded" type:

entity Employee {
    /**
    * The firstname attribute.
    */
    firstName String,
    lastName String,
    email String,
    phoneNumber String
}

entity Office {
    name String,
    phone String
}
**embedded** Address {
    street1 String,
    street2 String,
    city String,
    state String,
    zip String
}

relationship OneToOne {
    Employee{address} to Address
    Employee{workplace} to Office
}

relationship OneToMany {
    Office{address} to Address
    Office{staff} to Employee
}

I know ivangsa is apprehensive about modifying the JDL in this manner. I'm all for it >:-)

But definitely open to alternative suggestions

[jdubois]

Modifying the JDL can be quite complex -> @MathieuAA you are the boss on this part, what do you think?

[ivangsa]

yes,
I've have removed this altogheter and now it's much simpler and with fewer implications

Using @DBRef for relations makes everything else compatible right away and it makes sense ..

For many use cases @Field would be prefered but tweaking that would be real easy...
for microservices straightforward and gui interfaces would need tweaking anyway..

With this trouble out this is almost finished,
I expect this to be ready in the next few days..

Looking back I'm surprised how easy to implement this was..

Thankyou Randeep for your insight as well!

[randeepbydesign]

Sorry, I just want to clarify that what you're saying is that _only_ DBRef would be supported, and that there would be no subdocuments?

[MathieuAA]

@randeepbydesign I don't get the **embedded** part.

[MathieuAA]

Could it work (in your scenario) without it?

[randeepbydesign]

By embedded we are referring to an entity that would not have a controller or DAO. It would just be represented as a subdocument in the main entity. Let's say we had a Person Entity, an Office Entity, and an Address Embeddable (or whatever you would want to call it) in our JDL.

The Office would have One to Many Persons. Office would have one to many address. And Person would have one to one address.

Only Office and Person would be an Entity in this scenario, and have a Document representation in the MongoDB datastore. The address would only exist as a subdocument:

Office: {
staff: [{}, {}], //DBRef to Person Collection
address: [{"street": "123 main st.", "zip":"11111"}, {street: "2244 Anywhere Dr.", "zip":"11111"}]
}
Person: {
address: {{street: "333 Main Rd.", "zip":"11110"}
}

There is currently no way that I know of to specify the Address entity in the JDL and have the generated code reference it short of creating a new type to exist alongside Entity. Let me know if I am wrong! I have a bit of time where I can work on adding this to the syntax if need be.

[MathieuAA]

I know what embedded is, but after looking at the JDL with the "embedded" pseudo-annotation I wondered if it was the same thing.If we're set to have this information in the JDL, I'd put it in the relationship (it's the most logical place).I can try experimenting on that.---Le 10 sept. 2018 02:06, Randeep Walia [e-mail] a écrit :By embedded we are referring to an entity that would not have a controller or DAO. It would just be represented as a subdocument in the main entity. Let's say we had a Person Entity, an Office Entity, and an Address Embeddable (or whatever you would want to call it) in our JDL.
The Office would have One to Many Persons. Office would have one to many address. And Person would have one to one address.
Only Office and Person would be an Entity in this scenario, and have a Document representation in the MongoDB datastore. The address would only exist as a subdocument:
Office: {
staff: [{}, {}], //DBRef to Person Collection
address: [{"street": "123 main st.", "zip":"11111"}, {street: "2244 Anywhere Dr.", "zip":"11111"}]
}
Person: {
address: {{street: "333 Main Rd.", "zip":"11110"}
}
There is currently no way that I know of to specify the Address entity in the JDL and have the generated code reference it short of creating a new type to exist alongside Entity. Let me know if I am wrong! I have a bit of time where I can work on adding this to the syntax if need be.

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

[randeepbydesign]

I had considered that but I don't think it will work because it doesn't explicitly block entities and daos from being created. I guess we could check to see if no entities belong to the left side of a relationship and infer that status but that seems kind of hacky and might cause other problems

[ivangsa]

I've created a pull request for this feature

I think it is complete, it's not as ambitious as the first proposal but I regard it as a fairly good starting point
it permits generate any supported entity relations that later can be easily reconfigured, either as a @Field or a @DBRef to suit specific

There is one Travis test that is not passing, but it seems that it is because it depends on jhipster-core to be merged before..

Let me know if there is any further action on my side at this moment

[jdubois]

Thanks a lot everyone!
I guess this will require quite some time to review... And I know I'll be a bottleneck here, mostly given that we have 45+ opened tickets at the moment (and I have lots of work in parallel). So if anybody wants to do some review here, he's most welcomed. I can also ask for help on Twitter, as I guess many people will be interested.
Anyway, that's awesome work, very impressive!

[randeepbydesign]

Thank you Julien for your feedback. I would like to take a crack at extending the JDL to support a new type. I'm going through the core jhipster-core and generator-jhipster projects now to see where to insert the scalpel(s) to make it happen. I hear that there may be some hesitation with this, so if anyone feels like I might be making a mistake or wasting my time (or if you just want to offer some guidance!) please reach out to me so we can talk.

[nonamejx]

Hello,

Could you please update the docs as well ?
https://www.jhipster.tech/managing-relationships/

The relationships are now available when we choose to use MongoDB.

Thank you so much for your awesome work 🥇

[pigalon]

Hello,
many thanks for this great contribution.
I have one question about jdl test files from jhipster-mongodb-sample-projects/models-jdl/
Which jdl is the good ones ?
I took the last Jhipster version 5.4.0, and the orders-model-embedded.jdl doesn't work because "The type 'PaymentDetails' is an unknown field type."
Thanks in advance

[ivangsa]

hi @pigalon, orders-model-embedded.jdl is the only one that is deprecated, I will delete it shortly
you can use orders-model.jdl and append some of the other as options

[pigalon]

Hi ivangsa,
thank you for your answer, indeed the orders-model.jdl works perfectly.
Relations with Mongodb model is useful.
However I want to use a mix relation and embedded objects, so I've reused your data model and I've added new class product and add a relation between ordered item and product.
Ordered Item become just an embedded object in customer order, I've just dropped the annotation @DBRef and it perfectly works.
Thanks again.

[tchlyah]

Hi @ivangsa, @randeepbydesign,

Did you guys drop the embedded entities ? It is an important option for Document databases!

[gclayburg]

Hello,

Could you please update the docs as well ?
https://www.jhipster.tech/managing-relationships/

The relationships are now available when we choose to use MongoDB.

Thank you so much for your awesome work

I agree @nonamejx , it would be nice if the documentation there were updated. What I found is that a OneToOne relationship does work but it generates the java code with a @DBRef

This can be manually changed in the generated java code if you need the relationship to be embedded instead. All you need to do is remove the @DBRef annotation. Maybe this is also possible with JDL, but it isn't obvious.