Quarkus: mongodb-panache indexes

[diogocarleto]

Hi guys, hope you are doing well.

I've implemented a local feature that uses MongoOperations to ensure my indexes on mongodb at the application startup, but it is not elegant :)

I'd like to suggest some like we have seen in Spring or even on Morphia.
Some like @Indexed, @CompoundIndexes, @Indexes

What do you think?

[loicmathieu]

@diogocarleto I was thinking about MongoDB migrations some days ago (like Flyway) and didn't achieve to make any decision about it (I didn't find a simple tool that allow textual - like JS script - migrations).
And in fact, we need MongoDB migration script essentially for index creation ! So adding a way to defines index via annotations and create them automatically can do the trick.

But, if we compare what we have for Hibernate, we can run SQL migrations (Flyway) or strctured migrations (Liquibase), and arbitrary SQL at startup (import script). Maybe we need to define a strategy for MongoDB instead of rushing to implement what you asked. If at some point we feel that index creation is not enought we will maybe have two ways to create indexes (via annotations and some migration tool that we integrate at this time).

So, yes, a good idea, but maybe there is more to do about this ;)

Maybe @evanchooly have some thought about this ?

[r2liquibase]

Liquibase supports MongoDB now. Check it out and let us know what you think: https://github.com/liquibase/liquibase-mongodb.

[evanchooly]

Well, I need to admit up front a slight bias. I'm the maintainer of Morphia and so when I see more and more ODM features getting added to mongodb-panache I get twitchy. There's already Morphia and spring-data etc. so I'm leary of duplicating this functionality all over again.

Now specifically on the subject of indexes, there's some debate as to whether the ODM should manage that at all. Given the choice, i would integrate liquidbase's indexing support. These kinds of problems are solved problems and I don't really relish the thought of reimplementing them inside quarkus if we don't need to.

Given that panache is tightly bound to hibernate we can only go so far with external framework integration but where we can we should integrate rather than implement.

[diogocarleto]

Hi guys, how are you?

Thanks for your answers.

@loicmathieu I've got your point, I'm taking a look in integration tools like flyway/liquibase for MongoDB, to try get back here some ideas to evolve this.

Best,

[MisterRnobe]

Hi

I've got the same need of automatic index creation. Liquibase seems multipurpose solution but it's not integrated to quarkus. I think it's a good idea to implement this integration and let users configure it with application.properties.

For example Spring has integration with flyway and it can be configured with spring config file. I find it quite convenient and easy to use.

[diogocarleto]

I was checking some frameworks and one that get my attention is this.
https://www.mongock.io

I’m going to do some tests with this.

[loicmathieu]

For a quick heads up, this issue is still on my radar but I didn't find the perfect solution yet :(

• Mogock is interested but not yet very popular.
• Mongobee is more popular but not under active maintenance.
• Liquibase support is really young and didn't have any clear documentation yet (I didn't find any migration example!), and moreover uses db.eval() wich is not supported anymore in MongoDB 4.4
• Flyway didn't support MongoDB migrations.

So, due to MongoDB 4.4 not supporting db.eval() anymore, we cannot integrate liquibase which should have been the natural choice as Quarkus already support Liquibase.

Untill we decided which tool to include, this enhancement request will be _on-ice_.

@diogocarleto if you have success with Mongock, don't hesitate to give feedback on this issue and send us a small example project.

[evanchooly]

Why would db.eval() be needed at all? Index creation has explicit commands that can be run via the driver.

[loicmathieu]

@evanchooly most existing mongodb migration tools uses db.eval() to avoid mandating Java migrations.
I agree that you can do without them but it's the way it is done so Liquibase is a no-go.

We can always support an @Indexed annotation as it was requested if we didn't find a migration tool that satisfies us but we agree that it is not a good solution. Moreover, when transactions lands in, they will need the collection to be created prior to using it so we will definitly need a MongoDB migration tool.

[diogocarleto]

Hi @loicmathieu, I think this week I’ll have a prove of concept of this.
Leveraging this, I don’t think it is the properly place to ask, I can move the next question to another thread without problems, so let’s go to the e question :)
I think we should have in MongoDB Panache an optimistic lock feature, just like we have in JPA. What do you think? It makes sense?

Best.

[loicmathieu]

@diogocarleto yes, this should be a separated thread.

And how would you perform such lock ? MongoDB didn't provide any lock facility like SELECT FOR UPDATE on RDBMS databases.

[evanchooly]

Most solutions involve something like an @Version annotation to track concurrent modifications. Both Morphia and Spring Data support that approach. But before we go too far down either road (indexes and versioning) we should really consider if we want to build out such infrastructure and thus introducing yet another mapping framework inside of quarkus or just build on top of another existing framework. I have a personal bias toward Morphia (as i'm the maintainer) but there are others we could integrate rather than rebuilding that world.

[albert0815]

We would also be very happy with some kind of mongo db migration tool to be integrated with quarkus. Also optimistic locking is a feature which we need.

For the migration we did a POC using Mongock, which we setup in a startup listener which worked fine. But we also will have another look at Liquibase.

@loicmathieu Where did you find the information that liquibase is using db.eval? For me it sounds different, in their repo readme they are writing: "This is a Liquibase extension for MongoDB support. It resulted as an alternative to existing MongoDB evolution tools. Majority of them are basically wrappers over db.eval shell method that is deprecated staring from MongoDB 4.2." (https://github.com/liquibase/liquibase-mongodb)

[loicmathieu]

@albert0815 I found the information that liquibase uses db.eval in this very statement that your copy here: __Majority of them are basically wrappers over db.eval shell method that is deprecated staring from MongoDB 4.2__.

But I check the code, and they didn't use db.eval, so it may be a good candidate.

[avdev4j]

Hi @albert0815
I'm working on JHipster Quarkus Blueprint and I would like to support the MongoDb option. I can make it works (with panache) but I struggling a bit for migration feature. We initialize the db with such data like users and authorities.

I see you did a POC using Mongock, is this POC is available somewhere on GitHub? What about the Liquibase extension, have you tested it?

I'm working on a sample app, I will be happy to share it soon but I need to make it works and especially be able to init my db.