Mongoose: How to share the connection pool across multiple databases

[seasonhuang]

In node-mongodb-native, I can share the connection pool across multiple databases:


var mongodb = require("mongodb"),
mongoserver = new mongodb.Server(host, port, server_options),
db_connector = new mongodb.Db(name, mongoserver, db_options);

db_connector.open(callback);


And then I use db_connector.db(name), this returns a new db instance that shares the connections off the previous instance but will send all commands to the database name . This allows for better control of resource usage in a multiple database scenario.

HOW CAN I DO THIS IN MONGOOSE?

[aheckmann]

It's not presently supported.

On Sep 25, 2012, at 7:05 PM, Rex Huang [email protected] wrote:

In node-mongodb-native, I can share the connection pool across multiple databases:

var mongodb = require("mongodb"),
mongoserver = new mongodb.Server(host, port, server_options),
db_connector = new mongodb.Db(name, mongoserver, db_options);

db_connector.open(callback);

And then I use db_connector.db(name), this returns a new db instance that shares the connections off the previous instance but will send all commands to the database name . This allows for better control of resource usage in a multiple database scenario.

HOW CAN I DO THIS IN MONGOOSE?


Reply to this email directly or view it on GitHub.

[baoshan]

+1
Multi tenancy application with multiple database on a same mongod instance can benefit from shared connections.

[baoshan]

If a node.js server needs to connect to 20+ database in a same server, and using a node.js cluster with 4 process, then the connections be will easily over 500 (the default PoolSize is 5). This scenario is not rare for a multi-tenancy architecture.

Could somebody kindly provide a review of my tweak? The tweak reuses connections of the same server (ReplicaSet not implemented yet).

https://github.com/baoshan/mongoose/commit/e192bc94cb19f4c5d7b9e9cb52343bf6e82a178e

[aheckmann]

@baoshan yeah we should expose this somehow. i'm not convinced yet that caching it as in your commit is the right approach, but i haven't looked into what it would take yet.

[Reggino]

+1

I'm looking for a way to have models over different databases (multi tenancy application). The source of Mongoose.prototype.model suggests that passing a connection is possible when compiling a new Model. But this feature is undocumented and i have feeling i will run into some issues when using this in production :-)

Update: I've switched to Don Park's approach https://github.com/donpark/mongeese

[morphar]

+1

I have tried nearly everything I could think of - including using mongeese... No luck...
I'm in a situation, where I would like to have several "sets", each having a connection to a different database and the same schemas, except I need some of them to be slightly different.

The biggest problem being, I need to talk to the correct database, based on the settings in my wrapper class and hold default schemas + the altered versions.

@Reggino is that what you are doing with mongeese? I can't seem to make it work.

[morphar]

O.k... I have just gotten my case to work - though I haven't checked for the amount of connections...
It can be hard to find useful examples in the documentation, but with a days worth of tinkering, testing and being stubborn as a donkey, I have finally figured out, how to make it work...

So in case anybody else needs this, this is what works for me now (in it's most basic form):

var mongoose = require('mongoose');
var db1 = mongoose.createConnection('mongodb://127.0.0.1/test1');
var sites1 = db1.model('site', new mongoose.Schema({ … }));

var db2 = mongoose.createConnection('mongodb://127.0.0.1/test2');
var sites2 = db2.model('site', new mongoose.Schema({ … }));

sites1.find(…)…
sites2.find(…)...
sites1.find(…)...
sites2.find(…)...

[Reggino]

@morphar Yes, that is what Mongeese does. Where mongoose is designed as a Singleton instance by default, mongeese is a 'hack' to be able to have multiple instances. Advantage over your solution is that you may use a second mongoose-object with it's own default connection.

[aheckmann]

yes, the mongoose module is itself an instance of mongoose.Mongoose. creating more instances is definitely an option but not really what this ticket is about. this ticket is for reusing the existing connection pool.

[morphar]

(Feeling totally newbie here...)
@Reggino O.k... Thanks for clearing that up for me! :) I will have a look at it again.
@aheckmann No, I can see I've misunderstood the difference between what I was trying to do and what the ticket was really about. Thanks for clearing that up too :)

[morphar]

I still think it would be nice to be able to share the connection pool - but I can imagine that would be harder in the case I wrote above.
I can see that the case above creates a total of 10 connections - that's a bit high, when I will probably end up using at least 20+ databases...
Maybe I should just look into forking the project and maybe actually end up contributing some code ;)

[aheckmann]

working on a solution for 3.6.

[aheckmann]

will be in rc1

[aheckmann]

after further review we're pushing this to after 3.6.0. too big of a change this late.

[Reggino]

I don't want to stress things, but is there an ETA for this feature? I really need to know this for future plans on our multitenant application.... Thanks

[aheckmann]

No ETA but I'd like it in v3.7. Last I researched it the driver behavior
was a little wonky (events were not always fired as expected) so before
anything is built on it we should make sure the works properly. It might
work now, it's been a while.

That said, the best way to get this implemented in a timely fashion is
submitting the pull request :)

On Monday, May 6, 2013, Tim de Koning wrote:

I don't want to stress things, but is there an ETA for this feature? I
really need to know this for future plans on our multitenant
application.... Thanks

—
Reply to this email directly or view it on GitHubhttps://github.com/LearnBoost/mongoose/issues/1124#issuecomment-17525697
.

Aaron
@aaronheckmann https://twitter.com/#!/aaronheckmann

[aheckmann]

closed. https://github.com/LearnBoost/mongoose/pull/1601

[manast]

I would just like to add a comment regarding multi tenancy and mongoose. Seems to me that mongoose is not well designed for multi tenant applications since the models are hard coupled to the database. The API is designed for single databases, therefore you can do something like this:
User.findById(id), but in reality what you need is db.findById(User, id). So even if the former looks nicer it is in practice much less flexible and very difficult to work around.
It is also worth mentioning that to make things worse, this coupling between the database and the model is performed implicitly, since you use the mongoose instance to create the model, and at that point you may not have created any connection yet, so it creates the "illusion" that models are decoupled from the database, but in fact they are not...

[baoshan]

I don’t think mongoose is not suitable for a multi tenancy deployment.
Multi-tenancy means multiple tenants share a single application server (and/or a single mongod), but they can hold separate database.

At least, no body stop us from architecting our application into a one-databse-for-one-tenant design.

On Aug 13, 2014, at 4:13 PM, Manuel Astudillo [email protected] wrote:

I would just like to add a comment regarding multi tenancy and mongoose. Seems to me that mongoose is not well designed for multi tenant applications since the models are hard coupled to the database. The API is designed for single databases, therefore you can do something like this:
User.findById(id), but in reality what you need is db.findById(User, id). So even if the former looks nicer it is in practice much less flexible and very difficult to work around.

—
Reply to this email directly or view it on GitHub.

[manast]

@baoshan it may not be impossible, but if you aim to make a modular application I believe it is not suitable :).

[baoshan]

TL;DR
It all depends on how much time and effort you want to spend on (correctly) architecting your application.

@manast when I say modular, I mean separate of concern and independently testable. Both goals are achieved for several projects. Some of them are multi-tenancy.

From a data-governance point-of-view, one client (tenant) should have a separate database. Multiple clients' databases all share the same logic (that's what a service means, right?), so they share the same schemes (but not the same models), the same middlewares, and the same connection pools.

[manast]

@baoshan I agree in mostly everything you say. But I think it that what you would also need is to be able to share the Model, not model instances, but the Model class itself. Otherwise it does not scale. For instance I don't think it is efficient to create the model from a schema every time a user accesses your service... sure you could cache all the models in every user session, but seems to be overcomplicated and unneeded.

[Reggino]

In my scenario (imagine a REST-full data-service) every tenant (imagine an API key) has it's own schemas (and thus Models). As part of the service, a tenant can configure the schema/model himself. There is shared logic however in how e.g. validation is done and some business logic is handled.

Shared "Models" (or even shared schemas) may be useful in some scenarios but it doesn't fit the bill everywhere. So i guess that's just a matter of how your app is organised.

[baoshan]

@manast I think we're agree with each user/tenant hold their own separate database, are we?

And I believe each user's database are isolated totally, which means mostly query only access a single database, am I right? Otherwise we're not talking about multi-tenancy, we're talking about multiple users of a single tenant's application.

The current problem is you do not want to instantiate a set of new models when the user(tenant) use your application, am I right?

You can cache the models into a models pool. which will share a same connection pool, so it's resource / memory efficient. So you do not need to instaitiate new models (that may be time consuming).

Do you think this is an acceptable solution? What's your suggestion if you feel that's not optimal?

[manast]

@baoshan I think we agree on the same things.

Regarding my proposal, it is simple, just decouple the Model from the database. So when you need to access a Model in a given database you just do something like: db.findById(User, userId)
In that way you don't need to cache anything, you just create the models once from their schemas and reuse them for all your tenants.

What do you think?

[baoshan]

@manast I think I get your idea. According to your proposal:

1. we have to cache the db objects, otherwise, a tenant hit your server, we have to make a new db objects?
2. the new proposed api lose its simplicity for other developers who do not need to suffer from our problem
3. caching models for different tenants is no worse than caching db objects in case they don't lead to new connections.

Have I missed something?

[manast]

@baoshan

1. db objects are very lightweight, just telling which database to use, it can share the same connection to the database server as the rest (if they use the same database server). And they will be much less objects to cache.
2. It will loose a bit of its simplicity, it would be possible to keep the old api though.
3. It is worse because, a) you need to cache more of them, so it needs more memory, b) it is more difficult to generalize, for example I could have a middleware that returns the correct db for the current request, and then I can just use my models without caring on anything else.

[baoshan]

@manast

I agree with you that caching db objects are more lightweight for the number of objects. But I'm not sure if it will save a lot of memory because I made no investigation on that. If it a model simple wraps the db objects and with little instance properties, I think the memory saving is barely margin. As I said, I'm not sure, it may save a lot of memory, it may not.

Currently, I have a simple middleware (express middleware here) which return the cached models according to the subdomain (the subdomain denotes the tenant).

Then, for the 99 percents of my code, I have a set of models which is bounded to the requesting tenant.

For my projects, one app server only serves tens of tenants, and the solution fits my current requirement and doesn't compromise my developing experience.

I guess your application server needs to serves hundreds of thousands of tenants, am I right? Then you have to some test I think.

[manast]

@baoshan I am trying to create multiple database connections with mongoose according to #1601 but this functionality does not seem to exists anymore. How do you create your cached models per database as you mentioned above?

[manast]

@baoshan ah, ok. They just changed the api but it is still done with createConnection.

[memelet]

@manast, Have you found any way to use mongoose for your use case. In our case we have hundreds of tenants that all use the same schema. If I understand from the above, the proposed solution is to create thousands of models then cache them myself somehow?

[manast]

@memelet Yes, we have been running with this solution for a few months now. we have like max 10 tenants so there has not been any performance issue. Basically what you need to do is this:

  var db = mongoose.createConnection(uri);
  db.Schema = mongoose.Schema;

And then access every model with:

var User =  db.models.User;
etc

So it requires a new mongoose connection, unless mongoose multiplexes the connection it will result in several connections to mongodb, but still, having a few hundreds should be feasible, connections are not that expensive anyways.

[manojsethi]

@manast Can you please provide a little detailed example or if you have found any better approach now?