Mongoose: Problem with connection to db , queries hang. Reverting to mongoose 4.6.6 solves it

[dariiiannn]

Mongoose 4.7.7

Do you want to request a feature or report a bug?
I want to report a bug

What is the current behavior?
All request towards the db just hang and do not return anything

If the current behavior is a bug, please provide the steps to reproduce.

const ODconnection = mongoose.createConnection(config.mongo.uri, config.mongo.options);
ODconnection.on('error', err => {
  logger.error(err);
});
module.exports.ODconnection = ODconnection;


var connection = require('../../config/connections').ODconnection;
var ProgramSchema = new mongoose.Schema({...});
connection.model('Program', ProgramSchema);

All queries hang and return nothing
Program.find() will just hang

Fixed if I downgrade to [email protected]
The change in 4.7.7 is a dependency version update [email protected] to [email protected]. I am guessing that is was is causing the issue

Please mention your node.js, mongoose and MongoDB version.
mongoose 4.7.7 node 6.9.2 MongoDB 3.2.3

[vkarpov15]

What does your URI look like? Please make sure you remove any sensitive info. The big change in muri 1.2.0 was passing username and password through decodeURIComponent

[actionnick]

This is also happening to me.

[gfzabarino]

Happening to me as well. Maybe you meant switching to 4.7.6 @dariiiannn?

[gfzabarino]

My URI is something like:

mongodb://admin:[email protected]:27017,circolo-production-shard-00-01-qpdcn.mongodb.net:27017,circolo-production-shard-00-02-qpdcn.mongodb.net:27017/circolo?ssl=true&replicaSet=Circolo-Production-shard-0&authSource=admin

What happens is that the server hangs when connecting to the database. Rolling back to 4.7.6 makes it work.

[vkarpov15]

@gfzabarino does your password have any non-alphanumeric characters, like %?

[gfzabarino]

No, just alphanumerical characters.

[gacyun]

Confirmed, we have the same issue, after upgraded to 4.7.7, server pm2 process will be 100% CPU usage:

we are trying to trace this with strace on our server, step by step to positioning this problem, and finally confirmed caused by mongoose, this is the system call status when this happening, very like an infinite loop:

we are using MongoDB HA mode with replica set failover, the connection URI like:

mongodb://***:****@ze1a3dbb36ef5642.xxx.net:3717,ze1a3dbb36ef5641.xxx.net:3717/admin?replicaSet=mgset-1405217

after we go back to 4.6.6, thanks @dariiiannn :

[gfzabarino]

@dariiiannn @sonicgao May I ask the reason of why you went back to 4.6.6? Doesn't it work for you with 4.7.6?

[szenti]

@gfzabarino maybe they were upgrading from 4.6.6.

At our team, we have the same issue: 4.7.7 causes high CPU and increased RAM usage. After having downgraded to 4.7.6 the problem is solved.

[gfzabarino]

Yes, that was my experience as well.

[szenti]

Unfortunately we can't reproduce it (yet) by running tests locally, only after deploying our service to our staging environment.

[jrsalgado]

@sonicgao same problem here getting same results, actually i noticed that the app freezes after 2 hours even on high cpu instances

[gacyun]

@gfzabarino 4.7.6 working well, the only broken version here is the latest.

[gacyun]

Another point is the process is not completely not working, it's working but very slow until cpu to 100%, then the process died.

[szenti]

I've just tried to reproduce the problem by setting the version directly to the commits between the release 4.7.6 and the 4.7.7. The first version that produced the error was commit _chore: upgrade mongodb -> 2.2.18_ (f47260b002c4accc6f2b77b8042012d352c81618). I was able to temporarily fix it by clicking on _Reset Dependency Cache_ on CodeShip. Our service started to work again, but the next commit (_chore: actually bump to 2.2.19_, e4fb16cdf0abd5b847df0b8979d019c64c5c86a9) caused error again. This time _Reset Dependency Cache_ couldn't help it.

The used Node version is 6.9.1.

[tommarien]

Maybe related to https://github.com/christkv/mongodb-core/issues/156, i think mongodb 2.2.22 fixes it, well apparantly not entirely. Indeed it's related to mongodb >=2.2.17 in combo with replication

[jrsalgado]

mongoose = 4.7.8, still have this issue

[dariiiannn]

@gfzabarino sorry for the late reply, we reverted to that because if was the latest version that previously worked for us.

Connection string looks like mongodb://user:[email protected]:27017,xxx.xxx.com,xxx2.xxx.com,xxx3.xxx.com/db-dev?replicaSet=rs0

[sitegui]

Like many already pointed out, both 4.7.8 and 4.7.7 have this problem, while 4.7.6 does not.

For us, the hanging was not always reproducible. After restarting the application, we observed like a 60% chance it would hang all reads and writes to DB from that point on. In the fraction that worked, the process would hit 100% CPU after some time.

We tested both against MongoDB 3.2.11 and 3.4.1

[sitegui]

It seems to be a mongodb driver bug 🐞

[email protected] uses [email protected] while [email protected] uses [email protected].

Forcing [email protected] to downgrade to [email protected] fixes the problem for me.

This JIRA issue seems to be related: https://jira.mongodb.org/browse/NODE-917, but I'm not totally positive.

[szenti]

@sitegui I can confirm that. I've had forked the mongoose and downgraded mongodb to 2.2.16 in it, then changed our project's dependency to use my fork. It worked without any problems.

@vkarpov15 is it possible to temporarily downgrade the mongodb version to 2.2.16 until the mongodb issue gets sorted out? Thank you in advance.

[vkarpov15]

@szenti does 4.7.9 have the same problem? Currently on 2.2.22. Would rather not downgrade unless 2.2.22 has trouble.

[szenti]

@vkarpov15 4.7.9 seems to have fixed the problem for our service, thank you. If I encounter any further problems, I'll let you know.

[abondoa]

I know this issue is closed and maybe I should create a new issue instead, if that is the case let me know, and I will do that instead.

We are experiencing this problem on all versions > 4.7.6, including 4.7.9 and the newest 4.9.1.

I might add that we are seeing the error sporadically - sometimes our service will startup just fine and connect to mongo as expected, other times we get the error, similar to what @sitegui reported. I have not been able to notice any pattern. Haven't seen any suspicious CPU activity though. When the problem occur I have noticed that the events 'connected' and 'open' does not seem to be fired. When connecting to a single mongo (when developing locally), there is no problem at all.

Our URI string looks like this:
mongodb://db.xxx.co:41930,db.xxx.co:41931/masterDb

We are connecting through mongoose.createConnection, because we have multiple database connections that will be created after the masterDb and will have to be kept open concurrently.

We specify username and password along with other options in the options object when calling createConnection.

Our current workaround is to shrinkwrap the native mongo driver back to 2.2.16 (since we already started using feature only available in >= 4.8). As suggested by @sitegui.

Anyone has any input or better alternatives?

Node version: v6.9.5
Mongo: 3.2.11
Mongoose: >= 4.7.7 (tested on 4.7.7, 4.7.9, 4.8.0, 4.8.6, 4.9.1)

[abondoa]

Seems the trigger to our problem was that we were using connectWithNoPrimary: true in our configuration. changing that to false, seemed to fix it.

[marcusjwhelan]

So why is this closed @vkarpov15 ? still have his issue in mongoose 5

[marcusjwhelan]

Even Without any password. Just login in with the URL. Had to switch back to 4.6.6. I was at 5.0.1 or something

[vkarpov15]

@marcusjwhelan please provide code samples, there's a lot of potential explanations for this issue, some of which are just expected behavior, so without more information we don't have any way to help.

[marcusjwhelan]

@vkarpov15

config = {
mongodb: "mongoDBurl",
viewsPath: "./lib/views",
logging: {
  name: "name",
  streams: [
    {
      stream: process.stdout,
      level: 'info'
    },
    {
      stream: new BunyanSlack({
        webhook_url: "hookUrl",
        channel: 'alerts',
        username: 'name'
      }),
      level: 'error'
    }
  ]
},
port: 3000
}

Then in the app.js file that starts the application

mongoose.connect(config)

node - 8.11.1
mongodb - 3.0.15
Ubuntu - 14.x

[vkarpov15]

@marcusjwhelan those aren't mongoose connection options. Are you using mongoose.connect(config.mongodb) perhaps? And, if so, can you show your connection string with the sensitive details (username, password, host names) omitted? I want to see the general format of the connection string.