Mongoose: 4.7.1 => 4.7.2 : MongoError connection timed out

[dcolens]

When testing [email protected] / [email protected], I get:

MongoError: connection 2 to XXXXXX timed out
at Function.MongoError.create (/node_modules/mongodb-core/lib/error.js:29:11)
at Socket. (/node_modules/mongodb-core/lib/connection/connection.js:186:20)
at Socket.g (events.js:286:16)
at emitNone (events.js:86:13)
at Socket.emit (events.js:185:7)
at Socket._onTimeout (net.js:333:8)
at tryOnTimeout (timers.js:224:11)
at Timer.listOnTimeout (timers.js:198:5)' } 'connection 2 to XXXXXX timed out

this does not happen with [email protected] / [email protected].

More details:

• query: aggregate with allowDiskUse(true).read('secondaryPreferred').cursor()
• timing: the error happens about 35s after sending the query (before first results are received)
• node version: v6.3.0
• npm version: 3.10.9
• connect options:

{
 "socketOptions": {
    "socketTimeoutMS": 240000,
    "keepAlive": 10000,
    "connectTimeoutMS" : 30000
} 

• query:

[
    { $match: {
            _id: { $lt: oid },
            signature: { $exists: true }
        }
    },
    { $group: { 
            _id: { myid: '$myid', signature: '$signature' },
            count: { $sum:  1 },
            myid: { $first: '$myid' },
            docs: { $push: { _id: '$_id', name: '$name' } }
        }
    },
    { $match: {
            count: { $gt : 1 }
        }
    }
]

[silentjohnny]

I'm experiencing the same problem, [email protected], [email protected]

• setting socketTimeoutMS to higher values or 0 makes no difference
• setting connectTimeoutMS to higher values or 0 makes no difference
• no "close" or "reconnected" event is fired, only a "timeout", which results in a Mongoose error in the callback

Rolling back to [email protected] / [email protected] solves the issue, i.e., does not cause timeouts

From [email protected] to 2.2.12 the timeout event listener changed from "once" to "on", maybe this has something to do with it?

Or should we post this as an issue in the mongodb native driver?

[k14hatake]

@silentjohnny Getting the same issue with [email protected]

[vkarpov15]

I'd post an issue in the mongodb native driver, mongoose doesn't do much in the way of initial connection.

[vkarpov15]

Here's the mongodb driver jira

[tomgrossman]

Weird. I am experiencing the same issue but with [email protected] and higher which are using [email protected]
With [email protected] and [email protected] it does work!.
My aggregate query is without any options so I don't know, maybe that's the difference between me and the original issue @dcolens .

Anyone opened an issue to the native driver? so I can follow it...
Right now I'm staying with [email protected]

[silentjohnny]

[email protected] updated to [email protected] which includes a fix for monitoring operations timing out and generating a timeout event on running operations @1284917

Soon Mongoose updates to [email protected], I will give it a try.

[steve-p-com]

Same issue for me, and I'm not even sure if it's just long running aggregates that are timing out, or if it's random and not a real timeout at all. It seems like it times out almost straight away sometimes, but other times it runs for 30 seconds or so. In this particular case, I have four aggregate.execs running at once.

Reverting to [email protected] and [email protected] fixes it so I'm doing that for now.

[varunjayaraman]

This should hopefully be fixed in the next release since mongoose 4.7.7 uses [email protected]. Either way, this isn't really a mongoose issue like @vkarpov15 said but more to do with mongodb. I'm going to close this for now, but if you still get the same issue after the next release, you should open up a ticket or a PR on mongodb's end.

[steve-p-com]

I think there may be more to this issue than meets the eye. I've had to revert to 4.4.20 to completely stop these aggregate timeouts occurring under all circumstances. 4.7.x and 4.6.x both had bogus very quick timeouts under some circumstances of multiple concurrent executions. I skipped 4.5 and went straight back to a known good version (4.4.20) which we are using in production, and that has completely solved the problem, but TBH I'm still mystified by what is happening. I'm sorry I don't have any further data to add, but I'm not sure if it is solely a problem with the version of the native mongodb driver that mongoose is using, or if there's more than one contributory factor at work.

[flosky]

Having the same problem with [email protected] and [email protected]. Timeouts are really weird. This one is pretty short (around 5s) and very reproducible but other queries take longer than 5s and do not time out

[steve-p-com]

I'm certain the timeouts are often bogus. In one scenario I was looking at I started four simultaneous aggregate.exec operations which take around 30 seconds. One timed out in 8 seconds, one timed out in 20 seconds, but the other two completed normally in 30 seconds. Repeating caused a different timeout order, but always a similar result. I suspect timeouts are incorrectly being set on the wrong connections, or are not being removed when an operation completes so that they fire unexpectedly on a subsequent exec. Something like that anyway. Unfortunately I've no time at the moment to pursue it.

[silentjohnny]

[email protected] seems to have solved the timeouts for me. @steve-p-com like you are suggesting, the timeouts in 4.7.2 were indeed bogus and caused error events being emitted on the wrong connection, this was solved by [email protected] and so [email protected] should work fine in your case. Did you run a test with that version?

Haven't tested 4.7.9 or 4.8.0 yet, maybe [email protected] introduced a regression, which would explain the issue @flosky is experiencing. Does that timeout also happen on [email protected]?

[steve-p-com]

I have retested with 4.8.1 today, and have not had any issues with unexpected timeouts. I removed all npm caches and yarn caches first, then zapped node_modules, then reinstalled everything. So far, so good.

[thenitai]

Happening for me with 4.8.1 as well. I went back many versions on Mongoose and MongoDB and get the issue all the time.

[varunjayaraman]

@thenitai do you have more context / can we see how you set up your mongoDB connection?

[thenitai]

@varunjayaraman I found that our domain error code is catching the error and thus restarts our app. However, the issue is still there.

Not quite sure what you need to see in the set up of our connection. Do you mean the connection options? If so, they are:

var dbOptions = {
            db: { native_parser: true },
            server: {
                auto_reconnect: true,
                socketOptions: {
                    keepAlive: 1, 
                    connectTimeoutMS: 300000,
                    socketTimeoutMS: 300000
                }
            }
        };

[gitawego]

I've same problem as well. even with latest version v4.8.4

just found the prob. in my case, it's because the server memory usage is too high (always 95%~99%), after increased memory, no problem anymore.

[adrienbaron]

I'm seeing this issue too on my server with Mongoose 4.7.7, MongoDB 3.4.1 and Node 4.7.2. When memory usage is high on my server, timeouts seems to occur randomly in series:

CosWebsite-27 MongoError: connection 42 to 127.0.0.1:27017 timed out
CosWebsite-27     at Function.MongoError.create (/home/cos/cos/node_modules/mongodb/node_modules/mongodb-core/lib/error.js:29:11)
CosWebsite-27     at Socket.<anonymous> (/home/cos/cos/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:186:20)
CosWebsite-27     at Socket.g (events.js:260:16)
CosWebsite-27     at emitNone (events.js:67:13)
CosWebsite-27     at Socket.emit (events.js:166:7)
CosWebsite-27     at Socket.wrapped (/home/cos/cos/node_modules/newrelic/lib/transaction/tracer/index.js:183:28)
CosWebsite-27     at Socket.wrappedEmit [as emit] (/home/cos/cos/node_modules/newrelic/lib/transaction/tracer/index.js:220:46)
CosWebsite-27     at Socket._onTimeout (net.js:333:8)
CosWebsite-27     at _runOnTimeout (timers.js:537:11)
CosWebsite-27     at _makeTimerTimeout (timers.js:528:3)
CosWebsite-27     at Timer.unrefTimeout (timers.js:597:5)
... times 6
CosWebsite-27 GET /clans/compas-c-r-98QP9J2G/members 500 10569.300 ms - 9893
CosWebsite-27 GET /players/1000-99JQVQ9VU 500 12388.484 ms - 9849
CosWebsite-27 GET /players/R0YUPPRR/profile 500 8204.622 ms - 9857
CosWebsite-27 GET /players/UG8YJUJY/profile 500 4622.819 ms - 9857
CosWebsite-27 GET /clans/next-state-P8RYGQYV 500 11526.859 ms - 9861
CosWebsite-27 GET /clans/YY2CCUVV 500 6755.380 ms - 9817

As you can see the requests none of the request lasted more than 12s. My mongoose connector is set up this way:

const mongoDB = {
    uri: "mongodb://127.0.0.1:27017/XXX",
    options: {
        host: '127.0.0.1',
        port: '27017',
        database: "XXX",
        compression: false,
        server: {
            poolSize: 5,
            auto_reconnect: true,
            socketOptions: {
                socketTimeoutMS: 0,
                connectTimeoutMS: 0
            }
        },
        promiseLibrary: Promise
    }
};

Any idea what could be going wrong?

[kekedaine]

@adrienbaron Me too!!!
someone please help me.

[flosky]

Has someone files a bug report to the developers of the native mongo driver?

[artofspeed]

does this issue still exist in 4.9.1?

[hellivan]

Yes, issue still exists in 4.9.1....

[email protected]
[email protected]
[email protected]

[drawrowfly]

Had same issue today after updating to the latest mongoose version, than i downgraded back to 4.7.1 and everything is working perfect!

[flosky]

The issue has been unresolved for some time now.
What should we do to fix this?
At some point I would like to update to the recent version but I can only do that when this has been resolved.

[centigrade-thomas-becker]

I updated Mongoose from 4.4.17 to 4.7.9 about a month ago. Since then a daily cronjob has errored three times now, always with a mongo connect timeout error.

I'll switch to Mongoose 4.7.1 and see if this helps.

edit: the error message

2017-04-19 03:09:23: You have triggered an unhandledRejection, you may have forgotten to catch a Promise rejection:
MongoError: connection 261 to localhost:27017 timed out
    at Function.MongoError.create (/var/www/someproject/node_modules/mongodb-core/lib/error.js:29:11)
    at Socket.<anonymous> (/var/www/someproject/node_modules/mongodb-core/lib/connection/connection.js:186:20)
    at Socket.g (events.js:286:16)
    at emitNone (events.js:86:13)
    at Socket.emit (events.js:185:7)
    at Socket._onTimeout (net.js:333:8)
    at tryOnTimeout (timers.js:224:11)
    at Timer.listOnTimeout (timers.js:198:5)

[adrienbaron]

@centigrade-thomas-becker I had the same issue for more than a month in production, I switched back to 4.7.1 yesterday, no more issues!

[zqWu]

same problem occur @ 4.7.7

[centigrade-thomas-becker]

With Mongo 4.7.1 there are fewer problems than with 4.7.9, but I still get the occasional error message when saving to the MongoDB:

2017-04-30 03:09:32: Error: connection timeout
    at .<anonymous> (/var/www/someproject/node_modules/mongoose/lib/drivers/node-mongodb-native/connection.js:168:17)
    at emitTwo (events.js:106:13)
    at emit (events.js:191:7)
    at listener (/var/www/someproject/node_modules/mongodb/lib/db.js:1791:14)
    at emitOne (events.js:96:13)
    at emit (events.js:188:7)
    at .<anonymous> (/var/www/someproject/node_modules/mongodb/lib/server.js:270:14)
    at g (events.js:286:16)
    at emitOne (events.js:96:13)
    at emit (events.js:188:7)
    at .<anonymous> (/var/www/someproject/node_modules/mongodb-core/lib/topologies/server.js:322:12)
    at emitOne (events.js:96:13)
    at emit (events.js:188:7)
    at .<anonymous> (/var/www/someproject/node_modules/mongodb-core/lib/connection/pool.js:271:12)
    at g (events.js:286:16)
    at emitTwo (events.js:106:13)
    at emit (events.js:191:7)
    at Socket.<anonymous> (/var/www/someproject/node_modules/mongodb-core/lib/connection/connection.js:175:10)
    at Socket.g (events.js:286:16)
    at emitNone (events.js:86:13)
    at Socket.emit (events.js:185:7)
    at Socket._onTimeout (net.js:333:8)
    at tryOnTimeout (timers.js:224:11)
    at Timer.listOnTimeout (timers.js:198:5)

The error occurs on a Linux VM running Ubuntu 15.04

[xeroxoid]

Same here with:

├─┬ [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected]

mongo version @ 3.2.11

on AWS EC2.

Any update/workaround? Would keepAlive: true have any positive effect?

[ht2]

Not sure if this helps anyone, but in our case we noticed that we were passing in our timeout params (socketTimeoutMS etc) in through options.server. This was _wrong_! We are using a replica set and as such our options should have as below:

In our case the application actually needed the ability to connect to either a replica or normal server, so we did:

const serverOptions = {
  poolSize: 100,
  socketOptions: {
    socketTimeoutMS: 60000
  }
};

mongoose.createConnection(dbpath, {
  server: serverOptions,
  replset: serverOptions
});

Hope that helps someone!

[RemeAjayi]

Hello Everyone,
I am using mlab to host my database. I get this error
Mongo DB connectionerror: {[MongoError: connection 0 to ds155841.mlab.com:xxxxx timed out'}

[cyrilchapon]

@RemeAjayi I think this has nothing to do with this ticket.
You should try / create another one after digging into issue.

@ht2 you saved my day ❤️

[cyrilchapon]

Is there any viable solution to original issue ?

[cyrilchapon]

In case it helps anyone, I experienced this issue when parallelizing 2 brutal find() queries (using Promise pattern).

Turning

Promise.all([
  col1.find({longQuery: true}),
  col2.find({longQuery: true})
])
.spread(function(result1, result2) {
  //stuff
});

into a sequential approach

col1.find({longQuery: true})
.then(function(result1) {
  return Promise.all([
    result1,
    col2.find({longQuery: true})
  ]);
})
.spread(function(result1, result2) {
  //stuff
});

Made it work

[gustvao]

@cyrilchapon thought it was working for me but your sequential approach did not solve my issue :/

it is intermittent.. sometimes it works, sometimes it times out..

[cyrilchapon]

Why this silent close ?

[centigrade-thomas-becker]

A follow-up to my previous comment https://github.com/Automattic/mongoose/issues/4789#issuecomment-298849907

I moved the offending database from a virtual machine to a docker container with brand new linux, nodeJS and mongoDB. I created a stress test that does nothing but truncate and refill the database all day. I found that about 1 in 6000 test runs result in a timeout error.

[gustvao]

I had I guess the same experience after my last comment, forgot to mention.

But updating node to its one of its latest versions along with all the other deps apparently did the trick.

Now with the comment above I believe it is a matter of legacy error

[mzahidriaz]

I was also having the same issue with mongo 3.6.2 and mongoose 4.9.2
I fixed it by passing additional connection parameters

MONGO_URI=mongodb://user:[email protected]:27017/dbname?keepAlive=true&poolSize=30&autoReconnect=true&socketTimeoutMS=360000&connectTimeoutMS=360000

[MinhNguyen41092]

@mzahidriaz Thank you a lot your solutions works for me but can you explain or refer a link to how you chose those parameters to solve this problem?

[vkarpov15]

@MinhNguyen41092 right now the only docs are the MongoDB driver docs http://mongodb.github.io/node-mongodb-native/3.1/api/MongoClient.html#connect . we'll add details about this to http://mongoosejs.com/docs/connections.html

[binki]

For those experiencing the intermittent errors, I had those too on mongoose-4.13.11 with mongodb-2.2.35 mongodb-core-2.1.19 in a particular environment. Various queries at different points in our testsuite, though mostly testing one particular part of our app which has heavier queries and does do things in parallel IIRC, were all getting this error message. And the whole test suite does not even take 15s to run, so there’s no way I should have been hitting the built-in default 30s timeouts in the first place:

     MongoError: connection 0 to localhost:27017 timed out
      at Function.MongoError.create (node_modules/mongoose/node_modules/mongodb-core/lib/error.js:29:11)
      at Socket.<anonymous> (node_modules/mongoose/node_modules/mongodb-core/lib/connection/connection.js:200:20)
      at Socket._onTimeout (net.js:448:8)

This only happened in our CI test build server. I couldn’t reproduce it on my dev machine.

(In my own system (private repo, this is for my own reference in case ever need to dig this up again), this error shows up at least in build numbers are 358, 356, 355, 352).

After turning on useMongoClient: true, the tests seem to pass reliably in the CI environment so far. I have had three successful runs with just that change whereas many prior sequential runs fail due to the weird timeout error.

I think @thenitai @steve-p-com @adrienbaron were experiencing the weird bogus timeouts too. I think the error was happening to you guys before 4.11 was out which even brought useMongoClient: true as an option. Can you tell if that made the difference for your workloads?

[SergeyVatz]

[email protected]
[email protected]
[email protected]
Have the same problem, 1.5M errors in roll bar for a couple of days, with what it can be connected? How resolve?

[vkarpov15]

@SergeyVatz see response to your question on #5376