Mongoose: Model.find() is not returning all documents

[gasteve]

With a particular model, I have ~120 documents. When I run model.find({}, function(err, list) {…}); some documents for that model are not returned. If I add a clause to the query, I can make it return certain matching documents that are not returned with an empty query. The problem can also occur when I specify a query parameter that matches many documents (one or two are missing from the returned results).

I'm curious if others have experienced this problem and how they fixed it. Is it a mongoose issue, the mongo native module, or mongodb itself? What did others do to fix it?

I am running mongoose 1.3.7 and mongodb 1.8.1 on Ubuntu Linux.

[aheckmann]

are you using a sparse index?

[gasteve]

I am not using a sparse index, but I do have indexes.

I have 3 indexes for this schema…all three fields are of the type String, 2 are unique indexes and 1 is non unique... they are defined in the schema as follows:

field1: {type: String, unique: true},
field2: {type: String, unique: true},
field3: {type: String, index: true},

Note that "theModel.find({}, function(err, list) {…})" omits some of the documents for this model…however if I look up the document using one of the unique indexes, it finds returns it (so I know it's there). I don't believe the document will be returned if I query using one of the non indexed fields. So, it's behaving like the document appears in the index, but that it is not in whatever document collection that is being maintained for this schema as a whole. Is there any way to rebuild such a collection?

[gasteve]

Actually, with further testing, I see that in some cases if I add a clause using a non-indexed field, more of the documents are returned…in one case, with a clause "{activated: true}" I'm getting 101 documents back, but with an empty clause, I only get 82…so there are a significant percentage of documents that are missing from the result set when an empty clause is used.

[aheckmann]

care to share your schema and queries?

[gasteve]

They are rather large and uninteresting…the relevant info is here. One question though…over time we've added fields to the schema…we've not removed any fields or indexes. We didn't do anything special to migrate instances in the DB. Could this cause these symptoms we're experiencing? Do we need to perform some kind of maintenance in the DB when changing the schema?

[aheckmann]

no maintenance should be necessary.

I can't help any further if more information is not provided b/c I cannot reproduce.

[gasteve]

I upgraded to the latest mongoose, mongo plugin, and mongodb and I'm still having this issue. One thing I've noticed is that I am getting the following message in the mongodb console:

Wed Nov 23 13:42:09 [conn5] getMore: cursorid not found …

Also, I've confirmed that there are 166 documents for the collection using the mongo command line tool…however, the find is only returning 101 documents.

[aheckmann]

So the mongo shell is returning 166 docs but mongoose only returns 101? If so, can you test the mongodb-native driver directly by running it like so:

model.collection.find({}, callback)

the docs returned won't be instances of your model but this will let us see if its a -native issue or a mongoose issue.

also, some googling makes me think it may be timing out? http://groups.google.com/group/mongodb-user/browse_thread/thread/88246dffd68dd117/8efb39c093e5c60e?show_docid=8efb39c093e5c60e

if thats the case you may wanna try Model.find(query, { timeout: 999999 }, callback)

this is a tricky one. hopefully we nail it.

[gasteve]

Ok, I couldn't get model.collection.find({}, callback) to work…the second arg to the callback isn't an array of results…not sure what it was. However, I went to the mongo-native documentation and copied used this example (substituting my db and collection and bumping up the limit on results):

var mongodb = require('mongodb');
var server = new mongodb.Server("127.0.0.1", 27017, {});
new mongodb.Db('test', server, {}).open(function (error, client) {
  if (error) throw error;
  var collection = new mongodb.Collection(client, 'test_collection');
  collection.find({}, {limit:10}).toArray(function(err, docs) {
    console.dir(docs);
  });
});

I get 166 results when I do this and I do not get any messages about cursorid on the mongodb console…it doesn't appear that mongo-native is the culprit. I think I'll have a look at what the mongoose code is doing in the find() method and see if can spot anything. I can't help but think I must be doing something atypical, or else others would surely have encountered this.

Thanks for you help.

[aheckmann]

ah yeah, forgot about -native returning cursor. cool, let me know

[gasteve]

ok…I'm sneaking up on it…I switched to using the model.find().each() protocol to see if I hit the problem there as well. I did. Every time, after the same record, I get the cursorid not found output in the mongdb console and I get only 101 records. I looked at the code for the each() method and noticed the use of process.nextTick with the next function of the method. I decided to try it without using nextTick() and voila, I get all 166 records (and no message in the mongodb console). So, there must be something about the use of nextTick() that is causing mongodb to get confused about the cursorid (maybe it's a nodejs bug…searching).

[aheckmann]

sounding more like a driver issue. wish I had a dump of your collection to test against this weekend. care to send it to me? aaron.[email protected]

[aheckmann]

I mean "sounding more like a mongoose issue".

[gasteve]

Can't provide the data…it has sensitive customer information. I have a few more clues though…and I think it is in the -native module. I went back to the find() method and couldn't find any similar use of nextTick …so I dug further into -native and it appears that the initial result is returning 101 documents…with that result, it gets a cursorId…it then tries to do a getMore(), but mongo doesn't appear to like the cursorId and silently fails…so the query just appears to return 101 results, but in reality, there was a failure. If I manually tweak things such that the initial query requests 1000 results, then it returns the full set. I don't have a good explanation as to why nextTick affected the behavior of each() …I suspect it's a bug in the handling of the cursorId and it only manifests itself when getMore() is need. You might be able to replicate it by forcing the numberToReturn to a low number in Cursor.prototype.generateQueryCommand (in cursor.js in the -native module). But, as I said, it seems like it's definitely a -native issue, not mongoose.

[gasteve]

One other clue…I tried the iterative method again…with and without the use of nextTick() (in mongoose's query.each() method)…again when nextTick is used, I get 101 results and the bad cursorId message in the mongo console. When I removed nextTick(), I get the full 166 results…however, I noticed that in both cases the initial call fetches 101 results and in both cases, getMore() is being invoked. But, in one case, mongo is happy with the cursorId and the other it chokes. This must be some kind of concurrency issue causing the cursorId to get clobbered or closed. More digging.

[gasteve]

Ok..this might be a mongoose issue after all. What I've discovered is that the problem goes away if I put a delay after connecting to the DB. I think what might be happening is a timing issue between the initial connection establishment (and the querying and index setup) and my query. If I give the mongoose schema related queries a bit of time to run before my query runs, everything seems ok. I wonder if some of the initial mongoose setup is clobbering open cursors? If that's the case, maybe an event "initialized" or something could be triggered to indicate it's safe to start interacting with the DB. Also, if something like that is the case, it would be good to check to see whether cursors associated with other client processes get clobbered (that would be bad). The context in which I'm using this is command line utilities that I run against a production DB. If those utilities result in cursors in the main production server processes to get clobbered, that would not be good.

[aheckmann]

1) there is an event fired after mongoose connected.

var db = mongoose.createConnection(...)
db.on('open', callback)

2) when first connecting, mongoose ensures your indexes are created. I'll need to double check but it may be a race condition between this and the query that trips up mongodb.

so I can be certain I have the same setup for my tests, are you on mongoose 2.3.13 and mongodb 2.0.1?

[gasteve]

Yes, I'm on 2.3.13 and 2.0.1. I do wait on the 'open' callback…but it seems things are still happening after that (like ensuring those indexes). I suspect ensuring those indexes causes open cursors to get closed. If that's the case and it closes all open cursors for all connections (even other processes that are connected), I would suggest adding a way to connect to the DB without ensuring the indexes (in my case I would only do that when I bounce the main server process). If it only affects the cursors for the current DB connection, then I think all that's necessary is to make sure the 'open' event doesn't fire until after the indexes have been ensured (otoh, you might still want to provide control over it…index creation could take significant time in some cases and not always be necessary depending on what you're doing).

[aheckmann]

try commenting out the indexes in your schema and test it. that should narrow things down a bit since ensureIndex won't get called at startup.

On Nov 24, 2011, at 7:48 AM, [email protected] wrote:

Yes, I'm on 2.3.13 and 2.0.1. I do wait on the 'open' callback…but it seems things are still happening after that (like ensuring those indexes). I suspect ensuring those indexes causes open cursors to get closed. If that's the case and it closes all open cursors for all connections (even other processes that are connected), I would suggest adding a way to connect to the DB without ensuring the indexes (in my case I would only do that when I bounce the main server process). If it only affects the cursors for the current DB connection, then I think all that's necessary is to make sure the 'open' event doesn't fire until after the indexes have been ensured (otoh, you might still want to provide control over it…index creation could take significant time in some cases and not always be necessary depending on what you're doing).

---

Reply to this email directly or view it on GitHub:
https://github.com/LearnBoost/mongoose/issues/609#issuecomment-2863715

[gasteve]

That worked…I only commented out the indexes on the collection in question (though that doesn't necessarily mean cursors for other collections wouldn't get reset…since this is a timing issue). I'm going to look around in the mongodb documentation and see if it mentions anything about this.

[gasteve]

I did a little more testing…I've noticed that one of my indexes (String/unique) always rebuilds…I have two other indexes (both String, one unique and the other non unique)…neither of those two indexes ever rebuilds…I've tried enabling just one at a time and it's always that one index that gets rebuilt, never the other two (going to try and find out why an index would always get rebuilt). So, it seems this is only an issue when an index actually gets rebuilt as a result of ensureIndex().

[aheckmann]

Since we use ensureIndex it will attempt creation at each startup if it doesn't already exist. It must be failing for some reason, possibly a dup in the collection. If it fails, an error should be getting emitted on the connection, but it sounds like it may not be?

[gasteve]

There wasn't any indication of a failure in the console, however I just tried creating the index interactively via the mongo shell and discovered that indeed there is a duplicate and that caused it to fail. So, I guess for mongoose, the only suggestion I would have is to a) not fire the 'open' event until the ensureIndex() operations have completed and b) provide an option to open a connection without ensuring indexes (since I suspect the cursors for other clients could get clobbered by an index build and in a production environment you may need more control over when indexes do get built).

Thanks again for all your advice.

[aheckmann]

https://jira.mongodb.org/browse/SERVER-4462

[diversario]

Was this ever resolved in Mongoose or driver? I'm experiencing something similar to this.

I have a collection with the following model schema:

  var schema = new Schema({
    'userId': { 'type': String, 'required': true, 'index': { 'unique': true } }
  , 'props': [{
      'name': { 'type': String, 'index': true }
    , 'array': { 'type': [String], 'index': true }
  }]
  }, {'strict': true});

This model will be updated often and sometimes concurrently by multiple people. When running tests I found that find() returns null, null when I check for existing document with particular userId (so I know if I need to create one or to update existing doc). If I then run the same test on database from previous test run I get error 11000 about unique IDs and such.

I tried running mongoose.connection.db.ensureIndex('users', {'userId': 1}, {'unique': true, 'sparse': false}, function () {}); before find() but that didn't change anything.

[aheckmann]

its neither. see the MongoDB bug report above. its caused by the unique index creation failing and blowing away the current query cursor and happens at app startup.

[nmccready]

I can confirm this problem / bug on OSX and Ubuntu 12.04. Bug within MongoDB that is :P

[mitar]

I can confirm that indeed this is the problem of setting an unique index on field which is not unique. The issue is that there is no error shown about that. But yes, query then returns 101 results.

[aheckmann]

Please upvote/comment on the mongodb ticket to help raise its priority.

[mitar]

I did.

[ramseydsilva]

I can confirm, this is the case for me as well. If a unique index is set after creation of entries with non-unique/non-existent fields, the cursor doesn't return all values. It works if you narrow the query by querying on non unique fields.

[silverbucket]

Can also confirm that I'm only receiving 101 records when a limit is not specified, either with a simple find() or with conditions.

[totty90]

Any news? I'm making tests and the indexes are run after the queries or inbetween. I found that the open event is ran after the indexes are ensured.
http://mongoosejs.com/docs/api.html#connection_Connection

Is this issue closed already?

Edit: The indexes are not ensured when the open event is fired.

[vkarpov15]

@totty90 the server ticket unfortunately is still open. You can either disable index creation and just create them manually before your tests run, or you can wait for the Model.on('index') event to start your tests.

[totty90]

I've made a script to wait for the index event to fire. https://github.com/totty90/mongoose-connect

[vkarpov15]

Instead of using Model.on('index'), in mongoose 5 you can do Model.init().then(), Model.init() returns a promise that is fulfilled after index build is done, or immediately if autoIndex is disabled.

[MajidJafari]

For me, it was, part of documents timestamp fields type was Double and the other part of documents were in IsoDate format. So Mongoose filtered them out. Using aggregate instead of find saved me.