Mongoose: Cursor operator in Aggregate function not working..

Created on 15 Sep 2014  路  15Comments  路  Source: Automattic/mongoose

Hi,

I have the latest mongoose 3.8.16 with mongodb 2.6.4

I am trying to do cursor operation on an aggregate pipeline.

My function looks like,

var where = {  "storeId": storeId };
MongoBasket.aggregate()
                            .match(where)
                            .group({
                                "basketProducts": {
                                    "$push": "$basketProduct"
                                },
                                "basketIds": {
                                    "$push": "$basketId"
                                },
                                "storeId": {
                                    "$addToSet": "$storeId"
                                },
                                "consumerIds": {
                                    "$push": "$consumerId"
                                },
                                "basketTransactionCardTotal": {
                                    "$push": "$basketTransactionCard"
                                },
                                "basketTransactionCashTotal": {
                                    "$push": "$basketTransactionCash"
                                },
                                "totalTip": {
                                    "$push": "$tip"
                                },
                                "_id": {
                                    "year": {
                                        "$year": "$completed"
                                    },
                                    "month": {
                                        "$month": "$completed"
                                    },
                                    "day": {
                                        "$dayOfMonth": "$completed"
                                    },
                                    "hour": {
                                        "$hour": "$completed"
                                    }
                                }
                            })
                            .sort({_id: 1})
                            .cursor({
                                batchSize: 200000
                            })
                            .exec(function(err, result) {
                                 console.log(err, result);
                            }, req.messageInstance);

I printed the raw mongodb query that is built by mongoose (using mongoose.set('debug', true)) which looks like

baskets.aggregate([
    {
        '$match': {
            storeId: 11
        }
    },
    {
        '$group': {
            _id: {
                hour: {
                    '$hour': '$completed'
                },
                day: {
                    '$dayOfMonth': '$completed'
                },
                month: {
                    '$month': '$completed'
                },
                year: {
                    '$year': '$completed'
                }
            },
            totalTip: {
                '$push': '$tip'
            },
            basketTransactionCashTotal: {
                '$push': '$basketTransactionCash'
            },
            basketTransactionCardTotal: {
                '$push': '$basketTransactionCard'
            },
            consumerIds: {
                '$push': '$consumerId'
            },
            storeId: {
                '$addToSet': '$storeId'
            },
            basketIds: {
                '$push': '$basketId'
            },
            basketProducts: {
                '$push': '$basketProduct'
            }
        }
    },
    {
        '$sort': {
            _id: 1
        }
    }
]){
    cursor: {
        batchSize: 200000
    }
}

Please note that cursor operator is outside the aggregate () enclosure

]){
    cursor: {
        batchSize: 200000
    }
}

Shouldn't it be like

],
{
    cursor: {
        batchSize: 200000
    }
})

My hand written raw mongodb query for mongo shell works really well and here is how it looks

db.baskets.aggregate([
    {
        "$match": {
            storeId: 11
        }

    },
    {
        "$group": {
            "_id": {
                "year": {
                    "$year": "$completed"
                },
                "month": {
                    "$month": "$completed"
                },
                "day": {
                    "$dayOfMonth": "$completed"
                },
                "hour": {
                    "$hour": "$completed"
                }
            },
            "totalTip": {
                "$push": "$tip"
            },
            "basketTransactionCashTotal": {
                "$push": "$basketTransactionCash"
            },
            "basketTransactionCardTotal": {
                "$push": "$basketTransactionCard"
            },
            "consumerIds": {
                "$push": "$consumerId"
            },
            "storeId": {
                "$addToSet": "$storeId"bas
            },
            "basketIds": {
                "$push": "$basketId"
            },
            "basketProducts": {
                "$push": "$basketProduct"
            }
        }
    },
    {
        "$sort": {
            _id: 1
        }
    }
],
{
    cursor: {
        batchSize: 2000000
    }
})

Is it a bug or am I missing something?

If someone could shred some light on it would be much appreciated.

Many thank,

Karthik

Most helpful comment

@Ra1da35ma well that's embarrassing, looks like we closed out #2955 by mistake. Thanks for pointing that out, re-opened and will be fixed :+1:

All 15 comments

I have the same problem with the same versions.

This works:

return Rx.Observable.fromPromise(Meeting.aggregate(pipeline).exec());

This doesn't work:

return Rx.Observable.fromPromise(Meeting.aggregate(pipeline).cursor({ batchSize: 1000 }).exec());

Thanks for adding it to your milestone.

Many thanks,
Karthik

Hi, Any progress on this?

Some progress, but not much unfortunately. We've figured out what's wrong, just haven't had the time to fix it yet.

+1. Any progress on this?

So this issue is closed, what sort of problem are you having?

This works:

Model
.aggregate(pipeline)
.exec(function () {
  console.log(arguments);
})

And this doesn't: (never calls back)

Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec(function () {
  console.log(arguments);
})

Also, the following doesn't return a promise:

Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec()

Am I missing something?

Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec()

returns a cursor, as specified in 4.0 release notes. Admittedly this is a not-well-designed API, it's something that was thrown in at the last minute as a stopgap. The right way to do it would probably be something like this:

var stream = Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec().stream();

stream.on('data', function(doc) {
  // ...
});

See mongodb driver cursor docs

Ok cool, thanks for that! For the callback approach, the docs currently say this:

screen shot 2015-05-03 at 08 46 58

This is incorrect, right?

Yep those docs are incorrect, thanks for pointing that out

Anyone looking at this may also be interested in https://github.com/Automattic/mongoose/issues/3160 which outlines another bug about getting an undefined response from exec when a cursor is used.

@vkarpov15 its almost 2018 and the docs are still misleading. someone help 馃

@Ra1da35ma well that's embarrassing, looks like we closed out #2955 by mistake. Thanks for pointing that out, re-opened and will be fixed :+1:

I returned to the version mongoose": "4.13.7 and everything is normal .. using mongo 3.6

This works:

Model
.aggregate(pipeline)
.exec(function () {
  console.log(arguments);
})

And this doesn't: (never calls back)

Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec(function () {
  console.log(arguments);
})

Also, the following doesn't return a promise:

Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec()

Am I missing something?

This Answer + This One:

I returned to the version mongoose": "4.13.7 and everything is normal .. using mongo 3.6

Solved my problem. Thanks!

Was this page helpful?
0 / 5 - 0 ratings

Related issues

varunjayaraman picture varunjayaraman  路  3Comments

CodeurSauvage picture CodeurSauvage  路  3Comments

jeneser picture jeneser  路  3Comments

gustavomanolo picture gustavomanolo  路  3Comments

lukasz-zak picture lukasz-zak  路  3Comments