Hi,
I have the latest mongoose 3.8.16 with mongodb 2.6.4
I am trying to do cursor operation on an aggregate pipeline.
My function looks like,
var where = { "storeId": storeId };
MongoBasket.aggregate()
.match(where)
.group({
"basketProducts": {
"$push": "$basketProduct"
},
"basketIds": {
"$push": "$basketId"
},
"storeId": {
"$addToSet": "$storeId"
},
"consumerIds": {
"$push": "$consumerId"
},
"basketTransactionCardTotal": {
"$push": "$basketTransactionCard"
},
"basketTransactionCashTotal": {
"$push": "$basketTransactionCash"
},
"totalTip": {
"$push": "$tip"
},
"_id": {
"year": {
"$year": "$completed"
},
"month": {
"$month": "$completed"
},
"day": {
"$dayOfMonth": "$completed"
},
"hour": {
"$hour": "$completed"
}
}
})
.sort({_id: 1})
.cursor({
batchSize: 200000
})
.exec(function(err, result) {
console.log(err, result);
}, req.messageInstance);
I printed the raw mongodb query that is built by mongoose (using mongoose.set('debug', true)) which looks like
baskets.aggregate([
{
'$match': {
storeId: 11
}
},
{
'$group': {
_id: {
hour: {
'$hour': '$completed'
},
day: {
'$dayOfMonth': '$completed'
},
month: {
'$month': '$completed'
},
year: {
'$year': '$completed'
}
},
totalTip: {
'$push': '$tip'
},
basketTransactionCashTotal: {
'$push': '$basketTransactionCash'
},
basketTransactionCardTotal: {
'$push': '$basketTransactionCard'
},
consumerIds: {
'$push': '$consumerId'
},
storeId: {
'$addToSet': '$storeId'
},
basketIds: {
'$push': '$basketId'
},
basketProducts: {
'$push': '$basketProduct'
}
}
},
{
'$sort': {
_id: 1
}
}
]){
cursor: {
batchSize: 200000
}
}
Please note that cursor operator is outside the aggregate () enclosure
]){
cursor: {
batchSize: 200000
}
}
Shouldn't it be like
],
{
cursor: {
batchSize: 200000
}
})
My hand written raw mongodb query for mongo shell works really well and here is how it looks
db.baskets.aggregate([
{
"$match": {
storeId: 11
}
},
{
"$group": {
"_id": {
"year": {
"$year": "$completed"
},
"month": {
"$month": "$completed"
},
"day": {
"$dayOfMonth": "$completed"
},
"hour": {
"$hour": "$completed"
}
},
"totalTip": {
"$push": "$tip"
},
"basketTransactionCashTotal": {
"$push": "$basketTransactionCash"
},
"basketTransactionCardTotal": {
"$push": "$basketTransactionCard"
},
"consumerIds": {
"$push": "$consumerId"
},
"storeId": {
"$addToSet": "$storeId"bas
},
"basketIds": {
"$push": "$basketId"
},
"basketProducts": {
"$push": "$basketProduct"
}
}
},
{
"$sort": {
_id: 1
}
}
],
{
cursor: {
batchSize: 2000000
}
})
Is it a bug or am I missing something?
If someone could shred some light on it would be much appreciated.
Many thank,
Karthik
I have the same problem with the same versions.
This works:
return Rx.Observable.fromPromise(Meeting.aggregate(pipeline).exec());
This doesn't work:
return Rx.Observable.fromPromise(Meeting.aggregate(pipeline).cursor({ batchSize: 1000 }).exec());
Thanks for adding it to your milestone.
Many thanks,
Karthik
Hi, Any progress on this?
Some progress, but not much unfortunately. We've figured out what's wrong, just haven't had the time to fix it yet.
+1. Any progress on this?
So this issue is closed, what sort of problem are you having?
This works:
Model
.aggregate(pipeline)
.exec(function () {
console.log(arguments);
})
And this doesn't: (never calls back)
Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec(function () {
console.log(arguments);
})
Also, the following doesn't return a promise:
Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec()
Am I missing something?
Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec()
returns a cursor, as specified in 4.0 release notes. Admittedly this is a not-well-designed API, it's something that was thrown in at the last minute as a stopgap. The right way to do it would probably be something like this:
var stream = Model
.aggregate(pipeline)
.cursor({ batchSize: 1000 })
.exec().stream();
stream.on('data', function(doc) {
// ...
});
Ok cool, thanks for that! For the callback approach, the docs currently say this:
This is incorrect, right?
Yep those docs are incorrect, thanks for pointing that out
Anyone looking at this may also be interested in https://github.com/Automattic/mongoose/issues/3160 which outlines another bug about getting an undefined response from exec when a cursor is used.
@vkarpov15 its almost 2018 and the docs are still misleading. someone help 馃
@Ra1da35ma well that's embarrassing, looks like we closed out #2955 by mistake. Thanks for pointing that out, re-opened and will be fixed :+1:
I returned to the version mongoose": "4.13.7 and everything is normal .. using mongo 3.6
This works:
Model .aggregate(pipeline) .exec(function () { console.log(arguments); })
And this doesn't: (never calls back)
Model .aggregate(pipeline) .cursor({ batchSize: 1000 }) .exec(function () { console.log(arguments); })
Also, the following doesn't return a promise:
Model .aggregate(pipeline) .cursor({ batchSize: 1000 }) .exec()
Am I missing something?
This Answer + This One:
I returned to the version mongoose": "4.13.7 and everything is normal .. using mongo 3.6
Solved my problem. Thanks!
Most helpful comment
@Ra1da35ma well that's embarrassing, looks like we closed out #2955 by mistake. Thanks for pointing that out, re-opened and will be fixed :+1: