Mongoose: Cast to ObjectId failed for value on {field_object_id: {$in:[stringId1, stringId2, []]}}

[franky-continu]

Current behavior is that if I have a schema with an array of object ids referenced to another collection:

privacy: [{
    type: mongoose.Schema.Types.ObjectId,
    ref: 'Category',
    index: true,
    required: true
   // removed this line from a comment below...// default: []
  }],

And I want to filter all content for my categories and the public ones, in our case content that does not have a privacy_category setting. i.e. an empty array []

We currently query that with an or query

{"$or":[
   {"privacy": {"$size": 0}},
   {"privacy": {"$in":["5745bdd4b896d4f4367558b4","5745bd9bb896d4f4367558b2"]}}
]}

I would love to query it by only providing an empty array [] as one the comparison options in the $in
statement. Like this, which mlab has confirm is possible in mongodb

"privacy_locations":{
   "$in": ["5745bdd4b896d4f4367558b4","5745bd9bb896d4f4367558b2",[]]
}

(edited later: forgot to mentioned that this query, works from the console without using mongoose)

This throw cast error

{"message":"Error in retrieving records from db.","error":{"message":"Cast to ObjectId failed for value \"[]\" at path \"privacy_locations\" for model \"Article\"","name":"CastError","stringValue":"\"[]\"","kind":"ObjectId","value":[],"path":"privacy_locations"}}

That way I can avoid the $or query statement of an empty array size: 0 and the options in the $in statement and just have one in statement.

If there is a better way, I'm open to change that as well.

I just need to say, give me the content that match these privacy settings (A, B, C) and also give me the content that does not have any privacy settings.

(mongoose version 4.13.2)

[JasonCust]

Quoting from the referenced contributing guidelines when creating a new issue:

If you have a question about Mongoose (not a bug report) please post it to either StackOverflow, or on Gitter

You are more likely to get a response much sooner on either of those while not adding to a backlog of bugs/features that the maintainers need to sift through here.

[franky-continu]

forgot to mentioned that this query, works from the console without using mongoose
but throws this error in mongoose:

{"message":"Error in retrieving records from db.","error":{"message":"Cast to ObjectId failed for value \"[]\" at path \"privacy_locations\" for model \"Article\"","name":"CastError","stringValue":"\"[]\"","kind":"ObjectId","value":[],"path":"privacy_locations"}}

[franky-continu]

It is a bug report. $in [stringid_tobecasted, stringid_tobecasted, []] should cast object ids out of all the strings but not the [] (empty array)

[JasonCust]

It's not a bug. You are attempting to set a default for a field with type ObjectId to an empty array which will fail casting to an ObjectId as correctly reported.

This should be a question on SO or similar about how to properly set a default for an array field.

[franky-continu]

Is not because of the default. I added that later. And it is the schema definition.

Given sample data:

db.emptyarray.insert({a:1})
db.emptyarray.insert({a:2, b:null})
db.emptyarray.insert({a:2, b:[]})
db.emptyarray.insert({a:3, b:["perm1"]})
db.emptyarray.insert({a:3, b:["perm1", "perm2"]})
db.emptyarray.insert({a:3, b:["perm1", "perm2", []]})
> db.emptyarray.find({b:[]})
{ "_id" : ObjectId("5a305f3dd89e8a887e629ce0"), "a" : 2, "b" : [ ] }
{ "_id" : ObjectId("5a305f3dd89e8a887e629ce3"), "a" : 3, "b" : [ "perm1", "perm2", [ ] ] }
> db.emptyarray.find({b:{$in:[]}})
> db.emptyarray.find({b:{$in:[[], "perm1"]}})
{ "_id" : ObjectId("5a305f3dd89e8a887e629ce0"), "a" : 2, "b" : [ ] }
{ "_id" : ObjectId("5a305f3dd89e8a887e629ce1"), "a" : 3, "b" : [ "perm1" ] }
{ "_id" : ObjectId("5a305f3dd89e8a887e629ce2"), "a" : 3, "b" : [ "perm1", "perm2" ] }
{ "_id" : ObjectId("5a305f3dd89e8a887e629ce3"), "a" : 3, "b" : [ "perm1", "perm2", [ ] ] }
> db.emptyarray.find({b:{$in:[[], "perm1", null]}})
{ "_id" : ObjectId("5a305f3dd89e8a887e629cde"), "a" : 1 }
{ "_id" : ObjectId("5a305f3dd89e8a887e629cdf"), "a" : 2, "b" : null }
{ "_id" : ObjectId("5a305f3dd89e8a887e629ce0"), "a" : 2, "b" : [ ] }
{ "_id" : ObjectId("5a305f3dd89e8a887e629ce1"), "a" : 3, "b" : [ "perm1" ] }
{ "_id" : ObjectId("5a305f3dd89e8a887e629ce2"), "a" : 3, "b" : [ "perm1", "perm2" ] }
{ "_id" : ObjectId("5a305f3dd89e8a887e629ce3"), "a" : 3, "b" : [ "perm1", "perm2", [ ] ] }
> db.emptyarray.find({b:{$in:[[]]}})
{ "_id" : ObjectId("5a305f3dd89e8a887e629ce0"), "a" : 2, "b" : [ ] }
{ "_id" : ObjectId("5a305f3dd89e8a887e629ce3"), "a" : 3, "b" : [ "perm1", "perm2", [ ] ] }

[JasonCust]

@franky-continu If you want to continue this conversation then post on SO and I will be happy to explain there why this is NOT a bug but an issue with your default definition.

[franky-continu]

Never mind, I need a solution or a way to make it work, not a reason why is not a specific category of not working.

In the end is not working. it would be very easy to include NULL and [] emtpy arrays in the list of fields that can be added to ANY comparison for the $in statement.

Specially given that is the behavior of the $in statement which is being reduced by this casting issue. Very easy to say Cast x y and z if not null or array, ant that would be the end of it.

Thank you so kindly for all the time and replies, please forgive me if this was not your cup of tea or bug :-)

[JasonCust]

👍

The solution is simple enough and stems from an apparent slight misunderstanding. However you choose to proceed is totally up to you but the offer is there if you choose to accept it or not.

[franky-continu]

What is it? O great mystery-man. I shall create your query in SO, the only place on earth you may type such a SIMPLE thing, hope is not adding an ALL option :-)

[franky-continu]

https://stackoverflow.com/questions/47858674/mongoose-match-element-or-empty-array-with-in-statement

[varunjayaraman]

@franky-continu if you want, you can do an $or and the other condition can be { $exists: false }

[vkarpov15]

I'm somewhat surprised that mongoose even allows saving a doc where something that should be an array of ObjectIds actually saves if one of the elements is []. Just because the mongo shell allows something doesn't mean mongoose does, because as a general rule of thumb if you define something to have type ObjectId in mongoose, that means in MongoDB it will either be an ObjectId, null, or undefined. Saving an array where you expect an ObjectId sounds like a bug...

[franky-continu]

FIRST:
Is not about SAVING. It's about QUERYING.

SECOND:

I don't want to save an [] into an array of ObjectIds.

I want to compare an Empty Array var a = []; to another empty array var b = []; and have them match.

Document A defines field a in Mongoose as being an array of ObjectIds

a [{
    type: mongoose.Schema.Types.ObjectId,
    ref: 'Category',
    index: true,
    required: true,
    default: []
  }]

A.a can be [ObjectId-A, ObjectId-B] or A.a can be [] an empty array.

If I query mongoose/mongo by saying {a: []} it will actually works and bring me all document A instances where a is an empty array.

When in mongo you use $in you can specify that at least one element of a is found in the array supplied to the $in label, or that any of the elements supplied in the $in label are equal to (in this case) a.

That means that in mongo you can compare an element of an array or the element being equals which in turn results in being able to compare arrays that are empty to an empty array found within the elements supplied in the array assigned to the $in label

Again: I don't want to save, I want to query, compare, match. As you can with $in

[franky-continu]

@varunjayaraman of course. That is how it works now. But it is highly inefficient.

If I add a label ALL to categories and create a system by which [] is actually [ObjectId-ALL] and I always query privacy by saying $in ALL, any-other-option it works and works faster.

Again, the $in statement compares if field X is in any of the elements provided to $in.
In the exception that X is an array, it looks for any of its elements but it sould still check NULL or [ ] or even [ObjectId-A, ObjectId-B, [ObjectId-C, ObjectId-D], ObjectId-F].

In the last example for the given array to $in, it should compare A, B, then the Whole Array C-D, then C, then D, then F...

And I believe in Mongoose, because it is (IMHO) prematurely and forcely, casting to OID all arguments sent to statement $in, it fails by casting error, which perhaps should even be a non-error. If I look for all Strings equal to 5 maybe in the context of a query it shouldn't need to throw an exception. Furthermore, it is the field in the document that I have define as being of a certain type, not the arguments I pass to an $in statement.... And Furthermore [] can be of ANY type, including 'Category' (in this case), and so does null, null can be of ANY type, so throwing an exception because you turned null into "null" (String word) and then tried to cast string "null" word into an ObjectId is a bit far fetched as an argument for Type consistency. Neither String word "null" nor string word brackets "[]" were sent to the options of $in.

I sent a primitive value of null and an array both should be able to be compared, and Are able to be compared as I mentioned previously, I can do a query by {a: null} or {a: []} (and mongoose does not cast them to ObjetId before the comparison as it does in $in) but I cannot do {a: {$in: [null, []]}} because it is previously transforming null and [] to string "null" and string
"[]" and then casting them to ObjectId, and that removes a big portion of the functionality of $in by removing the possibility of $in to check whole values and not just elements within them.

[vkarpov15]

Ah ok, I see, if I'm reading correctly you're not looking to store a doc with privacy: [[]], you're looking to store privacy: [] and query for that. My confusion came from this code:

privacy: [{
    type: mongoose.Schema.Types.ObjectId,
    ref: 'Category',
    index: true,
    required: true,
    default: [] // <-- this means each element in the array has a default value of `[]`
  }],

Do this instead:

privacy: {
  type: [{
    type: mongoose.Schema.Types.ObjectId,
    ref: 'Category',
    index: true,
    required: true
  }],
  default: [] // <-- this is unnecessary, because arrays always have `[]` by default, but this is an example
},

This $in: [] code isn't something I've seen before, but it is a really neat trick, I think we can support that just for $in.

[franky-continu]

Oh I see, I'll correct that.

It works perfectly in mongo but mongoose pre casts everything in $in to objectid which is arguably correct, except that it doesn't take into consideration null or empty [].

[vkarpov15]

null should cast fine, it's just the [] that's a little weird. We'll investigate implementing this next week, thanks for your patience in communicating this issue :+1:

[franky-continu]

It should, but it doesn't. I believe it was my first error, but I pasted only the one with []

[varunjayaraman]

@vkarpov should we keep this labeled as abug or is it more of an enhancement?

[franky-continu]

+1 for being either an enhancement or a bug. Technically speaking it is a wrong implementation of behavior of the $in label (function) but it could be argue that because mongoose purpose includes giving schemas, it shouldn't work as it does in mongodb.

But, casting $in argument (elments) without flexibility it could go against the flexible nature of mongoose which does give schema and structure but it does so within the flexible nature of a functional language and the objectively smart way it is implemented.

Either enhancement or bug will greatly increase current $in state and capabilities as passing strings, nulls, objectids, and [] empty arrays is possible and would be awesome if it becomes possible.

[vkarpov15]

@franky-continu the plan right now is to just allow [] and just for $in because of the syntax you mentioned. No other types.

@varunjayaraman yeah, enhancement is more correct :+1:

[franky-continu]

100% Agreed, fantastic news

[hirengevariya]

// Create Geo Location Schema
const GeoSchema = new mongoose.Schema({
type: {type: String, default: 'Point'},
coordinates: {type: [Number], index: '2dsphere'}
});

// Create Uer Location Schema
const UserLocationSchema = new mongoose.Schema({
name: {type: String},
geometry: GeoSchema,
createdAt: Date,
updatedAt: Date

});

newUserLocation: function (req, res) {

const {name, geometry} = req.body;

// Create user location data
const userLocationData = {
  name,
  geometry
};
console.log('New User Data', userLocationData);

UserModel.create(userLocationData).then((userCreateRes) => {
  res.json({status: true, data: userCreateRes, message: 'User location successfully added'})
}).catch(() => {
  res.json({status: false, message: 'User location data failed!'})
})

},

message:
'Cast to Embedded failed for value "[72.591760]" at path "geometry"',
name: 'CastError',
stringValue: '"[72.591760]"',
kind: 'Embedded',
value: '[72.591760]',
path: 'geometry',
reason: [MongooseError] } },
_message: 'userLocation validation failed',
name: 'ValidationError' }

Please give me the solution for this.

[vkarpov15]

@hirengevariya it looks like the geometry property on req.body is an array, not an object