Mongoose: Temporary (virtual) properties on model instance

[whitecolor]

Sometimes I need to have on a model instance some temporaty property that is not in DB but for example I want to send to the client.

The scenario:

say I a have a Parent Model and Child model, I store them in separeate collections and only Child model has reference to it's parent

Parent.findOne({..}, function(err, parent){
    Child.find({parent: parent.id}, function(){err, children}{
        // now I want to send to the client parent instance with children as nested array
          parent.children = children
          res.send(parent)
    })
})

the problem is that parent.children = children won't work if children property is not defined on Parent schema. But its only temporary property and should not be saved to DB, it is not reasonable (and just incorrect) to define it on Parent schema.

The workaround I found is to add the followin virtual peroperty to Parent model:

parentSchema.virtual('children').get(function(){
    return this.__children
}).set(function(val){
    this.__children = val
})

I thought maybe something like that could work for such proerties:

javascript // that would allow setting ```children``` but do not store it in DB parentSchema.virtual('children')

What do you think?

[vkarpov15]

Virtuals are the correct way to do this. The method you described with .get() and .set() seems pretty concise, no?

[whitecolor]

Well I didn't try get/set. Will it work for non-schema properties, and won't it place "illegal" data in db by chance?

[vkarpov15]

I don't quite understand the last question. Can you please elaborate?

[whitecolor]

It seems set/get doesn't work for me.

One more time: I want to `set/get property that is NOT defined on schema. Its is only temporary. The property should be processed with toObject, toJSON, but should not be saved to DB. So it is like virtual or terporary. Look at my code example in OP.

[vkarpov15]

Both of the below options work for what you're trying to do as far as I can tell:

var Schema = mongoose.Schema;

mongoose.connect('mongodb://localhost:27017/gh2642');

var parentSchema = new Schema({ name: String });

var Parent = mongoose.model('gh2642', parentSchema, 'gh2642');

var p = new Parent({ name: 'test' });

p.children = { a: 1 };

console.log(JSON.stringify(p.children));

p.save(function(err, p) {
});

and

var Schema = mongoose.Schema;

mongoose.connect('mongodb://localhost:27017/gh2642');

var parentSchema = new Schema({ name: String });

parentSchema.virtual('children').
  get(function() {
    return this.__children;
  }).
  set(function(v) {
    this.__children = v;
  });

var Parent = mongoose.model('gh2642', parentSchema, 'gh2642');

var p = new Parent({ name: 'test' });

p.children = { a: 1 };

console.log(JSON.stringify(p.children));

p.save(function(err, p) {
});

In both cases children property isn't saved to the DB. If you want the children property included with toObject() and toJSON(), you need a virtual and pass virtuals: true to toObject() or toJSON().

[whitecolor]

Your first code example (where you do not define virtual property children) won't work with toJSON/toObject even if virtuals: true presents.

That is what about I'm talking. It only works if you have a definition for virutual property like this:

parentSchema.virtual('children').
    get(function() {
        return this.__children;
    }).
    set(function(v) {
        this.__children = v;
    });

The issue was about gettring rid of this exessive code.

[whitecolor]

Again about this issue I do not think it is resolved.

yes this code

p.children = { a: 1 };

works in 3.9 (though it seemd not to work in 3.8 p.children was undefined after assignment). But anyway with this approach children is not going to get into toObject or toJSON result. So the only way to make it getting there is to define the property as virtual field:

parentSchema.virtual('children').
    get(function() {
        return this.__children;
    }).
    set(function(v) {
        this.__children = v;
    });

So why not allow this with just one line, that would make all the stuff internaly

parentSchema.virtual('children')

?

[vkarpov15]

Too much magic under the hood IMO - what if you have a schema path called __children? You could utilize the fact that mongodb doesn't allow you to have field names that start with $ in >= 2.6, but mongoose should still support mongodb 2.4.

It would be pretty easy to implement this with a plugin though, if it really bugs you.

[whitecolor]

what if you have a schema path called __children

yes I undertand it, __children its just my local implementation, mongoose could implement it some how internaly with more correct approach. Any way if you don't like it, not a big problem.

[vkarpov15]

The general idea is good, but I think its tricky to get it right in the general case. Avoiding schema paths, etc. will make this more trouble than it's worth to trim 6 SLOC.

[vko-online]

Does virtual solves this problem?

[whitecolor]

only using such hack:

parentSchema.virtual('children').
    get(function() {
        return this.__children;
    }).
    set(function(v) {
        this.__children = v;
    });

[techjeffharris]

@vkarpov15, is it safe to use __?

More specifically, do documents (or their prototypes) have any internal/non-enumerable properties that start with __ ?

[vko-online]

Is it possible to pass nodejs/express request req inside virtual property?

parentSchema.virtual('children').
    get(function() { //how can we invoke req here?
        return this.__children;
});

[techjeffharris]

@vko Sure it's _possible_, but probably not the best idea as you would need to use some scoping magic and define your schema within a particular express route handler or param middleware... You're asking a specific question that is more relevant to function scoping than mongoose virtuals...

http://ryanmorr.com/understanding-scope-and-context-in-javascript/

[vko-online]

@techjeffharris is there any demo? I really want this but not sure how it's done.

[vkarpov15]

@techjeffharris I think it should be safe

@vko-online not really possible, I would advise against trying to do that.

[techjeffharris]

@vkarpov15 sounds like it will work for now. Thank you!

I'll investigate what it would take to add a schema method called temporary or something that would essentially create virtual properties that are stored in a per-document property like __temp. This would allow for officially sanctioned (and anticipated) virtual properties while keeping the internal __ namespace clean.

[whitecolor]

@techjeffharris you can use ____ (4 underlines) :D

[techjeffharris]

@whitecolor Haha

[vkarpov15]

Just make sure you don't use $__ :p Hopefully ES6 symbols will be helpful for this...

[arindrajitb]

I found a alternate way to solve the problem. As its JavaScript adding property on the go is not a problem. The problem is here with the toJson or toObject. you can overcome that bu over riding those methods.
i used this ( as suggested in http://stackoverflow.com/questions/11160955/how-to-exclude-some-fields-from-the-document).
UserSchema.methods.toJSON = function() {
var obj = this.toObject()
delete obj.passwordHash
return obj
}

This is better over mongoose transform. I believe wont work in the scenario when you delete the backing field the virtual will lose the source of information. therefore will not show-up in JSON.

[whitecolor]

Hey, for thouse who intrested you can check my current implementation for virtual async fields api.

https://github.com/whitecolor/mongoose-fill

I think mongoose.js actually should have something like this in the core. Intresting in you oppinion.

[vkarpov15]

@whitecolor sweet module! Way to go, I really like the plugin's API and I'm looking forward to using it. But looks like it isn't up on npm yet? https://www.npmjs.com/package/mongoose-fill

[vko-online]

@whitecolor this __fill.fill.fill looks really weird, just sayin

[whitecolor]

@vkarpov15 published it.

@vko-online its in dev mode so convetions and terms in the code may seem not clear.

__fill - is actually is object that contains info about what should be filled in particular query
__fill.fill - is ref to filler rules that should be applied (that are determined with mySchema.fill API)
__fill.fill.fill - is fill method of filler rules (it is not mentioned in docs, as I don't use it at currently, and not sure if it is needed at all)

We use it in our projects and we like api that it provides and allows to fill/combine mongoose objects with any kind of data, not only form current mongoose DB, but also from any async service.

[techjeffharris]

@whitecolor perhaps my usage of virtual properties is deferent a bit unorthodox compared to yours, but this module seems IMHO a bit of a complicated workaround to simply not store an array of child IDs. If the parent had an array of refs to the child IDs, one could simply populate the array of children and only select the name and age properties.

I'm using virtual properties to attach data (for instance, socket.io "sockets" for a user—each page refresh end and creates a new socket) which is expires when the process closes (or sooner) and is thus worthless to store in the db, but is nonetheless useful to have associated with a user model.

I'm writing a glorified chat server that caches all users from db at startup, then adds references to each of the users sockets (open windows with the same user session) to the document instance. If he serve restarts or closes, the sockets close and new sockets are made when the sever is back up and each window reconnects.

I suppose it's a more SQL-like way of doing things, stirring the parent ID in the child rather than storing child IDs in the parent, but I feel like using refs and populating the parent is effectively the same. Though, admittedly, I may be missing something.

I'm glad that you made something that works for you, nonetheless 😄

[whitecolor]

@techjeffharris
Well storing IDs consistently is additional task. Some cases really do demand it, for example when you need to keep some scpecific order of nested items and have ability of changing that order (you just swap, insert, remove IDs), etc. Sometimes using array of refs probably it can be aslo justified by perfomace requirements (when stored IDs act like index).

But often such solid connections are really exessive and add additional potential breaking point that you really want to avoid. In my oppinion if you can avoid exessive linking (coupling) in you architecture - you shoud (or must) avoid it. Yes it is some kind of SQL normalized structure way, but it simplifies things by reducing coupling and makes design more flexible.

[jloveridge]

@techjeffharris Storing the parentId on the child perfectly fine in both SQL and NOSQL worlds. Especially since unbounded arrays is an anti-pattern. Eventually you could exceed the 16MB document limit for a MongoDB document. For your particular case I would use a virtual populate.

parentSchema.virtual('children', {
  ref: 'Child',
  localField: '_id',
  foreignField: 'parentId',
});

Then when you want the children for any given parent you would simply do:

Parent.findOne({..}).populate('children').exec(function(err, parent) {
    if (err) {
        next(err);
    } else {
      res.send(parent.toObject({virtuals: true});
    }
});
// or using promise syntax
Parent.findOne({...}).populate('children').exec()
  .then((parent) => res.send(parent.toObject({virtuals: true}))
  .catch(next)

Also note that you can omit the {virtuals: true} if you set the schema to default to including virtuals on toObject (default is false).