Mongoose: Ability to add properties that aren't defined on the schema

[adamreisnz]

I have a use case where I read some documents from the database and then do some calculations on their data, after which I'd like to append this data to the documents as new properties.

In Mongoose, this is not possible unless the properties are defined in the schema. E.g. Mongoose will not allow you to add unknown properties that you haven't defined on the schema in advance.

So the "normal" solution would be to convert the documents to plain objects or JSON with toObject or toJSON first.

This works fine in most cases, but in my specific case I have defined several methods on the document model as well, and I'd like to retain the ability to call them for as long as I can. Converting the documents to plain objects or JSON obviously discards the methods, making it impossible to use them further down the data processing chain.

The only apparent work around would be to change the order of data processing or store both the models and JSON objects separately, but this is a bit of a drag if you're passing data around via promises or the req object.

I'm wondering if it would be hard to implement a new flag for Mongoose schema's which when enabled, will in fact let you add "new" or unknown properties to a document. This data can be ignored when a document is saved or validated, but appended when a document is converted to JSON or plain object.

Another use case for it that I encountered is to append some temporary meta data to my documents. Again, with the data processing example, I'd like to maybe find some additional data/information about my documents, and attach this to them temporarily for later use, again without losing access to my document methods.

Thoughts?

[vkarpov15]

Is strict mode sufficient for your use case? Sounds like exactly what you're looking for.

[adamreisnz]

Hmm, I'll have to experiment with it. I don't need to save the new properties to the database, merely pass them on via the object, and have them appear in toJSON data.

Will also see if doc.set() is what I need.

[vkarpov15]

Hmm so mongoose doesn't validate data coming out of the database, if you put additional fields in the db they should show up in the mongoose docs and in toJSON() output.

[adamreisnz]

I don't think they do, I have had to access them via document._doc.someProperty, because they wouldn't get attached to the actual model based document, and as a result, also didn't show up in toJSON() output. I assumed this is a feature of Mongoose. E.g.:

Model
  .findOne({...})
  .then(document => {
    document.nonSchemaProperty; //undefined
    document._doc.nonSchemaProperty; //defined
  });

[vkarpov15]

Hmm if that's the case, you could just add virtuals for your temporary meta properties, then they would show up with toJSON() if you configured it to do so.

[adamreisnz]

I don't know, if I create a virtual setter, does that allow me to set properties on the document which don't exist?

E.g. if I want to set doc.someMetaProp to a specific value, and assuming someMetaProp doesn't exist in the schema, how would I use a virtual setter to do that? I thought that setters could only be used to modify properties that exist on the schema.

So:

personSchema.virtual('someMetaProp').set(function (value) {
   this.someMetaProp = value; //don't think this will work, or at best create an infinite loop
});

[vkarpov15]

Nope it's perfectly valid for a virtual to set a property that isn't on the schema (modulo strict mode).

var personSchema = new Schema({
  name: String
});

personSchema.virtual('test').
  get(function() { return this._test; }).
  set(function(v) { this._test = v; });

var Person = mongoose.model('Person', personSchema);

var p = new Person({ name: 'Val', test: '123' });

console.log(p.toObject({ virtuals: true }));

$ node gh-4614.js 
{ name: 'Val',
  _id: 5838cc4ccf01b36af4082291,
  test: '123',
  id: '5838cc4ccf01b36af4082291' }
^C
$ 

With the default strict mode, you can set whatever properties on the object you want, they just won't get persisted to the db.

[adamreisnz]

Alright will play around with it a bit, thanks!

[adamreisnz]

@vkarpov15 as a follow up to this one, is it possible to have only some virtuals appear in toJSON output? I have some virtuals which I use only for internal purposes, and which I don't need to appear in JSON, but I'm going to add a few virtuals using your above suggested approach, which I would like to have appear in the JSON. Can I somehow specify which virtuals get included and which don't?

[vkarpov15]

Not supported atm. Best bet would be to just delete the properties you don't want to show up in a transform.

[adamreisnz]

Ok, yes that's what we're doing most of the time.

[maxpaj]

I employed this strategy:

1. Add whatever properties you need to the document. In this example customProp is not defined in the schema.
   User.findById(id).then((user) => { user.customProp = "test"; return user; })

2. Before sending data back to the client, convert the object to JSON and add the custom properties from the document to the JSON object. Here with spread notation:
   return response.status(200).json({ customProp: user.customProp, ...user.toJSON() })

I guess you could even add a table of these custom variables onto the document instance, then convert that entire table to properties on the JSON object. So you might have:

user.customProps = {
   foo: "abc",
   baz: 123
}
return response.status(200).json({ ...user.customProps, ...user.toJSON() })

This way, you don't have to work with a JSON object all the way up through the stack while having custom properties added along the way.

I also looked into if there was something you could to with the transform function on toJSON definition in the schema, but didn't get that to work the way I wanted it to.

toJSON: {
   virtuals: true,
   transform: function (doc, ret, options) {
      return { ...doc.customProps, ...ret };
   }
}

http://mongoosejs.com/docs/api.html#document_Document-toObject

Thought on this?

[vkarpov15]

@maxpaj what issue do you have with the toJSON transform approach? At first glance, that looks like it should work.

[maxpaj]

@vkarpov15 I don't think there's anything wrong with it, I just haven't had time to confirm that it works. I tried another implementation, then came up with this other approach when I wrote my comment.

Try it out!

[darshanan24]

@vkarpov15 hi, can i know how to use Mixed schema type without assigning any property name(key )?

for example: This is a valid mongoose schema:

const mongoose = require('mongoose');

const cubeSchema = mongoose.Schema({
_id:mongoose.Schema.Types.ObjectId,
cube: {type:mongoose.Schema.Types.Mixed, required:true}
})

module.exports= mongoose.model("Cube", cubeSchema);

the output for above code is: // just taking some random json

 {
        "cube": {
            "name": "JdbcSourcennector",
            "configuration": {
                "connector.class": " io.confluent.connect.jdbc.JdbcSourceConnector",
                "tasks.max": "1",
              }
        },
        "_id": "5bc06f086e25d246029de1de"
    }

But i would like to Post the data and get the data without propertyname(key).
for example something like this:

const mongoose = require('mongoose');

const cubeSchema = mongoose.Schema({
{type:mongoose.Schema.Types.Mixed, required:true}
})

module.exports= mongoose.model("Cube", cubeSchema);

Expected output:
// without the cube(key name)

{
            "name": "JdbcSourcennector",
            "configuration": {
                "connector.class": " io.confluent.connect.jdbc.JdbcSourceConnector",
                "tasks.max": "1",
              }
}

Is there any way available for the way i mentioned above? Thanks

[vkarpov15]

@darshanan24 so you're trying to make the whole document mixed, with no casting or validation? You can just try:

const cubeSchema = mongoose.Schema({}, { strict: false })

[unckleg]

@vkarpov15 What should I do when I have custom field that needs to be attached to mongoose fetched object on run ? (Without executing @toObject method).

In virtuals we can't define asnyc/await functions, because of that there is no chance to resolve above described.

[vkarpov15]

@unckleg you can use populate virtuals if you need to pull a doc from another collection. Or you can just define a method on your schema.

[spencerlevitt]

Hey @vkarpov15 @adamreisnz - Just wanted to tack on my solution here using underscore js:

const modifiedObject = _.mapObject(originalObject, function (ogObject) {
        return {
          ...ogObject,
          customProp: customPropValue,
        };
      });
res.send(modifiedObject._doc);

The ._doc provided me exactly what I wanted (the original object with just the custom prop)