Mongoose: Feature request: Force mongoose returns to respect model's fields.

[samhagman]

An issue that we are having while using Mongoose is that we have a single MongoDB that has collections (let's use People as our example) which hold documents that contain lots and lots of fields but applications we build on top of this database only care about a subset of these fields. This is a problem in the following cases.

(Note: This is a contrived example, please do not comment about how my database is setup incorrectly.)

The "Select *" Problem:

// "Model" inside MongoDB AKA what the documents inside MongoDB look like

{
  firstName: String,
  lastName: String,
  SalaryInfo: [ SalaryInfo ]
}

We have one application that uses this database for say a public person directory. This application will NEVER EVER have to care about the salary info or even know that it exists in the actual database.

// Actual model in our node Mongoose code
// This is the PersonSchema which then becomes a Person Model
{
  firstName: String,
  lastName: String,
}

But in our application the Person model doesn't actually represent the model that we have just created, as shown here:

Person.find({}, function(err, personDocs) {
  console.log(personDocs);

  /*
  Prints:
  {
    firstName: 'Joe',
    lastName: 'Schmo',
    salaryInfo: { ... }
  }
  */
});

In this case the only reason Person is there is to know which collection to search on. Shouldn't Mongoose respect the model and only return _what is actually defined in the model_?

The "No Documented Projection Options" Problem

This same problem is even more apparent with functions that return a document after some action has been taken. For example: findOneAndUpdate, findOneAndRemove, etc will all return to you the document (optionally the new one) inside it's callback. But here also the Model's fields are not respected, although you can pass { projection: { a: 1} } to the options but this is not a documented feature and not even as fully featured as other projection options so I am wary of using it. But either way shouldn't Mongoose respect the model and only return _what is actually defined in the model_?

Proposed Changes

1. The Model object should append a select statement to any query that does not specify what fields to return. This select statement would be created from what is found in model.schema.
2. The Model object should do the same exact thing but for .findOneAndX() and similar calls.

[vkarpov15]

So this is a pretty core assumption to mongoose - that any data you pull out of the database is safe. I'm open to changing this behavior if there's a lot of demand. Otherwise, this is suggestion is something that would be pretty easy to implement as a plugin.

[samhagman]

Yes @vkarpov15 this could be an easy plugin by using a bunch of pre hooks, schema.eachPath(), and query.selected(). My one problem when going down this path was Model#remove and possibly Model#save. I will modify the example in the docs slightly to show the problem I thought I might have.

person.remove(function (err, person) {
  if (err) return handleError(err);

  // Will print the entire document it finds in MongoDB, correct?
  console.log(person);

  Person.findById(person._id, function (err, person) {
    console.log(person) // null
  })
})

If this does indeed return the full document then I am not sure how I can use a plugin to correct this case as well.

Will this return the full MongoDB document like I suspect?

If so is there any way to modify this behavior in a plugin?

[vkarpov15]

The second param to person.remove()'s callback is just person, sending a remove() to mongodb doesn't actually return the document(s) it removes. Save has the same behavior, it just does callback(null, this);

[samhagman]

@vkarpov15 Oh okay! Thank you that makes sense, I guess my last question is for Model.findById() and Model.findByIdAndX().

According to the Mongoose guide and the API documentation for Mode.findByIdAndX() middleware is not supported. In these cases how would you go about building a plugin without pre hooks?

[vkarpov15]

findByIdAndX is a thin wrapper around findOneAndX, and findById is a thin wrapper around findOne. There are pre hooks for findOneAndUpdate and findOne. Use those.

[samhagman]

Oh awesome! The API docs should probably mention that middleware does work on findByIdAndX by using the findOneAndX and findOne hooks.

Thank you for your help and responsiveness, I will post a link to the plugin I create to solve this and close it.

[michwii]

:+1: @samhagman !

In our project we are working in a continuous delivery model.

Therefore we need to be able to upgrade the version of our API without impacting our previous consumers.

To do so we decided to have several instance of our API (one for each version) but only one mongoDB database.
The documents present within the database are shared across versions.

Maybe an example could clarify our situation :
Let's assume with have currently 2 versions of our API in production.

V1 model : {a : String, b: String, c: String}
V2 model : {a : String, b: String, c: String, d: Number}

If the V2 insert a document, I want to make sure that V1 do not return the "d" property.
This is why if Mongoose would have respected the schema present in V1 and ignored the properties coming from other versions, we would have been able to do easily continuous delivery / deployment without impacting our previous consumers.

Do my example make sense ?

Is it possible to add another option to the method toObject() ? This property could be "strict : true"...

Thank you for your help =D

[michwii]

For those who add the same problem as us. We forced Mongoose manually to respect the Schema when returning an object coming from the DB using this peace of code :

var Test = new Schema({
    id                              : ObjectId,
    a                               : String,
    b                               : String,
},  {strict: true});

var filter = function (doc, ret, options) {
  Object.keys(ret).forEach(function (element, index) {
    if(doc.schema.paths[element] == undefined){
      delete ret[element];
    }
  });
}

if (!Test.options.toObject){
    Test.options.toObject = {};
}
if (!Test.options.toJSON){
    Test.options.toJSON = {};
}
Test.options.toObject.transform = filter;
Test.options.toJSON.transform = filter;

You just have to replace the Test variable with the name of your Model.

Have a nice day :smiley:

[samhagman]

I have finally built a plugin that does basically what I need, although I can see ways in which it could be improved. This is the link to the NPM module.

And I will post the code below as it is fairly simple:

/**
 * @module strictModelPlugin
 */


/**
 * When projection/selection options are not specified, this plugin forces Mongoose to only return fields that were
 * specified on the model the query is called on.  AKA Fields in MongoDB but not in the Model are not returned.
 * @memberof module:strictModelPlugin
 * @param {Schema} Schema - Mongoose Schema object
 * @param {object} [options] - Options object to configure plugin behavior
 * @returns {Schema}
 */
module.exports = function StrictModelPlugin(Schema, options) {

    // Setup options
    var opts = options || {};
    var allowNonModelQueryParameters = !!opts.allowNonModelQueryParameters;
    var allowNonModelSelectionParameters = !!opts.allowNonModelSelectionParameters;

    /**
     * Holds all the model's fields (paths)
     * @type {Array}
     */
    var paths = [];
    Schema.eachPath(function(pathName, schemaType) {
        paths.push(pathName);
    });

    /* Attach function to every available pre hook. */
    Schema.pre('count', function(next) { restrictSelect(this, next); });
    Schema.pre('find', function(next) { restrictSelect(this, next); });
    Schema.pre('findOne', function(next) { restrictSelect(this, next); });
    Schema.pre('findOneAndRemove', function(next) { restrictSelect(this, next); });
    Schema.pre('findOneAndUpdate', function(next) { restrictSelect(this, next); });
    Schema.pre('update', function(next) { restrictSelect(this, next); });

    function restrictSelect(query, next) {

        //==================================================================
        //
        //               CHECK QUERY PARAMETERS FOR FIELDS NOT IN MODEL
        //
        //==================================================================
        if (!allowNonModelQueryParameters) {
            var queryConditions = query.getQuery();
            var queryFields = Object.keys(queryConditions);

            // Go through each query field and see if it is in the Mongoose model's Schema
            for (var queryFieldIndex = 0; queryFieldIndex < queryFields.length; queryFieldIndex += 1) {
                var queryField = queryFields[ queryFieldIndex ];

                if (paths.indexOf(queryField) === -1) {
                    throw new Error('Attempting to query on a field that is not listed in Mongoose model: ' + queryField);
                }
            }
        }


        //================================================================
        //
        //               CHECK PROJECTION FOR FIELDS NOT IN MODEL
        //
        //================================================================

        var projectionFieldMap = query._fields || {};
        var projectionFields = Object.keys(projectionFieldMap);
        var numSelectedPathIndexes = projectionFields.length;
        var newSelectQuery = {};

        // If there are no fields selected, select all of the model's fields
        if (numSelectedPathIndexes === 0) {
            query.select(paths.join(' '));
            return next();
        }
        else if (query._fields[ projectionFields[ 0 ] ] === 1) { // Query was Inclusion Projection

            // Go through each requested field and only include paths in Schema in new select query
            for (var projFieldIndex = 0; projFieldIndex < projectionFields.length; projFieldIndex += 1) {
                var projField = projectionFields[ projFieldIndex ];

                if (paths.indexOf(projField) > -1) {
                    newSelectQuery[ projField ] = 1;
                }
                else if (!allowNonModelSelectionParameters) {
                    throw new Error('Attempting to project on a field that is not listed in Mongoose model');
                }
            }

            query._fields = newSelectQuery;
            return next();

        }
        else { // Query was Exclusion Projection

            // Iterate through Schema paths and add all but the excluded ones to new select query
            for (var pathIndex = 0; pathIndex < paths.length; pathIndex += 1) {
                var path = paths[ pathIndex ];

                // If this path wasn't excluded by select parameters, add to new query
                if (projectionFieldMap[ path ] !== 0) {
                    newSelectQuery[ path ] = 1;
                }
            }

            if (Object.keys(newSelectQuery).length !== (paths.length - projectionFields.length) && !allowNonModelSelectionParameters) {
                throw new Error('Attempting to project on a field that is not listed in Mongoose model.');
            }

            query._fields = newSelectQuery;
            return next();
        }
    }

    return Schema;

};