Mongoose: Schema plugins not applied to subdocuments

[adamreisnz]

I have the following Mongoose plugin to handle some common JSON conversion:

'use strict';

/**
 * Default toJSON implementation for mongoose schema's
 */
module.exports = function toJsonPlugin(schema) {

  //NOTE: this plugin is actually called *after* any schema's
  //custom toJSON has been defined, so we need to ensure not to
  //overwrite it. Hence, we remember it here and call it later
  let transform;
  if (schema.options.toJSON && schema.options.toJSON.transform) {
    transform = schema.options.toJSON.transform;
  }

  //Set toJSON handler
  schema.options.toJSON = {
    transform(doc, ret) {

      //Delete version and set ID
      delete ret.__v;
      if (ret._id) {
        ret.id = ret._id.toString();
        delete ret._id;
      }

      //Call custom transform if present
      if (transform) {
        transform(doc, ret);
      }
    }
  };
};

This works great, and is applied on all my models. However, sub document schema's don't seem to inherit the behaviour for some reason.

For example, I have:

let NoticeSchema = new Schema({
  title: String,
  type: String,
  message: String,
  startDate: Date,
  endDate: Date,
  isEnabled: Boolean
});

This is a schema for a sub document, e.g.:

let ClubSchema = new Schema({
  //...
  notices: [NoticeSchema],
  //...
});

However, the outputted JSON for the club includes _id's for the notices, and I have to explicitly define a toJSON handler on the notice schema for it to work as intended:

NoticeSchema.options.toJSON = {
  transform(doc, ret) {
    ret.id = ret._id.toString();
    delete ret._id;
  }
};

Is this intentional or a bug? I would assume plugins for schema's apply to all schema's, also sub document schema's.

This is with 4.5.0.

[adamreisnz]

Hi @vkarpov15 , could you confirm if this is indeed a bug? I need to know if I should create transform functions for each of my sub schema's now, or if I can wait for a fix to come out. Thanks!

[vkarpov15]

Yeah this is expected behavior, not a bug. Unless you explicitly pass options to toJSON(), like toJSON({ transform: ret => { /* ... */ } }), toJSON() will use the options from that doc's schema. In your case, unless you explicitly do NoticeSchema.plugin(toJsonPlugin), the transform on ClubSchema won't run on notices.

[adamreisnz]

@vkarpov15 not sure if I follow.

The plugin is applied application wide, e.g.:

mongoose.plugin(require('../plugins/mongoose/to-json-plugin'));

And not just to the ClubSchema.

That means that the plugin works for all top-level model schema's, but _not_ for sub document schema's. Shouldn't global plugins (that are attached to mongoose via mongoose.plugin) be expected to work on _all_ schema's, including sub document schema's?

It's counter intuitive for it not to happen, and it's annoying that you have to re-apply global plugins explicitly for sub document schema's which results in unnecessary code duplication.

Is there a particular reason global plugins are not applied to sub document schema's?

[vkarpov15]

Ah I see. The reason is that mongoose.plugin() doesn't apply plugins to subschemas. Womp.

[adamreisnz]

Thanks for addressing! 🎉

[pbsis]

This fix/change has caused us some unexpected behaviour and I'm not sure if it's because we're mis-using global plugins. We're using it to globally add a "common" subdocument to all our top-level schemas. This was working fine until this fix came along, but it's now adding it everywhere - including the sub-documents of the common schema we're trying to add. Is there any way to tell from inside the plugin that the schema is a "sub-document" and not add the common schema? Otherwise does this now mean that using a global plugin isn't a viable way to add something to every top level schema? For now we'll probably have to go back to manually adding the common schema and stop using a global plugin.

[adamreisnz]

@pbsis I've also had to make a change to one of my plugins to accommodate for this change. You do have access to the schema from inside the plugin, so you could probably check if it's a top level document or sub document. I'm not sure if there's a mongoose property to indicate that a schema is a sub document, but I would imagine there should be some way of telling them apart.

[pbsis]

I couldn't see a way to identify a child schema directly, so I ended up with the following:

if (!schema.$isChild) {
    schema.add({
        // my common schema
    });
}
schema.childSchemas.forEach(e => e.$isChild = true);

Not a particularly elegant solution but it does the trick.

[vkarpov15]

Hmm good point actually, this change can be backwards breaking. The general idea was that "global plugins" are advertised as plugins applied to all schemas, but as it stood they were only applied to schemas that were being used for models (and potentially applied multiple times).

The tricky bit is that the distinction between "child schema" and "top level schema" doesn't really exist because a schema can easily be both.

Another potential short-term solution would be to write your own wrapper around mongoose.model() that applies plugins for you. Any other ideas are most welcome

[fyockm]

Anyone here ever heard of semver?

"Hmm good point actually, this change can be backwards breaking."

[adamreisnz]

I don't think it warrants a major bump if you fix a problem/bug which some people were using as an undocumented feature...

The plugin system is advertised as being global, so you would logically expect it to apply to all of your schema's -- including child schema's.

[vkarpov15]

My general rule of thumb is if I don't change any existing tests then we're good for a patch release. Semver is subtle because one man's bug can very well be another man's feature, like in this case. I'm sorry for the trouble but in this case the new behavior is what mongoose is going to support going forward. Global plugins apply to _all_ schemas.

[pbsis]

Definitely a case of us using a bug as a feature! Certainly not the first time it's happened over the years, but the good thing about open source is you can quite easily get to the root cause.

I'm having second thoughts about using global plugins to add sub-schemas because as mentioned above, you could have a schema used as both a top-level in one model and a child in another. If I'm going to tell my developers that certain schemas have to be considered as only "top level" then I might as well also tell them to add the common sub-schema to all these top-levels.

I'm fairly happy with my above workaround for now. The only dirty part is I'm using "childSchemas" which feels like a private API which a mongoose developer may decide to change the name of later. The most elegant solution I can think of would be for the plugin method to pass a "level" property to indicate at what level in the model the plugin is being applied (so I could ignore any calls for levels greater than zero).

[Jeff-Lewis]

IMHO, I think the confusion lies in users think of a Schema as those specifically defined with new Schema() in user code and the implementation of Mongoose appears to create a schema for every property defined as an object on any Schema.

We found this out implementing a global plugin recently and ended up with a combination of trying to flag child schemas to ignore, which doesn't cover all cases...

_.forEach(schema.childSchemas, childSchema => {
    childSchema.options.__isChildSchema = true;
});

... and possibly using custom Options in the schema definition, which defeats the purpose of a global plugin not touching the existing defined schemas.

Is there a way to have Mongoose mark or flag User created schemas vs. Mongoose created schemas during compilation? I think that would allow at least global plugins to filter out the sub-doc child schemas easier.

[vkarpov15]

@Jeff-Lewis I think mongoose only creates a new schema implicitly for document arrays. We can definitely mark those schemas with a property to make it easier to filter those out

[vkarpov15]

@Jeff-Lewis in 4.6.3 schemas created implicitly for document arrays will have a $implicitlyCreated field.

[Jeff-Lewis]

@vkarpov15 awesome! Does $implicitlyCreated work for all implicitly created schemas?

[vkarpov15]

So there's only one case where schemas are created implicitly, when you create a document array with inline schema declaration:

var schema = new Schema({
  arr: [{ str: String, num: Number }] <-- equivalent to `[new Schema({ str: String, num: Number })]`
});

And this is where the prop gets added.

[adamreisnz]

@vkarpov15 has anything changed recently that might have reintroduced this bug?
I am applying my plugins globally but they don't seem to work again with subdocuments.

[vkarpov15]

@adamreisnz not that I know of. Can you open up a new issue with code samples please?

[adamreisnz]

@vkarpov15 done! See #5690 for a reproducible test case