Mongoose: Add support for populating denormalized sub schema's

[adamreisnz]

@vkarpov15 I couldn't find this documented nor was I able to find a solution for it on the Slack channel, so I wanted to raise this use case.

Let's consider the following simplified schema for a Subscription which is linked to a Member. Due to application design and business rules we opt to store subscriptions in a separate collection and link them to a member via a reference. In addition, we denormalize some data (first and last name of the member) to limit the number of read queries needed for this common information.

So the schema's would look something like this:

//Main member schema, with all member properties
const MemberSchema = new Schema({
  firstName: String,
  lastName: String,
  email: String,
  roles: [ ... ],
  //etc
});

//Member sub schema, for use in the subscription schema
const MemberSubSchema = new Schema({
  _id: {
    type: Schema.Types.ObjectId,
    ref: 'Member',
  },
  firstName: String,
  lastName: String,
});

//Subscription schema
const SubscriptionSchema = new Schema({
  startDate: Date,
  endDate: Date,
  member: MemberSubSchema,
});

So now when we load subscription documents, we have access to the member _id as well as firstName and lastName directly, e.g:

Subscription
  .findOne(..)
  .then(sub => {
    sub.member.firstName; //Available, already on document
  })

But, if we would like to use population to populate additional fields on the member, this becomes slightly awkward.

The documentation illustrates that if I were to have normalized data, e.g.:

const SubscriptionSchema = new Schema({
  startDate: Date,
  endDate: Date,
  member: {
    type: Schema.Types.ObjectId,
    ref: 'Member',
  },
});

I can use population by populating the member field, since it contains a reference to the Member model and Mongoose can find it's way around that:

Subscription
  .findOne(..)
  .populate({
    path: 'member',
    select: 'firstName lastName email',
  })
  .then(sub => {
    sub.member.firstName; //Available, populated
    sub.member.email; //Available, populated
  })

However, in our case we already have some properties of the member present, so how do we leverage population to add additional fields to the object?

The problem is that Mongoose doesn't know how to populate the member path because there is no reference to the model (it is hidden in member._id).

So one thing we can do is to populate the _id field as documented:

Subscription
  .findOne(..)
  .populate({
    path: 'member._id',
    select: 'email',
  })
  .then(sub => {
    sub.member.firstName; //Available, already on object
    sub.member._id.email; //Populated, but ugly
    sub.member._id._id; //Ugh...
  })

Another option would be to manually populate the data by performing the queries and constructing the final document ourselves.

Neither option is very appealing though since there is already a built in population engine, so I'd like to suggest adding support for being able to populate denormalized documents if they contain a reference to a model in one of their properties, by being able to specify this property as one of the options.

For example, something like this:

Subscription
  .findOne(..)
  .populate({
    path: 'member',
    select: 'email',
    ref: 'member._id',
  })
  .then(sub => {
    sub.member.firstName; //Available, already on object
    sub.member.email; //Available, populated from Member collection as reference by member._id
  })

This will instruct Mongoose to look at the path member._id to find out what reference to use for population, yet append the requested fields onto the existing member sub document and not onto member._id.

Moreover, this will add additional flexibility, as we could theoretically populate paths based on other fields external to the path to be populated, e.g. memberId:

Subscription
  .findOne(..)
  .populate({
    path: 'member',
    select: 'email',
    ref: 'memberId',
  });

Thoughts? Feasibility? Already possible via some undocumented method?

[vkarpov15]

Always the problem with populating is how mongoose tracks changes to the populated docs. So let's say you populate member and then change email and then save(), mongoose would have to know somehow that email is a populated field.

That makes me think that the right solution for this would be virtuals and allowing you to define getters for populated virtuals. Here's a really hacky way of doing it:

var personSchema = new Schema({
  name: String,
  email: String
});

var personSubSchema = new Schema({
  name: String
});

var subscriptionSchema = new Schema({
  person: personSubSchema
});

subscriptionSchema.virtual('$person', { ref: 'Person', localField: 'person._id', foreignField: '_id', justOne: true }).
  get(function() {
    if (this.$$populatedVirtuals && this.$$populatedVirtuals['$person']) {
      return this.$$populatedVirtuals['$person'];
    }
    return this.person;
  });

var Person = mongoose.model('Person', personSchema);
var Subscription = mongoose.model('Subscription', subscriptionSchema);

Person.create({ name: 'Val', email: '[email protected]' }).
  then(val => Subscription.create({ person: val })).
  then(sub => Promise.all([
    Subscription.findById(sub).populate('$person'),
    Subscription.findById(sub)
  ])).
  then(res => {
    res.forEach(res => {
      console.log(res.toObject({ virtuals: true }));
    })
  }).
  catch(error => {
    console.error(error);
    process.exit(-1);
  });

In the above example, the $person virtual would contain either the populated person or the embedded person data if it wasn't available. You just need to reach into the internals and use $$populatedVirtuals, which isn't ideal.

[vkarpov15]

So I found a solution that works well in the existing paradigm. The tricky bit is that mongoose getters are executed in reverse order, so you need to do an unshift:

var personSchema = new Schema({
  name: String,
  email: String
});

var personSubSchema = new Schema({
  name: String
});

var subscriptionSchema = new Schema({
  person: personSubSchema
});

var virtual = subscriptionSchema.virtual('$person', {
  ref: 'Person',
  localField: 'person._id',
  foreignField: '_id',
  justOne: true
});

virtual.getters.unshift(function(v) { // <-- getters are executed in reverse order, so this executes after the default getter for virtual populate
  return v || this.person; // <-- parameter to getter is the value returned from the previous getter, so if the virtual has a value then v will be defined, otherwise falsy
});

var Person = mongoose.model('Person', personSchema);
var Subscription = mongoose.model('Subscription', subscriptionSchema);

Person.create({ name: 'Val', email: '[email protected]' }).
  then(val => Subscription.create({ person: val })).
  then(sub => Promise.all([
    Subscription.findById(sub).populate('$person'),
    Subscription.findById(sub)
  ])).
  then(res => {
    res.forEach(res => {
      console.log(res.toObject({ virtuals: true }));
    })
  }).
  catch(error => {
    console.error(error);
    process.exit(-1);
  });

[adamreisnz]

@vkarpov15 that's a cool work around. The only drawback is that you have to use a $person property instead of person, but I guess there's no way around that, is there?

Thanks for looking into it.

[vkarpov15]

Well you could do something like this instead:

var subscriptionSchema = new Schema({
  _person: personSubSchema
});

var virtual = subscriptionSchema.virtual('person', {
  ref: 'Person',
  localField: '_person._id',
  foreignField: '_id',
  justOne: true
});

if you want to avoid the $. You just need to make sure the virtual doesn't conflict with the existing property.

[adamreisnz]

Yep, that sounds reasonable, but one way or another you have a property either as a virtual or in the database that looks kind of ugly with a prefix/suffix. Unless you find a creative name like personBare and person. I'll experiment a bit with it, it's not a big deal. Thanks!

[vkarpov15]

No worries, good luck and let me know if you come up with any good ideas. I like the current virtual solution because it works with existing mongoose concepts, but if there's a way to do this that has a lot of advantages I'm all ears :+1: