Mongoose: Calling populate on embedded doc of embedded doc

[dmmalam]

Hey,
I have the following schemas ( simplified for issue)

User = new Schema
login:String

A = new Schema
emb: [B]
comments: [Comments]

//and following embedded schemas

B = new Schema
comments: [Comments]

Comments = new Schema
creator:
type: ObjectId
ref: 'User'
message: String

Now while populate works on A
eg A.find().populate('comments.creator')

it doesn't work on the double nested embedded doc
eg A.find().populate('emb.comments.creator')

Any ideas?

[eneko]

The way I am doing that currently is populating the nested items and then iterating through them to populate the children. I was planing on implementing a solution for populating multiple nested levels, but it get very tricky.

[paulshapiro]

Hi eneko,

How are you doing that?

When you do your first populate, you get returned the nested items but when you run through them to populate their children, are you able to save the populated children? It never sticks when I try to do it.

Some code would be awesome.

Thanks,
Paul

[paulshapiro]

@eneko nevermind, just needed .toObject() ;)

[eneko]

Actually Paul, I figured populate() works on queries that return multiple elements, so there is no need to iterate through the child objects. But yes, you are right, you need to call .toObject if you want the properties to stick.

Here is an example:

// Ideally should be Parent.findOne().populate('children').populate('children.grandchildren').run();

function loadParentWithChildrenAndGrandChildren(parentId) {

  // Load parent without children references (children is array of ObjectId)
  Parent.findOne({ id: parentId }, { children: 0 }, function(err, parent) {
    if (err || !parent) return next(new Error("Parent not found: " + parentId));

    // Load children for this parent, populating grandchildren (no need to load parent reference)
    Children.find({ parent: parent._id }, { parent: 0 })
      .populate('grandchildren', [], { })
      .run(function(err, children) {
        if (err) return next(new Error("Could not load children: " + parentId));

        var result = parent.toObject();
        result.children = children;
        next(null, result);
      });
  });

}

[paulshapiro]

Ah, thanks for your detailed answer. Your mongoose calls are more specific than mine and I could learn from your approach. As it happens my particular application required a recursive populate, so for now I may be stuck doing this manually.

I end up doing something like this.

var viewWithId_forDisplay = function(id, callback) {
if ( !id || typeof id === 'undefined' || id.toString().match(app_utils.emptyReg) ) {
    throw new Error('Bad user ID.');
}

View.findById(id)
.run(function(err, obj) {
    if ( err || !obj || !obj.subviews || !obj.subviews.length ) return callback(err, obj);

    obj = obj.toObject();

    // recursive subview fetch
    async.map( obj.subviews, viewWithId_forDisplay, function(err, results) {
        obj.subviews = results;

        return callback(err, obj);
    } );

})
}

[dmmalam]

Does this now work in the latest mongoose release?

[dbounds]

So I think this is the issue I'm experiencing. Each Comment object has an embedded User object. Comments are embedded as an array in Activity. A query for activities (as below) does not populate the User object on each comment.

var get_activities = function (callback) {
  var _this = this
  Activity.find({}, [], {sort:{ _id: -1 }})
  .populate('user')
  .populate('comments')
  .run(function (error, activities) {
    callback(error, activities)
  })
}

var User = new Schema({
    id: { type: String, required: true, lowercase: true, index: { unique: true } }
  , email_address: { type: String, required: true, index: true }
  , name: { type: String, required: true }
  , first_name: { type: String,  required: true } 
  , last_name: { type: String, required: true }
  , user_name : { type: String, required: true, lowercase: true, index: { unique: true } }
  , avatar_url: { type: String, required: true }
  , bio: String
  , following: [{ type: Schema.ObjectId, ref: 'User', index: { unique: true } }]
  , followers: [{ type: Schema.ObjectId, ref: 'User', index: { unique: true } }]
})

var Activity = new Schema({
    id: { type: Number, required: true, index: { unique: true } }
  , user: { type: Schema.ObjectId, ref: 'User', required: true }
  , recipients: [{ type: String, index: { unique: true } }]
  , type: String  
  , body: { type: String, required: true }
  , timestamp: { type: Date, required: true }
  , likes: [{ type: Schema.ObjectId, ref: 'User', index: { unique: true } }]
  , comments: [{ type: Schema.ObjectId, ref: 'Comment' }]
})

var Comment = new Schema({
    id: { type: String, required: true, index: { unique: true } }
  , timestamp: { type: Date, required: true }
  , body: { type: String, required: true }
  , user: { type: Schema.ObjectId, ref: 'User', required: true } 
}) 

[dethe]

Is there any plan for populate to support nested paths to handle cases like this?

[aheckmann]

yes we'd like to at some point. it could quickly become a performance issue but yeah we should support it.

[dethe]

For nested objects I need to iterate over them manually, loading them, which seems like more of a performance issue and clutters up my server code. So I'm +1 for making my job easier ;-)

In the meantime I will continue, and also address it by splitting sections of the page into separately loadable objects that don't require such deeply nested data on each call.

[Qard]

I'd be all for something like this;

Parent.findById(1).populate('child').run(function (err, parent) {
  parent.child.populate('grandchild', function (err) {
    console.log(parent.child.grandchild)
  })
})

It'd actually be pretty handy to have populate available on the document like that. You could keep diving as far as you need.

[aheckmann]

@Qard you read my mind.

[eneko]

@Qard that is how mongoose-relationships works. I used that module for a while at first, but ended up doing things manually. Maybe Mongoose should integrate it into the core, or at least parts of it like calling populate on arrays.

[diversario]

Has anything like https://github.com/LearnBoost/mongoose/issues/601#issuecomment-3258317 been implemented yet?

[mikeuduc]

I'm wondering the same thing. I'm trying to figure out how to populate embedded doc child references. @dbounds I have an analogous models. Have you resolved this issue: https://github.com/LearnBoost/mongoose/issues/601#issuecomment-3088564. I can't find mongoose-relationships

[mikemoser]

The information in this thread has been very helpful in understanding the current state of .populate(), the current limitation related to nested population of grandchildren and how .toObject() in combination with manual lookup can be an option to workaround it.

I'm wondering if there is a technique we could use that does not require .toObject(). I ask b\c after we populate our model and the grandchildren we'd still like to have a Document and proper types so we can use functions like parent.childarray.id(x) or modify the values and call .save().

Any help here would be greatly appreciated.

note: We've tried using parent.set('child.grandchild', value), however that seems to cause some issues with the integrity of the Document and we are no longer able to read values from that path (error is Invalid ObjectId when trying to read parent.child or parent.child.grandchild).

[aheckmann]

related: http://stackoverflow.com/questions/11137239/querying-nested-embedded-documents-with-mongoose

[vartana]

@aheckmann Any ideas on this yet?

[aheckmann]

been busy. i want it in 3.0 final

[vartana]

@aheckmann Awesome cant wait. It's will be really useful in real applications. But I do want to give you kudos for the great job on what you have accomplished so far.

[jsalonen]

Great awesomeness! I will use this feature as soon it is available.

[JoshuaGross]

Hey folks, I wrote my own layer around Mongoose that supports sub-population. It's not architecturally ideal since it's a wrapper around a wrapper, but it does support subpopulate the way we want it to work. Is anyone interested in seeing it if I clean it up and release it?

[jsalonen]

Definitely interested! I'm not sure if I'm the right person to do the patching though, but any ideas on how to work around this issue right now are very welcome!

[JoshuaGross]

Right now it's a hack, albeit a pleasant hack from my perspective :) Hopefully it can be used to patch Mongoose but it's a monkey-patch at the moment. I'll write some more tests, document my code, and come back here within the next week.

[hackfrag]

Will this be added in version 3.0 ?

[aheckmann]

@hackfrag yes

[farhanpatel]

Is this in 3.0 or is it still being added. If it is could we have an example please :)

[aheckmann]

it is not in 3.0. it will be added in an upcoming minor release.

[stowns]

I'm having an incredibly tough time trying to hack this functionality. Do you have any estimate as to when this release might be available? I will ditch my ugly code and work on other areas of my application if true support is in the near future :) . And might I add, thanks for the awesome library.

[JoshuaGross]

Stowns, my hack is working nicely and I now feel good enough about releasing it. I'll get back to you within 24 hrs - need to test against Mongoose 3 first

-- Joshua Gross
Christian / Web Development Consultant / BA Candidate of Computer Science, UW-Madison 2013
414-377-1041 / http://www.joshisgross.com

On Aug 13, 2012, at 6:56 PM, stowns [email protected] wrote:

I'm having an incredibly tough time trying to hack this functionality. Do you have any estimate as to when this release might be available? I will ditch my ugly code and work on other areas of my application if true support is in the near future :)

—
Reply to this email directly or view it on GitHub.

[madhums]

@JoshuaGross hey can you put your hack in a gist? my code is becoming too messy and ugly without this feature...

[JoshuaGross]

Hey @madhums, @stowns, @aheckmann, @farhanpatel, @hackfrag, @jsalonen, etc. My hack (only tested on Mongoose 2.7) is here: https://github.com/JoshuaGross/mongoose-subpopulate

Hope it's helpful.

[jsalonen]

WOW Joshua! Thanks a million!

I'm definitely gonna take a look it that right away!

[stowns]

thanks @JoshuaGross , will check this out later today

[alejonext]

That were to happen if any of the documents does not have the field to the "populate" is requesting to make the search? Would have errors? Technically there would be no error, but will be shown this documen(s)t?

[chovy]

Do nested populate() calls work yet? It looks like there has been some activity, but I was unable to determine how to do it.

I have:

List.findById(req.params.id).populate('items').populate('items.user').exec(fn);

user is a deep nested...is it possible to populate?

[JoshuaGross]

@chovy check out https://github.com/JoshuaGross/mongoose-subpopulate

[JoshuaGross]

@aheckmann can you comment on this? I've been using mongoose-subpopulate in production for quite a while to do subpopulates.

[chovy]

mongoose-subpopulate does not work with express 3.

[aheckmann]

I have a branch started but been pretty busy. Its on its way.

On Wed, Oct 3, 2012 at 11:37 PM, Anthony Ettinger
[email protected]:

mongoose-subpopulate does not work with express 3.

—
Reply to this email directly or view it on GitHubhttps://github.com/LearnBoost/mongoose/issues/601#issuecomment-9131991.

Aaron
@aaronheckmann https://twitter.com/#!/aaronheckmann

[JoshuaGross]

@chovy do you mean mongoose 3? I use mongoose-subpopulate with express 3.

[chovy]

Yes, mongoose 3 I meant.

[whito]

@aheckmann - Looking forward to see your branch!

[chovy]

I could not figure out how to use mongoose-subpopulate

[donnut]

+1 for this feature

[jechenique]

+1 for this one

[norman784]

There is a way to populate an array child of an array?

var notifications = new Schema({
    _id             : { type : ObjectId }
    , from          : { type : ObjectId, ref: 'user' }
    , status                : { type : Number, default : 1 }
    , created_at            : { type : Date }
    , updated_at            : { type : Date }
});

var applications = new Schema({
    _id                     : { type : ObjectId, required : true, ref : 'application' }
    , notifications                 : [notifications]
});

var schema = new Schema({
    name                        : { type : String, required : true }
    , email                     : { type : String }
    , applications                      : [applications]
    , created_at                        : { type : Date }
    , updated_at                        : { type : Date }
});

var User = module.exports = mongoose.model('user', schema);

What I want to accomplish here its to get the name of the notification sender

// works like a charm
User.findOne({_id:'me'}).populate('applications');

// doesn't work
User.findOne({_id:'me'}).populate('applications.notifications.from');

Or if there is another way that do another query to get all the user names manually?

[chovy]

you would need to make "from" an array like you're doing with notifications. The best thing to do is fetch the user for each notification in your callback.

[norman784]

yes, Im just doing the population manually, bull will be cool if populate works with deep references too...

[andyburke]

+1, was hoping I would get to the bottom and see this was in and I just wasn't doing it right. :(

[winduptoy]

+1, any timeline estimate for this feature? It would be insanely helpful.

[aheckmann]

@winduptoy looking like 3.6

[braunsquared]

+1 here too.

More importantly, we should at least be able to populate the children ourselves without having to call toObject() on the parent object first. I can understand the desire to keep the field types consistent after the set call, but mongoose already breaks this rule with the populate function to start with. We should be able to set a referenced field to the complete object without mongoose changing it to merely the mongo id.

Other than this snafu, love mongoose.

[koenoe]

I'm so looking forward to this feature.

[burakkilic]

+1 I also need this feature...

[Mikxail]

+1 for this future

[jedireza]

+1 - looking forward to this one

[ssmereka]

+1 This would be awesome

[alejonext]

+1 That came to software problems, but if you put another symbol would help, for example

User.findOne({_id:'me'}).populate('applications$notifications.from');
// Arrays $
// Simple document point "."

So do not avoid problems ... No?

[chovy]

+2

[Max-Might]

+1 This would be helpful.

[polanm]

+1 That would be super good of you to do.

[skotchio]

plese release that feature in 2013

[devniel]

+1 I need this right now, it will be very useful.

[JoshuaGross]

For those of you that need this right now, you might want to check out mongoose-subpopulate. I've been using it for several months: https://github.com/JoshuaGross/mongoose-subpopulate (I'm the maintainer)

[aheckmann]

Here's the plan for populate in 3.6: https://github.com/LearnBoost/mongoose/pull/1292

[dexcell]

+1 We need this. Hopefully released soon, (because the issue already started year ago)

[alejonext]

I think by popular vote, won this proposal!

[aheckmann]

done. https://github.com/LearnBoost/mongoose/wiki/3.6-pre-release-notes

[jsalonen]

@aheckmann Thank you a lot!!

Also love lean() too so this seems really superb so far!

[andyburke]

Excited for 3.6, this is a good fix, thanks!

[jechenique]

Thanks mr!

On Mar 5, 2013, at 5:26 PM, Andy Burke [email protected] wrote:

Excited for 3.6, this is a good fix, thanks!

—
Reply to this email directly or view it on GitHub.

[donnut]

@aheckmann Great, thank you!

[alejonext]

Thanks!!! Great Job!!!

[flockonus]

That's awesome! I believe it can help with my case. How should I do in this scenario:

var UserSchema, WineRating, WineSchema, mongoose;

UserSchema = new mongoose.Schema({
  wine_ratings: {
    type: [
      {
        type: mongoose.Schema.Types.ObjectId,
        ref: 'WineRating'
      }
    ]
  }
});
mongoose.model("User", UserSchema);

WineRating = new mongoose.Schema({
  wine: {
    type: mongoose.Schema.Types.ObjectId,
    ref: 'Wine'
  }
});
mongoose.model("WineRating", WineRating, 'wine_ratings');


WineSchema = new mongoose.Schema({
  name: String
});
mongoose.model("Wine", WineSchema);

mongoose.model("User").findById(user._id).populate('wine_ratings.wine').exec(function(err, user) {});
/*
gets exception: 
TypeError: Cannot call method 'path' of undefined
    at search (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/model.js:1830:28)
    at search (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/model.js:1849:22)
    at Function._getSchema (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/model.js:1856:5)
    at populate (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/model.js:1594:22)
    at Function.Model.populate (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/model.js:1573:5)
    at Query.findOne (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/query.js:1633:11)
    at exports.tick (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/utils.js:393:16)
*/

Is there something wrong? I am on "3.6.0rc0"

[skotchio]

I have the same issue: https://github.com/LearnBoost/mongoose/issues/1377

[yangsu]

@flockonus @vovan22 I ran to the same issue. This is how I resolved it.

[joeytwiddle]

I was trying to get post.populate("comments comments._creator") working in our project, and had a little success with the following tweaks.

It doesn't seem to work for deeper queries, and the first change breaks some of the existing tests, but I hope it may be of interest for anyone working on this.

https://gist.github.com/joeytwiddle/6129653

[joeytwiddle]

Since my patch kinda sucks, I have tried to contribute by writing a test case instead! https://github.com/LearnBoost/mongoose/pull/1603

[merhawie]

joeytwiddle, i am having the exact same problem and I hope you get it fixed - I am trying to recursively populate but I get the can't find path error too. I don't know if this is by design ...

[joeytwiddle]

Yes the infamous 601. It is by design, at least for the moment. The latest Mongoose release does support deep population, but _only within one schema_.

In our project we need to do deep population across different models quite often, so we have written a helper function:

https://gist.github.com/joeytwiddle/6129676

This allows you to populate descendants of one document to any depth you like! It still requires one extra callback after you have fetched the document, but only one. For example:

deepPopulate(blogPost, "comments comments._creator comments._creator.blogposts", {sort:{title:-1}}, callback);

I hope the use of doc.constructor to get the model will not prove problematic in the future!

_(Thanks to Sunride and the German government!)_

[merhawie]

@joeytwiddle - thank you so much! It worked like a charm. Now I will try to figure out a way to have it do n-deep populate.

[xizhao]

Any updates on this?

[digitaljohn]

@aheckmann I understand your theory on this, but I do disagree that it is an anti-pattern.

Sometimes you do need to perform expensive operations, its inevitable in complex systems and you do need to collect disparate documents together. For example, you may want to do one expensive operation when an document VERY VERY rarely changes and cache the results for performance.

Why do you believe one level of population is 'ok', but more than this is a sign of an anti-pattern?

[buunguyen]

I know this is an old thread, but I've just created a plugin that make it very easy to populate models at any level of depth. I'm posting here in case anyone's interested: https://github.com/buunguyen/mongoose-deep-populate.

The usage is very straightforward, for example:

post.deepPopulate('votes.user, comments.user.followers, ...', cb);
Post.deepPopulate(posts, 'votes.user, comments.user.followers', cb);

Please check out the plugin repo for more information.

[lbeschastny]

@buunguyen nice job!

[joaom182]

Not working for me.

models/missionParticipation.js

var deepPopulate = require('mongoose-deep-populate');
var mongoose = require('mongoose');
var Types = mongoose.Schema.Types;

var missionParticipationSchema = new mongoose.Schema({
        user: {
            type: String,
            default: ''
        },
        mission: {
            type: Types.ObjectId,
            ref: 'Mission'
        },
        images: [{
            type: Types.ObjectId,
            ref: 'Image'
        }]
    }, {
        toJSON: {
            getters: true,
            virtuals: true
        },
        toObject: {
            getters: true,
            virtuals: true
        }
    });

    missionParticipationSchema.plugin(deepPopulate, {
        whitelist: [
            'images',
            'mission',
            'mission.images.poster',
            'mission.images.banner'
        ]
    });

var MissionParticipation = mongoose.model('MissionParticipation', missionParticipationSchema);

module.exports = MissionParticipation;

services/missionParticipationService.js

MissionParticipation.find({user: userID}).deepPopulate('mission.images.poster mission.images.banner').exec(function (err, missionParticipationsDocs) {
    // do the magic.
});

And i get this error on console

TypeError: Object #<Query> has no method 'deepPopulate'

[Swissbite]

@joaom182
I did not yet use deepPopulate, but recording to the docs on http://npm.taobao.org/package/mongoose-deep-populate I would assume that the correct call should be:

MissionParticipation.find({user: userID}, function (err, participations) {
  MissionParticipation.deepPopulate(participations, 'mission.images.poster mission.images.banner', function(err) {
    if (err) {
      //handle it
      return void 0;
    }
    //do your magic stuff. with participations, which are populated in place in the examples 
  })
})

Regards

[joaom182]

I found another way, but I'm concerned about the performance, I try to make a comparison.

The other way uses the async module

MissionParticipation.find({
   user: userID
}).populate('mission').exec(function (err, missionParticipationsDocs) {
   if (err)
      return; // handle error

   async.forEach(missionParticipationsDocs, function (mp, callback) {
      mp.mission.populate('images.poster', 'images.banner', 'prize', function (err, result) {
         callback();
      });
   }, function (err) {
      // forEach async completed
      if(err)
         return; // handle error
      resolve(missionParticipationsDocs);
   });
});

[buunguyen]

@joaom182 you're a bit too fast :). Although I added code to bring deepPopulate to Query, I delayed pushing a new version on NPM so that I could test a bit more.

I've just pushed the new version (0.0.7). So this syntax you used should work after you update the dependency:

MissionParticipation
  .find({user: userID})
  .deepPopulate('mission.images.poster mission.images.banner')
  .exec(cb);

[joaom182]

@buunguyen Awesome!

[vkarpov15]

Can y'all open future issues in the mongoose-deep-populate repo please? Makes everyone's life a little easier :)

[andersLindqvist]

@buunguyen Awesome! The best thing that happened to my project and You made my day! Thanks!