Mongoose: When i use select with populate in find query, the document returns with unexpected parameters.

[Alex-Verevkine]

it seems that "populate" overrides "select" function

e.g: the returns documents contains both "select" & "populated" fields,
however, i expect to see only the fields that in the "select"
(for sure it was like that in previous versions)

Find Query:

VModel.findById(vId)
            .select('fRef')
            .populate([{
                path: "vJ",
                model: "JModel",
                populate: {
                    path: "bmarks",
                    model: "BmarkModel"
                }
            }, {
                path: "fRef"
            }, {
                path: "cand"
            }, {
                path: "sources",
            }, {
                path: "candInfo",
                populate: [{
                    path: "infoTabs.tabRows",
                    model: "CandInfoQModel",
                    select: "-__t -__v",
                    populate: {
                        path: "requirements",
                        model: "RequirementModel",
                        select: "-__v -__t"
                    },
                }, {
                    path: "infoTabs.customTabRows",
                    model: "CandInfoCustomQModel",
                    select: "-__t -__v",
                    populate: {
                        path: "requirements",
                        model: "RequirementModel",
                        select: "-__v -__t"
                    }
                }],
            }])
            .exec((err, requestedV) => {
                    console.log("requestedV",requestedV)
})

Expected Result :

requestedV:{
    _doc:{
        _id:ObjectID,
        fRef:{
            ...
        }
    }
}

Actual Result :

requestedV:{
    _doc:{
        _id:ObjectID,
        candInfo:[...],
        cand:[...]
        fRef:{
            ...
        },
        sources:[...],
        vJob:{
            ...
        }
    }
}

So the select didn't effect on response document and document returned with all properties from populate query.

Tested with:
Node 8.10.0
Mongoose 5.1.2
MongoDB 3.6.5

[lineus]

@Alex-Verevkine Can you please also add your relevant schema and Model creation? It will help us reproduce the issue. This example shows the select effecting the result as expected:

6546.js

#!/usr/bin/env node
'use strict';

const mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/test');
const conn = mongoose.connection;
const Schema = mongoose.Schema;

const otherSchema = new Schema({
  val: String
});

const schema = new Schema({
  name: String,
  other: {
    type: Schema.Types.ObjectId,
    ref: 'other'
  }
});


const Other = mongoose.model('other', otherSchema);
const Test = mongoose.model('test', schema);

const other = new Other({
  val: 'xyz'
});

const test = new Test({
  name: 'abc',
  other: other._id
});

async function run() {
  await conn.dropDatabase();
  await other.save();
  await test.save();

  let pop = await Test.findById(test.id)
    .select('other -_id')
    .populate('other', 'val -_id');


  console.log(pop);
  return conn.close();
}

run();

Output:

issues: ./6546.js
{ other: { val: 'xyz' } }
issues:

[Alex-Verevkine]

Hi @lineus,
For recap,
it seems that "populate" overrides "select" function

e.g: the returns documents contains both "select" & "populated" fields,
however, i expect to see only the fields that in the "select"
(for sure it was like that in previous versions)

Here is the code example with schemas and models.

const mongoose = require("mongoose");
const connection = mongoose.connection;
const Schema = mongoose.Schema;

const innerTestSchema = new Schema({
    title: String,
    desc: String
});

const testSchema = new Schema({
    title: String,
    inner: {
        type: Schema.Types.ObjectId,
        ref: 'innertest'
    }
});


const InnerTest = mongoose.model('innertest', innerTestSchema);
const Test = mongoose.model('test', testSchema);

const inner = new InnerTest({
    title: "inner",
    desc: "inner document"
});
const test = new Test({
    title: "test",
    inner: inner._id
});

(
    async () => {
        await mongoose.connect('mongodb://localhost/test');
        await connection.dropDatabase();
        await inner.save();
        await test.save();

        const result = await Test.findById(test._id)
            .select('title')
            .populate({
                path: "inner",
                model: "innertest"
            });

        console.log("result", result);
        return connection.close();
    }
)();

The Actual result that i receiving contains nested document object:

{
    _id: 5b2bae0cf60cc35a4bbc270c,
    title: 'test',
    inner: {
        _id: 5b2bae0cf60cc35a4bbc270b,
        title: 'inner',
        desc: 'inner document',
        __v: 0
    }
}

I am expecting result only with fields that i passed in select, it means that result need to be returned without 'inner' object.

Expected Result Example

{
    _id: 5b2bae0cf60cc35a4bbc270c,
    title: 'test',
}

[Alex-Verevkine]

@lineus , @vkarpov15
After some investigation i found this mongoose patch that caused my issue.

In my opinion if select query isn't contains some property, there is no need to return it in response object, even if it was provided in populate.

i mean, i want to select what i put in "select", seems very straightforward.

[lineus]

Thanks for clarifying @Alex-Verevkine. I'll dig into this ASAP.

[vkarpov15]

@Alex-Verevkine this is expected behavior since mongoose 4.11, if you populate() a field we'll add that in to your projection if your projection is exclusive. We assume that if you populate() you want that property in the result.

Can you elaborate on what your use case is for this behavior?

[Hyperblaster]

I noticed this also. Previously we had really small fast queries, but since the patch the returned data is far larger as it includes all the populated fields, where as it used to only include the fields specified in the select.

Is there a way that we can change it back selectively? With middleware?

We have default populates setup for each of our models, but then allow the user to request which fields they want to return to optimise their queries.

[vkarpov15]

@Hyperblaster can you provide an example of what your code looks like? I want to help but hard to do without a general idea of how your code is using populate and select

[Hyperblaster]

Sure thing @vkarpov15 ,

I basically have a switch statement that decides which fields are relevant for which kinds of content stored in our database, so when the requesting user hits our API we by default populate the key bits of information that are most likely needed,

However for listing pages (Where we request for example, the last 500 documents of a type) we usually select only the fields relevant to keep the request size down and only return whats needed rather than the whole article.

This used to work and allow the user to specify the fields to select, but now if you specify a field to be populated, it adds itself into the select list.

//We already establish the model to use and the criteria above
var myMongooseQuery = ModelName.find(criteria);

/////////////////////////////////

switch (typeName) {
    case 'article':
        myMongooseQuery.populate('author relatedLinks', 'title other fields')
    break;
    case 'product':
        myMongooseQuery.populate({
            path: 'storefront',
            select: basicFields,
        });
        myMongooseQuery.populate({
            path: 'tiles',
            select: basicFields, //'title _type color bgColor',
            match: {
                status: {
                    $nin: ['deleted', 'draft']
                }
            },
        })

        myMongooseQuery.populate({
            path: 'products',
            select: basicFields + 'data.publicData', //'title _type color bgColor',
            match: {
                status: {
                    $nin: ['deleted', 'draft']
                }
            },
        })
        break;


}

/////////////////////////////////

//If the user specified that they only want say 'title' and 'date'
if(req.query.select && req.query.select.length) {
    //Then tell mongoose to only select those fields
    myMongooseQuery.select(req.query.select);
}

/////////////////////////////////

myMongooseQuery.exec(function(err, results) {
    if(err) {
        return res.send(err);
    }

    //Then we send the results
    return res.send(results);
})

I actually understand the decision behind this, and i'm sure that years ago when i was first starting out with Mongoose i was probably complaining that 'populate' fields didn't automatically select. But i built our whole system around the way it used to work and it's making a lot of our queries waaay larger than they need to be since that change.

I could write a large amount of code to try and intersect the specified populate and select fields, but it will be an enormous amount of rewrite for our app.

If there is a way to switch the behaviour per query, or via middleware it would be greatly greatly appreciated. The other option for us currently would be to downgrade back to V3, which i'm not keen on.

PS: Mongoose is such a damn good product, thanks for your work on it!

[gabayto]

Hi,
i actually strongly agree with @Alex-Verevkine,
i also think that the "fix" PR was done because of a miss-understood issue.

"select" should do the selection. much more strait straightforward.
and enable agility by client to select what field they want (more easily)

[vkarpov15]

Any objection to us adding an option for this at the schema level? This fix has definitely caused more headache than we thought, but undoing it will be messy.

[Hyperblaster]

For me that would be fine as i would just enable it across all my schemas.

But is it difficult to do it as middleware?
Or per query, something like this (can’t think of a good property name haha)

Model.find()
.lean()
.select(fieldString, {unlessPopulated:true})
.exec()

Var options = {onlyIfSelected:true}
Model.find()
.populate(fields, select, options)

Model.find()
.populateBehaviour(‘include’)

At the schema level solves my problem but per query might give more flexibility for others

On 26 Aug 2018, at 2:42 am, Valeri Karpov [e-mail] wrote:

Any objection to us adding an option for this at the schema level? This fix has definitely caused more headache than we thought, but undoing it will be messy.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[mamsoudi]

Hi,
Just like @Hyperblaster, we're facing the very same issue. The problem is that we let users decide which properties they want to retrieve. Let's assume we have a schema that looks something like this:

export const FinancialDocumentSchema = new Mongoose.Schema(
    {
        officialTitle: {
            type: String,
            required: true,
        },
        financialCode: {
            type: String,
            required: true,
        },
        registrationNumber: {
            type: String,
            required: true,
        },
        customer: {
            type: Mongoose.Schema.Types.ObjectId,
            ref: "Customer",
        },
        participants: [
            {
                type: Mongoose.Schema.Types.ObjectId,
                ref: "Participant",
            },
        ],
    },
    {
        timestamps: true,
    },
);

Now user can ask for financialCode, officialTitle, customer.title in the query param of our API.

The problem is that I try to conditionally populate customer. Basically what I do, is that I construct a query then I'll check if customer is amongst the properties user has asked for and if it was, I add populate to the query for customer. But this way population overrides the selected properties and instead of customer.title populates and returns all properties in customer.

After lots of researching I ended up using a custom function that basically does what select is supposed to do after I've populated schemas I want.

I guess what @Hyperblaster is proposing might be a solution to this.

[Hyperblaster]

Legendary! Thanks valeri

On 15 Sep 2018, at 12:24 am, Valeri Karpov [e-mail] wrote:

Closed #6546 via 950d223.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.