Strapi: Option to exclude relations in REST

[Mat-thieu]

What is the expected behavior?
There is currently no way to exclude relations. They're all included by default and I can't imagine this is something you want often, it also creates far more complex queries with larger responses, resulting in lesser performance overall.

My suggestion would be to exclude relations by default and only include them when they're requested through a filter (query parameter) or when specifically included by default during attribute creation.

Something like this;
/categories?include=products
Or with nested relations
/categories?include=products.reviews
Or with multiple top level relations and nested relations
/categories?include=products.reviews,vendors

[derrickmehaffy]

While I'm not terribly in agreement with nothing by default, this does make sense.

So I'll add my :+1:

[FaustTheThird]

I made something similar (my REST requests are returning only requested fields and requested fields of relations). I can share the sources of that with you if you want

[Mat-thieu]

@FaustTheThird Sure I'd be interested to see it

[FaustTheThird]

@Mat-thieu Unfortunately I don't know how to connect with you through GitHub, please write to my E-Mail: [email protected]. Don't want to fill this issue with flood :)

[derrickmehaffy]

@FaustTheThird Just upload the code to a github Repo and share a link here.

[FaustTheThird]

@Mat-thieu
So here's a little example that may be useful for this task:
in _api\utilsservices\utils.js:_

extractObjectFields: (object, fields) =>
  {
    let output = {};
    object = object.toJSON ? object.toJSON() : object;
    fields.map(function (val) {
      let field = val.split('.');
      if(field.length > 1) {
        output[field[0]] = {};
        output[field[0]][field[1]] = object[field[0]][field[1]];
      }
      else
      output[field[0]] = object[field[0]];
    });
    return output;
  },

And here's the example of using this method:

find: async (ctx) => {
    let fields = [];
    let output = [];
//Determine if our query has 'include' param, save it to variable and exclude from query (else it will determine 'include' as a table column
    if (ctx.query.include) {
      fields = ctx.query.include.split(',').map(function (val) { return val.trim(); });
      ctx.query = _.omit(ctx.query.toJSON ? ctx.query.toJSON() : ctx.query, ['include']);
    }

    if (ctx.query._q) {
      data = await strapi.services.example.search(ctx.query);
    } else {
      data = await strapi.services.example.fetchAll(ctx.query);
    }
    //if we need to extract fields from result
    if(fields) {
      data.forEach(
         function (object) {
         output.push(strapi.services.utils.extractObjectFields(object, fields));
      });
     return output;
    }
return data;
},

I'm currently at my job so I can't write more flexible code (like using arrays of relations etc.) anyway I hope that this may be helpful for you to achieve what you want.

[Mat-thieu]

@FaustTheThird Thanks for sharing, but I'm wondering if this does actually prevent the server from querying the relation on the database?

Also @derrickmehaffy Do you happen to know if the GraphQL request only selects the field from the database that your request?

[derrickmehaffy]

@Mat-thieu GraphQL is very specific and you can go pretty deep. If you have the graphQL plugin installed you can go to localhost:1337/graphql and use the playground to test

[Mat-thieu]

@derrickmehaffy Yeah I tried it out, GraphQL seems great!
I was just wondering if in the core it's requesting all fields with all relations from the database and then it filters the field you requested with GraphQL afterwards

[derrickmehaffy]

Honestly not sure there, I know very little about GraphQL and had never used it before I started working with Strapi.

[maturanomx]

I was just wondering if in the core it's requesting all fields with all relations from the database and then it filters the field you requested with GraphQL afterwards

You can add a debug attribute on you database configuration file:

config/environments/developer/database.json

{
  "defaultConnection": "default",
  "connections": {
    "default": {
      "connector": "strapi-hook-bookshelf",
      "settings": {
      },
      "options": {
        "debug": true
      }
    }
  }
}

Hopefully, you can see enough to answer that question. Let us know!

I should agree that the default behavior should be including the relations just when explicitly request for them.

[lauriejim]

You can add autoPopulate: false in your model to not populate it.

https://github.com/strapi/strapi/blob/master/packages/strapi-generate-api/templates/mongoose/service.template#L25

You can also make a condition or add params in this function to give relations you want populate.

[Mat-thieu]

@maturanomx Useful, thanks! I just tested it and it appears that regardless of the attributes in Graphql, it's requesting everything. So Graphql is really only transforming/filtering the response but has no effect on the database queries it seems.

I think this might really take a hit on performance if your database grows and has a lot of relations, in the background everything related is queried and retrieved.

@lauriejim Thanks, that's good to know. I think a global toggle for autopopulate with an out-of-the-box option to include related resources each request would be sufficient for this issue, what do you think?

[kamalbennani]

@Mat-thieu I agree with you buddy, in the GraphQL context we're fetching extra things that we are not supposed to fetch, the purpose of this PR https://github.com/strapi/strapi/pull/1948 is to make sure 1st that in GraphQL context only the fields that you request and the relation that you ask for are the only things that are being fetched, 2nd to give you this possibility to easily implement the behavior you asked for in the REST API

[mnlbox]

You can also make a condition or add params in this function to give relations you want populate.

@lauriejim can you guide about this? I can't find anything in docs about this.

[Mat-thieu]

@kamalbennani Great, thanks! Looking forward to the feature!

[Mat-thieu]

@kamalbennani So does this disable fetching everything by default or do you still have to disable autopopulate on each model?

[mnlbox]

@Mat-thieu I disable it for one model and it's working well but I want to disable only specific relation (not all relation)

[kamalbennani]

@Mat-thieu in the REST API I think that the default behavior is to fetch all the relations related to a given model but in GraphQL no extra fetching will be done unless you request it via the fields that you ask for or if you filter on a given relation (in this case only the relation used in the filter will be populated)

[kamalbennani]

@Mat-thieu I think that the feature that you requested has its place in strapi, once my PR is merged we can easily implement it

[Mat-thieu]

@kamalbennani Sweet, thanks again

[Aurelsicoko]

You can use two features to return the correct result:

• autoPopulate: false in the model's attributes: it won't populate but it will return the IDs.
• private: true in the model's attributes: it will populate (except if you also set autoPopulate to false) and it won't return this field.

So, if you use both properties, you can avoid the unnecessary population and hide the fields 👍

[derrickmehaffy]

@Aurelsicoko I think as a footnote here, selecting specific relations based on the need and also going more than 1 level deep also applies

[derrickmehaffy]

You suggestion is more geared towards global application of this :stuck_out_tongue:

[mnlbox]

@Aurelsicoko based on your guideline is it possible to disable only specific relation in specific endpoint?
For example I want to have a summary of the models items with /articles that contain title, text and want to have full attribute list for example title, text, author (it's a relation) in /articles/full.

[Aurelsicoko]

@mnlbox Not at all, it will be applied on every request. You need to edit the ORM's queries in the services to customise the results to fit with your needs.

[abdonrd]

Also interested in includes/excludes fields!

[Aurelsicoko]

I created a new public card on Product Board, feel free to give me more insights. I think this feature can be very useful and easily implemented (https://portal.productboard.com/strapi/c/31-exclude-include-fields-from-query-request).

I added more details about my thoughts:

• Fetch every relation:
  /categories

• Exclude the product relation but returns the others:
  /categories?exclude=products

• Or only include the products relation:
  /categories?include=products

• It should also work with nested relations:
  /categories?include=products.reviews

• Or with multiple top-level relations and nested relations:
  /categories?include=products.reviews,vendors

I'm closing the issue to keep the repository as clear as possible, but feel free to comment on the issue or on Product Board.

[Mat-thieu]

@Aurelsicoko I like it, but I think having an option to exclude all relations would be nice also, something like . ?exclude=* perhaps?

[Aurelsicoko]

@Mat-thieu To me, it means that we should exclude everything. We will use the same filter to exclude fields from the query. It means that we can exclude simple field and relational field.

GET /categories?exclude=products,name&include=author.articles

[sajjad-shirazy]

ffff, it's 3 days i'm facing memory out of stack on my server because of the autoPopulate.

[Aurelsicoko]

You can easily disable the autoPopulate feature by setting the autoPopulate property to false for the fields which don't need to be populated.

[AlexanderTserkovniy]

Some useful information:

1. setting autoPopulate to false will cause an issue when you want to filter by this relation in alpha-26.

curl http://localhost:1337/contentpages?brands.name=COM

{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

{ MongoError: $regex has to be a string
    at Connection.<anonymous> (/Users/strapi/cms/node_modules/mongodb-core/lib/connection/pool.js:443:61)
    at Connection.emit (events.js:189:13)
    at Connection.EventEmitter.emit (domain.js:441:20)
    at processMessage (/Users/strapi/cms/node_modules/mongodb-core/lib/connection/connection.js:364:10)
    at Socket.<anonymous> (/Users/strapi/cms/node_modules/mongodb-core/lib/connection/connection.js:533:15)
    at Socket.emit (events.js:189:13)
    at Socket.EventEmitter.emit (domain.js:441:20)
    at addChunk (_stream_readable.js:284:12)
    at readableAddChunk (_stream_readable.js:265:11)
    at Socket.Readable.push (_stream_readable.js:220:10)
    at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
  ok: 0,
  errmsg: '$regex has to be a string',
  code: 2,
  codeName: 'BadValue',
  name: 'MongoError',
  [Symbol(mongoErrorContextSymbol)]: {} }

1. Setting the "private": true will just break the entire application.

[2019-04-25T20:15:08.925Z] info File changed: /Users/strapi/cms/api/contentpages/models/Contentpages.settings.json
[2019-04-25T20:15:08.932Z] info The server is restarting

[2019-04-25T20:15:14.156Z] debug ⛔️ Server wasn't able to start properly.
[2019-04-25T20:15:14.159Z] error Cannot read property 'replace' of undefined
TypeError: Cannot read property 'replace' of undefined
    at isPrimitiveType (/Users/strapi/cms/plugins/graphql/services/Aggregator.js:21:22)
    at extractType (/Users/strapi/cms/plugins/graphql/services/Aggregator.js:143:10)
    at getFieldsByTypes (/Users/strapi/cms/plugins/graphql/services/Aggregator.js:254:5)
    at _.reduce (/Users/strapi/cms/plugins/graphql/services/Aggregator.js:72:26)
    at /Users/strapi/cms/plugins/graphql/node_modules/lodash/lodash.js:914:11
    at /Users/strapi/cms/plugins/graphql/node_modules/lodash/lodash.js:4911:15
    at baseForOwn (/Users/strapi/cms/plugins/graphql/node_modules/lodash/lodash.js:2996:24)
    at /Users/strapi/cms/plugins/graphql/node_modules/lodash/lodash.js:4880:18
    at baseReduce (/Users/strapi/cms/plugins/graphql/node_modules/lodash/lodash.js:911:5)
    at Function.reduce (/Users/strapi/cms/plugins/graphql/node_modules/lodash/lodash.js:9683:14)
    at getFieldsByTypes (/Users/strapi/cms/plugins/graphql/services/Aggregator.js:68:12)
    at generateConnectionFieldsTypes (/Users/strapi/cms/plugins/graphql/services/Aggregator.js:253:27)
    at formatConnectionGroupBy (/Users/strapi/cms/plugins/graphql/services/Aggregator.js:285:19)
    at Object.formatModelConnectionsGQL (/Users/strapi/cms/plugins/graphql/services/Aggregator.js:448:25)
    at models.reduce (/Users/strapi/cms/plugins/graphql/services/Resolvers.js:244:42)
    at Array.reduce (<anonymous>)
    at Object.buildShadowCRUD (/Users/strapi/cms/plugins/graphql/services/Resolvers.js:38:17)
    at Object.generateSchema (/Users/strapi/cms/plugins/graphql/services/Schema.js:141:36)
    at Function.initialize (/Users/strapi/cms/plugins/graphql/hooks/graphql/index.js:145:78)
    at /Users/.nvm/versions/node/v10.15.3/lib/node_modules/strapi/lib/hooks/index.js:52:31
    at after (/Users/.nvm/versions/node/v10.15.3/lib/node_modules/strapi/lib/hooks/index.js:153:39)
    at /Users/.nvm/versions/node/v10.15.3/lib/node_modules/strapi/node_modules/lodash/lodash.js:9999:23
    at Strapi.once (/Users/.nvm/versions/node/v10.15.3/lib/node_modules/strapi/lib/hooks/index.js:164:17)
    at Object.onceWrapper (events.js:277:13)
    at Strapi.emit (events.js:189:13)
    at Strapi.EventEmitter.emit (domain.js:441:20)
    at onFinish (/Users/.nvm/versions/node/v10.15.3/lib/node_modules/strapi/lib/hooks/index.js:44:12)
    at _.forEach (/Users/strapi/cms/node_modules/strapi-hook-mongoose/lib/index.js:628:13)

Strapi:
v3.0.0-alpha.26

Plugins:
content-manager
content-type-builder
email
graphql
settings-manager
upload
users-permissions

Also might be because some fields in MongoDB are named with big letter in the beginning, like Name or Content

Hope it helps.

[uF4No]

@AlexanderTserkovniy same issue here with beta 11. Applying private to a relation attribute inside the model crashes the app with error Cannot read property 'replace' of undefined

[BriantAnthony]

Tip: as of January 2020, you are able to pass an empty object and empty array which allows you to exclude all relations.

const { data } = await strapi.services.YOUR_MODEL_NAME.find({}, [])

Optionally, if you want to populate only certain fields:

const { data } = await strapi.services.YOUR_MODEL_NAME.find({}, ['FIELD_NAME1', 'FIELD_NAME2'])

[ningacoding]

by doing ['FIELD_NAME1', 'etc']

const entities = await strapi.services.inventory.find({ user: user.id }, ['name', 'location']);

im getting:

MissingSchemaError: Schema hasn't been registered for model "inventory".

by removing find's second param, everyting works.

[derrickmehaffy]

@ningacoding we need your model file to understand how your relations are setup.

[ginishsharma]

by doing ['FIELD_NAME1', 'etc']

const entities = await strapi.services.inventory.find({ user: user.id }, ['name', 'location']);

im getting:

MissingSchemaError: Schema hasn't been registered for model "inventory".

by removing find's second param, everyting works.

Tip: as of January 2020, you are able to pass an empty object and empty array which allows you to exclude all relations.

const { data } = await strapi.services.YOUR_MODEL_NAME.find({}, [])

Optionally, if you want to populate only certain fields:

const { data } = await strapi.services.YOUR_MODEL_NAME.find({}, ['FIELD_NAME1', 'FIELD_NAME2'])

@BriantAnthony Hi, I'm using the query you mentioned above and it is working fine with relation type fields but it's not working with Dynamiczone field

For eg. I have this dynamiczone field
"extra_options": { "type": "dynamiczone", "autoPopulate": false, "components": [ "extra-options.file-upload", "extra-options.photo-gallery" ] },

when I try this in the query its working fine and get all two components

entities = await strapi.services.model.find(ctx.query , ['extra_options'])

but if I need only "extra-options.file-upload" then it's not working for this my query is

entities = await strapi.services.model.find(ctx.query , ['extra-options.file-upload'])

[PaRoxUs]

https://github.com/strapi/strapi/blob/master/packages/strapi-generate-api/templates/mongoose/service.template#L25

Just a note, there is still no reference to the autoPopulate parameter into the documentation

[Mat-thieu]

Reading this back I still stand by excluding everything out of the box.

All in all, having implicit includes might initially feel like you're saving fractions of time, but on the long run it doesn't, with every relation added you need to think about where it's automatically outputted and make sure all usages of the updated routes don't output the extra relation if this is not desired. With explicit relations everything is as clear as day, for every request you can see exactly which relations will be used, making it easier to reason about in code, resulting in easier debugging.

I reckon adding a prototyping flag in the global config or build command would be ideal, this would mean that everything's on debug mode and all top-level relations are always returned.

[matrixersp]

const { data } = await strapi.services.YOUR_MODEL_NAME.find({}, [])

It would be great if this also works for create({...}, []), it'll save an extra query when creating a new record.

[omaksi]

I agree with OP here, I'd stick to the logic of an SQL SELECT and include columns as needed. Return empty if none specified, a * to return all. I wouldn't even go with the prototyping flag, the * pretty much covers the cases.

[wahengchang]

is this feature on the production ? i really need it

[derrickmehaffy]

Not currently @wahengchang you will need to customize your controllers and the population array (or use GraphQL)

[u007]

example working on my end:
api/product/services/product.js

?_exclude[]=relation1

module.exports = {
  find(params, populate) {
    // support for params _exclude=[] to exclude field
    const exclude = params._exclude
    console.log("service find", params, populate)
    var fields = [];
    if (exclude) {
      fields = ['relation1', 'relation2']
      for (var c = 0; c < exclude.length; c++) {
        const field = exclude[c];
        const pos = fields.indexOf(field);
        if (pos >= 0) {
          fields.splice(pos, 1);
        }
      }
      populate = fields
      delete params._exclude
    }

    return strapi.query('display').find(params, populate);
  },

};