Json-api: Proposal: Limit Inclusion of Related Documents

[ZenCocoon]

Developing our new API following json-api proposal, I feel the desire to add a limit to how many related documents we include. This could bring some nice flexibility and performance improvements on the requests.

For example, a blog post have many comments, but we only want to get the last 3 comments, we could use:

GET /posts/1?include=comments:3

For more complexe requests, it could end up like:

Include authors from last 3 comments
GET /posts/1?include=comments:3.author

Return last 3 comments and there respective authors
GET /posts/1?include=author,comments:3,comments:3.author

Return last 3 comments and all comments authors
GET /posts/1?include=author,comments:3,comments.author

How does that sound?

[steveklabnik]

I'm not totally opposed, but I find this syntax kinda ugly.

[butsjoh]

I would stick with the same semantics as sort which uses either sort or sort[DOCUMENT_TYPE]. Do you thank that would work?

[ZenCocoon]

@steveklabnik Glad you find the idea interesting, any semantic in mind that would feel cleaner?
@butsjoh How would you apply the sort semantic to inclusion limits?

[butsjoh]

A quick stab at it, still to discuss of course. A limit value with a - in front means "last x of" and a normal value "first x of". I agree there is some difference between the two. Using your examples:

For example, a blog post have many comments, but we only want to get the last 3 comments, we could use:

GET /posts/1?include=comments&limit[comments]=-3

For more complex requests, it could end up like:

Include authors from last 3 comments

You put the limit on comments and it should not give you back more than 3 authors. If authors would have a type of people (like in the json api doc) then i could be that in the linked section there could be more then 3 if other related documents use people as a type.

GET /posts/1?include=comments.author&limit[comments]=-3

Return last 3 comments and there respective author

For me this is no different then the previous. You only express in the include that comments should be seen as well.

GET /posts/1?include=comments,comments.author&limit[comments]=-3

Return last 3 comments and all comments authors

This one is indeed not easy to express if you take the "type" into consideration. For sort this is not really an issue because it's about sorting all of the documents of the given type. Here you could have a limit applied to a type which is linked by two different relations (e.g: comment has author and posts has creator which both have type people).

so specifying all the people like below does not really express that you would need as it would probably also add the creator of the post to it

GET /posts/1?include=comments,comments.author&limit[comments]=-3&limit[people]=+

could work but expressing "comments.author" in limit seems to be not so nice.

GET /posts/1?include=comments,comments.author&limit[comments]=-3&limit[comments.author]=+

so ... let's indeed start discussing it because according to me it seems their should be a suggestion to express this.

I agree with @steveklabnik that the format you are proposing is not that easy to parse and with my initial reply i just wanted to refer to sticking with normal query params in the url.

[ZenCocoon]

Indeed, the query params format is much more common, scalable and clear. Don't know how I could miss that ;-)

Interesting issue on how to express the inclusion limit.

[gr0uch]

I've been watching this issue, and I've a few concerns:

• What might be the default behavior regarding how many resources can be included? Include all or limit by default? Obviously trying to include all related resources wouldn't be scalable.
• How to paginate with respect to included resources?

[dgeb]

@ZenCocoon I would also solve the problem with params like @butsjoh suggests. However, the negative limits aren't needed - just specify a sort order (which is covered in the spec).

@daliwali I'd like to address your questions with some options for pagination in the spec post-1.0.

[ZenCocoon]

@daliwali Good point, negative limits are to be done by the sort order. The point of default / pagination is also important to be covered as you mention. What's your position and ideas on this?

@dgeb I think that this entire feature can be dealt with, post 1.0, some more important maters are to be taken care of before, including #208

[dgeb]

Pagination is now covered in the spec. The reserved page query param can be used to express limits (e.g. page[limit]=5).

[NinoFloris]

Sorry to dig this up again but what was decided surrounding the original issue of limiting included documents?
Is this just a server-side decision right now? No client-side control at all?

E.g. If I have a resource type posts with the relationship mentions that links to people type resources. How would I:

1. Page the amount of relationships that I return per resource type? Something like /posts/1?page[limit]=10&page[mentions;limit]=5&page[mentions;offset]=0? Where the ;would separate the operation to a relationship path.
2. Page the amount of included documents that I return? As I might want all the relationship identifiers for a specific type but I would only want a select amount of included resources. e.g. the proposed /posts/1?include=mentions:3 would almost work but ideally you would probably use something like /posts/1?include=mentions;page[limit]=3 where the ; would now again separate operations to a relationship path and , would still have its original meaning. This would probably have to be limited to a subset of operations as for instance you cannot sort on included resources without getting into inter-included sort order issues.
3. Finally how would this look if I wanted to limit the returned relationships and/or included resources for resources that were included for the primary data relationships? I guess this would be through relationship paths with syntax equal in functionality to that shown above right?
   /posts/1?page[limit]=10&page[comments.likes;limit]=5&include=comments.likes;page[limit]=5

Is this something that has been looked at and discarded as 'too much' or 'unpractical' (which I would understand) or is this new terrain?

[masterspambot]

I think that limitation of included resources is not in scope of client side implementation....

[dgeb]

I will reopen this issue for further discussion since it's possible that we might want to make some more specific recommendations here.

[koraykoska]

Compound document pagination should be really considered to be added to the specifications as included resources are currently not really usable as they can either be limited without the possibility to request additional ones or include all related resources which would just end up being very inefficient and not very scalable.

[wimleers]

Restricting includes to only the nth or -nth resource referenced by a relationship also came up as a feature request in the Drupal implementation of JSON API today: https://www.drupal.org/project/jsonapi/issues/3006217.