Json-api: Dealing with millions of items in relationship

[gnewton]

It is not clear to me from the spec how to implement the scenario where an item has millions of relationship links. Clearly this cannot be included in a single payload (ok, not 'cannot' more like 'ideally for most use cases'). I have looked at some related issues #509 but it does not clarify things for me.

Thanks,
Glen

[beauby]

Do you mean a million of related resources (within the data key)? Or actually a million of links (within the links key)?

For the former, you can provide a link instead of the linkage data, and offer paginated access to the related resources in an other endpoint.

For the latter, I fail to see how that could happen?

[dvogel]

I think @gnewton's question is getting at the implications of how relationships reject side-loading.

If the related resource is implemented by an external app, the app receiving the side-loading request could proxy upstream (almost always a bad idea) or it could reject the side-loading request with a 400 response. That case is pretty clear.

However, if the primary and related resources are implemented in the same app, then the primary consideration for whether you would want to support side-loading is whether the payload size would grow sufficiently large as to require pagination. The obvious options are all pretty disappointing:

• It would be poor practice to return a 400 depending on the number of records associated with the primary resource record of a particular request. For example, a request to /posts/1/comments?include=posts.comments returning 200 while /posts/2/comments?include=posts.comments returning 400.

• Publishing an API that supported side-loading for a particular relationship and having to augment it to no-longer support side-loading as the size of your data grew would be a disappointing development. For example, if comments on posts were limited to 100 but the cap was removed, allowing the side-loading response size grow too large.

• The conservative commitment seems to be rejecting side-loading for all relationships for which the record count either could grow unbounded or is limited by business logic only. This is what I choose to do, but it really seems to push the usefulness of side-loading below the break-even point when deciding whether to support side-loading at all.

[beauby]

How about setting a global pagination scheme? With links it should be
transparent to the client.

On Wed, 11 Jan 2017 at 22:38, dvogel [e-mail] wrote:

I think @gnewton https://github.com/gnewton's question is getting at
the implications of how relationships reject side-loading.

If the related resource is implemented by an external app, the app
receiving the side-loading request could proxy upstream (almost always a
bad idea) or it could reject the side-loading request with a 400 response.
That case is pretty clear.

However, if the primary and related resources are implemented in the same
app, then the primary consideration for whether you would want to support
side-loading is whether the payload size would grow sufficiently large as
to require pagination. The obvious options are all pretty disappointing:

-

It would be poor practice to return a 400 depending on the number of
records associated with the primary resource record of a particular
request. For example, a request to
/posts/1/comments?include=posts.comments returning 200 while
/posts/2/comments?include=posts.comments returning 400.

-

Publishing an API that supported side-loading for a particular
relationship and having to augment it to no-longer support side-loading as
the size of your data grew would be a disappointing development. For
example, if comments on posts were limited to 100 but the cap was removed,
allowing the side-loading response size grow too large.

-

The conservative commitment seems to be rejecting side-loading for all
relationships for which the record count either could grow unbounded or is
limited by business logic only. This is what I choose to do, but it really
seems to push the usefulness of side-loading below the break-even point
when deciding whether to support side-loading at all.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/json-api/json-api/issues/1120#issuecomment-272002941,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACHPYjHSsYKDWan51oJTVlfmhbrXVEAjks5rRUvbgaJpZM4K5itC
.

[jamesplease]

@gnewton , I think the pagination section of the spec provides enough information to solve this problem:

1. The page query parameter is reserved for pagination
2. Included resources _can_ be paginated: they just need the pagination links

Let's say you want to fetch article with ID of 3, which happens to be rather popular, and has 3 million comments. For efficiency, you only want to load the first 50 comments, and the authors of those first 50 comments. You could do something like this:

/article/3?include=comments.author&page[comments][number]=1&page[comments][size]=50

If that seems ugly, it could be page[comments.number]. Or you could do something else, too. I would just pick whatever syntax that lets you convey pagination for individual included resources, and then be consistent about it.

Does that help at all?