Graphene-django: Avoid calling count unless it is required

[crucialfelix]

It is checking the count for every query, but isn't using it for anything AFAICT except hasNextPage

DjangoObjectType and DjangoConnectionField

[4.490] SELECT COUNT(*) AS "__count"
FROM "nsproperties_apt"

[0.159] SELECT "nsproperties_apt"."id", ...

This is a massive performance hit on postgresql. If I am selecting only 10 items and I don't ask for the count then it should not call count. Even if the number of records is < 10 it still calls count.

For hasNextPage you can just iterate one past.

Related to #162 - it would be nice to get totalCount when we need it, but only pay the performance price then.

[mjtamlyn]

You may find that https://github.com/photocrowd/django-cursor-pagination helps here, I wrote it to do "true" cursor style pagination, and have a version of DjangoConnectionField which uses it instead. Note that requires an explicit, unique ordering of the objects to work.

[hyusetiawan]

@mjtamlyn how would django cursor pagination help? I am missing something here, could you provide an example on how to use it?

[mjtamlyn]

The current version of Connection paginates using the index within the list you're currently at, and calculates the total count to evaluate whether there is another page after the current one. Cursor pagination identifies ordering data from the field, and loads "the first n after the ordering value x". By loading n+1 items, you know whether there is at least one item afterwards, but you have no idea whether there is 1 or 10000. This style of query is also much more efficient for the database when loading much later pages.

More details of the approach in a blog post by David Cramer

[danpalmer]

To clarify here, that pagination library is good, but currently graphene-django does not provide a way to customise pagination behaviour or the PageInfo type.

It would probably be possible, but it would require a lot of hacks and would likely be very brittle to upgrades in the library. I suggest that we wait for customisable pagination support.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[crucialfelix]

Here is the alternate DjangoConnectionField we've been using: https://gist.github.com/crucialfelix/d14506cdb9d3a04427a72aadcad30af4

This avoids calling len() or .count() on the QuerySet which has very
poor performance on PostgresQL. The value is only used to determine if there
is another page.

NOTE: This does NOT support the GraphQL args "before" or "last"

[sliverc]

If relay doesn't require a length to create a slice like proposed at https://github.com/graphql-python/graphql-relay-py/issues/16 a fix for graphene-django would be easy.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ledil]

What is the official solution here?

[macdja38]

@mjtamlyn Any chance you could make your version of DjangoConnectionField that uses django-cursor-pagination public? I'm in a similar situation and it would be super useful.

[crucialfelix]

It's in the gist linked from the comment above: https://gist.github.com/crucialfelix/d14506cdb9d3a04427a72aadcad30af4

[macdja38]

@crucialfelix forgive me if I'm mistaken but that looks like it uses offset based pagination?

[crucialfelix]

Yes it does— the same way as the original DjangoConnectField does.
It's implements the approach @mjtamlyn was describing in https://github.com/graphql-python/graphene-django/issues/177#issuecomment-322413008

For relay the offset is encoded in the after hash: cursor_to_offset(after_cursor)

[macdja38]

@crucialfelix mjtamlyn had mentioned they have a version of DjangoConnecionField true cursor based pagination, as opposed to offset based. That's what I was interested in.

[stickperson]

@crucialfelix how do you go about using SimplifiedDjangoConnectionField? I tried using it instead of DjangoFilterConnectionField, so I used DjangoFilterConnectionField instead of DjangoConnectionField. That didn't work, so I dug into the code a bit and found this bit of code that uses DjangoFilterConnectionField.