Urql: Add a way to fetchMore and merge results (pagination / infinite scrolling)

[jrobber]

First, super neat library. Great to see something less bloated out there.

It would be great if it worked a little closer to Apollo's Query component and allowed the ability to fetchMore. The executeQuery function provided by the Query callback is fine for refreshing, but is insufficient for infinite scroll scenarios.

[kitten]

This will be out-of-scope for urql itself but I'm working on a normalised cache that will be a drop-in replacement for the built-in document cache.

It's going to be a little different from Apollo's approach, but will be as seamless as changing variables and expecting a full list for instance ✨

Currently, I'd recommend anyone approaching this problem to wrap the useQuery hook or Query component. Even a naive approach would simply vary the variables, collect the data and manually merge it in a local components state

[jrobber]

Thanks for the quick response. I'll do that for now.

[kitten]

@jrobber Let me know if you need any help! The current urql package will likely not include this, but it'd be great to start a "recipes" docs section that explains such an approach.

I'd suspect that even with a normalised cache, some would still love to opt-in to the more basic approaches for the sake of bundlesize and simplicity

[jrobber]

I got something simple working by adding an intermediary component between the query and my component that merges the value in componentWillRecieveProps, but I don't feel like it's going to be a very good recipe. Adding an extra component to store and merge data adds a lot of extra props passing just to achieve the basic scenario for fetching more pages. It also requires that once you render the intermediary component that you have to ensure it stays rendered or you lose your aggregated data array. (IE - I had an if statement that showed a loading spinner on fetching, like in the url docs. I never saw it because the query was fast, but it was enough to cause the intermediary component to unmount and remount and lose everything)

It's quite possible someone smarter than me could do it better/cleaner. But as long as the only hook into getting query results is in the render callback we won't be able to store or modify data on the same component (can't update state from render method). Meaning we have to insert an intermediate, and that makes things messy.

[kitten]

@andyrichardson has picked this up and we're looking to provide two solutions for this that will cover different use cases, but will also overlap:

• A preferable, simple API that also works with the document cache that knows how to "fetchMore" and "merge results". This API might be built into useQuery or a separate hook, but will be very similar to Apollo's fetchMore
• A built-in way to merge pages of a paginated field into one list in the normalised cache that I'm working on, which is a similar solution to connection directives in Apollo manipulating the cache key

[rafamel]

Hi @kitten, just wondering whether the new cache is still being actively worked on?

[kitten]

@rafamel it is indeed but I haven't had much time recently. I'll see whether I can get it done next week. You can follow its development here: https://github.com/kitten/urql-exchange-graphcache

[rafamel]

@kitten That's great news!! I'm willing to bet on urql if that's on the near roadmap. I love the minimal core philosophy and the exchanges-based architecture. It's easily extensible and getting a grasp of what goes on behind the scenes is fairly simple. It does lack a few key features but I believe between the new cache and some minimal investment on exchanges development this can be all powerful :D. It would be great to have a section on the docs to list recommended externally developed exchanges for certain features, as they come. I'm personally planning of getting some going.

[stearm]

@kitten any news about this topic? :)

[JoviDeCroock]

@stearm work is being done on the graphcache, we're putting a lot of thought into this so we can get it right. This would solve the pagination issue

[stearm]

Amazing! Thank you, I would like to be helpful if possible, do you have any topic in which I can give a contribution?

[JoviDeCroock]

Hey,

While testing our new graphCache with a real-world application I encountered this issue.

Maybe the solution used there can help you out:

import React from 'react';
import { useQuery } from 'urql';

const usePreviousValue = (track) => {
  const ref = React.useRef(track);
  React.useEffect(() => {
    ref.current = track;
  });
  return ref.current;
}

const usePagination = (query, keyword, limit = 20, initialFrom) => {
  const [result, setResult] = React.useState([]);
  const [from, setFrom] = React.useState(initialFrom);

  const [{ fetching, data, error }] = useQuery({
    query,
    variables: { limit, from },
  });

  const prevFetching = usePreviousValue(fetching);

  React.useEffect(() => {
    if (prevFetching === true && fetching === false && !error) {
      setResult(d => [...d, ...data[keyword]);
    }
  }, [fetching, prevFetching, data, error]);

  const fetchMore = React.useCallback(() => {
    setFrom(s => s + limit);
  }, [limit]);

  return [{ data: result, error }, fetchMore, data && data.hasMore];
}

export default usePagination;

[kitten]

So as @JoviDeCroock is saying, there's basically two solutions that we think are sufficient to be able to close this issue:

• Implementing a usePagination hook or a component for pages being infinitely appended
• Use @urql/exchange-graphcache and create a resolver & page updater like what you can do with Apollo

I think either solutions aren't necessarily patterns that we'd build in, but there'll be more guides around both! For now I'll close this issue. Happy to answer any questions about this on Spectrum though :) https://spectrum.chat/urql

[ivan-kleshnin]

For the interest of future readers...

The above solution didn't work for me out of the box on server. I suppose @JoviDeCroock simplified his code a bit. Here is a version of mine with some rearrangments:

import React, {useCallback, useEffect, useState} from "react"
import {useQuery} from "urql"

let usePreviousValue = (track) => {
  let ref = React.useRef(track)
  React.useEffect(() => {
    ref.current = track
  })
  return ref.current
}

let usePagination = ({query, indexKey, totalKey, variables}) => {
  let {limit, offset: initialOffset} = variables

  let [offset, setOffset] = useState(initialOffset)

  let [result] = useQuery({
    query,
    variables: {...variables, limit, offset},
  })

  // should be here to immediately pick up the first (cached) result 
  let [index, setIndex] = useState(result.data ? result.data[indexKey] : [])
  let [total, setTotal] = useState(result.data ? result.data[totalKey] : null)

  let prevFetching = usePreviousValue(result.fetching)

  useEffect(() => {
    if (prevFetching == true && result.fetching == false && !result.error) {
      setIndex(prevIndex => [...prevIndex, ...result.data[indexKey]])
      setTotal(result.data[totalKey])
    }
  }, [prevFetching, result.fetching, result.error, result.data])

  let fetchMore = React.useCallback(() => {
    setOffset(prevOffset => prevOffset + limit)
  }, [limit])

  return [
    {data: {[indexKey]: index, [totalKey]: total}, fetching: result.fetching, error: result.error},
    fetchMore
  ]
}

export default usePagination

I used total instead of hasMore. This part is entirely subjective and should mirror your API conventions / preferences. Mine currently consists of two separate queries like

Server (API)

extend type Query {
  post(where: PostWhere): Post!
  posts(where: PostWhere, orderBy: PostOrderBy, offset: Int, limit: Int): [Post!]!
  postsTotal(where: PostWhere): Int!
}

Client (browser)

let queryPosts = `
  query (
    $where: PostWhere!
    $orderBy: PostOrderBy!
    $limit: Int!
    $offset: Int
  ) {
    posts(where: $where, orderBy: $orderBy, limit: $limit, offset: $offset) {
      id slug
      shortTitle title
      createdAt
    }
    postsTotal(where: $where)
  }`

I intentionally don't follow Relay spec to avoid copy-paste thinking and learn from my own mistakes.

[JoviDeCroock]

@ivan-kleshnin we now have an out of the box solution if you are using graphcache

[ivan-kleshnin]

Yeah. But document-based cache was a selling point of URQL to me (and other GraphQL newbies I guess). I'd like to have more basic recipes. I've personally grown to hate ApolloClient because of state links and code spaghetties that are necessary to keep normalized caches in-sync.

Maybe URQL's version of normalized cache is different in practice – I dunno.
Gonna try it, once my bad imprints will expire :)

[roshangm1]

@ivan-kleshnin we now have an out of the box solution if you are using graphcache

Do you have working example for simple pagination ?

[ivan-kleshnin]

Yes. But I'm most probably switching back to Apollo-Client so I don't care about publishing it.

[roshangm1]

Yes. But I'm most probably switching back to Apollo-Client so I don't care about publishing it.

You can at least publish the gist somewhere so that it would help other people who is using this library instead. What is the main reason of switching back to apollo for you ?

[ivan-kleshnin]

you can at least publish the gist somewhere so that it would help other people who is using this library instead.

I'll see if that's possible.

What is the main reason of switching back to apollo for you?

Not enough information available: docs, examples, searchable questions and answers...
I work alone with GraphQL at my projects and/so I don't have enough time, or health for that matter, to lurk through such big code bases each time a question arises. Apollo 3.0 looks more similar to URQL, by the way – the are picking up some good design decisions.

Nevertheless, I keep tracking the progress of this project.
Pretty sure, I'll give at another look in a year or so.

[JoviDeCroock]

Hey @ivan-kleshnin

I'm sorry to hear that, if you got any pointers as to what parts of the docs we can improve, what kind of examples you want to see those would be more than welcome. This way we can improve this part of the project in the future.

[ivan-kleshnin]

https://blog.logrocket.com/exploring-urql-from-an-apollo-perspective/

Apollo has a lot of API that urql does not. For example, Apollo gives you direct access to the cache. That can be really useful if you are working against a GraphQL schema that does not return the types needed for cache invalidation.

For example ^ this. I've found I sometimes really need a direct cache access and I don't even know if URQL allows it.

The core of the current docs seems to be centered around streams and library extensions which a casual user exploring the possibilities of the library won't, probably, find very important. Introductory chapters, instead, are lacking content (context, design decisions, etc). Just my IMO.

If you ask me, I would even go as further as removing all non-hook API examples. 99% of users exploring URQL are probably using hooks and that HOC alternatives take a lot of effort (first for you to write, than for us to scroll-down & ignore 😞 ).