gatsby-source-graphql re-use data across multiple pages without multiple fetch

Created on 26 Nov 2019 · 18Comments · Source: gatsbyjs/gatsby

Hey all, a question about GraphQL in Gatsby and if it is possible to do the following

Assuming a GraphQL API endpoint is fetched using gatsby-source-graphql, is it possible to call the API from multiple pages where the data is identical, only once

For example, a type 'Promotions' is required across multiple pages, but the structure and data returned is the same, in the current way we are obtaining data, Promotions is essentially being asked for 3 times, we want to ask for the promotions once, and then plug those in to the schema where they are needed to then pass on via props in the normal way...

SomePageQuery {
    promotions {
        slug
    }
    uniqueFieldForIndexPage
}

SomeStaticQuery {
    promotions {
        slug    
    }
   uniqueFieldForStaticQuery
}

SomeGatsbyNodeCreatePagesQuery {
    promotions {
        slug
    }
    uniqueFieldForDetailsPage
}

This is a very simplified example, however what you are seeing is that a page (Promotions index) a separate page (Lets say, Games Index where we want to show some promotions there using the static query as an example) and some programatically generated pages (Promotion Details for each promotion) - currently in this format Gatsby is making a POST request to the API endpoint for EACH page, which means we are fetching the same promotions 3 times per build. Is it possible to call the promotions only once, and somehow then add them to the schema for each page during the build process?

stale? GraphQL question or discussion

Source

ArcaneTSGK

All 18 comments

When Gatsby builds its GraphQL server from data sources (like gatsby-source-graphql) it does not fetch that data multiple times depending on how often it is used. It fetches it once from your endpoint, then builds each page from its internal GraphQL server that exists during the build command. So in your case, you don't need to worry about Gatsby 'making a POST request to the API endpoint for EACH page'. That is not happening.

Now, if you want to keep from repeating the same fields for the query multiple times (ie writing promotions, slug, etc in the same way multiple times). You can look here for how to use fragments with GraphQL.

graysonhicks on 26 Nov 2019

Thank you for your response, and whilst I understand that would make sense, this is not the case, atleast from what I can see.

Let me expand upon my example -

On our API endpoint I can see multiple POST requests are being made, for every individual query that is written in the gatsby application source code, a POST request is sent to the API.

Consider these queries

index.tsx

MyAPIQuery {
  managedContent {
    homePage {
      promotions(count: 3) {
        slug
      }
    }
  }
}

promotions.tsx

MyAPIQuery {
  managedContent {
    promotionsIndexPage {
      promotions(count: 100) {
        slug
      }
    }
  }
}

I can see a post request for both at my endpoint.

Any further advice would be much appreciated, thank you

ArcaneTSGK on 27 Nov 2019

cc @freiksenet @vladar

LekoArts on 27 Nov 2019

What does the gatsby-config.js file look like? Do you have any queries happening in gatsby-node.js?

graysonhicks on 27 Nov 2019

I have only included relevant code from the config

gatsby-config.js

const path = require(`path`);
const fs = require("fs");
const { buildSchema } = require("graphql");

let url = "http://localhost:50070/graphql";
let authorization = {
  Authorization: `Bearer ${process.env.MYAPI_TOKEN}`
};

...

,
    {
      resolve: "gatsby-source-graphql",
      options: {
        typeName: "MyAPIQuery",
        fieldName: "MyAPIQuery",
        url: url,
        headers: authorization,

         createSchema: async () => {
          const sdl = fs.readFileSync(`${__dirname}/schema.graphql`).toString();
          return buildSchema(sdl)
        },
      },
    },

gatsby-node.js

exports.createPages = async ({ actions, graphql }) => {
  return new Promise((resolve) => {
    //++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    //  Generic Page Template
    //++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    return graphql(`{
      MyAPIQuery {
        managedContent {
          siteContent {
            genericPages {
              metaContent {
                description
                keywords
                title
              }
            primaryContent
            pageHeading
            secondaryContent
            secondaryTitle
            slug
          }
...

      data.genericPages.forEach((pageData) => {
          actions.createPage({
            path: `/${pageData.slug}`,
            component: path.resolve(`src/templates/genericPage/genericPage.tsx`),
            context: {
              pageData: pageData,
            }
          });
      });

resolve();

...

Please note in gatsby-node.js there are also queries for other generated pages, such as promotions details pages (so a page for each individual promotion) ...

So in essence maybe this makes my specific use case clearer, we have a page under /pages called promotions.tsx, this has a page query which contains

metaContent { .... }
otherPromotionIndexPageTypes...
promotions {
  slug
  ...otherproperties
}

In gatsby-node.js we have the details pages

metaContent {...}
promotions {
  slug
  ...otherproperties
}
promotionDetailPageTypes...

Please let me know if I am able to make it clearer or provide a better example

Thank you

ArcaneTSGK on 27 Nov 2019

Hm, I don't see why it would be hitting the API more than once. It should only be hitting it during the 'source nodes' step. Anything after that (as long as there aren't duplicates in the config which is what I was looking for), should only be querying against the internal server. Sorry, not sure why you are seeing that behavior. Are you seeing this during both develop and build or just one?

graysonhicks on 27 Nov 2019

Have only tested this against gatsby develop so far, I will try gatsby build and post the results here today.

ArcaneTSGK on 28 Nov 2019

It is the same with gatsby build, multiple post requests can be seen at the API endpoint, essentially, one request per query on the site.

I'm not 100% certain but it seems to be attempting to read the data (call the API) at the createPages step, as there is one query in particular it struggles with, whereby if it takes longer than 30 seconds for the API to respond with data it times out (this is a restriction imposed by the API), and this only ever happens on the createPages step, and it is a query within createPages that is timing out at the API level.

ArcaneTSGK on 28 Nov 2019

@ArcaneTSGK gatsby-source-graphql plugin uses Apollo Link to fetch data from the remote source.

By default, the plugin creates a simple apollo-link-http. But you can provide your own link (see the example with createLink option in plugin docs).

So you could take one of the existing link implementations or create your own with optimizations for your schema.

vladar on 3 Dec 2019

Using Apollo Link shouldn't cause hits for each page query as I understand it. Am I wrong on that @vladar ? As noted here that should still only happen once during the sourceNodes step, not every time that query as used. If that were the case, you could technically have data that did not match (ie fetch for one page query happening a few seconds before another and a data field changed on the same piece of content).

graysonhicks on 3 Dec 2019

@graysonhicks Yes but it runs every page query separately. So if two pages Foo and Bar have the same field baz in their query - this field will be requested twice anyway - once in Foo query and once in Bar query.

The issue description is about having baz requested only once (maybe in Foo query and then dedupe it from the Bar query).

vladar on 3 Dec 2019

Right, but the issue describes this duplicate request happening multiple times to the _remote data source_ not the Gatsby internal GraphQL server that exists at build time. If I have a source plugin that gets images from Instagram, it should:

get those images _once_ during sourceNodes
add nodes to internal GraphQL server
any page queries get baz as many times as needed, but only against the internal server

This issue is saying essentially that the data source (instagram.com in this example) is getting hit with a request for each page query, versus just the one request during sourceNodes. Gatsby should not be hitting the external data source for each Page query. It should be hitting the internal GraphQL server per Page query. This issue says they are seeing a request to the external data source via Postman for each Page query. That sounds like a problem.

graysonhicks on 3 Dec 2019

This is not the case now. Plugin gatsby-source-graphql doesn't source data from the remote GraphQL API into the internal Gatsby store. It uses makeRemoteExecutableSchema from graphql-tools to "proxy" queries to a remote during running page queries step. So this data bypasses gatsby store.