Gatsby: Gatsby app combined from many sources

[zanehiredevs]

I am creating a site that is pulling from 2 graphql sources, contentful and an external product endpoint and then I'm manually combining the data and doing some transform work then passing it into the default template as data props for all subsequent pages.

This is working fine but I'm doing the combining and transforming in the default template javascript file which means it's processing it with every render/page load. I would rather do this at build time and inject/seed it into all pages.

Where is the best place to do this? I see a lot of hooks in gatsby-node but not sure where and how something like this should happen? Any examples of working with data at build time and injecting it into the default template?

Thanks

[jonniebigodes]

@zanehiredevs you can start by taking a look in here and here.

Also if you want, take a look at the blog starter for instance, where you can see how the data is fetched and passed to the template itself via the context prop which will be available as pageContext in the template component, all done at build time.

If you want i can cook up a small example on how to fetch data and populate the pages with multiple sources in gatsby-node.

[zanehiredevs]

@jonniebigodes nice thanks for the links, I'll give it a shot today and report back!

[jonniebigodes]

@zanehiredevs take a look at here, i've created a minimal gatsby site whose content will be sourced from contentful and another graphql endpoint/source. Take a look at the gatsby-node.js to check on how to fetch the data and create and populate the pages based on the data recieved and how is injected into the page. Two points of notice are gatsby-node.js and src/templates/dummy-template.js to see how the actual mechanism works.

[zanehiredevs]

@jonniebigodes nice thanks that helps a lot! I was able to follow the example and pull from multiple data sources and inject it using the context successfully like your example. But now I am hitting another wall.. I need to be able to merge the data together before it renders based on the page slug being loaded. Another words, run through the contentful query result and match up any products with the external product, merge it together to one object etc. I already have all the merging code my problem is I can't get the contentful data property pulled in gatsby-node because you can't use fragments so it doesn't know anything about the product fragment or other things like contentful image and so on..

So now I am thinking I will just inject the product data and match it at run time since I really don't know any other way to get around this limitation of not being able to use fragments.

[jonniebigodes]

@zanehiredevs with that in mind it seems that some adjustments are in order. One thing to keep in mind is that with gatsby you have more than way to achieve the same result.

If you have a way to correlate the contenful data to the external graphql api it should be too dificult, you can use page queries in your template with variables and fetch the data you need. The query will execute at build time and will be populated and with that you can use the fragments. Like for instance in here.

If you want to go with the route of getting all the data in gatsby-node.js and if you can and you don't mind, please create a reproduction following these steps so that it can be looked at in more detail. I can update the example to at least fetch what i think is the minimal information for contentful images to work with gatsby-image, but asides from that a reproduction would help out.

[universse]

@zanehiredevs You can try https://www.gatsbyjs.org/docs/node-apis/#onCreateNode

exports.onCreateNode = async ({
  node,
  actions: { createNode },
  createContentDigest,
  cache
}) => {
  switch (node.internal.type) {
    case '1st SOURCE':
      await cache.set(node.id, node)
      break

    case '2nd SOURCE': {
      // I'm assuming node.id is the same between both your sources
      const createdNode = await cache.get(node.id)

      // function merge is your own logic
      const newNodeData = merge(createdNode, node)

      // type can be anything
      const type = 'mergeNoded'

      createNode({
        ...newNodeData,
        id: node.id,
        parent: null,
        children: [],
        internal: {
          type,
          contentDigest: createContentDigest(newNodeData)
        }
      })
      break
    }
  }
}

Then when creating pages query from mergeNodes.

[jonniebigodes]

@universse that approach would work in some form, but and i assume that @zanehiredevs is using gatsby-source-graphql to fetch the second portion of the data from the external api, that will be combined with the contentful plugin would lead into some issues as the the source-graphql will take ownership of the node type used for the external api call.

I tried a similar approach with to combine them first and then tried a second approach just to download the images used in the reproduction locally so that they could be used with gatsby-image.

Modifying my gatsby-node.js with the following code:

exports.onCreateNode = async ({
  node,
  actions: { createNode },
  createContentDigest,
  cache,
}) => {
  console.log(node.internal.type)
  if (node.internal.type==='GraphQLSource'){
    console.log(node)
  }
}

leads to the following:

SitePage
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
SitePlugin
Site
ContentfulContentType
ContentfulBlogPost
ContentfulBlogPost
ContentfulBlogPost
ContentfulBlogPost
ContentfulBlogPost
ContentfulBlogPost
ContentfulBlogPost
ContentfulBlogPost
ContentfulBlogPost
ContentfulBlogPost
ContentfulBlogPost
contentfulBlogPostRtbodyRichTextNode
contentfulBlogPostRtbodyRichTextNode
contentfulBlogPostRtbodyRichTextNode
ContentfulAsset
ContentfulAsset
ContentfulAsset
ContentfulAsset
ContentfulAsset
ContentfulAsset
ContentfulAsset
ContentfulAsset
ContentfulAsset
ContentfulAsset
ContentfulAsset
ContentfulAsset
contentfulBlogPostBodyTextNode
contentfulBlogPostBodyTextNode
contentfulBlogPostBodyTextNode
contentfulBlogPostBodyTextNode
contentfulBlogPostBodyTextNode
contentfulBlogPostBodyTextNode
contentfulBlogPostBodyTextNode
contentfulBlogPostBodyTextNode
contentfulBlogPostBodyTextNode
contentfulBlogPostBodyTextNode
contentfulBlogPostBodyTextNode
MarkdownRemark
MarkdownRemark
MarkdownRemark
MarkdownRemark
MarkdownRemark
MarkdownRemark
MarkdownRemark
MarkdownRemark
MarkdownRemark
MarkdownRemark
MarkdownRemark
File
ImageSharp
File
ImageSharp
File
ImageSharp
File
ImageSharp
File
ImageSharp
File
ImageSharp
File
ImageSharp
File
ImageSharp
File
ImageSharp
File
ImageSharp
File
ImageSharp
File
ImageSharp
GraphQLSource
{ id: 'b70d8c13-94e2-5d69-8f4b-42209d97b53a',
  typeName: 'RMAPI',
  fieldName: 'rickAndMorty',
  parent: null,
  children: [],
  internal:
   { type: 'GraphQLSource',
     contentDigest: 'd2e52ae6a50eeaa16c78bd555e42af0f',
     ignoreType: true,
     counter: 98,
     owner: 'gatsby-source-graphql' } }
SitePage
SitePage     
SitePage
SitePage
SitePage
SitePage
SitePage
SitePage
SitePage
SitePage
SitePage
SitePage
SitePage

I checked the graphql endpoint to try and introspect the data and data types and it lead me to believe that some extra work would be necessary, probably in schema stiching or something else i missed.

[muescha]

BTW: that's a good topic for a doc or receipt

[jonniebigodes]

it could be i have no issues with it, but probably avoid gatsby-source-graphql and pair up contentful plugin with another plugin, or two diferent plugins alltogether.

[zanehiredevs]

@jonniebigodes not sure what you mean how to implement

avoid gatsby-source-graphql and pair up contentful plugin with another plugin, or two diferent plugins alltogether.

We are using both of those plugins but what do you mean by "pair it up" with another plugin? How would one accomplish that?

[universse]

For gatsby-source-graphql sources, can identify using node.fieldName instead since it's an option that uniquely identify each graphql source, configured in gatsby-config.js.

[universse]

So I guess the solution is something like this

// gatsby-config.js

module.exports = {
  plugins: [
    ...
    {
      resolve: 'gatsby-source-graphql',
      options: {
        typeName: 'SWAPI',

        // take note of fieldName
        fieldName: 'source2',

        url: 'https://api.graphcms.com/simple/v1/swapi',
      }
    },
    ...
  ]
}

// gatsby-node.js
exports.onCreateNode = async ({
  node,
  actions: { createNode },
  createContentDigest,
  cache
}) => {
  switch (node.internal.type) {
    case '1st SOURCE': // contentful source for eg
      await cache.set(node.id, node)
      break

    case 'GraphQLSource': {
      // node.fieldName is from the config above
      if (node.fieldName !== 'source2') return

      // previously created and cached node from 1st source
      // I'm assuming node.id is the same between both your sources
      const createdNode = await cache.get(node.id)

      // function merge is your own logic
      const newNodeData = merge(createdNode, node)

      // type can be anything
      const type = 'mergeNoded'

      createNode({
        ...newNodeData,
        id: node.id,
        parent: null,
        children: [],
        internal: {
          type,
          contentDigest: createContentDigest(newNodeData)
        }
      })
      break
    }
  }
}

[jonniebigodes]

@zanehiredevs sorry for the confusion, in my earlier comment i was responding to muescha and i forgot to include one thing in my comment, and it was for simplicity purposes and avoid confusion in the recipe/doc.

[zanehiredevs]

@universse that does seem feasible, I'll play with that idea and report back thanks for the help guys!

[universse]

You may need to consider cases where source 1 nodes are created after source 2 e.g. because of a slow network request.

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!