Gatsby: Is there a way to query/gain access to other markdown files/nodes when resolving a Graphql query?

[luczaki114]

Is there a way to query/gain access to other markdown files/nodes when resolving a Graphql query?

I have pages as markdown files with a front matter field for a list of widget names. These widgets are markdown files of their own with a bunch of front-matter fields that I will use as props in a react component.

I have all of the widgets in a separate folder and am not directly using them to create pages. What I would like to do is create a query that functions like this:

{
    allMarkdownRemark(filter: { fileAbsolutePath: {regex : "/src\/pages/"} }) {
        edges {
            node {
                excerpt(pruneLength: 400)
                html
                id
                frontmatter {
                    templateKey
                    path
                    date
                    title
                    // This frontmatter widgetlist has the names of the markdown files that I need to resolve the widgetList on the node.
                    widgetList {
                        widget 
                    }
                }
                widgetList {
                    widget {
                        widgetStyle
                        isCenter
                        isFullWidth
                    }
                }
            }
        }
    }
}

I am currently stuck because I have the names of the widgets that are supposed to be on each page in the front matter, but to resolve the widgetList type, I need to to find the markdown nodes in question in the page-components folder.

I used the code from gatsby-transformer-remark to get started creating my custom plugin in plugins/markdown-extender. Gatsby-transformer-remark has a file extend-node-type.js which I have been modifying. But a lot of this code is completely foreign to me other than the Graphql bits. @KyleAMathews would you be able to shed some light on this so I can start digging in a better direction? That would be much appreciated!

Here's a link to the repo:
https://github.com/luczaki114/Lajkonik-Gatsby-NetlifyCMS-Site

[calcsam]

What you want is a mapping between widgets and your frontmatter. You can set this up in your gatsby-config.js file:

https://github.com/gatsbyjs/gatsby/blob/master/www/gatsby-config.js#L7-L9

[kripod]

Is there a more detailed explanation about this? I would like to map markdown files to other markdown files by two distinct attribute names. How could that be done? @KyleAMathews even some clues would be highly appreciated.

I could only find this code snippet regarding the issue: https://github.com/gatsbyjs/gatsby/blob/751d3cf5e3dadbd06daa59f87090b124ec3f5a76/packages/gatsby/src/schema/infer-graphql-type.js#L199-L253

[pieh]

I'm not sure if I understand correctly but You can do something like this:

In frontmatter link to other markdown file (that's for single link - you can do array or object if you need more):

---
title: New Beginnings
date: "2015-05-28T22:40:32.169Z"
linkedMakdownFile: "../hello-world/index.md"
---

and then query:

  markdownRemark(<your_filter_here>) {
    html
    frontmatter {
      title
      linkedMakdownFile {
        childMarkdownRemark {
          frontmatter {
            title
          }
        }
      }
    }
  }

If that's what you want then no additional configuration is needed

[kripod]

@pieh Sounds great, but what I would like to achieve is similar to the AuthorYaml solution.

books/lorem-ipsum.md:

---
title: "Lorem ipsum"
date: "2015-05-28"
author: John Doe
---

Book plot

authors/john-doe.md:

---
title: John Doe
birthdate: "1979-01-02"
---

Author introduction

Those two should be connected by Book.author -> Author.title.

[pieh]

Oh, we currently only map to ids. But we can make it work with some additional custom code in gatsby-node.js:

// we use sourceNodes instead of onCreateNode because at this time plugins
// will have created all nodes already and we can link both books to authors
// and reverse link on authors to books
exports.sourceNodes = ({ boundActionCreators, getNodes, getNode }) => {
  const { createNodeField } = boundActionCreators

  const booksOfAuthors = {}
  // iterate thorugh all markdown nodes to link books to author
  // and build author index
  const markdownNodes = getNodes()
    .filter(node => node.internal.type === `MarkdownRemark`)
    .forEach(node => {
      if (node.frontmatter.author) {
        const authorNode = getNodes().find(
          node2 =>
            node2.internal.type === `MarkdownRemark` &&
            node2.frontmatter.title === node.frontmatter.author
        )

        if (authorNode) {
          createNodeField({
            node,
            name: `author`,
            value: authorNode.id,
          })

          // if it's first time for this author init empty array for his books
          if (!(authorNode.id in booksOfAuthors)) {
            booksOfAuthors[authorNode.id] = []
          }
          // add book to this author
          booksOfAuthors[authorNode.id].push(node.id)
        }
      }
    })

  Object.entries(booksOfAuthors).forEach(([authorNodeId, bookIds]) => {
    createNodeField({
      node: getNode(authorNodeId),
      name: `books`,
      value: bookIds,
    })
  })
}

and in your gatsby-config.js use this mapping config:

  mapping: {
    'MarkdownRemark.fields.author': `MarkdownRemark`,
    'MarkdownRemark.fields.books': `MarkdownRemark`,
  },

and example query:

{
  markdownRemark(frontmatter: {title: {eq: "New Beginnings"}}) {
    id
    frontmatter {
      title
    }
    # author 
    fields {
      author {
        frontmatter {
          title
        }
        # all books of author
        fields {
          books {
            frontmatter {
              title
            }
          }
        }
      }
    }
  }
}

[kripod]

@pieh Thank you for all your efforts! 😊 I'm wondering whether the developer experience could be improved, though...

[pieh]

There's always room for improvement for sure. I'm currently working on schema related things and will add this to my list (why that list is only growing and not getting smaller 😠 ).

Way to to be able to specify on what field to we should link nodes would indeed be great as current mapping is best suited for json/yaml data (where we define id ourselves) or for programmatic solution (like one I pasted above). It would be great to do something like:

'MarkdownRemark.fields.author': {
  type: `MarkdownRemark`,
  fieldToJoinOn: 'frontmatter.title'
}

[kripod]

Also, I think it would be a good idea to add optional path selectors (probably regex/glob) for nodes with the type MarkdownRemark.

[KyleAMathews]

It's not documented yet but there is a way to add mappings directly between nodes e.g. in gatsbyjs.org, we map from an author field in markdown to an authors.yaml file https://github.com/gatsbyjs/gatsby/blob/36742df34648f6a392f5b37d297399266a76e047/www/gatsby-config.js#L7

[pieh]

Right, but mapping currently only tries to link on node ids and apart from non json/yaml type sources ids are not user defined but are generated, so I didn't suggest to use it.

This is something I will try to tackle with my schema adventures to allow to define fields we want to use to link on. Simple cases (like one I presented above with pseudo mapping config) are straight forward to implement, but I didn't yet consider how to handle cases when fields we want to join on are not single objects but f.e. arrays or there are linked nodes in the mix. There is also certain problem that we are certain that ids are unique and so this 1-1 mapping (or N-N if field is array of ids). When we would join on other fields we don't know if it will be 1-1 or 1-N - so this is propably something that would need to be another configurable option if we want to get list or first found item

[thomasheimstad]

@pieh Great explanation on how to map the books to the author. How would you go about to add multiple authors per book? Doing a

    node.frontmatter.authors.forEach(author => {
        const authorNode = getNodes().find(
            node2 =>
              node2.internal.type === `MarkdownRemark` &&
              node2.frontmatter.title === author
          )
    ...etc...

works in the sense that the following query shows each book

    fields {
        slug
        authors {
          frontmatter {
            title
          }
          # all books of author
          fields {
            books {
              frontmatter {
                title
              }
            }
          }
        }
      }

But the authors query doesn't show all authors, only the last in the array of authors in the frontmatter:

    authors:
      - Author1
      - Author2

[pieh]

@thomasheimstad

We would only need to change code in gatsby-node.js (just note that I didn't have time to test, so might be bugged, but You should propably get at least idea from it):

// we use sourceNodes instead of onCreateNode because at this time plugins
// will have created all nodes already and we can link both books to authors
// and reverse link on authors to books
exports.sourceNodes = ({ boundActionCreators, getNodes, getNode }) => {
  const { createNodeField } = boundActionCreators

  const booksOfAuthors = {}
  const authorsOfBooks = {} // reverse index

  // as we can have multiple authors in book we should handle both cases
  // both when author is specified as single item and when there is list of authors
  // abstracting it to helper function help prevent code duplication
  const getAuthorNodeByName = name => getNodes().find(
    node2 =>
      node2.internal.type === `MarkdownRemark` &&
      node2.frontmatter.title === name 
  )

  // iterate thorugh all markdown nodes to link books to author
  // and build author index
  const markdownNodes = getNodes()
    .filter(node => node.internal.type === `MarkdownRemark`)
    .forEach(node => {
      if (node.frontmatter.author) {
        const authorNodes = node.frontmatter.author instanceof Array 
          ? node.frontmatter.author.map(getAuthorNodeByName) // get array of nodes
          : [getAuthorNodeByName(node.frontmatter.author)] // get single node and create 1 element array

        // filtered not defined nodes and iterate through defined authors nodes to add data to indexes
        authorNodes.filter(authorNode=> authorNode).map(authorNode => {
          // if it's first time for this author init empty array for his books
          if (!(authorNode.id in booksOfAuthors)) {
            booksOfAuthors[authorNode.id] = []
          }
          // add book to this author
          booksOfAuthors[authorNode.id].push(node.id)

          // if it's first time for this book init empty array for its authors
          if (!(node.id in authorsOfBooks )) {
            authorsOfBooks[node.id] = []
          }
          // add author to this book
          authorsOfBooks[node.id].push(authorNode.id)
        })
      }
    })

  Object.entries(booksOfAuthors).forEach(([authorNodeId, bookIds]) => {
    createNodeField({
      node: getNode(authorNodeId),
      name: `books`,
      value: bookIds,
    })
  })

  Object.entries(authorsOfBooks).forEach(([bookNodeId, authorIds]) => {
    createNodeField({
      node: getNode(bookNodeId),
      name: `authors`,
      value: authorIds,
    })
  })
}

[thomasheimstad]

@pieh Brilliant, thank you! Can confirm, your code works great. Updated the query to something like this:

      fields {
        slug
        authors {
          frontmatter {
            title
          }
          # all books of authors
          fields {
            books {
              frontmatter {
                title
              }
            }
          }
        }
        books {
          frontmatter {
            title
          }
          # all authors of books
          fields {
            authors {
              frontmatter {
                title
              }
            }
          }
        }
      }

[guilhermedecampo]

I cleaned up a bit for my needs that is Gatsby + NetlifyCMS = <3

My collections

  - name: "pages"
    label: "Pages"
    files:
      - file: "src/content/pages/home.md"
        label: "Home page"
        name: "home"
        fields:
          - {label: "Template Key", name: "templateKey", widget: "hidden", default: "home"}
          - {label: "Body", name: "body", widget: markdown}
          - label: "List of works"
            name: "works"
            widget: "list"
            fields:
              - {label: Work, name: work, widget: relation, collection: works, searchFields: [title, explanation], valueField: title }

  - label: Works
    name: works
    folder: src/content/works
    create: true
    fields:
      - { label: "Template Key", name: "templateKey", widget: "hidden", default: "work" }
      - { label: Name, name: title, widget: string }
      - { label: Explanation, name: explanation, widget: markdown }
      - {label: "Featured image", name: "featuredImage", widget: "image"}
      - label: "Images"
        name: "images"
        widget: "list"
        fields:
          - {label: Image, name: image, widget: image }
          - {label: "Row(starts at 1)", name: row, widget: number }
      - label: "Tags"
        name: "tags"
        widget: "list"
        fields:
          - {label: Tag, name: tag, widget: relation, collection: tags, searchFields: [title], valueField: title }
  ```

// we use sourceNodes instead of onCreateNode because
// at this time plugins will have created all nodes already

exports.sourceNodes = ({ boundActionCreators: { createNodeField }, getNodes, getNode }) => {
// iterate thorugh all markdown nodes to link page to works
const { homeNodeId, workNodeIds } = getNodes()
.filter(node => node.internal.type === MarkdownRemark)
.reduce(
(acc, node) =>
node.frontmatter.templateKey && node.frontmatter.templateKey.includes('home')
? { ...acc, homeNodeId: node.id, homeWorks: node.frontmatter.works.map(item => item.work) }
: node.frontmatter.templateKey &&
node.frontmatter.templateKey.includes('work') &&
acc.homeWorks.includes(node.frontmatter.title)
? { ...acc, workNodeIds: [...acc.workNodeIds, node.id] }
: acc,
{ homeNodeId: '', homeWorks: [], workNodeIds: [] },
)

createNodeField({
node: getNode(homeNodeId),
name: 'works',
value: workNodeIds,
})
}

```

Hope helps someone. Best!

[samyilias]

It's not documented yet but there is a way to add mappings directly between nodes e.g. in gatsbyjs.org, we map from an author field in markdown to an authors.yaml file

https://github.com/gatsbyjs/gatsby/blob/36742df34648f6a392f5b37d297399266a76e047/www/gatsby-config.js#L7

@KyleAMathews That worked great for me , it's not mentioned in the docs but I had to use the gatsby-transformer-yaml in order to make it work