Gatsby: [gatsby-transformer-remark] [RFC] gatsby-transformer-remark should reflect/expose sourceInstanceName from gatsby-source-filesystem

[sw-yx]

Summary

we encourage people to name things in gatsby-source-filesystem, but then barely use them in practical gatsby-transformer-remark usage. everyone does hacky stuff like this: https://github.com/gatsbyjs/gatsby/issues/1634#issuecomment-388899348 and regexing on filepath (filepath! regex!!) which means the value of naming things is pretty meaningless.

as described in #1634 we want the ability to filter by namespace and sort by remark's fields in the same layer. there's probably some computer science theorem that says why these logic layers belong together, all i know is i've seen it enough that it makes a ton of sense and is a core design principle of graphql.

Basic example

If the proposal involves a new or changed API, include a basic code example. Omit this section if it's not applicable.

Motivation

Why are we doing this? What use cases does it support? What is the expected outcome?

if gatsby is to scale well, in particular, gatsby themes, we have to make namespacing a lot easier and a lot more of a norm. this small step fixes a glaring hole in a key part of the ecosystem that causes a nasty spike in the learning curve as beginners progress to heavier usecases. i as a fulltime js person with 2 yrs exp on gatsby and nothing better to do can figure it out. it still annoys me to no end. i can't imagine what beginners think of this.

obviously i am completely ignorant of lower level design and technical constraints. but lets please discuss this because i dont yet think it is a big ask and would make sourceInstanceName mean a lot more.

[stefanprobst]

You could use the schema customization APIs to add a field that resolves to the parent File node. For convenience, you could wrap this up in a custom field extension:

// gatsby-node.js
exports.createSchemaCustomization = ({ actions }) => {
  actions.createTypes(`
    type FileInfo {
      base: String
      birthTime: Date
      mtime: Date
      relativeDirectory: String
      relativePath: String
      sourceInstanceName: String
    }
  `)
  actions.createFieldExtension({
    name: 'fileInfo',
    extend() {
      return {
        type: 'FileInfo',
        resolve(source, args, context, info) {
          // If this is only ever used with a direct File parent,
          // you could call `getNodeById` instead of `findRootNodeAncestor`
          // const fileNode = context.nodeModel.getNodeById({ id: source.parent })
          const fileNode = context.nodeModel.findRootNodeAncestor(
            source,
            node => node.internal && node.internal.type === 'File'
          )
          if (!fileNode) return null
          const {
            base,
            birthTime,
            mtime,
            relativeDirectory,
            relativePath,
            sourceInstanceName,
          } = fileNode
          return {
            base,
            birthTime,
            mtime,
            relativeDirectory,
            relativePath,
            sourceInstanceName,
          }
        },
      }
    },
  })

  // You can then use the `fileInfo` extension anywhere you need access
  // to the parent File node, e.g.
  actions.createTypes(`
    type Mdx implements Node {
      fileInfo: FileInfo @fileInfo
    }
  `)
}

[sw-yx]

thank you. i understand there are any number of ways to customize this to get what i want. i’m trying to make it default.

[stefanprobst]

the question is what "default" means in this case and what needs to go where. e.g. we can include a fileInfo extension in core, but it would still be up to plugins/themes to use this, since we cannot assume in core that nodes derive from files.

slightly related to namespacing: i think it would also help if transformer plugins would expose stuff on resolver context (which they can now) to make it easier to compose custom types. a theme could then very easily define the shape of a markdown type - a transformer plugin would not simply transform one node type into another, but provide hooks into the transformation pipeline when composing custom types.

[sw-yx]

ok! i fully support having a nullable fileInfo extension in core!

[freiksenet]

I don't like this idea as it is. I think the core issue is that our parent based search is very limited. I don't think solution in this case is to copy data from parent.

Instead, I propose that for types that have specified types in @childOf, we provide search and sort fields for fields inside the types specified. For mime types, we assume that the type is File. We don't include conflicting fields if there are any fields like that among possible parents.

actions.createTypes(`
  type Mdx implements Node @childOf(mimeTypes: ['application/mdx'])  { # or @childOf(types: ['File'])
    id: ID!
  }
`)

[sw-yx]

sounds interesting. is this childOf directive documented somewhere? cant find it in the docs.

[stefanprobst]

Better making use of @childOf definitely makes sense here -- currently it only affects the parent (which gets child/children fields) but not the child.

sounds interesting. is this childOf directive documented somewhere? cant find it in the docs.

unfortunately not yet

@sw-yx

Also, an easier way than the above that works with what is currently in core is:

// gatsby-node.js
exports.createSchemaCustomization = ({ actions }) => {
  actions.createTypes(`
    type Mdx implements Node {
      fileInfo: File @link(from: "parent")
    }
  `)
}

[sw-yx]

man, directives is some black magic. 😅 i would strongly strongly appreciate docs and some nice examples cc @shannonbux @marcysutton

i trust you all know what to do, i'm just trying to advocate from the POV of a frequent user :)

[stefanprobst]

yeah schema customization docs are a bit of a mess since everything has been dumped into one overlong file here. we should split this up, regroup into recipes, and probably also create a new using-schema-customization example project

[stefanprobst]

also. for explanation: the link extension takes two arguments: by and optionally from.

the by argument sets the foreign-key and defaults to the id field. so in the above case: map the field value on Mdx.fileInfo to File.id.

since the Mdx.fileInfo field is empty, we can use the from argument to say: don't map the current field value to File[by], but take the value from another field, in this case parent which holds a string id for a File node.

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[sw-yx]

maybe we should turn this issue into a schema customization/ directives docs issue?

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!

[programmieraffe]

Workaround: A way of querying by sourceInstanceName (found at https://github.com/sroertgen/homepage/blob/master/src/pages/projects.js)

export const query = graphql`
  query {
  allFile(filter: {sourceInstanceName: {eq: "projects"}, internal: {mediaType: {eq: "text/markdown"}}}, sort: {fields: childMarkdownRemark___frontmatter___date, order: DESC}) {
    edges {
      node {
        id
        childMarkdownRemark {
          frontmatter {
            date(formatString: "DD MMMM, YYYY")
            title
          }
          excerpt
          fields {
            slug
          }
        }
      }
    }
  }
  }`

[sw-yx]

closing since i no longer use gatsby