Gatsby: [plugin-netlify-cms] Crawl admin config to allow non-existent optional fields in GraphQL queries

[austincondiff]

Summary

Currently, if optional fields are defined in the Netlify CMS config (config.yml) but not yet used, querying for those fields breaks the build because of the nature of how Gatsby generates it's GraphQL queries. I propose that we crawl the config and generate hidden or ghost entries so this does not happen.

Problem

Even the field they are in use currently, at some point someone might come along and decide that they no longer want a value for that field. Once saved in the CMS, it would effectively break the build as well causing them to wonder why their page is not updating because they build broke and the change was not deployed. If it is left in this state, each subsequent change would not be reflected as well leaving me to try to hunt for which change broke the build.

Solution

I propose that the Nettlfy CMS plugin crawl through config.yml and generate a dummy hidden entry for all collections and lists using all available fields so Gatsby's generated GraphQL queries retains all Netlify CMS fields even though they may not be currently populated.

Basic example

I have a "Page" file collection. In a page I have a field called sectionLink. In my graphql query, I include it and reference it in my code. I add a conditional to say if sectionLink does not exist (is null), do not show this code block.

Initially, I am not able to include sectionLink in my query until it is used in at least one markdown file.

Later after the site is built, and colleague to my client (who is unaware not to delete certain fields) comes along to edit the site. He no longer wants to see a link on this page, so he removes the value for sectionLink in Netlify CMS and saves the page. The build then breaks and the change is not reflected because the new version will not deploy due to the broken build.

After this change, anyone would be able to add and delete what they want without fear of catastrophic repercussion.

Motivation

So developers can build full websites that can be handed over to clients and be confident that it will not break due to a seemingly innocent change in the CMS.

Related links

• https://github.com/netlify/netlify-cms/issues/2426
• https://github.com/gatsbyjs/gatsby/issues/2392
• https://github.com/gatsbyjs/gatsby/issues/6800
• https://medium.com/stories-from-upstatement/in-the-trenches-with-gatsby-contentful-netlify-53a7fea23d37

[sidharthachatterjee]

CC @erquhart

[erquhart]

I believe @talves was looking into something like this. That said, it really does seem like something that should be handled within Gatsby as this issue is not at all specific to Netlify CMS, as shown in the related issues that you linked.

[talves]

Yes, this should be handled by Gatsby. Specifically a Gatsby source plugin! 🙂. Sorry, I missed this issue.
The problem is everyone is creating Gatsby sites with inferred data in their graphQL and expect it to pick up the fields. This is an issue, because it can be solved by creating the types in the schema.

Anyway, good news. I am writing a source plugin right now that will resolve a dynamic schema based on a config. I am putting the final touches on it now.
Good and Bad news: it will not self heal data that has been broken, but will help to debug what the issue is by browsing the graphql tool and see the culprit easier.

The schema will be complete with all fields and not missing inferred fields that are not required, so less errors on fields not showing up in frontmatter. Another feature of the plugin will be to source the nodes with the data from the build, so you do not have to setup gatsby-source-filesystem plugins for all your collections.

[austincondiff]

@erquhart I think the reason they haven’t addressed it is because you can theoretically define your own schema in gatsby-node, however doing this would be redundant when using Netlify CMS as you have already done this in config.yml.

[talves]

@austincondiff Shawn was talking about a Gatsby solution not Gatsby the team, I believe. Specifically a solution like I've been working on.

BTW: You can actually create a Schema for your data in Gatsby in gatsby-node. That is the reason I wrote the plugin, because it is a daunting task and in some cases harder than creating the netlify-cms config. The plugin will allow someone to harness the work from creating the netlify-cms config into creating a graphQL schema in Gatsby for you dynamically.

[erquhart]

Yeah I like that better than creating ghost entries.

[austincondiff]

@talves I’d have to agree. Creating a schema dynamically based on your config would be excellent, much better than ghost entries. That was simply a bandaid I am using currently. How far along are you? Do we have an ETA? Will this be a separate plugin or part of gatsby’s netlify-cms plugin?

[talves]

@austincondiff Unfortunately, I don't have an ETA. Since it is not a paid gig, real work will come first. I work on it as I have time. 😜 It's planned to be it's own plugin for now.

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contributefor more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!

[Pearce-Ropion]

Hi, I was just scrolling and found this and just wanted to mention that I wrote my own implementation of this.

Essentially its a script that runs before gatsby starts that imports the config (ours is defined using manual init using JS) and then iterates (recursively) through each folder/file collection and creates a sample-schema.yml file that contains a sample value in the correct folder location. Then, when gatsby starts, it will read all the sample-schema.yml files as valid items in the collection. Each sample file also has an additional field schema which is set to true.

Then when querying for data, you can just filter out the schema file. So, if for example, you have a folder collection called blog-posts, your query might look like this:

query {
  allBlogPostsYaml(filter: { schema: { ne: true } }) {
    title
    body
  }
}

Here's the code if anyone wants to use it:
https://gist.github.com/Pearce-Ropion/eec2416696fdf0764023b8306d7c46ef
And the associated README:
https://gist.github.com/Pearce-Ropion/f0e92f1befc164aa4fd2c0a1e330fd94
And the webpack config to compile it:
https://gist.github.com/Pearce-Ropion/3ed3135fa4d7cf43fe55d2a5cee8c0ae
If you use this webpack config, just make sure that your CMS config doesn't import any React or CSS.

Feel free to pm me if you have any questions.