Gatsby: [Discussion] Feature: GraphQL schema snapshots for all data sources to solve undefined/empty data issues

[jsanchez034]

Description

Currently there are issues with GraphQL schemas produced from data sources where at the moment gatsby develop or gatsby build is executed the data shape is incomplete, parts are in an empty state or in a different type than they would be if the source data was filled out. Below are a few example issues..

• #3009 -- WordPress data source with some properties set to false when empty even though they should be an object or null
• #1517 & #2881-- Contentful data source with some empty/undefined fields

It would be great if as the data shape evolves on the data source side, you could create test pages or pieces of content that are fully filled out, meaning no empty fields. Then on the Gatsby side you could run a cli command called something like gatsby snapshot-schemas which would fetch the current data sources, run the source data through there regular plugin data normalization paths, run the data through the existing infer GraphQL schema code and then finally at the end take the schemas generated and save them off to a folder in /src called schemas.

On subsequent builds Gatsby could skip over inferring of the GraphQL schemas when it sees schemas defined in /src/schemas. These schema snapshots could then committed into a sites repo and allow for data shape changes to require new snapshots instead of just data source changes. These schema snapshots open up many possibilities such as the validation of incoming data shape changes. Schema snapshot diffs could be shown in Gatsby CLI as well when gatsby snapshot-schemas is ran again once initial schemas have been saved.

I would love some feedback on the idea itself from the contributors of the various source plugins. If the idea passes the smell test, I would like some feedback on what format the snapshots should be saved to. Maybe the GraphQL schema language using something like gestalt-graphql would be nice.

[m4rrc0]

It seems like an interesting idea since it should work regardless of the source. It would definitely work for my usecase.
While I am pretty confident about simple fields, I am wondering how it would work for assets though.

[jsanchez034]

Good question about assets. I think for assets most data sources represent them as a string with a URL or path to the asset. The schema snapshot would just represent the type as a string with the URL/path or an asset object which has a string property with the URL/path.

[jquense]

Seems like an interesting idea, though why not skip the extra indirection of a snapshot and and define the schema fully? Like if we're going go a head and make fully complete data example, it seems like not much more work to directly specify the gql schema while also being clearer?

[KyleAMathews]

@jquense that's what I'm thinking. You could either run a command to "snapshot" the dynamically generated schema which would write out a file with the schema in normal graphql form or you could directly write the same form yourself.

[jsanchez034]

Yeah I definitely like the idea of keeping it open to write your schema from scratch along side the "snapshot" feature. The benefits I see from the "snapshot" feature is the ability to ease yourself into learning the GraphQL schema syntax and reducing the amount of boilerplate code needed to get up and running with Gatsby and any source plugin.

I am currently looking into the internals of Gatsby's schema inferring logic in ..
https://github.com/gatsbyjs/gatsby/tree/master/packages/gatsby/src/schema
And getting familiar with how GraphQL schemas are defined. I will try to create a proof on concept PR soon.

[jquense]

I totally understand easing users into a new thing, however I think plugins are a bit different. Authors are generally expected to understand gql already and users shouldn't generally have to mess with anything other than querying when consuming a source plugin.

That said tho the schema language for defining said schema is probably less complicated than querying, in terms of getting up to speed. I think it'd be time better spent allowing plugin authors to provide a .graphql file than working on snapshot ting a comprehensive data object, which may not even be enough to properly define the schema wholly.

I would like to mention to that manually defining the schema is _already_ possible and supported via the gatsby api, albeit with graphql-js not the terser schema language.

[jsanchez034]

I understand that plugin authors should know gql already and users
shouldn't need to mess with schemas. But the issue is that the schema
produced from inferring from the source data is many times incomplete based
on the current state of the source data (see the issues I mentioned in the
opening of this issue). So the way I see it is in order for users not to
really mess around with having to write there schema's from scratch, they
could set up the source data to complete, take a snapshot of the schema and
then have Gatsby use that snapshot going forward. This way the user avoids
having to write schemas from scratch in order to solve empty/undefined data
causing the graphql queries to fail.

Also could you elaborate a little more on how " I think it'd be time better
spent allowing plugin authors to provide a .graphql ..." would work?

On Thu, Dec 28, 2017 at 8:27 AM Jason Quense [e-mail]
wrote:

I totally understand easing users into a new thing, however I think
plugins are a bit different. Authors are generally expected to understand
gql already and users shouldn't generally have to mess with anything other
than querying when consuming a source plugin.

That said tho the schema language for defining said schema is probably
less complicated than querying, in terms of getting up to speed. I think
it'd be time better spent allowing plugin authors to provide a .graphql
file than working on snapshot ting a comprehensive data object, which may
not even be enough to properly define the schema wholly.

I would like to mention to that manually defining the schema is already
possible and supported via the gatsby api, albeit with graphql-js not the
terser schema language.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/gatsbyjs/gatsby/issues/3344#issuecomment-354288169,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAuc_6X_eEWEf281KUspXxB1NuPZPeC4ks5tE5clgaJpZM4RNLJn
.

[jquense]

So I think we are missing each other on where this should be solved :) in the WordPress example, I think rather than the plugin completely relying schema inference it should fully specify fields that may exist, e.g. all the nullable fields. Then a user won't run into the problem at all, because gatsby will have a complete schema for the datasource.

In cases where the plugin author can't possibly know what all the fields are I still think that, while more of a learning curve, it'll end up being more accurate and less error prone for the user to provide the rough schema vs fake data to be inferred as a schema. Yes you need to learn more up front (and we should provide good documentation for it) but you avoid users needing to understand the internal workings of how gatsby infers schemas. For example, if a user wants to specify a date field, it's (I think) simpler to learn to directly specify a date vs know to provide an iso compatible string that will be inferred as a date. Plus a lot of other features like connecting two data nodes, or specifying nullablility, is only really doable with a schema definition vs a data object

[m4rrc0]

What about 2 different plugins? One that allows us to easily specify the schema in a file and another one to generate that file from a specific source.
So the source might be different... For example a contentful space where every field is complete might be the source to generate the schema. Then the schema can be used on any contentful space that has the same architecture. And of course we might still tweak the schema manually if needed.
In any case it seems like plugin no1 is the first step. We might still decide when it is done what to do next.
@jquense is there somewhere an example of a manually defined schema? I didn't know it was possible.

[jsanchez034]

I like the idea of two plugins. I think the API @jquense is talking about
is https://www.gatsbyjs.org/docs/node-apis/#setFieldsOnGraphQLNodeType. But
I think it only allows to add fields to a schema right? And not overwrite
what was inferred.

On Thu, Dec 28, 2017 at 10:06 AM M4rrc0 [e-mail] wrote:

What about 2 different plugins? One that allows us to easily specify the
schema in a file and another one to generate that file from a specific
source.
So the source might be different... For example a contentful space where
every field is complete might be the source to generate the schema. Then
the schema can be used on any contentful space that has the same
architecture. And of course we might still tweak the schema manually if
needed.
In any case it seems like plugin no1 is the first step. We might still
decide when it is done what to do next.
@jquense https://github.com/jquense is there somewhere an example of a
manually defined schema? I didn't know it was possible.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/gatsbyjs/gatsby/issues/3344#issuecomment-354302287,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAuc_4fqrocUXxBPkFsK2tFn2VVDduv3ks5tE66AgaJpZM4RNLJn
.

[jquense]

Yeah I think you are right it doesn't all overwriting :/

I'm generally in favor of plugins to solve issues but I think this sort of thing may need to handled in the core api to strike the right usability notes (I could totally be wrong tho). In my head the ideal api would be just to point gatsby at a .graphql file and it does the rest, but that may not as nice as I imagine. I think it's all probably worth exploring in plugins tho if possible before possibly moving into core... I of course defer to what Kyle thinks since he's the boss :)

[jsanchez034]

Ok cool I will do some exploring around the first plugin for defining
schemas as .graphql files. I'm interested to hear Kyle's input too :)
Thanks for the great discussion everyone!

On Thu, Dec 28, 2017 at 10:31 AM Jason Quense [e-mail]
wrote:

Yeah I think you are right it doesn't all overwriting :/

I'm generally in favor of plugins to solve issues but I think this sort of
thing may need to handled in the core api to strike the right usability
notes (I could totally be wrong tho). In my head the ideal api would be
just to point gatsby at a .graphql file and it does the rest, but that may
not as nice as I imagine. I think it's all probably worth exploring in
plugins tho if possible before possibly moving into core... I of course
defer to what Kyle thinks since he's the boss :)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/gatsbyjs/gatsby/issues/3344#issuecomment-354306088,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAuc_6wNLHfte9zCZ2OqgirIMRadX609ks5tE7RTgaJpZM4RNLJn
.

[pieh]

@jsanchez034 did you look more into it? I will propably try to work on this next week and I'm interested if you did some research and can share your findings

[jsanchez034]

Hey Michal, I did some research and have some next steps in mind. I'll post
my findings when I get back on my machine in a few hours.

On Fri, Jan 5, 2018 at 7:16 PM Michal Piechowiak [e-mail]
wrote:

@jsanchez034 https://github.com/jsanchez034 did you look more into it?
I will propably try to work on this next week and I'm interested if you did
some research and can share your findings

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/gatsbyjs/gatsby/issues/3344#issuecomment-355699899,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAuc_3w7px_NJjL2EKl8G5TP1GJgQFVWks5tHrtNgaJpZM4RNLJn
.

[jsanchez034]

Hey @pieh, I started looking into how we can inject whole new schemas into the schema that Gatsby infers from the source data. So I figured a plugin could be created to add schemas to Gatsby. Below are my notes for this plugin...

Schema writer plugin

What should the plugin do?

• Take a schema and resolver and the resulting schema should be stitched into main Gatsby schema
  
  
  
  • Looks in src/schema for folders, each folder has a schema and resolver file
  
  
  • Plugin will take all files and run them through makeExecutableSchema of the graphql-tool npm package
  
  
  • Create a new Gatsby Node API function called something like mergeSchema that gets called before this line in the Schema creation file which uses graphql-tool Schema stitching | GraphQL Tools to stitch Gatsby produced schema with manually created schema from src/schema
  
  
• Should set some dummy sample data for the new schema using createNode
  
  
  
  • Use sourceNodes in gatsby-node.js file of plugin
  
  

Those are my rough notes on a schema writer plugin. The other plugin that @MarcCoet mentioned would be one where you could point the plugin to endpoints on a domain where data is set up to be in a complete state then schemas would be generated and saved off to src/schema.

Note: Schemas would be defined using GraphQL type language

[pieh]

Unfortunately I don't think graphql-tools will work here as it was meant to merge full schemas (that is both definitions and data/resolvers) and adding links between those schemas.

From documentation of mergeSchemas function (https://www.apollographql.com/docs/graphql-tools/schema-stitching.html#mergeSchemas) about passed schemas to merge:

schemas is an array of either GraphQLSchema objects or strings. For strings, only extend type declarations will be used. Passing strings is useful to add fields to existing types to link schemas together, as described in the example above.

We would rather want to use string option, as GraphQLSchema require to define at least 1 query (or it throw an error), but extending type doesn't allow to overwrite field type. Error is thrown when trying to overwrite existing field (conflict resolution function isn't called so nothing we can do about it unfortunately):

Field "frontmatter.title" already exists in the schema. It cannot also be defined in this type extension

As for second option (using GraphQLSchema) I "hacked" it to work by adding dummy query, but initial problem is I can't use types from "main" schema - we can't simply do this:

type frontmatter {
  featuredImage: File
  title: String
}

because File isn't defined in this additional schema. We could probably append builtin types so it would work but I feel like trying to "fool" this tool to do stuff it's not designed to do (at least at this point in time) is not the way to go.

I'll will try to evaluate 2 other approaches:

• search for or make a tool designed to modify already existing schema (that would be more generic graphql tool)
• or force user/plugin defined types on fields when creating schema (that would gatsby specific)

[pieh]

I've made some progress. Decided to go with gatsby specific approach for now (adding support for "forced types" in https://github.com/gatsbyjs/gatsby/tree/master/packages/gatsby/src/schema ) as I found it easier to work with than playing with graphql schema objects. I'll try to clean up my code a bit so it would be more readable and create pull request this week to discuss it further there.

Just little sneak peak of results (imo quite promising):
I've used https://github.com/gatsbyjs/gatsby-starter-blog and used frontmatter in posts to emulate some problems that can cause build errors caused by queries that don't match schema:

// post #1
---
title: Hello World
date: "2015-05-01T22:12:03.284Z"
featuredImage: "sasalty_egg.jpg" // badly formatted value that wouldn't allow to make this File type 
test: "some string" // incompatible value type with other posts (in other posts test field is object) - it would be skipped in schema
---

// post #2
---
title: My Second Post!
date: "2015-05-06T23:46:37.121Z"
test:
  floatField: 5.5
  intField: 99
  arrayOfStringsField:
    - "it"
    - "works"
---

// post #3
---
title: New Beginnings
date: "2015-05-28T22:40:32.169Z"
test:
  stringField: "string"
  boolField: true
---

Used type definitions:

type MarkdownRemark {
  frontmatter: frontmatter_forced_type
}

type frontmatter_forced_type {
  featuredImage: File
  title: ImageSharp
  test: TotallyCustomType
}

type TotallyCustomType {
  stringField: String
  floatField: Float
  intField: Int
  boolField: Boolean
  arrayOfStringsField: [String]
}

_Note:_ those don't have to be full type definitions (f.e. in MarkdownRemark definition I just force type of frontmatter field to not rely on automatic type naming)

Here's how frontmatter looks before using my changes (no test field, "bad" type of featuredImage field):

And after:

[jsanchez034]

Hi @pieh! Thanks for taking this on, your solution looks very promising! Sorry I couldn't contribute more to this issue in the past couple of weeks, I haven't had the time to dive into this. Let me know if there is anything I can help with.

[pieh]

If anyone is interested you can view my code changes https://github.com/gatsbyjs/gatsby/compare/master...pieh:schema_wip (after deleting some dead code and refactoring some stuff I had yesterday it's much less code than I thought :) ).

Don't have time to put PR for comments with good description and website repository used for testing, so figured I would just share my code for now. If anyone want to play with it - place gatsby-schema.gql file with schema definition in root directory of your project (along gatsby-node.js, etc.).

[m4rrc0]

Any chance this could make its way into a PR?
A lot of people would be glad to have this I think.

[pieh]

@MarcCoet more I dived into this code more I realized that my approach in that branch can be used only as proof of concept. I plan on working on this to have it done proper way but this a lot bigger task than I originally anticipated (as I was learning more about how currently schema creation is done and what features it offers I didn't earlier know about as I was coding stuff)

So currently schema creation looks like this:

Implementation is doubled for each source of types. My proof of concept (branch I linked) adds just another source and now we have it tripled:

It would be unmaintainable to just add this in current form.

What I want to do is so this looks something like that:

But this require some major refactoring/rework as current schema creation is designed with inferring types from data as only source of data. After that, adding another source would be trivial with knowledge I gained during development of my proof of concept.

I want to create RFC issue for this (I created flowcharts earlier this week :) ). Just need to get some spare time to actually commit to discussion this would start.

[Unforgiven-wanda]

@MarcCoet I personally have moved all my projects temporarily to react-static, as this feature is essentially a deal-breaker for me.
But I plan to transition back to Gatsby the second this is fixed.

[pieh]

I created RFC issue about refactoring ( https://github.com/gatsbyjs/gatsby/issues/4261 ), if anyone is interested to join discussion, please do so.

[m4rrc0]

Thanks for the update and the work @pieh . I'll keep an eye on the RFC issue.

[i8ramin]

Hey guys. Any updates on this? Currently a blocker/show-stopper for us to use Gatsby + Contentful and just wanted to know if there was any active development or if we should consider other options. Thank you.

[pieh]

@i8ramin There isn't active development yet, I plan to start it this week.

[nicolaslair]

Hi guys, just checking if someone made any progress with this issue - we just discovered it wehen dealing with optional / empty fields in Contentful > Gatsby - so it would be nice to have a solution - obviously :) Thanks in advance!

[nthndnn]

Hey @pieh, just wondered if anything has happened with the development? Would love to use Gatsby with Contentful for a project, however, this is a little bit of a blocker. cheers in advance!

[pieh]

Hey @nicolaslair and @nathanjdunn ! Sorry for lack of updates on this. I had to put this project on hold (...again) and focus my time on gatsby v2 for time being. I wish I had better news :(

[nthndnn]

Hey @pieh, will this issue be fixed in Gatsby v2?

[pieh]

This is not scheduled for v2 - this feature won't be breaking change so plan is to push v2 as fast as possible and then add this feature in one of v2 minor realeases

[dominicfallows]

@pieh and anyone else, I'm doing a messy noramlizer to make sure that there is always the same data model for GraphQL regardless of optional/null fields etc. Has anyone come up with a better workaround ahead of something more core being developed?

[pieh]

Pattern I've used was to create "dummy" entries (in CMS / markdown) that would have all optional fields filled and later when building gatsby site I would have filter in queries, so dummy nodes not show up in results.

[dominicfallows]

@pieh that is a good idea, thanks I think I'll give that a go

[dominicfallows]

For reference, I'm actually now doing this dummy entry workaround in my source plugin, which keeps the CMS team happy, and means the normalizing and exclusion of dummy entry is handled within the Gatsby code, rather than in the CMS or UI React components.

[Unforgiven-wanda]

@dominicchapman Care to create a demo repo for this?
or a Medium post?
or anything at this point

[m4rrc0]

Any news?

[m4rrc0]

Just seen @Undistraction's plugin https://github.com/Undistraction/gatsby-plugin-node-fields .

a simple, consistent way to manage the creation of fields on your nodes, with support for default values, transformations and validation of values

I am thinking this should be part of core... I'll definitely give it a try.

[drobinhood]

Any updates on this?

[pieh]

This will be resumed after v2 is released. I don't have bandwidth to handle v2 and this at the same

[supermoos]

Is there a suggested workaround until sometime after V2 when this would be worked on again?
Sounds weird if so many people using GatsbyJS has never had a need for optional fields. I'm using it with Contentful btw.

[westmark]

We're also using it with Contentful and this behaviour is driving us nuts. As we see it the Contentful plugin is mostly at fault here for (supposedly) constructing the schema from data received rather from the actual content type schema received over the API.

[pieh]

@westmark contentful plugin don't have power over this - it's just right now Gatsby doesn't allow plugins to set explicit types and schema is inferred from available data

As for workaround - if possible create dummy entry for each content type that will contain all optional fields and then filter that dummy entry out when you are creating pages or listings of entries. Dummy entry data will ensure full schema is built

[supermoos]

@pieh Thanks for the work-around suggestion for now :-)

[westmark]

@pieh I did not know that - thank you for clarifying. A follow-up question then, wouldn't it be possible for plugin authors to inject dummy values for missing data before passing it on to Gatsby? If Gatsby admits keys with undefined as value into the schema I would regard this as a far better workaround.

[ErisDS]

We're also running into this issue with gatsby-source-ghost. Our data schema has a lot of fields that can be legitimately returned as nulls, and are therefore omitted by Gatsby.

I 100% hear and understand that any work on this is paused until after than land of v2 (which I'm very much looking forward to). I wanted to flag up the Ghost source plugin as being another good example where this occurs & offer to help out if there is anything we can do whenever the time comes to resume this 🙂

[stefanoverna]

Same problem on https://github.com/datocms/gatsby-source-datocms 😄

[westmark]

Question: In V2, could we use this to avoid the "query on missing data"-problem?

https://next.gatsbyjs.org/docs/actions/#addThirdPartySchema

In other words, could the Contentful plugin use this function to inject the schema for a content model that has no data/entries yet?

[stoltzrobin]

@pieh have you had any time to refine this and start any development? 😁

[moreguppy]

Hello everybody, I'm quite new to GatsbyJS but have been using Contentful for a long time, and I just started trying out the framework and messing around with a bit of GraphQL. Ran into this issue quite quickly as it is often useful to have optional fields in Contentful.

However, I discovered if you are pulling down a collection (eg. allContentfulNews etc) and you have at least 1 entry in that collection which has content in all the fields (in my case I have an optional 'Expiry Date'), then it won't break the build and the entries with the Expiry Date empty returns as null. So as a workaround I'm planning on keeping this "template" post in, and hiding it on the frontend.

[shevchenkonik]

@moreguppy I got a similar solution. You create one ready page _Test_ and hide this page. For example:

allContentfulPageExample( filter: { title: { ne: "Test" } } )

[stoltzrobin]

@moreguppy @loeildes Yea, that's exactly how we also solved it in the meantime. But it makes it harder in the long run, every time you add a new content model/new field you need to create/update all "dummy" entries created.

[sami616]

creating dummy entries has worked for me as a temporary work around. not ideal for the reasons mentioned above, is this likely to be looked at any time soon?

[bgnz968]

still no fix in sight? we have a launch in 21 days

[pieh]

@bgnz968 It's very unlikely there will be proper fix (or rather feature) land in 21 days. I don't have any dates here. There is a lot of problems that need to be solved here - some related to gatsby builtin inferred schema, some to 3rd party stitched schemas ( https://github.com/gatsbyjs/rfcs/pull/11 ).

[metamas]

Figure I'll throw a log on the fire too... I also have to do the "dummy entry" work-around to avoid build breakage on my handful of projects that use gatsby-source-prismic. Like @stoltzrobin voiced: it's a good enough temporary fix, but it's hacky.

Schema snapshots feels like too many steps to get something that seems so basic and essential (i.e. optional data fields). But it sounds like sounds like that or explicitly writing the expected schema is the only way to allow optional data fields that are more complex than simple strings or numbers.

[jademh]

I'm also using dummy entries workaround to handle this but agree that it's v hacky. Would love to see a fix!

[Jerbach]

Also using @loeildes solution. Just adding my voice here hoping there will be a more optimal solution soon.

[Khaledgarbaya]

Hey Folks,
I haven't tested this but this might be a p[otential Clean workaround that does not require creating dummy content.

https://medium.com/@Zepro/contentful-reference-fields-with-gatsby-js-graphql-9f14ed90bdf9

[sami616]

@Khaledgarbaya I've not had much time to test your solution however, at a glance i can only seem to get this to by by spreading Node like this ?

... on Node {
    ... on FieldType {
     # ...
   }
}

[lkol]

Yeah, the workaround only works for "polymorphic" references, but not for normal optional fields

[hanoii]

Might be late to the party, also stumbled upon this on Drupal source plugin, and not only due to missing fields, but also missing entities altogether making fragments failing to work.

I believe @pieh's workaround is going to be enough but I want mention something on my mind.

I see the take of https://www.drupal.org/project/schemata using http://json-schema.org/ useful here. Maybe we can leverage this and have an optional way of providing a json-schema instead of infering it. How this json schema is exposed will be then partially dependent on the plugin.

[sami616]

It would be awesome to have a fix for this. Using placeholder content does work but feels hacky and makes me a nervous especially when you have nested references

[rexxars]

I have created a similar but slightly differently scoped issue in #10856 - how a source plugin can fix the issue of missing fields and node types given that a schema is available to inform Gatsby of what is missing.

[pieh]

@rexxars There is another ticket related to this - https://github.com/gatsbyjs/gatsby/issues/4261 . @stefanprobst did work on prototype that allow providing types - please check this comment in particular https://github.com/gatsbyjs/gatsby/issues/4261#issuecomment-442549881 - there is example of defining Node types and their shape (even if no data is available to infer from)

[sami616]

@pieh thanks! This looks promising!

[diegotrigo]

The original post is from Dec 2017. Starting to lose hope we'll ever find a proper solution.

[mattclough1]

I know this isn't a solution to the problem in this version of Gatsby, but Gatsby 2.0's gatsby-source-graphql has pretty much eliminated schema inference problems in my experience.

[stefanprobst]

Everyone: this is actively being worked on in #11480.

[stefanprobst]

We have an alpha version of our new Schema Customization API ready, and we'd love your feedback! For more info check out this blogpost.

[sami616]

this is _very_ exciting, thanks for all your hard work @stefanprobst !

[vermario]

I have seen the new API in preview: however it's not clear to me how that could be used to "snapshot" the existing graphql schema to solve the issue that was discussed in this ticket. I saw there is for example https://www.gatsbyjs.org/packages/gatsby-plugin-extract-schema/ with some simple code to extract the schema to a json file. Would the idea be that then something like that could be "fed again" into the gatsby schema?

(Crossing my finger because this is at the moment the biggest headache we are having in our project(s), using drupal + gatsby_source_drupal plugin) and it would be so great to be able to fix it cleanly). :)

[pieh]

@vermario It doesn't support creating schema snaphost just yet. But it should solve the issue better than creating snapshot. Once this is merged we will start implementing type definitions to plugins (including Drupal one). If Drupal has introspection endpoint (I'm not expert on drupal). Then we will be able to generate correct schema even without any data, so snapshot wouldn't be needed at all

[vermario]

@pieh sounds great: we will keep watching this space. In the meantime we are adding "test" content ( :/ ) and excluding the content from graphql queries at build time. Feels "dirty" :) Thanks!

[sami616]

Im running into an issue when using the new schema customisation with gatsby-source-contentful.

Defining a type where one of the fields points to a reference array always returns null

Ie:

type ContentfulPage implements Node {
   editorTitle: String
   title: String
   slug: String
   sections: [ContentfulSection] // im always null 
}

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

Thanks again for being part of the Gatsby community!

[moreguppy]

Im running into an issue when using the new schema customisation with gatsby-source-contentful.

Defining a type where one of the fields points to a reference array always returns null

Ie:

type ContentfulPage implements Node {
   editorTitle: String
   title: String
   slug: String
   sections: [ContentfulSection] // im always null 
}

@sami616 I found this approach worked:

// In gatsby-node.js
exports.sourceNodes = ({ actions }) => {
  const { createTypes } = actions
  const typeDefs = `
    type ContentfulPage implements Node {
      sections: ContentfulSection
    }
    type ContentfulSection implements Node {
      title: String
    }
  `
  createTypes(typeDefs)
}

It seems for reference fields, you have to define the shape of the model you are referencing as well

[imshuffling]

So I'm trying to work this out. I have a field of "Video" (that is not required in contentful) on a content type of "ContentfulHomepage" with a reference field called "Blocks" calling "ContentfulBlockHomepageBanner".

Works fine if video field has content... but falls over if null, so here's my stab at the empty field issue.

Query below.

My attempt...

Erroring out...

Can anyone point out where I'm going wrong here? Thanks.

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!

[silviopaganini]

am I the only one still having this problem?

[KyleAMathews]

Sorry we're working towards this. I'll set this as not stale so the bot stops closing it.

[moreguppy]

Im running into an issue when using the new schema customisation with gatsby-source-contentful.
Defining a type where one of the fields points to a reference array always returns null
Ie:

type ContentfulPage implements Node {
   editorTitle: String
   title: String
   slug: String
   sections: [ContentfulSection] // im always null 
}

@sami616 I found this approach worked:

// In gatsby-node.js
exports.sourceNodes = ({ actions }) => {
  const { createTypes } = actions
  const typeDefs = `
    type ContentfulPage implements Node {
      sections: ContentfulSection
    }
    type ContentfulSection implements Node {
      title: String
    }
  `
  createTypes(typeDefs)
}

It seems for reference fields, you have to define the shape of the model you are referencing as well

@silviopaganini I found this is a pretty good solution in the interim

[silviopaganini]

Yeah! trying this now, seems to work, but it's not scalable... I'm working on a huge website, if I have to create those for all non-required fields is not great, but works for now

[miraclemaker]

@moreguppy I'm trying this to fix a similar issue with gatsby-source-wordpress, but I get the following message when I add this code to gatsby-node.js:

Error: Schema must contain uniquely named types but contains multiple types named "wordpress__PAGEAcf".

My code:

exports.sourceNodes = ({ actions }) => {
  const { createTypes } = actions
  const typeDefs = `
    type wordpress__PAGEAcf implements Node {
      enter_content: String
    }
  `
  createTypes(typeDefs)
}

[miraclemaker]

Nevermind, for anyone experiencing a similar issue, this is the correct format ti fix it:

exports.sourceNodes = ({ actions }) => {
  const { createTypes } = actions
  const typeDefs = `
    type wordpress__PAGE implements Node {
      acf: wordpress__PAGEAcf
    }
    type wordpress__PAGEAcf implements Node {
      enter_content: String
    }
  `
  createTypes(typeDefs)
}

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!

[kalinchernev]

Re-opening and adding a not stale tag as of https://github.com/gatsbyjs/gatsby/issues/3344#issuecomment-505001979

[stefanprobst]

I'd be super grateful if someone could give the stuff in #16291 some real-world testing. Thanks!

[stefanprobst]

We have a open PR for schema lock-down in #16291. Would be cool if those still interested in seeing this land could share some feedback. Thanks!

[m4rrc0]

Hey @stefanprobst . Awesome work!
Sorry for the late reply. This has been stalling for so long that I got used to the workarounds and it was not a major problem anymore to me. It is so neat though that this is moving forward! :)
I gave it a quick try and it worked seamlessly! That is rare enough to salute. ;D
I am working on a big update on a project and I will have some more 'real world' testing in the coming weeks. I'll comment further in the PR.
Thanks a lot for the work!

[m4rrc0]

@stefanprobst any idea about #19210