Gatsby: createPages() does not re-run when files change

Created on 15 Jul 2017 · 11Comments · Source: gatsbyjs/gatsby

Recently switched to using createPages() in gatsby-node.js for some of my similar pages. Noticed that changes to my data do not trigger updates to my pages. createPages() seems to only be called once during startup. Would be nice if it was called again when my data changes.

More details:
I'm using gatsby-transformer-json and I'm querying that data with graphql in createPages(). I'm creating one page for each object found in my JSON file.

Seems like if people are creating pages based on graphql data, it will be important that changes to that data trigger new pages to be created.

I've attached a simplified version of my use-case.

gatsby-json-update-test.zip

Versions:
Gatsby 1.2.0
Node v6.10.2
Mac 10.12.5

Source

ahfarmer

Most helpful comment

@KyleAMathews AWESOME this clarifies everything. I'll take another stab at this tonight.

ahfarmer on 25 Jul 2017

👍2

All 11 comments

I'm exploring the wordpress-source plugin - "Seems like if people are creating pages based on graphql data, it will be important that changes to that data trigger new pages to be created" pretty much sums it up: if creating pages from any graphql data source is it the expected behaviour to have to run the build process again on save/ data change?

chrisk2020 on 17 Jul 2017

👍1

Hey! Thanks for bringing this up. This is definitely the plan. Was actually hoping to get this done before v1 launch but it's one of the features that got edited out of the launch plan and postponed.

Everything is setup already to do this. I even documented things for this haha https://www.gatsbyjs.org/docs/node-apis/#createPagesStatefully

Anyways, the algorithm will be basically

On a data change, wait for data transformation to finish and call createPage for every implementing plugin.
Compare the array of pages returned by this call to the previous list of pages. If there were any pages not recreated, call deletePage to remove them.

Should work well.

Lemme know if you want to take this on and I can give you some more pointers!

KyleAMathews on 18 Jul 2017

👍2

Would that be a (in my case [source-wordpress]) plugin level change or something 'deeper in the core'? Would love to crack this though just finding my feet with gatsby - so probably need some hand-holding.

chrisk2020 on 20 Jul 2017

@chrisk2020 it's a core issue but fairly tricky. Making this change requires understanding core well. Feel free to dive in though! Happy to help you along where can.

KyleAMathews on 20 Jul 2017

@KyleAMathews
I spent a little time looking into this but didn't get very far.

This is what I learned:

I see that the CREATE_NODE action will be emitted when a new node is created. That's great.
I can use apiRunnerNode to call createPages.
I see my gatsby-node.js API method createPages being called, and I see it in turn calling boundActionCreators.createPage. Also good.

And these are my issues:

boundActionRunners.createPage does not seem to have any effect when called after bootstrap is complete. New pages are not created/updated.
I'm not sure how to determine what pages each plugin created so that I can delete the ones that were not created (as you suggested) - apiRunnerNode doesn't provide a list of which bound actions were called does it?
An invariant violation is thrown in query-watchers.js. It seems that this file expects to find a src/pages/<component name> file for each page. This is probably the least of my concerns right now.

My work so far (as little as it is) is in this commit: https://github.com/ahfarmer/gatsby/commit/a9e32dd71a1775241aa4440cf886a9a5a5f1ccc7

Any light you can shed on these would be appreciated. Thanks!

ahfarmer on 22 Jul 2017

@KyleAMathews This is definitely something that will make its way into the gatsby-source-wordpress plugin

sebastienfi on 23 Jul 2017

@sebastienfi this is a core Gatsby concern that needs to work for everyone.

@ahfarmer You need to wait until all transformer plugins have finished running. For example, you could add a new markdown file to your site which triggers the creation of a new File node. If you tried to create a page based on Markdown at that point, nothing would happen.

Instead you want to wait for the somewhat poorly named API_RUNNING_QUEUE_EMPTY event which is emitted whenever the API queue goes to zero or in other words, all transformer plugins have finished running.

You'll also want to listen for new nodes being created as API_RUNNING_QUEUE_EMPTY will get called in response to other internal changes not related to nodes. So in other words, when a node is created, mark the pages state as "dirty" so you know to regenerate pages in response to API_RUNNING_QUEUE_EMPTY.

It doesn't matter which plugins created the pages. You'll just take a snapshot of the pages data before running createPages and then again after and compare the two to see which pages weren't recreated and then delete those pages.

KyleAMathews on 25 Jul 2017

Some code links

KyleAMathews on 25 Jul 2017

@KyleAMathews AWESOME this clarifies everything. I'll take another stab at this tonight.

ahfarmer on 25 Jul 2017

👍2

@KyleAMathews I'm getting really close here.

The last thing I have to do is the page deletion as you described. I'm not sure how to deal with createPagesStatefully. Pages created by that method will not have been re-created after I call createPages(), but they also should not be deleted, isn't that correct?

Is there a way to differentiate pages created statefully from those that were not?

Thank you!

BTW the changes I'm working with are here:
https://github.com/gatsbyjs/gatsby/compare/master...ahfarmer:topics/hot-create-pages
Any feedback on that would be great.

ahfarmer on 11 Aug 2017

❤1

Sorry to bring this up on such an old issue - but was there ever any solution to re-run createPages when source changes from something like gatsby-source-wordpress?