Gatsby: createPages uses stale cached data for "previous" and "next" pages

[johnridesabike]

Description

I noticed this on my own blog's "previous post" and "next post" links, and it's reproducible on gatsby-starter-blog. It may have broader implications beyond my use case as well.

If you have links to "previous" and "next" posts on each blog post's page, then those are not updated on each build. As you add posts, they remember their cached "previous" link, but are never updated with the "next." If a post is removed, then existing "previous/next" links are preserved. (And its page will still exist in the build.)

Steps to reproduce

I made a sample repo (almost identical to gatsby-starter-blog): https://github.com/johnridesabike/gatsby-prev-next

git clone https://github.com/johnridesabike/gatsby-prev-next.git
cd gatsby-prev-next
gatsby build

If you serve the built site, everything should be normal. The next part is where the problem happens:

mv test-post content/blog/test-post # add a new post
gatsby build
gatsby serve

Expected result

The build should use up-to-date data in regards to "previous" and "next" links.

Actual result

If you open the new built site, the new post ("Test Post") should be present. The previous post ("New Beginnings") does _not_ link to it. Clearing the cache fixes the problem:

gatsby clean
gatsby build
gatsby serve

The site should build as expected.

But now, if you remove the content/blog/test-post directory, and run gatsby build, "Test Post" won't appear on the index page but the "New Beginnings" post will still link to it (and the "Test Post" page will still exist in the build.)

Additional information

I played around a bit trying to figure out how to bust the stale data, but with no luck. One thing I noticed is that the createPages function uses this logic to generate the previous and next data:

const previous = index === posts.length - 1 ? null : posts[index + 1].node
const next = index === 0 ? null : posts[index - 1].node

I tried to update it to use a GraphQL query instead:

{
  allMarkdownRemark(sort: {fields: [frontmatter___date], order: DESC}, limit: 1000) {
    edges {
      node {
        fields {
          slug
        }
        frontmatter {
          title
        }
      }
+      previous {
+        fields {
+          slug
+        }
+        frontmatter {
+          title
+        }
+      }
+      next {
+        fields {
+          slug
+        }
+        frontmatter {
+          title
+        }
+      }
    }
  }
}

This had no effect. The stale cache was still used.

If it's not feasible to make the caching "smart" enough for this scenario, it would be also good if createPages had the ability to manually use or override the cache, as well as delete cached pages after their source data has been deleted.

Environment

  System:
    OS: macOS 10.15.6
    CPU: (6) x64 Intel(R) Core(TM) i5-8500B CPU @ 3.00GHz
    Shell: 5.7.1 - /bin/zsh
  Binaries:
    Node: 14.8.0 - /usr/local/bin/node
    Yarn: 1.22.4 - /usr/local/bin/yarn
    npm: 6.14.7 - /usr/local/bin/npm
  Languages:
    Python: 2.7.17 - /usr/local/bin/python
  Browsers:
    Chrome: 84.0.4147.125
    Firefox: 79.0
    Safari: 13.1.2
  npmPackages:
    gatsby: ^2.24.41 => 2.24.41 
    gatsby-image: ^2.4.15 => 2.4.15 
    gatsby-plugin-feed: ^2.5.11 => 2.5.11 
    gatsby-plugin-google-analytics: ^2.3.13 => 2.3.13 
    gatsby-plugin-manifest: ^2.4.22 => 2.4.22 
    gatsby-plugin-offline: ^3.2.23 => 3.2.23 
    gatsby-plugin-react-helmet: ^3.3.10 => 3.3.10 
    gatsby-plugin-sharp: ^2.6.26 => 2.6.26 
    gatsby-plugin-typography: ^2.5.10 => 2.5.10 
    gatsby-remark-copy-linked-files: ^2.3.12 => 2.3.12 
    gatsby-remark-images: ^3.3.25 => 3.3.25 
    gatsby-remark-prismjs: ^3.5.10 => 3.5.10 
    gatsby-remark-responsive-iframe: ^2.4.12 => 2.4.12 
    gatsby-remark-smartypants: ^2.3.10 => 2.3.10 
    gatsby-source-filesystem: ^2.3.24 => 2.3.24 
    gatsby-transformer-remark: ^2.8.28 => 2.8.28 
    gatsby-transformer-sharp: ^2.5.12 => 2.5.12 
  npmGlobalPackages:
    gatsby-cli: 2.12.66

[Fasani]

I think I have this exact same issue. I used starter blog and I also used the instructions from the Gatsby site regarding adding pagination and I also found that when I added blog post 5, blog post 4 was not updated with the new next link.

[johnridesabike]

I was focused on the previous/next links when I submitted this issue, but looking at it now I realize that the much more critical bug is probably the fact that deleted markdown pages are not deleted from builds (when the cache is used).

[vstoyanoff]

Same here. The data is not updated either. Opened an issue here https://github.com/gatsbyjs/gatsby/issues/28281

[vstoyanoff]

Have someone found a solution for this? I see the last comment is from 18 Aug. This is a big issue in my opinion, because if I want it to work I must opt-out of caching between builds. For smaller websites is not a problem, but for large ones it is really bad. I am using Netlify CMS with Markdown files so my clients want to make changes to some pages and those changes are saved, but in reality, they are not present on the built pages...

[pieh]

@vstoyanoff - I opened PR that is looking to solve this - https://github.com/gatsbyjs/gatsby/pull/28351 - for now it's draft, because there are some additional considerations to make, but you can follow that PR for the potential fix for this problem.

[vstoyanoff]

@pieh Thanks for the follow up! I will watch for solution in the meantime.

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜