Gatsby: [bug]: ☂️ umbrella issue for cache and over-eager persistence issues

[DSchau]

Description

In this umbrella issue, I'd like to provide a means for others to share any issues encountered regarding caching, specifically in instances when the solution to a problem you were facing was remediated by removing the cache (e.g. rm -rf .cache).

Issues that we have seen before (but generally have not been able to reproduce) can usually be classified as one of the following:

1. Remote content sourced by a plugin is available (e.g. in the database, CMS, etc.) but is _not_ available after a local re-build
2. A local plugin source code changed, but the change(s) do not seem to have taken effect
3. Local content (e.g. images, Markdown, etc.) were changed, but the changes do not seem to show up after a build

If any of these scenarios, or more broadly speaking, if you've ran into a problem where content wasn't appearing appropriately and deleting the cache solved this problem--then please chime in!

Intent

If we can reliably reproduce these types of issues and scenarios, we can harden our cache and improve the experience for _all_ Gatsby users. Specifically, if we can reliably reproduce we can fix the underlying issue _and_ then author end-to-end tests to ensure that the caching issue remains fixed.

Template

Please use the following template to report an issue so that we can most effectively debug the underlying issue.

Note: it is extremely important that you provide as much information as possible for the reproduction step. The more info and clearer the reproduction, the better we will be able to debug and ship fixes to these issues!

## Description

<!-- What led you to delete the cache? What was the exact problem that deleting the cache seemed to have fixed? -->

## Reproduction

<!-- Please, please provide detailed steps to reproduce. Ideally a Github repository where we can reproduce the issue(s) ourselves, as well. -->

## Environment Info

<!-- Run `npx gatsby info --clipboard` in the root of your Gatsby app and paste the output here -->

[mrispoli24]

I think I ran into this issue and after much frustration I started this issue but ended up closing it after deleting the public folder and getting everything to work. I do believe this sounds a lot like what others are experiencing though.

[nickytonline]

I built my site off of the Netlify CMS starter, https://github.com/nickytonline/www.iamdeveloper.com. ‪It has deleting the .cache as part of their build steps. See https://github.com/netlify-templates/gatsby-starter-netlify-cms/blob/master/package.json#L40‬

Not sure why, but they must have a good reason to delete the .cache.

I remember when I started off with another starter site, I had to delete the .cache frequently, but this was pre-Gatsby 2.

[keithmifsud]

Please bear in mind that I am not 100% sure this was the cause of the issue forcing me to delete the .cache directory but it did fix the issue as soon as I did. I have a json file with an array of objects. I decided to give each object an identifier key. Gatsby does not like this key name, maybe it conflicts with an internal key name. I renamed the identifier to name but the related changes were not being built until I deleted the .cache directory.

[endymion1818]

In working with several developers using GatsbyJS over the past 9 months I’ve only had this isssue come up once. A colleague was experiencing graphql errors because an API wasn’t being reached. We had recently changed how we fetch data (using gatsby-source-Wordpress) in Gatsby-config.js. When he updated his master branch, running the cli command “gatsby develop” still used the old settings. Deleting .cache solved the issue.

[oobidin]

I've been working with GatsbyJS last 6 months and I faced some issues when you update some local content (MDs and JSONs in our case) and it doesn't update. Actually, we've created the yarn clean command with the content: rm -rf public && rm -rf .cache to solve such kind of issues.

[trotzig]

We updated our content types in contentful (removed some types, added some). Removing the cache folder made the dev server start up again. Cc @axhamre.

[kylefraser]

We've run into an issue with contentful where, when we've added images via a rich-text, we've had to delete the .cache folder in order to query additional properties with GraphQL on the embedded asset.

For instance, we've added images, re-started the dev server, and were still unable to query image properties like

target {
   fields {
      file {
         url
      }
   }
}

deleting the .cache folder, solved this for us.

[vacas5]

I abstracted a lot of functions utilized inside my sourceNodes function into a separate file. Changes made in that file were not detected by cache busting, and so any time I changed the code in my abstracted file, I would have to clear the cache manually. Any way that Gatsby could detect changes in required files within gatsby-node.js would be advantageous to my use case.

[amankkg]

Not sure if it counts...

Recently, I had to press Clear cache and retry deploy button in Netlify's CI during this PR - that checks turned green only after cache clean.

It started to fail with adding of prismjs and its theme CSS import in gatsby-browser.js.

[u12206050]

Though it is undocumented it seems one can use Gatsby's cache object to set and get json objects. Data is stored in the .cache/cache/{plugin-name}/ directory.

In your plugin gatsby-node.js in one of the many apis include cache as an argument; eg.

exports.onPostBuild = async function(
  { cache, store, graphql },
  { some, plugin, options} ) {
    const cacheKey = 'some-key-name';
    ...
    let obj = await cache.get(cacheKey);

    if (!obj) {
      obj = { created: Date.now() };
    }
    ...
    obj.modified = Date.now();

    await cache.set(cacheKey, obj);
  }

[DSchau]

@u12206050 correct! A bit out-of-scope for this discussion though, right?

We'd love if you were able to rectify the "not documented" piece and document that API :)

[danoc]

Hi @DSchau! Thanks for making this.

Here's one that me and my team have been struggling with lately: https://github.com/gatsbyjs/gatsby/issues/11109

I spent some time debugging but couldn't figure it out.

[kbrgl]

Oops, unpinned this accidentally. Sorry!

(Perhaps GitHub should make repository actions more obvious 😛)

[aarondack]

This is a pretty systemic problem whenever adding a new image. Pretty much exclusively I have to rm -rf .cache/ and re-install. Sometimes I also just nuke the public/ folder. I know this is sort of already known, just figured that I'd share that I _have_ had this experience.

[MANTENN]

I have the same issue as well, except in my case the error appears with Contentful.

Unknown field 'heroImage' on type 'ContentfulBlogPost'. Source: document `HomeQuery` file: `GraphQL request`

By removing .cache folder, and rerunning yarn dev I am to launch and work on the project.

[DSchau]

@MANTENN sorry to hear that! Would you mind adding additional reproduction steps and editing your comment?

As it stands--there's really no way we can reproduce that, because reproduction steps weren't provided. Did something change in the Contentful schema that wasn't picked up? Did you change a query? Anything that you can provide helps us harden the cache and alleviate these issues.

Thanks!

[MANTENN]

@DSchau I have no idea why it occurs. I'll post the reproduction steps when I'll encounter the issue again

[MANTENN]

@DSchau False alarm, typo.

[m-allanson]

A potential cause of caching issues is mentioned here.

In short - if you create a node using createNode, without setting the parent field correctly, the node can be incorrectly garbage collected as part of the bootstrap process. Clearing the cache will correctly recreate the node.

[MANTENN]

Changing to draft status on Contentful would result in this error:

error Encountered an error trying to infer a GraphQL type for: "blog post___NODE". There is no corresponding node with the id field matching: "aa7f37c3-955c-5d5e-9024-0b7e98684e23"


  Error: Invariant Violation: Encountered an error trying to infer a GraphQL type for: "blog post___NODE". There is no corresponding node with the id field matching: "xx6f36c3-955c-5d5e-9026-0b7e98  654e23"

I assume this is an issue with the contentful plugin/module.

[MANTENN]

I also encountered the same error above by discarding a component change.

[kristojorg]

I have been having some issues with the cache as well, but on the other side. I have pretty bad internet for the time being, and having gatsby clear the cache every time I change a config is pretty frustrating. Rebuilds can take ~15 minutes. Wondering if we should cache different objects (images vs configs or built files) differently?

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[JonnyHayes]

This is happening to us a lot now. Message keeps changing -

Error: Invariant Violation: Encountered an error trying to infer a GraphQL type for: block > teaser___NODE.
Error: Invariant Violation: Encountered an error trying to infer a GraphQL type for: navigation___NODE.
Error: Invariant Violation: Encountered an error trying to infer a GraphQL type for: body___NODE.

We are also using contentful.

[mrispoli24]

Running into this issue again myself now, deleting the public folder and .cache folders don't seem to do the trick anymore. The images are there, I am willing to invite someone to the repository to check it out if you like but it seems that when working with static image files this happens a lot. They are present in the gh-pages branch but showing up broken on the deployed site.

[mrispoli24]

I figured out what was going on in the above case. I have a series of image files in a /static directory that are divided by folders. In my template I had the wrong case for a character so /static/Folder instead of /static/folder. In development and on local building this ran just fine, upon deployment to github this rightly broke. Not sure exactly how gatsby builds things under the hood but it would have been a faster catch had it broken in development mode locally. This might not have had anything to do with the cache but I think it's worth letting ya'll know in case this is a symptom or indicator of something here.

[smerth]

I wrote plugin gatsby-source-gh-readme to pull file content data from a user's github account, it does the job it should do but I need to clear the cache before running gatsby develop or most of the time the build fails. I added a comment about this here because I think it might be related

[mkaradeniz]

I'm running into trouble with gatsby-source-filesystem and remote assets.

I'm using gatsby-source-graphql to use GraphCMS in my project. I added a custom resolver to download the GraphCMS assets via createRemoteNode to use gatsby-image later.

gatsby-node.js:

exports.createResolvers = ({ actions, cache, createNodeId, createResolvers, store, reporter }) => {
  const { createNode } = actions;

  createResolvers({
    GCMS_Asset: {
      file: {
        type: `File`,
        resolve(source, args, context, info) {
          return createRemoteFileNode({
            url: source.url,
            store,
            cache,
            createNode,
            createNodeId,
            reporter,
          });
        },
      },
    },
  });
};

When first starting gatsby develop or gatsby build without having a .cache, it will download the asset(s) and this query works fine:

    query {
      gcms {
        assets(first: 1) {
          url
          file {
            childImageSharp {
              fluid {
                src
                originalName
              }
            }
          }
        }
      }
    }

the second time I start develop or build with the .cache present, I get an ENOENT error:

Unexpected error value: "failed to process https://media.graphcms.com/ASSET_HANDLE\nError: ENOENT: no such file or directo
ry, stat 'C:\\...\\.cache\\gatsby-source-filesystem\\a7762a6a4f2571864804231ff8b216e6\\ASSET_HANDLE
gR'"

The file does exist:

C:\..\.cache\gatsby-source-filesystem\a7762a6a4f2571864804231ff8b216e6\ASSET_HANDLE.jpg

The only thing that I do find odd, is that it's looking for the file ASSET_HANDLE not ASSET_HANDLE.jpg.

Deleting the .cache would work, but in the future I'll have about 3k images in my project, so downloading on every build would be a waste of bandwith and time.

Edit:
After some digging, I think my suspicion was right. The file can't be found because we don't get a file extensions from looking at the url, so ext is empty. When the file is not in cache, we get the ext from the buffer, but that will not be called when trying to get the file from .cache.

As a workaround I could add the ext: '.jpg' when using createRemoteFileNode, but that would limit me to only use one extension for all assets, which might not be what I want or, since I'm using GraphCMS in this example, I could get the original filename and get the extension from that, but that also doesn't seem optimal.

[DmacTorstar]

Having this issue, and detailed my experience in this ticket:
https://github.com/gatsbyjs/gatsby/issues/18030

[slaterbbx]

I think I have found that when using gatsby-background-image I have random .cache issues that are only on the client side when viewing the website, If I clear my local cach or turn off the local cache and then reload the pages, it will work fine after. Anyone have a similar issue?

Has me scared to do webhook based rebuilds on my project, If I do, can I just run gatsby clean then gatsby build to guarantee a clean build?

[sidharthachatterjee]

Let’s go ahead and close this for now since it’s been relatively quiet and people seem to have been hitting this less often than before.

Please open independent issues with reproductions for cases where caching breaks your gatsby build so that we can action those!

Thanks everyone!

[jonahallibone]

Unsure if this is an issue related solely to Gatsby, but I've had to use this command twice in the last two days. I am using the gatsby-source-contentful plugin, and have found that it will often cache images when content changes, leading to GraphQL throwing errors on those models. This appears to only happen with SVG images, but it's possible I am changing those most often.